the non existent REST interface...

What happens when you want to establish b2b communication so that you can exchange information with a cooperating legal entity, but there is only the web interface? What happens if there is an enormous amount of critical business data to download and manipulate every day but you are forced to manually use the web interface in person? 

This situation happened to me at least twice, and I see it coming once more in the near future.

The first time was back in late 2010... when the Greek government initiated some transparency laws most notably known as 'Δι@ύγεια' (diavgia - https://diavgeia.gov.gr/). Just a web interface, only manual interaction, and imagine that every single decision of every public authority (including payment orders) ought to be published there to be valid and executable... with metadata... and the decision itself in pdf format...without delay (αμελλητί)... and with many payments blocked, delayed or due, because of this artificially created, but real bottleneck.

It was easy to automate the creation of the pdf, and the collection of the related metadata for every payment decision, but then what? You had to manual insert the data on the web page, upload the pdf file, confirm the correctness of the post, and get back a signed pdf with a reference number called 'ΑΔΑ' (ada), generated by the website...That specific number is the proof of the submission. The more difficult part is to keep this process in sync, fearing the legal consequences of payment someone, not in lack of the official approval, but by not making the approval publicly available through 'Δι@ύγεια'.

The solution back then was to trick the website and imitate the operation of the browser inside our application using C#. All HTTP gets, posts and cookie handling. Even scraping the captcha image and presenting it to the user, so that the login process integrates in our application and feels more natural to the end user. Luckily the website has been tricked as expected, and our users could submit decisions and automatically store the resulting reference number with the click of a button at high speeds and even higher rates, with no submission errors. Actually many of them were never introduced to the manual process of submission, through the actual website. That was a win situation over a... non-existent REST interface.

A couple of years later there was a nice and shiny version upgrade of 'Δι@ύγεια' with an exemplary REST API, but we were forced at first to deal with the manual procedure just like a punishment, which was avoided.

The second time was a couple of months ago... when we had to import financial data to our accounting system... too many banking accounts to review and register and many credit card transactions to watch over too... this was a two face problem... although we managed to get the banking account transactions directly from the bank in the latest ISO20022 XML format and automate the process as we should, the credit card transactions was another story. We had to manually download the data every single day from two different sites to keep payments and customers in sync, ...with different credentials, through a web only interface...no REST, no Web Service, no alternative or maybe there is...

This time we tried another more flexible solution. The idea was to apply known tools from our testing and automation tool belt on this production facing requirement. Our main ingredients include Ruby, the programmer's best friend, and Capybara, the acceptance test framework for web applications with selenium-webdriver. This way we were able to drive a browser through the process of log into the web site with valid credentials and download the relevant files.