Robot Framework Test Automation – The Book

Packt Publishing recently released a new book about  “Robot Framework Test Automation” by Sumit Bisht. The book is a quick and easy 100 pages read that can be useful to those who find the user guide a bit to dry.

First chapter helps go through the installation steps but fails at giving a clear picture of Robot Framework ecosystem: a diagram shows a “testing tool” component being the one that actually target the SUT while Robot Framework is more on the user end side as a coordinator. This is just one way to use Robot (a way that is later described more in details in section about Robot+Selenium and Robot+Sikuli). But Robot can be used to test directly the SUT through its native libraries or some custom libraries.

Second chapter is quite clear about the different file format and the organization of the test portfolio. Third chapter treats the actual creation of test cases with some data about syntax and libraries, but it lacks some examples going from the easiest to more complex cases. Chapter four discuss Robot association with other testing tools (like mentioned before) and finally the last chapter helps generating standard and custom reports.

A topic that is not covered is who is the “ideal” Robot Framework user. It is implied in chapter three that “developer and stakeholders” will collaborate in writing the tests. My experience is rather than a tool like Robot Framework is a very good fit for a QA/testers team that is distinct from dev and stakeholders. For such a team it makes sense to have a specific tool/DSL for testing rather than coding the functional/acceptance tests in the langage of the product.

Globally though the book is a good read, it fails at being a real “missing manual” compared to what the User Guide already offers. The book would maybe have benefit from taking an alternate approach with much more example, like a cookbook, and more experience from the field. Anyway, thanks to Sumit and Packt for the effort in sharing some knowledge about Robot Framework!

How Google tests software

Just finished “How Google tests software” book written by some Googlers. It is very much worth reading. I have very few informations about how testing goes at Google but as far as I could tell it seemed quite an honnest account on their practices. Here are some informations/data I found to be worth sharing.

First, it seems that a lot of the google testing culture comes from Patrick Copeland who joined the company in 2005 (a bit late in their history as they started a couple of years before 2000). At that time they were 50 full time testers and the culture of test was quite low (and bad) to say the least. It took years to Patrick and his teams (including Alberto Savoia and James Whittacker, authors of the book) to reach their idea of how quality should be managed.

Even if the word “agile” does not appear in the book (probably not much used within Google), one can detect a lot of Agile practices at Google. The major ones beeing the full integration/collaboration between developers and testers, the production in the most continuous way (up to the deployment) and the goal to reach a pyramide-shaped automation portfolio (much unit test, few end-to-end).

One point that is very much stressed and not so spread is the fact that they decided that testers should be as good programmers as programmers themselves. Basically they first stated that programmers (software engineers) were going to test as much as all the other engineering people (everyone is in charge of quality). Second step was to change the name of the testers/QA to “software engineers in tests” and “tests engineers” to make them more equal to programmers. And finally those SET and TE where in charge of helping the teams to test/automate, not just test. Given this, it became clear that finding the right people for the job was quite tough. All the parts about the hiring process are interesting.

Another quite counter-intuitive fact is that the methods and tools are not so centralized in Google. They are going in this direction and the best tools/methods they came up with are quite popular inside (and even outside as the release some tools). But even basic opinions like “how much automation versus exploration” seem to differ from one test manager to another. The book is quite honest and humble about this and does not try to sell THE google way of testing.

The future of the test as seen by James Whittaker in the last chapter is quite unexpected to ! He figures that testing will disappear as a job because it will so much become everyone else responsibility (Product manager,  programmers…. customers ?) that the job will be diluted in the company.

Other misc facts :
– Google does not really think in term of dev/qa ratio. In their model with SET and TE, it does not make much sense
– the whole continuous integration system was very complex to set-up. Testing teams pushed a lot to have it and it became, of course, the backbone of their organisation
– they try to build automated test cases that are independent of the order in which they are run and that let the “system” in the same state they discovered it. This allow to run all the tests in parallel. This is a good practice I tried myself, and it appears to be not so easy to build and maintain.

Oh…. and my favorite paragraph about automation :

“The larger an automation effort is, the harder it is to maintain and the more brittle it becomes as the system evolves. It’s the smaller, more special purpose automation that creates useful infrastructure and that attracts the most software engineers to write tests.
Overinvesting in end-to-end automation often ties you to a product’s specific design and isn’t particularly useful until the entire product is built and in stable form. By then, it’s often too late to make design changes in the product, so whatever you learn in testing at that point is moot. Time that testers could have invested in improving quality was instead spent on maintening a brittle end-to-end test suite.”

Amen !