Just finished “How Google tests software” book written by some Googlers. It is very much worth reading. I have very few informations about how testing goes at Google but as far as I could tell it seemed quite an honnest account on their practices. Here are some informations/data I found to be worth sharing.
First, it seems that a lot of the google testing culture comes from Patrick Copeland who joined the company in 2005 (a bit late in their history as they started a couple of years before 2000). At that time they were 50 full time testers and the culture of test was quite low (and bad) to say the least. It took years to Patrick and his teams (including Alberto Savoia and James Whittacker, authors of the book) to reach their idea of how quality should be managed.
Even if the word “agile” does not appear in the book (probably not much used within Google), one can detect a lot of Agile practices at Google. The major ones beeing the full integration/collaboration between developers and testers, the production in the most continuous way (up to the deployment) and the goal to reach a pyramide-shaped automation portfolio (much unit test, few end-to-end).
One point that is very much stressed and not so spread is the fact that they decided that testers should be as good programmers as programmers themselves. Basically they first stated that programmers (software engineers) were going to test as much as all the other engineering people (everyone is in charge of quality). Second step was to change the name of the testers/QA to “software engineers in tests” and “tests engineers” to make them more equal to programmers. And finally those SET and TE where in charge of helping the teams to test/automate, not just test. Given this, it became clear that finding the right people for the job was quite tough. All the parts about the hiring process are interesting.
Another quite counter-intuitive fact is that the methods and tools are not so centralized in Google. They are going in this direction and the best tools/methods they came up with are quite popular inside (and even outside as the release some tools). But even basic opinions like “how much automation versus exploration” seem to differ from one test manager to another. The book is quite honest and humble about this and does not try to sell THE google way of testing.
The future of the test as seen by James Whittaker in the last chapter is quite unexpected to ! He figures that testing will disappear as a job because it will so much become everyone else responsibility (Product manager, programmers…. customers ?) that the job will be diluted in the company.
Other misc facts :
– Google does not really think in term of dev/qa ratio. In their model with SET and TE, it does not make much sense
– the whole continuous integration system was very complex to set-up. Testing teams pushed a lot to have it and it became, of course, the backbone of their organisation
– they try to build automated test cases that are independent of the order in which they are run and that let the “system” in the same state they discovered it. This allow to run all the tests in parallel. This is a good practice I tried myself, and it appears to be not so easy to build and maintain.
Oh…. and my favorite paragraph about automation :
“The larger an automation effort is, the harder it is to maintain and the more brittle it becomes as the system evolves. It’s the smaller, more special purpose automation that creates useful infrastructure and that attracts the most software engineers to write tests.
Overinvesting in end-to-end automation often ties you to a product’s specific design and isn’t particularly useful until the entire product is built and in stable form. By then, it’s often too late to make design changes in the product, so whatever you learn in testing at that point is moot. Time that testers could have invested in improving quality was instead spent on maintening a brittle end-to-end test suite.”
Amen !
