Should the test automation framework be written in the same language the application is being written or developed in?

Just tweeted an article published by Sauce Labs on “Reducing False Positives in Automated Testing” that takes the form of a Q&A about automation. I mostly agree with the article but there is one answers that bothers me :

Should the test automation framework be written in the same language the application is being written or developed in?

A: Yes. Creating the automation framework in the same language of application will help the development team in testing specified areas whenever there is any code change or defect fix.[…]

I would replace the “yes” by “it depends”. I also see value in using another langage to automate functional tests. Specially when you have a dedicated QA/tester team. Here are a couple of reasons:

  • if your company is producing different products in different langages, then it could be difficult for your QA team to keep up with those different langages for their automation. So using a single langage for the tests would ease QA life. That would also allow QA to share libraries/tools among different tests for different SUT.
  • hiring skilled QA is not easy. Finding people with good functional understanding/curiosity of the SUT *and* a good technical knowledge can be difficult. Asking them to code in a langage like Python/Ruby rather than coding in C, C#, Java etc. could be make it easier to achieve
  • using a different langage than the SUT is also a good way to force QA (or dev if they code tests) to take a new perspective on the SUT. If you test your Java product in Java, you might swim in the “integration tests” water and even when you try to black-box test your application, you might end up interacting with the SUT in a gray-box way.

In the end it really depends on your organisation, culture, people etc. but it is worth having a debate on that point. You might want to take a look at this other article on the same subject: Your automated acceptance tests needn’t be written in the same language as your system being tested

Automation of Functional Tests with Robot Framework at SoftShake 2013

Last 24-25th october I went to SoftShake conference in Geneva. The conference was very interesting and was happy to see a lot of source code and live coding/demos. I was given the chance to make a presentation about automations of functional tests with Robot Framework. Here follows some feedback on it.

The slides can be found bellow though it misses what I consider being the most interesting part: live coding.  To show practical examples of code, I wrote some tests with Jenkins as a SUT. Those are very simple tests but that helped people to get a better idea of what kind of test can be written with Robot. Those little tests can be found in Github. I used fswatch to have the tests be executed every time I would save the tests. So with a split screen code+robot it was looking like an infinitest setup.

Questions I got after the presentations are worth noting as they give some information about what people care/think when they discuss tests automation:

1) Is it possible to have variables in the middle of a Robot Framework keyword? Question arose because in the Robot code I showed and wrote, I always write :

keyword argument1 argument2
ex/ file should exist in folder    file name    directory name

where in fact I should use the facility offered by Robot to put argument inside keywords like described in the documentation. This helps ending up with tests that are easier to read. Here is a simple example:

*** test cases ***
Embedding arguments into keyword name
number 2 and 2 should be equal

*** Keywords ***
number ${x} and ${y} should be equal
should be equal ${x} ${y}

2) “With a tool like Robot Framework, are QA supposed to commit the tests in the SCM?”. Answer is yes as the tool makes the more sense when integrated in a continuous integration environment that will retrieve the tests in an SCM. But the question is worth asking because in my previous company, having QA using the SCM was not an obvious step to make. With some non-technical rather-functional QA we might have a group that is more at ease with Office documents and filesystems than with developer tools. Personally, I consider using the same tool is also a very good opportunity to make the QA and Dev team be closer. (further reading on this topic: “Technical artifacts including test automation and manual regression test scripts (if any) belong in the Source Control System versioned with the associated code” by Elisabeth Hendrickson)

3) is it possible to write keywords in Java rather than Python. Answer is yes, although I consider Python to be a good choice and more homogenous with the syntax used in Robot DSL.

4) finally, there were several questions about what “component tests” are and what entry points could be used in the SUT to write such tests. My definition of component tests is rather shallow: everything that is between unit tests and end-2-end tests! That’s enough an explanation for developer-type people who can envision a REST API for example. That is more complicated to grasp for non-technical QA or managers. I was, once more, surprised to see that for many people they were 2 obvious way to test a product: unit tests written by developers (white box testing) and gui-driven-tests written by QA (black box testing)… Some more evangelism to do for gray box testing I guess!

So this was good opportunity to share my experience of Robot and to be challenged with unexpected questions is a very good way to progress. So I might be back with other sessions in the future…


Robot Framework Introduction

Did a little presentation of Robot Framework at Human Talk Grenoble about Robot Framework. Quite difficult to sell that kind of tool to an audience into web development with no or few experience of tests. This is more easy when talking to bigger team working on more traditional (old) softwares. Have to understand more about the possibilities and constraint of web development to understand how Robot could be used for the business/functional logic of the web projects.