Create Jenkins Job for Robot Framework

Once you have created your first tests in Robot Framework, next step is to include those tests in your Continuous Integration (CI) System. Here I will show the different steps to do so in Jenkins.

Let’s assume you have


First we create a new job to launch our Robot tests:


Once the job is created, we configure it.

  1. set up Source Code Management for the source code of the tests:jenkins_svn
  2. set up a first “Build Trigger” on the success of the job that builds the SUT:trigger_build
  3. set up a second “Build Trigger” on changes in the Source Code of the tests:trigger_scm
    This way your tests will be launched either if there is a new build of the SUT or if your tests have changed. Second trigger is relevant because some modifications in your tests may have broken some of them, and you don’t want to wait for the next build of the SUT to find it out. In other words, when a test fails in Jenkins, it is good to know if this is a consequence of a change in the SUT of a change in the tests (if both changed, analysis will be tricker).
  4. get the artefact from the project that build your SUT so that your SUT is available from your Jenkins’ workspace where the Robot tests will be run. To do so you can either use the Jenkins Copy Artifact Plugin or write a piece of Batch/Shell script.screenshot-copy-artefacts
  5. The comes the step in which Robot tests are going to be launched. For this you create a “Execute Shell” Build step that contains, at least:
    pybot path/to/my_tests/

    and all the –variable, –include, –exclude etc. that you use to customise you run.
    One noteworthy command line option in the context of a CI server is –NoStatusRC, which force Robot’s return code to zero even when there is test that fails. This way the status of the Jenkins build can be driven by Robot Jenkins Plugin like you will see in final step.build_robot

  6. Finally, to have a more granular settings of the results of the tests, and keep a copy of the report/log of the test executions located in the Jenkins Server, you can use Robot Framework Plugin. Once the plugin is installed, it will be available in the list of “Post Builds Actions”. A simple configuration would be like this:pluginand after a couple of builds, the project page would look like that:plugin_project_page

Once this basic setup is working, you will find out many options in Jenkins and Robot Framework to get more value out of it. To give just one example, once the test portfolio becomes large and/or long, you might find out that this is not efficient to launch the full regression suite at once when there is a change in the SUT or the tests’ code. A good strategy is to have 2 Jenkins job. The first one (“smoke tests”) is running only a portion of the whole suite that runs quickly (say 5/10 minutes for example):

pybot path/to/my_tests/ --include smoke --exclude not_ready

and the second job (“full tests”) launches all the tests:

pybot path/to/my_tests/ --exclude not_ready

but is launched only when smoke tests are run successfully: chain_builds

so if your SUT or your tests have some essential feature (covered in the smoke test) broken, you will save your machine a “full test”, and, more important, the team have a quicker feedback on the quality of the SUT build.


What editor for Robot Framework test cases

Robot Framework home page lists a number of plugins to edit Robot Framework test cases along with Robot’s own editor RIDE. Here is a feedback based on my experience with some of those tools.

RIDE was my first choice four years ago when I started using Robot. We were a team of 20 quality engineers working on Windows machine. Some of us had a technical background and some others were more on the banking business side (we were producing a financial software). RIDE ended up being a good choice, specially for the non-tech people as it hided part of the grammar of test cases: Suite Setup, Tags, Library etc. are all input boxes in a GUI. The ability to launch test cases from Ride was also very handy and kept us away from the command-line (which most of the team was never using). Overall we had a very good experience with RIDE and I would recommend it in a similar context.

TextMate (with its Robot’s bundle) was the editor I switched to when I switched to Mac. There were 2 motivations to move away from RIDE. First one is that RIDE is not very slick on Mac (there are several issues opened for a while) and even install is a bit complicated with WxPython and Python versions collisions. Second motivation was that I joined a more technical company where manipulating the source code of the test was much more frequent (e.g. editing the test case on a remote VM via SSH with vi, merging changes done by another tester…). So I didn’t want anymore to have a GUI layer hiding the source code of the tests. I chose TextMate because it was free, lightweight and worked at once with Robot and SVN. After some time, I started to miss keyword completion and quick access to keywords source code though.

PyCharm is the editor I am using for a couple of weeks now. This time the switch was motivated by some limitation of TextMate (see before) and also by looking over my colleague’s  shoulder who were getting happier and happier with PyCharm. Looks like  JetBrains’s IDE is getting momentum in the Python  community as I hear/read more and more about it. There are currently 2 competing plugins for Robot Framework: Robot Plugin and Intellibot. Both are providing Syntax Highlighting, Code Completion, Jump to Source with some little differences. Best thing is to try them both and see which one fits best.

A side note about this editor topic. when I moved to PyCharm, the amount of syntax checking became one level higher than on previous editor, and I was bothered by the fact that all my TXT files were getting analysed by the plugin (making my non-Robot TXT files uneasy to read). So I changed the extensions of my Robot Test Cases and libraries from .txt from .robot. This way I can configure Robot Plugin to affect only my Robot files and not all the classical text files.