Once you have created your first tests in Robot Framework, next step is to include those tests in your Continuous Integration (CI) System. Here I will show the different steps to do so in Jenkins.
Let’s assume you have
- a Jenkins server
- a Software Under Tests (SUT) that is built by a job in Jenkins
- a portfolio of Robot Framework tests stored in an SCM repository (we will use SVN for this example)
First we create a new job to launch our Robot tests:
Once the job is created, we configure it.
- set up Source Code Management for the source code of the tests:
- set up a first “Build Trigger” on the success of the job that builds the SUT:
- set up a second “Build Trigger” on changes in the Source Code of the tests:
This way your tests will be launched either if there is a new build of the SUT or if your tests have changed. Second trigger is relevant because some modifications in your tests may have broken some of them, and you don’t want to wait for the next build of the SUT to find it out. In other words, when a test fails in Jenkins, it is good to know if this is a consequence of a change in the SUT of a change in the tests (if both changed, analysis will be tricker). - get the artefact from the project that build your SUT so that your SUT is available from your Jenkins’ workspace where the Robot tests will be run. To do so you can either use the Jenkins Copy Artifact Plugin or write a piece of Batch/Shell script.
- The comes the step in which Robot tests are going to be launched. For this you create a “Execute Shell” Build step that contains, at least:
pybot path/to/my_tests/
and all the –variable, –include, –exclude etc. that you use to customise you run.
One noteworthy command line option in the context of a CI server is –NoStatusRC, which force Robot’s return code to zero even when there is test that fails. This way the status of the Jenkins build can be driven by Robot Jenkins Plugin like you will see in final step.
- Finally, to have a more granular settings of the results of the tests, and keep a copy of the report/log of the test executions located in the Jenkins Server, you can use Robot Framework Plugin. Once the plugin is installed, it will be available in the list of “Post Builds Actions”. A simple configuration would be like this:
and after a couple of builds, the project page would look like that:
Once this basic setup is working, you will find out many options in Jenkins and Robot Framework to get more value out of it. To give just one example, once the test portfolio becomes large and/or long, you might find out that this is not efficient to launch the full regression suite at once when there is a change in the SUT or the tests’ code. A good strategy is to have 2 Jenkins job. The first one (“smoke tests”) is running only a portion of the whole suite that runs quickly (say 5/10 minutes for example):
pybot path/to/my_tests/ --include smoke --exclude not_ready |
and the second job (“full tests”) launches all the tests:
pybot path/to/my_tests/ --exclude not_ready |
but is launched only when smoke tests are run successfully: 
so if your SUT or your tests have some essential feature (covered in the smoke test) broken, you will save your machine a “full test”, and, more important, the team have a quicker feedback on the quality of the SUT build.
I have tried this localy where my pybot was installed. I was wondering where do I install it if jenkins is on a remote server and my tests are placed in git. Do I ssh on jenkins server and install Python – robot framework?
If Jenkins is on remote server then you will have to install Python and Robot Framework on this remote server (connect via SSH is one good way to do it). But more often, you would have different machines/VM, one for Jenkins Server and other to execute the jobs (slaves). In that case you would have to install Python/Robot on your slaves.
Any idea on how to setup the RF in Bamboo?
I have no experience with Bamboo so I can not help you.
But the question was raised in Robot Framework mailing-list.
You can take a look: https://groups.google.com/forum/#!topic/robotframework-users/L-k3w8UEu0U
Thanks for the nice introduction laurent!
I have installed both robotframework and jenkins. my test runs fine when i run it my local machine. but when I run my test using jenkins some test fails
we will need some more informations to help you. What is your test and what is failure?
I am having my tests stored in .txt format , I am not able to Run it with the use of Following Batch commad
pybot –timestampoutputs -l NetBankLogs.html -r NetbankReport.html -d Path to test\Payment_List.txt
what response do you get?
I think you need something after the -d which tells RobotFramework where to create your output files.
-d –outputdir dir Where to create output files. The default is the
directory where tests are run from and the given path
is considered relative to that unless it is absolute.
Maybe something like this:
pybot –timestampoutputs -l NetBankLogs.html -r NetbankReport.html -d ResultPath “Path to test\Payment_List.txt”
Hi I have a question regarding setting up my jobs on my Local Jenkins.
Do I have to setup my workspace for it to work?
I always setup the batch windows command to change the directory to where I place my Robot Framework scripts. I notice that when I do this and build the job it always fails because the filemask output xml cannot be found in the directory. Am not sure what I’m doing wrong.
hudson.AbortException: No files found in path C:\Program Files (x86)\Jenkins\jobs\Checkout Test Case\workspace with configured filemask: output.xml
Is there something else I should configure for my tests to run smoothly?
Thanks!
Do you use Jenkins-Robot Plugin?
If so, you have to set up “Directory of Robot output”.
Hi there.
Thanks. Should I put an absolute directory for this?
I placed a directory but it still failed. Also for some reason the windows batch command doesn’t seem to recognize pybot even when I installed the Robot Framework plugin.
C:\Users\MC\automated-tests\Robot Framework\testsuites>pybot –VARIABLE ENV:browserstack –VARIABLE COUNTRY:$Country Checkout.txt –outputdir Result/Test/$Country
‘pybot’ is not recognized as an internal or external command,
operable program or batch file
and then when creating the output it still has issues with the filemask
C:\Users\MC\automated-tests\Robot Framework\testsuites>exit 9009
Build step ‘Execute Windows batch command’ marked build as failure
Robot results publisher started…
-Parsing output xml:
Failed!
hudson.AbortException: No files found in path C:\Users\toledo-ni\Documents\Codebase\TIGER B2C\automated-tests\Robot Framework\Test\Result with configured filemask: Checkout
Hi all,
Firstly thank you for this post Laurent. I have a question about Robot Result Plugin. I’ve integrated it in my Jenkins automation job (Jenkins works on AWS). When I tried to open report.html from my local pc I get that error everytime. Do you have any iade about that?
Best regards.
Error
Opening Robot Framework report failed
Verify that you have JavaScript enabled in your browser.
Make sure you are using a modern enough browser. Firefox 3.5, IE 8, or equivalent is required, newer browsers are recommended.
Check are there messages in your browser’s JavaScript error log. Please report the problem if you suspect you have encountered a bug
sorry for the (very) late response.
Where you able to fix your problem.
> “When I tried to open report.html from my local pc”
are you trying to open the report from the Jenkins job page?
Do you have the Jenkins-Robot plugin installed?
Hello,
Thanks for your all efforts, here I am stuck with my Jenkins + Robot Framwork configuration. I have integrate my Automation script with Jenkins but it is not up and running also I am not getting any error code , After running I am just getting below message.
Started by user Test Admin
Building remotely on Slave in workspace D:\JENKINS\jenkins\workspace\
No emails were triggered.
[Fact24] $ C:\Windows\system32\cmd.exe -xe C:\Users\XXName~1.TUP\AppData\Local\Temp\hudson7217699004107166534.sh
Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.
D:\JENKINS\jenkins\workspace\Robot results publisher started…
-Parsing output xml:
Done!
-Copying log files to build dir:
Done!
-Assigning results to build:
Done!
-Checking thresholds:
Done!
Done publishing Robot results.
Build step ‘Publish Robot Framework test results’ changed build result to UNSTABLE
Email was triggered for: Always
Sending email for trigger: Always
An attempt to send an e-mail to empty list of recipients, ignored.
Finished: UNSTABLE
“unstable” is what you get when you have some test failures. Are the test executed? Are some of the test failing?
I want to use Jenkins just to initiate my Robotframework tests. I am not using any version control, what I have is a folder structure which contains my Robot scripts.
I want to trigger these tests nightly at a fixed time using jenkins and generate/view reports. Can you please advise?
then you don’t need to have the Version Control phase, and you can just launch a script in your Jenkins Job that launches “pybot” on your Jenkins slave.
Hi all
Iam new to OpenShift CI/CD. its contains jenkins/gogs/nexas/sonarcube(pipeline).
i need this Robot Framework plugin for our project requirement.
Any idea this plugin installation and usage
pls do needfull
Thanks in advance
Hi,
I have my Jenkins sitting on remote, and we do not have admin rights to install robot and selenium, but python is available on the Jenkins server. In this case, how do I achieve my execution?
Hi,
I want to run robot framework script on Centos server where Jenkins is installed but Python and Robot framework is not installed.
Without installing Python and Robot framework on Centos server is it possible that I can run robot script in Jenkins? if not, could you please provide me steps to install Python and Robot Framework on Centos.
Thanks
If you want to run your Robot Framework tests on your Centos Server you will need to install Python and Robot on this machine.
You can follow the instructions of the user guide: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#installation-instructions
I need to run my tests on Jenkins remote server(ubuntu). I understand that i have to download python and Robot on the server. How about all the other libraries that i an using in my tests? These libraries might change as we build more test so how does jenkin server know when to install what libraries? Also how do we manage selenium drivers on Jenkins server? Do i need to update drivers and libraries with every test run?
If you are using default Robot libraries you don’t have to download them, they come with Robot.
If you are using other Python libraries (e.g. Requests) you will have to install them on your Ubuntu box (using “pip” usually).
This whole install process should be included in a script that will be run before your tests are run.
So that you download/install everything you need once before launching all the tests.
Hi, and thanks for useful post! Have you ever tried/used Slack upload file plugin? If yes, did you manage to get it do the job?
Hi ! Please help me out. I install Robot framework plugin and it was successful. But I am unable to see Robot options in sidepanel or post build steps. What am I doing wrong?
Hi.. I have multiple tests placed within folders. I want to initiate a bulk run(Single folder which contains multiple folder with multiple test suites) for my robot scripts from jenkins. how can do it?
Hi Laurent,
I had problem with running my robot scripts in jenkins
Steps I followed
1.Installed python and robotfrwarework,selenium2Library in local
2.Added the robotframework plugin into jenkins’
3.Created my robot scripts
4.Added command shell
5.I got the below error when am trying build it
here is the error log i found from console output of jenkins
Importing test library ‘Selenium2Library’ failed: ModuleNotFoundError: No module named ‘SeleniumLibrary’
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/Selenium2Library/__init__.py”, line 19, in
from SeleniumLibrary import SeleniumLibrary
Hi..
I kept the Robot file and Jenkins in VM. In the robot script I kept the user ID and password as a dialog box (as user entry). WHne i ran Jenkins in my local ,the credentials popup is displayed in my VM. Could you please help on this.
Thank you for the explanaton n the jenkins side,
i am working on azure test plans and plan to import the test cases of robot framework.
Can you tell how can we import all the 2000 testcases in azure test plans and run it selectively on test cases basis in a pipeline
Hello,
Just wondering if you found an answer to your question?
Thanks,
Hi,
I have my tests in git and Jenkins is in a remote server. There is a Jenkins windows slave machine with python and required files. How do I execute my test in the Jenkins slave machine?
HI All,
I need one help, I don’t want to declare my project’s credentials in my automation project files.
Is there any way to keep the project login credentials in jenkins and at the time of execution fetch those credentials from jenkins and use them in project execution.
Thanks