Automated tests in Continuous Integration environment – Part 3

26 August 2011
Zbyszek Mockun
Frink_Cognifide_2016_HeaderImages_0117
As promised, here's the concluding post on how to use Selenium and Hudson together in a continuous integration environment and why, I believe time rules!

Integrate automated tests with CI environment

Integrating Selenium with Hudson is actually quite easy. You basically need to install the Selenium plugin for Hudson. Oh and yes, you will obviously have to install ant/maven and other applications that will be used to run test suites/rebuild tested application. For running automated test suites, I recommend Selenium RC should be used. Ant is used for running tests. As you can see integration is quite easy and does not require any specific knowledge. If we add some new features like screenshots we should write additional Ant targets for them (clearing directory, copy selenium reports/screenshots to archive, etc)

Selenium plugin gives two additional functions:
  • add link to Selenium Report - only for the latest run.
  • add Selenium Test Result Trend chart. Chart gather data for latest job runs and shows trends. We can observe how many test case failed, and the growing number of automated tests.
Hudson: Selenium plugin installed on Hudson, Selenium Report link and Selenium Test Results Trend chart

Time rules

It’s one of the most important subject when we think about integration our automated tests with CI. Before we start, please try to answer the following questions.
  • how often do we want to run/rebuild application?
  • how much time is needed to rebuild application and run automated tests
  • how much time can we spend on analyzing test results (regression tests) The answers to these questions are crucial for us.
How often is it worth rebuilding an application and running automated test cases? Clearly, as often as possible. Should it be after each commit? Or daily? It really depends how many developers are working on the project and how your application is specified. Generally, one integration daily should be enough and doing this at night is often a wise move. There could be some issues when your application is built by developers from all the world, but at times, the most important factor can be tester's time zone. If we schedule the build at night, everything should be finished before the testers come to work. This way, testers can check the results, reports bugs and start doing their daily tasks. If there are some critical issues, the scrum will be a good place to raise them.

The time needed to rebuild applications and run test suites is very important too. This should not exceed 12 hours. If it takes longer, you should divide the work on to smaller suites or choose important tests which are run daily, with all tests being run at the weekend. Getting everyday results and break work to check is disturbing for tester and we should avoid it.

The main test engineer's task during iteration is testing new features. If the test engineer spends 30 minutes each day dealing with issues found in automated test, it shouldn’t interfere with his daily tasks. Nor will it exceed the total time spent on regression testing at the end of the iteration. And all critical issues will have been found and fixed as they occur, or raised to client, rather than impeding delivery.

Should we use open source tools?

Before you say yes or no, take a moment to think about the other tools you already use and it should be fairly obvious that you don't really judge a tool based on whether it is open source or not but as to how well it addresses your requirements.  As shown above, we can integrate Selenium and Hudson quite easily and enhance them ourselves depending on our needs or surely,  find a suitable solution on the web. The good bit is that the money that would otherwise be spent on license fees can be invested into enhancing functionality available on open source tools. In most cases it will be cheaper and you will have exactly what you need but would advise that you evaluate your needs and perhaps, there is a combination of open source and commercial tools that provides the best solution.