Automated tests in Continuous Integration environment - Part 1

02 August 2011
Zbyszek Mockun

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.” Martin Fowler, Thoughtworks

Martin sums it up quite well - the idea is to integrate new pieces of software as fast as possible. Time frames are crucial. Integration should be divided into three steps:

  1. commit new functionality and build new application
  2. run unit tests
  3. run Integration/System tests

First and second steps should be run for each commit. They are very specific and shouldn’t take more than a few minutes. They can be done without any test (QA) engineer input.

The third point - integration and system testing - is more complicated. System tests usually need more time and test engineer influence. There are several open source tools such as Bamboo, Hudson and Cruise Control in the market that allow you to introduce Continuous Integration in your environment.

How important is to integrate automated tests with Continuous Integration environment?

Agile methodologies are based mostly on short iterations (one or two weeks). The question is how do you test efficiently? How can we be sure of the released software's quality? How do we manage/plan test that fit within budgets and resources?

If we think about automated tests, the question is: how often and when should we run automated tests?

At first thought, the answer is really easy - as often as possible and as quickly as possible.

Both answers suggest one solution: Continuous Integration.

CI allows us to run automated tests after each commit and send feedback with results to developers. Good automated tests should cover all functionality, or at least most of it. If we run automated tests only at the end of each iteration, there will probably be no time to fix any found issues. Not all issues are quick to fix, some need changes that can influence already implemented features or may require changes to the technical solution. What if a new feature has a bigger impact on the application that we thought? There is no time to change the new feature or ask the client for additional feedback. All of them suggest that regression tests should be performed continuously throughout the iteration. The question is how to automate regression tests, as often as possible, without draining QA team resources, which should be concentrated on new features.

The next post on the topic of Continuous Integration will deal with how you can automate functional tests using Selenium and using Hudson.