In the testers world automation is kind of like, for the want of a better phrase, "the icing on the cake". This area of quality assurance is certainly considered the most interesting and developmental. At Cognifide, we trust in automation. Although manual tests are necessary during the development process, constantly changing code deserves to be checked as a whole all the time and as automated tests of our web applications provide almost immediate feedback on the quality of the software we produce, it is just impossible to overlook. To meet customer expectations, we have developed a framework for desktop web application testing and proudly, it has been used successfully for some time. Unfortunately, nature abhors a vacuum. A little while ago, as you know only too well, internet applications for mobile devices gained in popularity (almost overnight for some). Fascinating as to how certain web pages, especially social networks, grew their network traffic from mobiles by over 50 percent last year (comScore). Seeing this, our customers have begun to demand websites that are compatible not only with desktop browsers, but also look good on glossy screens of touch phones as well. We had to face the problem: how to automate tests on mobiles and our answer was: WebDriver.
The world of test automation has been evolving and a number of mobile automation frameworks have been developed. Our choice of mobile automation framework was based on the following assumptions:
- popularity, which involves a developer community ready to share their knowledge,
- the position of the framework creator who will provide development and support in the future,
- the ability to reuse whole or parts of tests written in Selenium 2 for desktop browsers,
- compatibility with Continuous Integration tools such as Jenkins,
- support for a variety of mobile platforms in different versions,
- support for the upcoming versions of mobiles,
- the ability to perform tests both on real devices and their emulators,
- ease of writing, debugging and result analysis,
- purchase price and implementation costs in our company.
Within Cognifide, WebDriver was pretty much a natural selection. Previously, we were using Java Selenium 2 to automate tests of our web applications. The WebDriver project is very popular in the QA community and well supported as the main commiter and user is the search giant - Google. It gave us the ability to reuse previously written tests and solutions designed for desktop browsers. Moreover, the tests written in Selenium 2 that we previously used for automation could be easily ported to WebDriver because Selenium 2 makes use of WebDriver under the hood. To cut a long story short, we used the same common classes developed earlier and switched WebDriver to its mobile version. All of these using Java were built by Maven and are supported by TestNG or JUnit. However, it would be worth nothing if tests were performed remotely from a real separate laboratory environment. But is it possible to launch tests on a broad spectrum of real devices or at least on theirs emulators?
Multiple environments, single approach
The mobile market share is seriously split between a wide range of devices. In fact, the majority of them browse the web using the webkit engine to render web pages. Two main players: iOS and Android are fully supported by the WebDriver framework and have native drivers to run tests. Other devices e.g. RIM Blackberry have also got drivers developed but to be honest, I have not tried them yet. For other Webkit based browsers there always remains the Headless WebKit WebDriver which can emulate the pure WebKit environment. Recently, a new player has appeared on the ultra competitive mobile market, i.e. Windows Phone. It has a browser based on Internet Explorer 9 so until native WebDriver is developed, a desktop Internet Explorer driver with the proper resolution set can be used.
What about different versions of operating systems for mobile devices? Sometimes a client requires support for several versions of iOS or Android. It is very expensive to maintain such a variety of mobile devices. With WebDriver we have the ability to launch tests on a real device but also what is more important on their emulators with different versions of operating systems.
Another important issue is that emulators have given us the opportunity to be always up to date with mobile operating systems, even if the newest device is hardly available on the market.
Based on WebDriver we developed our own testing framework. We use it for functional tests and something we call the Automated Exploratory Testing (AET) tool.
In a nutshell, AET crawls pages from a source provided (csv file, site-map, etc.) in two modes. The first one conducted at the very beginning collects patterns of html, captures layout screens, etc. while the second one is a test mode. Still using the same URL source AET crawls the same pages and verifies their content and layout in comparison to gathered patterns. At the end, there is an HTML report generated.
It's a very powerful tool for regression and sanity checks. Our goal was to implement the same solution for mobile devices. Thanks to the chosen framework it could be done without much work. This approach has been used successfully in our projects (and it is obviously subject to continuous improvement but that's another interesting subject for another day). Below is an example of the layout validation. I must say that I am personally, quite impressed by it!
Worth mentioning is the fact that mobile versions of websites are simple and static in most cases. Testing this kind of site with the use of the AET tool with layout validation gives us a great test coverage rate. Now, when our framework is ready, we achieve this goal without effort in every single project involving mobiles. AET is also fully integrated with the continuous integration environment. In every project we have separate jobs responsible for firing different type of tests on various environments - see the screen-shot below. There are several Jenkins jobs responsible for launching tests of touch and smart page versions and one separate job for tests on iOS Safari.
Finally I've recorded a short demo of WebDriver in action. Enjoy:
|Step number||Step description||Expected outcome|
|1.||Enter about us page||about us page is displayed|
|2.||Enter services page||services page is displayed|
|3.||Enter clients page||clients page is displayed|
|4.||Enter contact us page||contact us page is displayed|
|5.||Enter blogs page||blogs page is displayed|
|6.||Search for quality assurance blogs||search results are displayed|
As an output of this sample test, a html report is generated. You can find a sample report below.
WebDriver provides a solid basis for building testing frameworks. It allows to run test cases created previously and designed for desktop web applications. Test automation on real devices is no big deal with WebDriver. What is more, it also supports mobile device emulators. If you want to automate your tests of mobile web sites on a wide range of devices I highly recommend using WebDriver.
Do not forget to let us know what you think about the your comment!
We're always looking for new faces to join the Cognifide family. Take a look at our available jobs.