Mobile testing - emulators or real devices

26 March 2012
Krzysztof Konopko
Frink_Cognifide_2016_HeaderImages_0117
Testing mobile applications and websites isn't new anymore. This post is reminiscent of the issues we faced when we started testing out web applications on mobiles.

Main goals

No compromises were going to be made with the functionality or design, i.e. the website had to work and look good for all visitors on mobile devices. The client provided us with an acceptable list of mobiles that they wished to support. We split it to two groups: touch screen and smart devices.

Now,  our front end developers could determine webpage behaviour and deduce separate styles for all the different mobiles that needed to be supported. The main question that arose early on from a testing perspective was : should we use real devices or would simulators suffice? Of course, we're never short of ideas and in this case, there were quite a few good ideas that merited an investigation. After a bit of reasearch, I found that there are many simulators for all of the main manufacturers such as:
  • Iphone - Xcode - Version of Apple’s powerful integrated development environment for creating great apps for Mac
  • Ipad - can be simulated using Xcode for all versions including iOS 5.0
  • Blackberry - BlackBerry Smartphone Simulators
  • Nokia - Nokia remote access , Nokia desktop emulators (S60,S40)
  • Android OS based phones - Android SDK
  • Windows Phone - Windows phone Developer tools
  • All other devices - MITE ( Mobile Internet Testing Environment) - desktop tool that lets you interactively test and verify mobile content by emulating 2,000 devices and 12,000 device profiles

 As you can see, with these emulators we could cover most of the devices available on the market but the key question that still remained was whether they were reliable enough to replace real devices? We got the answer to this question on the first day of testing. There are many examples which prove the theory that testing web page functionality and designs on an emulator cannot satisfy our clients. As an example, here's what we found on the BlackBerry 8520 :
  • top menu items take the entire space on device


On other simulators, there were situations like dysfunctional 'Back to top' links or issues with resizing images that do not appear on the actual device and to top it all, we encountered some OS specific issues...

If we use emulators for testing the site, we have to remember to check default emulator settings that reflect the real device behaviour such as resolution, abstracted LCD density, maximum VM application heap size, ram size, cache partition size and also settings inside particular mobile such as the JavaScript, default browser, operating system.
As a tester, you have to bear in mind bandwidth limitations. If you're wondering what tools to use, you can use Fiddler or NetLimiter tool to simulate GSM/GPRS network speeds. If you're lucky like we were and are able to use real devices, then here's a checklist:
  • valid sim card
  • wi-fi connection with limited bandwidth to simulate mobile network - this way we will be able to check if web page rendering time is short enough to satisfy your client
  • clear your cache memory when retesting issues
  • keep the mobile OS version up to date

Browsers vs devices

The next thing we had to consider was the list of browsers that we were to support. Most popular browsers  (Android & Safari) were supported by devices but remember, there is a user base that may opt to download browsers like the Opera Mini as it is free to download. Moreover, statistics prove that this browser accounts for a significant part of the total market. This obviously implied that we had to include the Opera Mini browser into our testing scope and begin supporting this particular browser to avoid further amount of work.
Mobile/Tablet Browser Market Share

Testing problems

If you're wondering what the really significant issues encountered during testing were, here's the top 3 issues:

  • typing/opening urls
    • easy for emulators (copy/paste)
    • not possible in mobiles - user has to provide url manually, the solution can be using phone bookmarks, short url services or sending urls to mobile using bluetooth connection
    • or we can simply create the page that contains some links
  • making screenshots for issue trackers
    • can be done in easy way on emulator
    • time consuming (require research) on real mobile
    • different screenshot tools for different mobile OS
    • tools that display mobile screen on computer screen - require connection mobile->PC
  •  request validations -eg. trackers requests
    •  in order to validate requests sent from mobile WiFi access point has to be configured to handle requests that comes from particular device
    • debug web page on real device (example: weinre)

Some concluding tips:

Emulators should definitely not be used to sign-off if you value your mobile user base. Why? The answer is quite simple : You may not spot issues that weren't discovered during the testing phase and it may really hurt when you check the site on the actual mobile device. By now, you're used to examples so here are a few more -

  • dysfunctional login functionality
  • invalid image rendering of the online store
  • broken links
  • performance issues : some BlackBerry phones don't load images on the page if they are bigger (more than 200kB)

Of course, using emulators is sometimes the only way to reach all types of mobiles : touch, smart and older phones and there is no budget to buy all of the real devices.

We recommend that emulators be used by web developers for the first couple of tests of new functionality or a new component design. From a QA perspective, they can be used for internal tests of base functionality. When there is sufficient budget - it is better to buy some of the main devices based on your analytics reports from providers such as Bango, Google Analytics or Omniture:

Mobile device statistics by Bango

Mobile device statistics by Bango, source http://bango.com

With this approach, your client and the users of those devices will be happy...for some time. Mobile market is growing so fast that sooner or later, you will have to think about using emulators again and maybe, just maybe, the emulators will evolve fast enough to be better!