The mission is a “test this” type of mission.

source: barcode.com

Application

http://www.barcodeart.com/artwork/netart/yourself/yourself.swf

Mission

Find out how the barcode is generated and report any bugs within 20 minutes.

Prerequisites

• 4 laptops
• pencils and paper for notes

The Dojo

We started with pairing and subject presentation. This time I was able to take a part in this dojo and wasn’t limited to being the organiser.

The flash application presents a question and expects an answer. User can move forward to the next question with “Next” link and move backward with clicking on “Back” link.

Until no answer is granted, “Next” link is not visible. After answering all questions, the barcode is generated based on all the questions. Application asks for:

• sex,
• age,
• nationality,
• height (in feet and inches or in centimetres),
• weight (pounds or kilograms),

The Dojo started and for next 20 minutes every teams were trying to solve the problem. Here are the conclusions that teams made:

• Sex is coded in first digit, 1 for male, 0 for female. This was pretty obvious and every team quickly reported this one,
• Age is coded in next 2 digits, directly as it was enter by user. We observe that maximum value we could enter was “99″,
• Location (or nationality) is coded on next 3 digits, the country codes were a mystery at first but as the teams noticed the codes starts from “001” value

  for the upper left flag and then continues in order. The UK flag is nineteen in a row and the code for UK is ”019”. For Poland the code is “062”,

• Height. I think this was most time consuming because most of us started with centimetres. Turns out that thinking in the “metric” system didn’t really help in solving this challenge.

  Finally, we resolved this that the height is coded in 2 digits in inches. So we enter 1 foot and 0 inches and this gave us the multiplier – “12”.

  So 5 feet and 5 inches gives up “65” inches and so is the value in barcode. Sadly, this was not so obvious for us!

• Weight. Coded in pounds on 3 digits with no mathematics. Entering “999″ gives “999″ in barcode and it was the maximum value,
• CRC – control checksum calculated based on the rest of the code. We didn’t take the challenge with this as the time ran out.

During the Dojo, we found some bugs:

• Moving back and then, hitting next unlocks next link: “Start application-sex:male-next-back-next”,
• Moving back to the start makes “location” dropdown empty: “Start app-sex:male-next-age:31-next-location:Luxemburg-back-back-next-next”,
• In height, when providing feet with no (empty) inches gives “00” in barcode. For example: 1 foot, empty inches result in “00”. 1 foot and 0 inches gives “12” barcode,
• Undefined variables – Bug has preserved its existence on screen attached at the end of this post. Going forward, backward and forward again gives “undefined” strings,

Most of the bugs are the result of one issue – the “next” link appearing.

Picture

Results table

Undefined error

Application

Light-bot, a flash game (http://www.funnygames.co.uk/light-robot.htm) – programme a little robot to “light” blue squares.

Light-bot game, first level

Mission Tasks:

1. Finish as many levels as you can.
2. Question: Does this teach you something?
3. Additional task: try to implement a recursive function that lights up and lights down the square.
4. Challenge: Try to finish the game with less than 205 commands (my result).

Time: 40 minutes

Testers: Cognifide QA Team

The Dojo

The main question asked while conducting this dojo was how such an exercise can help testers. Is it a waste of time or just a free time killer? Or will it really teach us something?

Obviously, it helps testers who develop tests because the main goal is to program the robot, but is it helpful for functional testers who don’t write code? This conversation led us to think what tasks really help us to do and our observations were:

• Task grouping – we can learn how to prepare to test, manage our time,
• Algorithm – developers create algorithms which we test
• Searching for patterns – functions are patterns, while testing we search patterns to identify and reproduce bugs,
• Developer view – we learn how to be a developer, what challenges and what problems he is facing (“you fixed it instead of refactor the code”),
• Logic – we’re trained to think logically,
• Inquisitiveness – sometimes you need to be very inquisitive to find some bugs, they hide well.

And the three main reasons to conduct such a dojo was:

• Creative thinking – testing needs creativity to be successfully conducted,
• Problem solving – as a testers, we are faced to problems which we are supposed to solve. We have to handle it.
• Building a team spirit

Summary

I was little afraid before conducting this dojo because this was not a standard event. I didn’t know how my fellow testers would react to such a (trivial at first glance) exercise. But it was OK, we went through the dojo without a hitch. Participants were engrossed by the game and concerned with finishing as many levels as they could but burdened by the time constraint, we managed to get to 10th level. Also, teams came up with an enhancement for the game – “a hot seat mode”. Here, every tester is allowed to make one move then the next one makes their move and so on.

This shows that there can be many ways to achieve one goal and most of the time, we have to choose only one. Such games would also illustrate the limits for us testers (time, knowledge, tools) and teach you how to overcome them. This, I believe could be a great example of team work. Hmm, I wonder if such an approach would result in any success. We will definitely try this out someday!

Unfortunately, on this dojo, we didn’t manage to complete all missions – implementation of recursive function were left for other time. And the homework is still to be done – 205 commands to beat. Anyone to match ?

Can you beat my result?

In my previous post, What’s a testing dojo, I described the purpose of having testing dojos and also described “how we do it” at Cognifide. Now, I’d like to share a detailed report from our exercises. So, here it goes.

The mission I’ve chosen for the first testing dojo was “Learn new approaches”. Recently, I’ve come across a couple of articles and posts on various blogs about test heuristics and thought the topic was interesting enough to be our first subject at Cognifide.

Introduction

The aim of these exercises is to gain both theoretical and practical knowledge of test heuristics. First, let’s define the term “test heuristics” and then, discuss some practical examples. Also, we will focus on usability heuristics based on Jakob Nielsen’s Ten Usability Heuristics whitepaper. Then, we will exercise usage of the test heuristics in pairs.

Application

An Adobe WEM solution – CQ – component, the Table. It allows you to create a table with the desired number of rows and cells. And also, allows you to set table’s header and footer.

CQ Table Component

CQ Table Dialog

Test Dojo Mission: Get familiar with cheat sheets. Pick up and use three heuristics. Try to use usability heuristics. Report any bugs, question and explore the product.

Time: Up to 30 minutes.

The Test Dojo:

We started with pairing, i.e. two testers collaborating on one laptop and had 4 teams working on the dojo mission. Initially, I handed a cheat sheet per person so that each pair would have some reference. And hey, it’s a cheat sheet for a reason, right? Next, we spent 5 minutes refreshing our memory about test heuristics – the training, theoretical part was conducted on previous QA Meeting a week ago. We identified the term “heuristics” as:

• Rules of thumb
• Educated guesses
• Common sense
• Problem solving

I revealed the aim and subject of exercise and all 4 teams started testing. For the next 30 minutes, all teams were discussing, questioning and exploring the product. Some of the identified heuristics and consequential bugs were quickly captured on the whiteboard but the main discussion was left for the end. The last 10 minutes were intended for a summary.

The teams identified following heuristics:

CRUD

This heuristic was used by almost all teams. Creating, Reading, Updating and Deleting was recognized as the most common and basic operation to be performed on all tested components.

Team 3, working on IE 8, found out that deleting the component doesn’t really delete all its sub items. They observed that it deleted only the main container, leaving child items intact. It turned out that if a certain row contains some data, it results in some serious errors on the published instance. The same operation performed by other teams who were using the Firefox browser revealed that the bug didn’t surface there. Instead, after deleting, the hoverbar appeared on the upper left corner of the browser. Refreshing the page (Web test navigation heuristics) resulted in the hoverbar disappearing. The teams pointed that multiple editing cells doesn’t change its value presented in component while in configuration dialog the value is proper. Problem with refreshing components content – as far as we know. Digging some deeper (testing wisdom Bug cluster) – we found out that this problem occurs in all cells except cells in first row. This row is always refreshed. Auto refreshing only first row would be some kind of consistency and standards heuristic (why only first row?). At first glance it looks like a bug.

So it seems like it was a combination of CRUD and Position heuristics. Another problem focused on the flow of issues found so far, leading us to move the component to other parsys with drag&drop. After moving, the data stored in component just disappeared. Page refreshing gave us some relief – data exists, though. Some of us questioned drag&drop capability and pointed this as a usability issue (consistency and standards) because there is no copy option available from hoverbar and Paste option is disabled.

Boundaries

A very well-known heuristic is Boundary Value Analysis. The teams identified this heuristic with the “Number of rows” and “Number of columns” properties. Exploring the boundaries revealed that for columns maximum value is 10 and for rows is 30. Entering values 11 and 31 was validated properly – suitable error message occurred. But entering “0″ value lead to component disappearing. Also entering floating point value (data type attack) here lead to Java error and strange component appearance. Another try with a very large number produced a rounding to floating point number (or maybe converting to scientific notation?). Teams also questioned field “Width” – accepting all characters (strings attack).

Usability

The configuration dialog needs to be resized to use it. Also teams pointed that correlation between controls is not clear and emphasized. It applies to, for instance, the footer tab.
The footer bar can be disabled (value set to “none”) but one can edit its text. Maybe better solution is to disable or hide rich text while footer is disabled? Some of teams complained about used language and complexity of the component pointing Aesthetic and minimalist design and Match between system and the real world heuristics in use here.
We identified also Consistency and standards usability heuristic issue. Property name “Table col stripe” should be changed to “Table column stripe” – there is already property “Number of columns” so we should stick to one naming convention.

Other issues

During the test session, some JavaScript errors have been found and performance issues risen. JavaScript error on IE8, just opened the page with component:

Webpage error details
Message: 'null' is null or not an object
Line: 857
Char: 2
Code: 0

Also, the performance issue occurs when the maximum number of columns and rows were selected. Generally, the more columns and rows, the slower the performance. On IE we get “Stop running this script” window message.

Conclusions

The aim was to gain some knowledge and the ability to name, mostly, already known and used test techniques. I believe that every team member will be more aware of activities he performs during testing. Issues named during the session have been raised in Jira and have been confronted with the project’s Tech Lead Bartosz Mordaka (text in italic font is Bartosz’s answer):

• Problem with deleting component (IE and FF) –
  

  “CQ issue. CQ does not handle such component structure that well. A fix can be provided: refresh the page after deleting component.”

• First row refreshing –“This is intentional. The first row cells are configured with product references, which are used by all cells in the column. Thus, it is needed to refresh all cells in that column after changing product reference. This is achieved by refreshing whole page (refreshing particular component is not trivial code-wise). For all other cells, it is enough to refresh the edited cell only, so no need to refresh whole page.” Won’t fix – it seems that we didn’t even scratch the surface here

• Not changing value –
  

  “CQ issue, already spotted earlier on other components as well. As this happens on IE authoring, and after manual page refresh fixes the issue, the workaround of automatic page refresh after editing is rather an overhead and is not suggested approach.”, Won’t fix

• Moving component to other parsys makes data disappear –
  

  “This happens due to complex structure of component – the cells, which don’t preserve data when moving to
  other parsys are actually dynamically generated but as FIXed component – thus making CQ hard time handling this while copying. A fix for that would be to refresh the page after copying/dropping the component.”

• Drag&Drop –
  

  “The “New…” and “Paste” actions come together and cannot be separated. So it’s either removing them both, or adding “Copy” and “Cut”. In this particular case, “New…” is required as of client’s request, so let’s add “Copy” and “Cut” (they can be safely added I guess, and component is supposed to be rather wideone).
  The thing with “New…” is also this: it is required in order to give ability to drop a component just before/after a component with the “New…” action. However, there is possibility to remove “New…” action from hoverbar, but preserve in context menu. So if the component is too narrow to have all mentioned actions, and then let’s remove New…, Paste, Cut, Copy and drag and dropping will still be possible.”

• Rows and columns accepting too many –
  

  “Server-side validation should restrict number to be 1-30 and no floating point numbers. As for client-side validation (dialog) let’s incorporate a standard validator, but live with its issue”

• Rows and columns accepts very big number and then converts it somehow –
  

  “Built in validators in dialogs are not perfect.. Although we stick to the rule that in such cases authors are responsible for putting reasonable values into dialog fields, this is planned to be dealt with as part of CQ Practice”

• Correlation between controls –
  

  “This is a nice idea, and has been spotted to be investigated in CQ Practice as part of CQP-92, which is custom implementation of mechanisms to manage dynamic field hiding/disabling etc. Once it’s done and approved, will be incorporated into our guidelines, but not ready at the moment.”

• Configuration dialog needs to be resized –
  

  “For sake of better authoring experience, let’s extract the last properties group from Header tab into separate tab, and increase the size of dialog as well.”

• Performance issue –
  

  “Not proud to say that this proves CQ weakness under IE. 300 components is too much for client-side interface but nothing really we can do about it at the moment.” Won’t fix,

• Resource error on publish – crème de la crème, the most important issue that we have found – making testing wisdom completely true – big bugs are often found by coincidence. Entering data into hanging rows after deletion of component produces some chaos in database and results in errors on publish. Maybe this is the ghost issue that has been poisoning life of our developers for some time especially in CQ Aftercare. Hard bugs but simple solution – page refresh after deletion will prevent this bug from emerging. But if it is already present – only digging in database can fix it.

So, if it would be all about numbers, we’ve found 15 issue whereof 6 issues have been marked as Won’t fix (2 of them are in CQ Practice already), 9 issues have been or will be fixed. But as we all know it’s not all about the numbers.

Pictures

TestHeuristicsWhiteBoard1

Testing Dojo Whiteboard

References

1. Test Heuristics Cheat Sheet – http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
2. http://en.wikipedia.org/wiki/Heuristics

In an ideal world, the QA team should have a security specialist. But what if this isn’t always possible? Well, even if we can’t do something perfectly (due to the lack of specialized knowledge), let’s at least do what we can. Anyone can really perform a basic security audit that will detect and eliminate so called “low-hanging fruits” (I’m referring to obvious security holes that could be exploited by an inexperienced attacker).

Even if you have absolutely no knowledge of security, you can use a wide range of tools available on the market that will help you scan for some potential problems.Of course, to use a tool however it would be good to have some basic knowledge so we know what we’re actually testing.

For starters, it may be a good idea to visit the OWASP Top 10 Project  that lists top security vulnerabilities for the previous year. Chances are that not all of them will be applicable to your project (i.e. if your website does not use any database then you are definitely not vulnerable to SQL injection).

Let’s go a step further and try to run a test using one of tools mentioned in the OWASP Top 10 Projects list. At Cognifide, our team uses both Netsparker and Acunetix to test against XSS holes and Wapiti to check for SQL injection issues. Both  have nice, user-friendly interfaces, but can only be run on the Windows system. Wapiti, on the other hand is cross-platform and can be run from the console which is quite important if you want to be running your tests using continuous integration tool (like Jenkins or CruiseControl).

Starting a scan in Netsparker is pretty straightforward and doesn’t require much explanation. After opening the program, the  “Start a New Scan” window appears automatically. If it doesn’t, press Alt+N or click “Start New Scan button” on the top left of the window. Generally, you can leave all the settings as they are. The only thing you should do is to make sure that a proper database is set in the “Advanced Settings” tab. In case that website requires authentication or if clicking some kind of disclaimer is necessary, one needs to set the cookie manually. You can do it in “Authentication” tab in “Custom Cookies” field. Simply type cookieName1=value1; cookieName2=value2;… etc. The report is very clear and you should not encounter any problems with interpreting results.

Now, let’s now move on to Acunetix. After you run the program, the “Scan Wizard” window should appear. If it doesn’t simply click “New Scan” button which can be found in the toolbar on the top. The wizard has a few steps, but again, they don’t require much description.

1. Scan type -> Scan single website -> provide base url of a website to be scanned
2. Select targers -> optimize for following technologies -> check options that apply for your project
3. Crawling Options -> leave all settings as they are (default)
4. Scan options -> Scanning mode -> I recommend to set it to “Extensive”, but this will result in greater amount of scanning time. Therefore “Heuristic” mode can be better in some situations.
5. Login – if site requires login or clicking on some kind of disclaimer click “Record Login Sequence” button. New wizar will appear, follow the steps to record login sequence. A new browser window will open within the program and you will be allowed to record all the necessary actions.

After completing all steps in the wizard, the scan will start automatically. Acunetix reports are rather vague but it’s still more readable than those from some open source tools. The report contains both exploited url and attack string. However, note that they are in different sections so in order to get the whole picture it’s advisable to look at HTTP Request Headers. Look at the picture below to get a better idea.

Example of Acunetix report

And  last but not least: Wapiti. Unfortunately, it does not have a user friendly GUI, but running a scan from console is very simple. You just need to type “wapiti.py http://your.site”. After the scan is done, report can be found in generated_report/index.html file.

As you can see, using vulnerability scanner is pretty easy and it can save you a lot of trouble in the future! Just remember: it’s only an automated scan and although it’s better than nothing, it cannot replace a professional audit!

If you have any questions on how to use the tools, please share the question so that everyone who shares the same thoughts, can benefit.