Testing Dojo – Test Heuristics
In my previous post, What’s a testing dojo, I described the purpose of having testing dojos and also described “how we do it” at Cognifide. Now, I’d like to share a detailed report from our exercises. So, here it goes.
The mission I’ve chosen for the first testing dojo was “Learn new approaches”. Recently, I’ve come across a couple of articles and posts on various blogs about test heuristics and thought the topic was interesting enough to be our first subject at Cognifide.
The aim of these exercises is to gain both theoretical and practical knowledge of test heuristics. First, let’s define the term “test heuristics” and then, discuss some practical examples. Also, we will focus on usability heuristics based on Jakob Nielsen’s Ten Usability Heuristics whitepaper. Then, we will exercise usage of the test heuristics in pairs.
An Adobe WEM solution – CQ – component, the Table. It allows you to create a table with the desired number of rows and cells. And also, allows you to set table’s header and footer.
Test Dojo Mission: Get familiar with cheat sheets. Pick up and use three heuristics. Try to use usability heuristics. Report any bugs, question and explore the product.
Time: Up to 30 minutes.
The Test Dojo:
We started with pairing, i.e. two testers collaborating on one laptop and had 4 teams working on the dojo mission. Initially, I handed a cheat sheet per person so that each pair would have some reference. And hey, it’s a cheat sheet for a reason, right? Next, we spent 5 minutes refreshing our memory about test heuristics – the training, theoretical part was conducted on previous QA Meeting a week ago. We identified the term “heuristics” as:
- Rules of thumb
- Educated guesses
- Common sense
- Problem solving
I revealed the aim and subject of exercise and all 4 teams started testing. For the next 30 minutes, all teams were discussing, questioning and exploring the product. Some of the identified heuristics and consequential bugs were quickly captured on the whiteboard but the main discussion was left for the end. The last 10 minutes were intended for a summary.
The teams identified following heuristics:
This heuristic was used by almost all teams. Creating, Reading, Updating and Deleting was recognized as the most common and basic operation to be performed on all tested components.
Team 3, working on IE 8, found out that deleting the component doesn’t really delete all its sub items. They observed that it deleted only the main container, leaving child items intact. It turned out that if a certain row contains some data, it results in some serious errors on the published instance. The same operation performed by other teams who were using the Firefox browser revealed that the bug didn’t surface there. Instead, after deleting, the hoverbar appeared on the upper left corner of the browser. Refreshing the page (Web test navigation heuristics) resulted in the hoverbar disappearing. The teams pointed that multiple editing cells doesn’t change its value presented in component while in configuration dialog the value is proper. Problem with refreshing components content – as far as we know. Digging some deeper (testing wisdom Bug cluster) – we found out that this problem occurs in all cells except cells in first row. This row is always refreshed. Auto refreshing only first row would be some kind of consistency and standards heuristic (why only first row?). At first glance it looks like a bug.
So it seems like it was a combination of CRUD and Position heuristics. Another problem focused on the flow of issues found so far, leading us to move the component to other parsys with drag&drop. After moving, the data stored in component just disappeared. Page refreshing gave us some relief – data exists, though. Some of us questioned drag&drop capability and pointed this as a usability issue (consistency and standards) because there is no copy option available from hoverbar and Paste option is disabled.
A very well-known heuristic is Boundary Value Analysis. The teams identified this heuristic with the “Number of rows” and “Number of columns” properties. Exploring the boundaries revealed that for columns maximum value is 10 and for rows is 30. Entering values 11 and 31 was validated properly – suitable error message occurred. But entering “0″ value lead to component disappearing. Also entering floating point value (data type attack) here lead to Java error and strange component appearance. Another try with a very large number produced a rounding to floating point number (or maybe converting to scientific notation?). Teams also questioned field “Width” – accepting all characters (strings attack).
The configuration dialog needs to be resized to use it. Also teams pointed that correlation between controls is not clear and emphasized. It applies to, for instance, the footer tab.
The footer bar can be disabled (value set to “none”) but one can edit its text. Maybe better solution is to disable or hide rich text while footer is disabled? Some of teams complained about used language and complexity of the component pointing Aesthetic and minimalist design and Match between system and the real world heuristics in use here.
We identified also Consistency and standards usability heuristic issue. Property name “Table col stripe” should be changed to “Table column stripe” – there is already property “Number of columns” so we should stick to one naming convention.
Webpage error details Message: 'null' is null or not an object Line: 857 Char: 2 Code: 0
Also, the performance issue occurs when the maximum number of columns and rows were selected. Generally, the more columns and rows, the slower the performance. On IE we get “Stop running this script” window message.
The aim was to gain some knowledge and the ability to name, mostly, already known and used test techniques. I believe that every team member will be more aware of activities he performs during testing. Issues named during the session have been raised in Jira and have been confronted with the project’s Tech Lead Bartosz Mordaka (text in italic font is Bartosz’s answer):
- Problem with deleting component (IE and FF) –
“CQ issue. CQ does not handle such component structure that well. A fix can be provided: refresh the page after deleting component.”
- First row refreshing –“This is intentional. The first row cells are configured with product references, which are used by all cells in the column. Thus, it is needed to refresh all cells in that column after changing product reference. This is achieved by refreshing whole page (refreshing particular component is not trivial code-wise). For all other cells, it is enough to refresh the edited cell only, so no need to refresh whole page.” Won’t fix – it seems that we didn’t even scratch the surface here
- Not changing value –
“CQ issue, already spotted earlier on other components as well. As this happens on IE authoring, and after manual page refresh fixes the issue, the workaround of automatic page refresh after editing is rather an overhead and is not suggested approach.”, Won’t fix
- Moving component to other parsys makes data disappear –
“This happens due to complex structure of component – the cells, which don’t preserve data when moving to
other parsys are actually dynamically generated but as FIXed component – thus making CQ hard time handling this while copying. A fix for that would be to refresh the page after copying/dropping the component.”
- Drag&Drop –
“The “New…” and “Paste” actions come together and cannot be separated. So it’s either removing them both, or adding “Copy” and “Cut”. In this particular case, “New…” is required as of client’s request, so let’s add “Copy” and “Cut” (they can be safely added I guess, and component is supposed to be rather wideone).
The thing with “New…” is also this: it is required in order to give ability to drop a component just before/after a component with the “New…” action. However, there is possibility to remove “New…” action from hoverbar, but preserve in context menu. So if the component is too narrow to have all mentioned actions, and then let’s remove New…, Paste, Cut, Copy and drag and dropping will still be possible.”
- Rows and columns accepting too many –
“Server-side validation should restrict number to be 1-30 and no floating point numbers. As for client-side validation (dialog) let’s incorporate a standard validator, but live with its issue”
- Rows and columns accepts very big number and then converts it somehow –
“Built in validators in dialogs are not perfect.. Although we stick to the rule that in such cases authors are responsible for putting reasonable values into dialog fields, this is planned to be dealt with as part of CQ Practice”
- Correlation between controls –
“This is a nice idea, and has been spotted to be investigated in CQ Practice as part of CQP-92, which is custom implementation of mechanisms to manage dynamic field hiding/disabling etc. Once it’s done and approved, will be incorporated into our guidelines, but not ready at the moment.”
- Configuration dialog needs to be resized –
“For sake of better authoring experience, let’s extract the last properties group from Header tab into separate tab, and increase the size of dialog as well.”
- Performance issue –
“Not proud to say that this proves CQ weakness under IE. 300 components is too much for client-side interface but nothing really we can do about it at the moment.” Won’t fix,
- Resource error on publish – crème de la crème, the most important issue that we have found – making testing wisdom completely true – big bugs are often found by coincidence. Entering data into hanging rows after deletion of component produces some chaos in database and results in errors on publish. Maybe this is the ghost issue that has been poisoning life of our developers for some time especially in CQ Aftercare. Hard bugs but simple solution – page refresh after deletion will prevent this bug from emerging. But if it is already present – only digging in database can fix it.
So, if it would be all about numbers, we’ve found 15 issue whereof 6 issues have been marked as Won’t fix (2 of them are in CQ Practice already), 9 issues have been or will be fixed. But as we all know it’s not all about the numbers.
- Test Heuristics Cheat Sheet – http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf