One of the questions I find hard to answer when I use exploratory test techniques in my project is,”How do you run regression tests? How do you prioritise them? Do you automate them? Do you document regression test cases?” And then a bunch of questions follow.
Let me define the context. I work closely with developers; we use either Scrum or Kanban - which ever best suits our project, product owner, business. We do not believe in individual metrics, blame games, stringent processes, and heavy documentation. All we care about is to deliver high quality software. This does not mean that there is no process, metrics or documentation. We have them all, but they are there to meet our needs and to help those who will come onboard to support/maintain the software.
At the start of a story, testers pair up with developers to develop feature files using SpecFlow. Some call it Acceptance Test Driven Development and others Behaviour Driven Development [I am yet to figure out the difference between the two] This exercise helps us think better and ask more questions to the product owner and business people. It also bridges the gap between testers and developers in understanding requirements and developing test ideas. To me, this is not just an activity of developing feature files, it’s that phase when testers and developers together learn a lot from one another. Developers start to appreciate test ideas from a tester, his/her thoughts, strategies etc. Testers learn a lot technically about design, logic, memory management and tons of other stuff. I strongly believe that this pairing up helps in bug prevention as well. These feature files developed in Gherkin Syntax are now binded to the code a layer below UI and checked in. The team can decide if these tests have to be run automatically every time a build is made or manually as needed.
This is one of my answers to regression checks. Its brilliant, isn’t it? All tests are automated at the start of a story. Even before a code of line is developed? It’s great, but it does come at a cost – what these tests can do is checking and not testing. Also it takes time for both developers and testers to learn to develop feature files together. We did fall into trap of developing bulky, long and hard to maintain feature files initially. However, we put in a lot of effort to make it light, maintainable and stable.
To me regression testing is regression checks & regression testing. I am not trying to bring in my own terminology here. I am just extending the idea from Michael Bolton ‘Testing v/s Checking’. With feature files we now have automated regression checks. I think it’s very important for a team to recognize the value in anything they have invested their time on. We discern the limitations of feature files and the value it brings. So we started documenting regression test ideas using mind maps.
We develop checklists in mind maps before we move a story to “done”. We call it ‘regression map’. As I write this, I think we need to have a better name for this document. Doesn’t the word ‘regression’ means decay, weak? A bit of investigation might help here.For now, let me stick to ‘regression maps’.
Regression maps can have one liners, couple words, screen shots etc which aid memorability when one looks at it. The regression map also includes paths, tests, ideas, risks and heuristics documented when we move a story to “done”. All this, when it’s fresh in our minds. At the end of a sprint or prior to a demo or when we need to run regression tests quickly, we refer to regression maps to drive our time boxed exploratory sessions.
This simple exercise of recording test ideas in a mind map helps us focus on critical tests and requirements when exploring a feature for regression. It is also the easiest document I have ever maintained. I do not want to get into more detail on how we report or maintain versioning among testers here. In addition to aiding regressing testing, regression maps help in “show and tell” demos. It’s also a great document to pass to support/maintenance team during handover.
With the regression checks and regression testing in place, our team now has a convincing story to share with our management. There is still a long way to go, however I think this is a good first step to move forward. What do you think?
Further Reading: http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/