Saturday, January 28, 2012

Regression Checks + Regression testing = Regression testing?!

One of the questions I find hard to answer when I use exploratory test techniques in my project is,”How do you run regression tests? How do you prioritise them? Do you automate them? Do you document regression test cases?” And then a bunch of questions follow.

Let me define the context. I work closely with developers; we use either Scrum or Kanban - which ever best suits our project, product owner, business. We do not believe in individual metrics, blame games, stringent processes, and heavy documentation. All we care about is to deliver high quality software. This does not mean that there is no process, metrics or documentation. We have them all, but they are there to meet our needs and to help those who will come onboard to support/maintain the software.

Regression Checks

At the start of a story, testers pair up with developers to develop feature files using SpecFlow. Some call it Acceptance Test Driven Development and others Behaviour Driven Development [I am yet to figure out the difference between the two] This exercise helps us think better and ask more questions to the product owner and business people. It also bridges the gap between testers and developers in understanding requirements and developing test ideas. To me, this is not just an activity of developing feature files, it’s that phase when testers and developers together learn a lot from one another. Developers start to appreciate test ideas from a tester, his/her thoughts, strategies etc. Testers learn a lot technically about design, logic, memory management and tons of other stuff. I strongly believe that this pairing up helps in bug prevention as well. These feature files developed in Gherkin Syntax are now binded to the code a layer below UI and checked in. The team can decide if these tests have to be run automatically every time a build is made or manually as needed.

This is one of my answers to regression checks. Its brilliant, isn’t it? All tests are automated at the start of a story. Even before a code of line is developed? It’s great, but it does come at a cost – what these tests can do is checking and not testing. Also it takes time for both developers and testers to learn to develop feature files together. We did fall into trap of developing bulky, long and hard to maintain feature files initially. However, we put in a lot of effort to make it light, maintainable and stable.

To me regression testing is regression checks & regression testing. I am not trying to bring in my own terminology here. I am just extending the idea from Michael Bolton ‘Testing v/s Checking’. With feature files we now have automated regression checks. I think it’s very important for a team to recognize the value in anything they have invested their time on. We discern the limitations of feature files and the value it brings. So we started documenting regression test ideas using mind maps.


Regression Testing

We develop checklists in mind maps before we move a story to “done”. We call it ‘regression map’. As I write this, I think we need to have a better name for this document. Doesn’t the word ‘regression’ means decay, weak? A bit of investigation might help here.For now, let me stick to ‘regression maps’.

Regression maps can have one liners, couple words, screen shots etc which aid memorability when one looks at it. The regression map also includes paths, tests, ideas, risks and heuristics documented when we move a story to “done”. All this, when it’s fresh in our minds. At the end of a sprint or prior to a demo or when we need to run regression tests quickly, we refer to regression maps to drive our time boxed exploratory sessions.

This simple exercise of recording test ideas in a mind map helps us focus on critical tests and requirements when exploring a feature for regression. It is also the easiest document I have ever maintained. I do not want to get into more detail on how we report or maintain versioning among testers here. In addition to aiding regressing testing, regression maps help in “show and tell” demos. It’s also a great document to pass to support/maintenance team during handover.

With the regression checks and regression testing in place, our team now has a convincing story to share with our management. There is still a long way to go, however I think this is a good first step to move forward. What do you think?

Further Reading: http://www.developsense.com/blog/2009/11/merely-checking-or-merely-testing/

12 comments:

  1. I think you're in the right way.
    I like the way Testers and Developers work together. I think that's the hardest step to give.
    Good article.

    ReplyDelete
  2. Regression map is something new and i like the idea, we too use mind maps but we are using to develop scenarios. could you please share any of the regression mind map so that i could try it in my project.

    ReplyDelete
  3. Thanks for the comments Luisa. Developers and testers working together can be magical :) I don't think it's as hard as we feel.

    @Pradeep When you say you use mind maps to develop scenarios, what do you mean by it? I do not have a dummy regression map right now & I cannot share the project one with you because of NDA. I will work on a dummy one soon.

    ReplyDelete
  4. We follow scrum methodology, we start picking from product backlog and so for every new functionality we identify possible alternatives that will interact with other functionalities so to make sure it doesn't create any breakages. Our mind map is kind of a checklist so to make things easy for us. We just implemented it so please let me know if I could use it in much better way.

    ReplyDelete
  5. Sharath, Very true about the regression checks concept. I have used mindmaps in various contexts. I have used it like a coverage tool personally which can tell me where am I really in my testing activity. I have created mindmaps showcasing the workflow of features and this could become training material or to demo it to someone in a visual manner.

    In one of the project we also made a mind-map of people involved in the project and shared it with all the team members. This list had their department name, person name, e-mail address, telephone number. So, it could be handy to get in touch with different teams and get things resolved quickly in case of any problems or any help required.

    I liked the idea of testers and developers working closely. Feature files was a good one.

    I also want to add this - I have been thinking of observation notes during SBTM acting as a regression checks too. Example: There was a different shade of color used for specific block and in next version it changed. So, these minute observations could be of help if they are recorded as observation notes in the session sheets.

    Instead of observation with respect to those kind of minute things with respect to GUI, I would use image compare as one of the solution.

    I would write how image compare check automation could add value to your project in a specific context in my upcoming blog post.

    Thanks for the nice blog post and looking forward for series of blog posts on this topic.

    -- Santhosh Tuppad

    ReplyDelete
  6. @ Pradeep - If I have understood you right you now use mind maps to document the flows, interaction between the modules right? That is very clever.

    @Santosh - mind map for people and their roles - Nice

    When you say ‘I have been thinking of observation notes during SBTM acting as a regression checks too.’ - will this not put burden on the tester running the test? he now will have to document a lot of stuff and in a way every one understands it. To me making notes when I am testing is for my reference. I use a lot of shortened names, typos, etc.

    Observation notes: Yes, you are right when observation notes can help us find a lot of problems. We move our observation notes from session sheet to mind maps and bind it to the story so we don’t miss it when we have to regression test.

    I have come across image comparison tools before. It is a good idea. But for our project there is not much UI :(. I hope to use it in other projects soon.

    ReplyDelete
  7. Yes, I use it to cover flows and in during exploratiry testing.I would be much obliged if you could share any template for regression maps.

    ReplyDelete
  8. @ Sharath,

    I call it "consistency checks" rather than regression checks. The consistency checks help us learn how different the current build is for the same tests that were also run on a previous build. My idea of these checks are automated ones.

    So after this comes the exploration part where we go off track, test for coverage for changes made, explore risks based on changes made to say, "Looks like this is how it works for this build"

    We both seem to be doing the same and calling it by different names, which is perfectly fine, as long as we recognize what we are doing and what value it is adding.

    Please write more and often and as good as this one.

    ReplyDelete
  9. @Pradeep Kota: I will email you one shortly

    @Pradeep Soundararajan - Thanks for the comment

    ReplyDelete
  10. Hi Sharath, Help me understand the topic, the title says
    Regression Checks + Regression Testing = Regression Testing??
    like X + Y = Y ??
    In my view regression is always checking only not testing as we check the existing functionality If they are still intact and not affected by the new code deployment. So most the old test sets were run either manually or automatically. There is less scope of any exploratory testing during regression as the functionality under test is already old and known.

    ReplyDelete
  11. @Prasanna - Like your question on the title of the post. I do not see + as a mathematical operator. In this context it means checking and testing should be part of regression testing.

    It's interesting when you say Regression testing is only checking. If that is case why do you call it Regression testing? and not regression checking? Secondly if you want to limit yourself to just checking how do you plan to overcome 'pesticide paradox'? Third point have you never found a performance or usability or security or data break when regression testing?

    ReplyDelete
  12. Comments in my own post: http://dancedwiththetester.blogspot.com/2012/03/re-regression-checks-regression-testing.html

    ReplyDelete