Oracle deficiency?


We were in the middle of a release. It was late and we had just found a bug, a show stopper for a feature. Ben then said I always wonder - what separates a good tester from an average tester? at most times I feel it's their ability to spot the important bugs at the right time. 

These words have struck with me ever since. And this happened at another site.

I observed "attack-name": "HEADER_COUNT_EXCEEDED" in an API response when I used postman to poke some of the APIs built. The error was not consistent but I just did not feel right about this. Checked with the team (lets call them teamA) testing the APIs and it appears they had never seen it before. So I checked with another team (lets call this teamB) who were consuming the APIs but building something different. It was the same response. I was aware that both teams used J-meter to test APIs and I was poking it via postman. Could that be a problem?

I built a small collection to iterate the API calls to check if its reproducible. I had 6 failures of the 10 calls. When I shared this observation with TeamA again. The recommendation was to use J-meter instead of Postman! Really? Am I the only one seeing a problem here?

Pinged my developer and asked if she was willing to pair/investigate and her reply was exactly what I was looking for - oh yea one of our remote devs spotted this yesterday but because it was not consistent we thought it was a transient issue. I love such words 'transient', 'buggered', etc it always to me mean there is something which cannot be explained logically and so could be a potential bug.

I showed her my collection and she was convinced to dig in. 15 min into tracing the logs we found out that there is a new rule set on WAF to block requests with more than X headers. Postman with accept version was sending X+1 headers and without the accept version, it was falling just under the WAF configured rule (X). This also helped us understand why J-meter never picked it up. From J-Meter, there was only X-2 headers sent, including the accept-version.

When we took this to the architect we learnt that a lot of headers passed are informational, but not essential. So a decision was made to strip out headers that are not required (except for what is required by WAF, audit and logging)

This observation could had been a potential show stopper!

Looking back at this I understand why the two teams did not spot it. It could very much be down to both using only J-meter to test APIs. But what really surprised me was the complete lack of - could that be a problem? This makes me wonder if its because of the lack of oracles?

Another learning for me from this was that - this bug could have not been picked at teams/companies who thrive for uniformity in their toolset, frameworks, processes, etc

Comments

Post a Comment

Popular posts from this blog

Exploratory testing, Session based testing, Scripted testing…concertedly

In my last assignment a mix of exploratory, session based and scripted testing helped won an award, appreciation by managers & stake holders at our organization. The quotes from our award certificate “ QA team has come up with some intelligent testing techniques to validate the product. The customer test team was not able to find even a single critical bug after delivery ” Though the award was a small one in terms of the award categories our organization have, it meant a lot to me and the team, because this was the first time the team used exploratory approach. The assignment was not pure exploratory or session based, in fact it was supposed to be scripted, but a mix of all three exploratory, session based and scripted testing helped us learn, enjoy, and meet our mission. The assignment started off with developing test cases. We wrote test cases based on the initial requirement document provided. We did come up with a large number of test cases, of which most were re-written, some
Read more

What do you do when you find a bug?

What do you do when you find a bug? Hold on do not answer it right now. Let me first set the context. You are part of an agile team. The developers sit opposite to you. You are testing a feature in a session. You are time bound and have the good habit of capturing notes when you test. When you now find a bug what do you do? Assuming that you have taken enough notes you go on to write a good bug description and may capture a screen shot, video of the bug and continue with your session. At the end of the session do you Open the defect tracking tool (wait for it to load), enter all the fields for a bug, upload necessary evidence and have the tool share the defects with your dev team? Or Open the defect tracking tool (wait for it to load), enter all fields for a bug, upload necessary evidence and then compose an email to the dev team with the bug id’s/descriptions? Or Walk to the developer share your notes; reproduce it if required, work together to identify the problem? I work in
Read more

Regression Checks + Regression testing = Regression testing?!

One of the questions I find hard to answer when I use exploratory test techniques in my project is,”How do you run regression tests? How do you prioritise them? Do you automate them? Do you document regression test cases?” And then a bunch of questions follow. Let me define the context. I work closely with developers; we use either Scrum or Kanban - which ever best suits our project, product owner, business. We do not believe in individual metrics, blame games, stringent processes, and heavy documentation. All we care about is to deliver high quality software. This does not mean that there is no process, metrics or documentation. We have them all, but they are there to meet our needs and to help those who will come onboard to support/maintain the software. Regression Checks At the start of a story, testers pair up with developers to develop feature files using SpecFlow. Some call it Acceptance Test Driven Development and others Behaviour Driven Development [I am yet to figure out the d
Read more