TestToTester

How do you measure and appraise testers?

- Do you use a tool to measure testers?
- Do you measure them based on the number of bugs they find?
- Do you measure them by the number of valid (acknowledged by development/product/ team) bugs they find?
- Do you measure them based on the x% variance with schedule for a testing effort?
- Do you measure them based on the x% adherence to configuration management practices?
- Do you measure them based on the Ratio of defects found during testing (vs) defects found post release?
- Do you measure them based on the x% system test coverage of the assigned functionality as measured through Reviews and Requirement Traceability Matrix?
- Do you measure them based on the Test Environment Utilization Rate?
- Do you measure testers based on how testers improve development teams?(http://blogs.msdn.com/james_whittaker/archive/2008/07/22/measuring-testers.aspx)
- etc

For all those who strive by metrics and incentive plans based on metrics read this fantastic article “Employees will always game incentive plans -- because the geniuses who design them don't anticipate how employees will respond” By Joes Spolsky

I thank Michael Bolton for sharing the above link by Joel in a discussion at Test Republic.

Right now I do not have “the/a” answer for the question of this post.

But, isn’t it also our responsibility to not persist/continue in such a blind system.


Further reading:
- Software Engineering Metrics: What Do They Measure and How Do We Know? by Cem Kaner
- Side Effects of Metrics/Statistics by Shrini Kulkarni
- Remuneration and Punished by Rewards by Jonathan Kohl
- Don’t Use Bug Counts to Measure Testers by Cem Kaner
- The Darker Side of Metrics by Douglas Hoffman

Disclaimer: All the blogs shared by me are my ideas, my thought, my understanding of the subject and does not represent any of my employer’s ideas, thought, plans or strategies.

Testing Interview Questions

Couple of month’s back I was invited for an interview at one of the big, reputed Indian service based company in Bangalore. This interview turned out to be an eye-opener for me at the state of testing in India.

Let me start with the conversation I had with HR of the company.

Me: When is the interview scheduled?
HR: 12:30 PM
Me: Could you please re-schedule it? Since it is a lunch time and I do not want the interviewers to be hungry.
HR: HaHaHa… no Sharath our interviewers work in shifts, so there will be no such issues.
Me: Ok then, I shall meet you at the location. Thanks.

Date of Interview

12:20 PM I submitted all the required documents to HR, at the interview location
12:50 PM HR informed me that the interview panel will be back after lunch by 2:00 PM.

A lesson taught by the mighty organization – “don’t show up on time for the interview” or “we do not value other’s time”

At around 2:10 PM I am escorted to the interview room by HR.

What followed was an enlightening conversation between me and the 2 interviewers (one of them over tele-con) in the room.

Interviewer 1: When do you do performance testing?
Me: Well, it depends on the context.
Interviewer 2: How can it depend on the context?
Me: If my mission it to test a program stores a minimum x number of packets per minute to help management take a decision on investing development effort - I shall be performance testing early.
If product development team seeks help to test the database for a certain number of transactions in the middle of a project to avoid bottlenecks towards the release – I would be performance testing in the middle of a product time line.
If my mission demands me to record the response time after a specific function – I shall performance test after functional testing.
Interviewer 1: Yes, I understand, but when do you normally do performance testing?
Me: :-)

Interviewer 1: Have you seen a use case?
Me: Hmm… yes, sir
Interviewer 1: How many test cases can you write from a use case?
Me: 0 to n
Interviewer 2: What?
Me: Yes, 0 to n, if a tester does not understand the use case – he might develop 0 test cases, if a tester understands the use case – he might develop “n” number of test cases.
Interviewer 1: Can you think of an average number?
Me: :-)

Interviewer 2: How do you determine the complexity of a test case?
Me: I do not know. I feel complexity of a module/feature under test is more important. Test cases developed for complex module is more important to me, when I compare with test cases developed for a comparatively simple module/feature.
Interviewer 2: Does not number of execution steps in a test case help you identify a complex test case?
Me::-), What if a logo is displayed across 10 different pages. A test case is developed to verify the result at 10 different pages. This test case shall have more steps when compared to other test cases, so does it make the test case more complex?
Interviewer 2: So, how do you know which test case is complex?
Me: :-)

Interviewer 1: Why do you always refer to context? I do not understand how testing is different from any other like development?
Me: Let’s consider an example: Test a program which add integers and returns a value. What will be your test approach? How many test cases will you have to test this?
- What if this program is used to display the number of sms’s in the inbox of a mobile application? Does your test strategy change? Or will it be the same?
- What if this program is used in a ICICI ATM machine? Does your test strategy change? Will you add more test cases? Or will it be the same?
- What if this program is used in a corrective laser eye surgery? Does your test strategy change? Will you add more test cases? Or will it be the same?
- What if this program is used to control a Dam (may be KRS)? Does your test strategy change? Will you add more test cases? Or will it be the same?
Interviewer 1: :-(

Interviewer 1: You use different testing terminologies like para-functional etc, which our customers will not be able to understand.
Me: :-)

Interviewer 2: We always refer Stability in Performance Testing, but you tend to use it for functional as well.
Me: :-)

Well, I was not offered and I acknowledge their decision. May be I did not meet their requirements, or may be I miss-understood their questions or may be my answers were pathetic or may be my attitude wasn’t right or may be....

But, I personally felt the interview was useless. I felt the two interviewers have never tested or understood testing or might have never looked beyond the company process and their campus walls. I might be wrong, but my oracles tell me it was “useless”. I request you all to help me by answering the questions put forward to me by the interviewers.

Further Reading: The (bad) state of Software Testing interviews in India by Pradeep Soundararajan

Disclaimer: All the blogs shared by me are my ideas, my thought, my understanding of the subject and does not represent any of my employer’s ideas, thought, plans or strategies.

Test with me @

Tweets