A common question that comes up when I conduct our proprietary path breaking testing training program MOST (Mind of a Software Tester) is how to estimate the number of test iterations. In my view, a good way to do that is to compute bug insertion rate and bug fix rate by the development team. Once this is done, you can easily estimate number of iterations to test.
Let me give you an example here;
You have been asked to estimate the number of more test iterations required. You are at the end of round one. You can find the usual bug insert rate by developers in your organization when they fix bugs. As well as find the usual bug fix rate (number of bugs that usually get fixed when you report 100 bugs). Thus in your organization if the bug fix rate is 50% and the bug insert rate is 10% (it is usually not this high), then this is how you shall calculate. Let us assume that the number of open bugs today at the end of iteration one is 100. Thus in round two, 50 bugs shall be open and 10 more shall be introduced. Thus, at the end of round two, you shall have 60 bugs. In round three, you shall have 30 bugs fixed and 6 introduced. Thus you shall be left with 36 bugs. Keep doing this calculation till you arrive at zero or one bug. That shall tell you the number of iterations.
Consider the following as well:
- You may be asked to estimate number of iterations at the beginning of the project and not at the end of round one. In that case, you shall estimate the number of bugs at the end of round one and perform the above calculation.
- Consider finding averages in your organization for different rounds. It is possible that your average bug fix and insert rates are higher in the initial rounds.
- You could apply further math to this basic idea and tailor this to your organization. For instance, Lloyd Raden from Grove Consultants recommends you to use nested rate as well.
- Note that this is not a pure statistical way. But, in our experience we have found it simple and practical to come up with an estimate on number of iterations than use SWAG (Scientific Wild Ass Guess).
My team is currently running a poll on LinkedIn to gauge how other people out there are going about test size estimation and I am due to publish a whitepaper on this topic shortly. I would like to welcome all of you to participate in this discussion here: http://linkd.in/rsty6o
Let me know your views
Krishna Iyer | CEO | Zen Test Labs