Couple of weeks back we held our first crowd sourced testing event “the Zentestathon” which turned out to be a memorable event. Testing was on a module of a LMS application. Like all things that happen for the first time there were good results and valuable experiences that we experienced and I wanted to share some things that could help you with your crowd sourcing strategy.
To ensure that the focus remains on testing and getting more number of defects, we discovered that maintaining a tight schedule is extremely critical to the success of the program. The schedule break up given below is what we found generating the optimum results
- Application training-10% of the time
- Exploring the application by the testers-15% of the time
- Understanding business rules and requirements- 10% of the time
- Testing-60% of the time
Some other observations include
- We realized that, what most testers were skilled at was context driven testing.
- Testers were able to suit themselves to the application and think of end to end scenarios with minimum amount of guidance
- They add tremendous value through their experience, mentality and the ability to transcend based on this application.
- In an extremely short time, they came up with various combinations of valid and invalid scenarios, which ensured test coverage.
- Percentage of critical defects resulted in 40% by the virtue that there were so many testers testing different scenarios, optimum test coverage was guaranteed.
- The more the merrier is something that goes well with this form of testing as scenario coverage improves
- Simulate end user scenario testing for internet bandwidth and other devices.
Through this initiative, we have validated our hunch and what we heard from a lot of people in the industry; that crowd sourced testing works best with applications that have a large global users base like mobile apps or games. We believe it will be time consuming and strenuous to involve external testers to crowd test for a product that requires a lot of internal communication with cross groups. All in all, this form of testing makes for an interesting venture and definitely has an exciting future ahead. We continue to watch what companies do globally in this space and innovate within this space. What have your experiences in crowd testing been?
Vijeethkumar Chathu | Test Manager | Zen Test Labs