Mistakes Still Prevalent in the 3 Decade Old Testing Industry

‘I make one mistake only once’ –the dream statement of most people, processes and organizations.

The software testing industry being an industry more than 3 decades old is still stuck in the spiral of some mistakes that should have been overcome before they resulted in grave mishaps. Although a lot has been written about the ‘classic mistakes in software testing’, I want to go ahead and highlight the mistakes that are still being overlooked at each phase of testing:

1. Requirements Phase:
We miss on acquiring articulate information. We don’t explore the requirements by using a requirements traceability matrix which is the most ideal system for maintaining the relationship of requirements to design, development, testing and release of software. An RTM also charters out the testable, non-testable and unfrozen requirements- those that have been finalized and signed off.

2. Test Plan
Anybody who has been in software testing knows what importance a test plan holds. It gives a detailed testing effort of the scope of testing, schedule, test deliverables, risks involved and release criteria. However, a number of things that could go wrong are unclear out of scope items and omitting of misplaced assumptions which could result in misdiagnosed estimations. Many a times, under business pressure to release, there is a high tendency to estimate fewer testing cycles which adversely affects the quality of testing.Test design and test data design are also an integral part of the test plan. Faults occurring in them could collapse the test plan to bits, which is possible if the test design is too detailed or too brief, if a wrong template is used or if the expected results are missing. Test data design can cause a major glitch if it is not centralized, rationalized, automatable or hard coded. Ensuring the above pointers makes the process efficient and provides high quality testing.

3. Test Environment
Establishing no separate test environment for testing would result in missing critical defects and the inability to cover business flows. Missing preconditions from the environment will further throttle the process from creating a production like environment which would overlook vital bugs.

4. Test Execution
Mistakes still occurring at the test execution stage would be not optimizing test cases for execution resulting in unnecessary effort which translates to delayed release time, a lack of smoke testing and not prioritizing test cases. Prioritizing test cases guarantees maximum coverage and depth in testing which could be compromised if not done.

These are some of the points that I think are very crucial, my next post will cover more about defect analysis and bug reporting

Zen Test Labs

Communication- A Key Skill to Excel at Testing

Enabling better communication is not a onetime activity. It requires continuous effort across the company. Good internal and external communication is extremely important to a business’s success. In order to work together effectively, there must be clear and coherent communication among all the departments.

Here are a few scenarios wherein communication gaps may arise and lead to poor quality:

1. Continuously Changing Requirements:
At times, requirement changes are implemented directly without updating the specification document. In such cases, there is a chance that the changed requirements remain untested or are tested incorrectly.
Any change in the requirements should be communicated correctly to all stake holders and it is necessary to update the specification document on a timely basis.

2. Configurations:
Lack of clarity from the stakeholders on the configurations to be tested can lead to wasted effort and extra work. Configuration testing can be expensive and time consuming. Investment in hardware and software is required to test the different permutations and combinations. There is also the cost of test execution, test reporting and managing the infrastructure.
Increasing communication between development, QA and stakeholders can help deal with these challenges.

3. Team Size:
When team sizes are large, some members of the team may miss changes in requirements or may not be communicated updates on activities in the project. This could lead to severe problems in the project or project failure.  Each team member should be abreast of the activities in the project through a log or through other means.

4. Changes in  Application Behavior not Communicated:
Continuous changes in the application behavior may lead to requirements being tested incorrectly.  All the functionality implemented in the application should be frozen while testing. If any changes are made to the functionality, they should be communicated to the testing team on a timely basis.

5. Unclear Requirements:
Complex requirements that contain insufficient data may be difficult to understand and therefore, may lead to improper testing. The functional/technical specification documents should be clear and easy to understand; they should contain a glossary, screenshots and examples wherever necessary.

The path to project success is through ensuring that small communication problems are eliminated completely before they build up, so that the message is delivered correctly and completely. Instead of discovering problems, we should figure out how to stop them from appearing in the first place.

Poonam Rathi | Test Consultant | Zen Test Labs

The Art of Test Automation

Test automation has evolved to become a strategic and integral part of the software development process. Most of us start our test automation careers with record and playback. Over time, some of us move to data-driven test automation, but very few of us move towards the core where in the principles of design and development are applied to test automation. Test automation is like developing a system where test cases are requirements. The depth of thinking and planning that goes into test automation before hitting the record button is similar to developing software

Over the last 10 years, I have seen multiple Fortune clients struggle with automation and some of them eventually getting it right. For some of the projects that failed, we had the best test automation resources and a very stable manual testing practice but in spite of all this there was a huge gap between what was dreamt and what actually got realized. Over the period, we realized that the planning process is a key component for successful test automation. In 65% of the projects that failed the planning process and sequence of steps followed were the reasons.

Based on my experience, the automation process is:

  • Why (Purpose)
  • When (Stable Setup and Manual Process)
  • Which (Tool Selection)
  • What (Test Case Selection)
  • How (Design)

We have written a detailed whitepaper “The Art of Test Automation” based on the test automation process above. Through this white paper, we have attempted to outline how to actually go about automating, planning, prioritizing and using better practices to ensure a lesser risk of complete failure in automation projects.

Some of the important test automation questions that this paper attempts to address:

  • Why automation fails in spite of having technical resources?
  • Is there a standard process to be followed for test automation?
  • When to start and stop automation?
  • Test selection criterion

Download “The Art of Test Automation” to read more about the ideal automation process.

Poonam Rathi | Test Consultant | Zen Test Labs

8 Steps to Improve Your Regression Testing Process

With business and user requirements perpetually in an evolutionary mode, I find that regression testing has become a key component of the software development lifecycle. As testers, we need to keep in mind that a constant change in the functionality of the application lends the system to vulnerabilities in the base functionality too. These vulnerabilities tend to creep in due to an oversight while adding new functionality, poor analysis of impact on interfacing/ integrating applications and many a times due to the fact that customizations are an unknown entity. Poor regression testing can not only result in poor software quality but also impact revenue and cause customer loss.
Based on many years of planning, creating and executing the Quality Assurance programs of multiple Fortune 500 companies, I suggest the following eight step methodology to improve any regression testing process.

Phase 1: Defining
Step 1: Objective Finding (OF) – Challenges and Goal Identification
This step answers one of the most important questions “Why is regression testing not effective in its current state?”

Step 2: Fact Finding (FF) – Data Collation and Analysis
During this stage, teams must trail defects found in the past to conduct a defect root cause analysis. An important part of this step is bug prediction analysis so that defect prone areas in the application can be found.

Step 3: Problem Finding (PF) – Problem Clarification and Statement
Once the results of Steps 1 and 2 are combined, the exact scope of the challenges to address is established. These refined objectives act as the equivalent of a “Requirements Document”.

Phase 2- Scoping
Step 4: Test Cases Finding (TF) –Coverage Gap Analysis
Gaps in test coverage are found based on the current test cases and the application functionality. Techniques to map test cases to requirements and testing techniques are used to identify missing test cases

Step 5: Test Case Centralization (TC) – Test Case Repository Creation
Ensure that all test cases are stored in a centralized repository and in an optimized manner. Each test case must have a clear objective, precondition, steps, expected result and test data.

Step 6 : Test Case Optimization (TO) – Maximum Coverage in Desired Time with Minimum Risk
Statistical techniques such as Classification Tree and Orthogonal Array may be used to run minimum number of test cases in a way where every business process/ function is covered at least once

Phase 3- Executing
Step 7: Reusing Test Components (RT) – A Modular Approach
Create business functions and test data in a way that they can be reused for building manual test cases. Automate the generation of descriptive manual test cases.

Step 8: Test Case Classification (TC) – Test Case Mapping
At this stage, test cases are grouped requirement wise, screen wise, module wise, etc. Small frequently used regression pack/suites are created.

We have written a detailed whitepaper ‘Progress Not Regress’ on improving any regression testing process. We would love to hear your thoughts on it!

Girish Nair | Sr. Consultant | Zen Test Labs

Top 10 Non Testing skills you never knew you needed!

There is a good amount of discussion on testing skills out there. But what about non testing skills for testers? In 2008, when Mukesh (Zen Test Labs’ CTO) and I had to choose a topic for what we shall present at the STARWEST Conference in the United States, we decided to do a talk on the “Top Ten Non Testing Skills” for testers.
Using our experience with training testers and building a testing company, we got our heads together on what in our opinion are the “Top Ten Non Testing Skills” that testers will find useful. We came up with a laundry list of skills. From that list we deleted any generic skill that any professional ought to have including learning skills, communication and presentation skills, leadership skills, initiative. planning & organizing skills, time control, self confidence, interpersonal skills, self control and even focus on quality. After all our focus was on non testing skills that impact testing!
The list and the premise for why we listed this as important for testers is given below:
1. Collaboration: Good testers not just communicate well, they actively collaborate with everyone from developers to project managers to business analysts.
2. Creativity: Good testers have the ability to think up of test cases that nobody else can think about. They practice lateral thinking in test case design and exploratory testing.
3. Experimentation: Good testers do common things uncommonly and keep experimenting with newer methods, strategies and tools.
4. Passion: Good testers are highly motivated and they never give up. They ask empowering and uplifting questions to themselves and others.
5. Alertness: Good testers have an eye for detail. They use every defect to unearth more defects.
6. Connecting the dots: Good testers have the ability to connect the dots. They understand the requirements, the big picture as well as the details required in the current phase of the project.
7. Introspection: Good testers are honest thinkers. They hold up a mirror to themselves and others.
8. Challenging: Good testers challenge the status quo. They challenge assumptions. They practice critical thinking and challenge claims or bias laden statements.
9. Prioritization: Good testers prioritize their tests, their metrics and their strategies. They have a virtual effort impact matrix going on in their hands constantly.
10. Not carrying testing home: Great testers stop being testers at home. They remind themselves to tune off from work when at home and can enjoy the colors of life, not just the defects!

So, that was the list of non testing skills useful for testers in our opinion. Anything else, you can think of?

Krishna Iyer | CEO | Zen Test Labs

Tug of War between BA’s and QA…Guess who gets stuck?

I have always wondered what the core responsibility of a business analyst is.
Googling it, which is not a great idea, threw up hundreds of responses! The more I analyzed, I realized that they form the bridge between business and technology – translating technical explanations into something that business understands and vice versa. On the one hand, a proficient business analyst performs testing better than those with a strong technology background, and on the other hand they are able to write use cases which can be easily understood by the technical team. They lose credibility if one side is favored against the other, even if it is because they come with one dominant background walking into the BA role. Due to this unique combination of skills, good business analysts are a ‘high in demand’ rare genre.

Since Quality Assurance is a big part of what BAs do, wouldn’t it be wonderful to have some tools which use business vocabulary instead of technical? Most QA related tools like Selenium and QTP demand technical knowledge which business analysts often lack. So much for the rapid progress of technology! What BAs need is a framework that deskills the entire process of test automation! Easier said than done, right?

Well, I have done some research on that front and while I have not come across a dedicated tool for BA’s, I have been exposed to some automation frameworks that use the concept of business scenarios based test case creation using an graphical interface. These frameworks claim automating testing at a click of button by dragging/ dropping business flows. They combine different approaches of automation like functional decomposition and keyword driven to give friendly user interface, are easy to maintain, and allow reuse scaling the entire testing operation. In fact some of them even claim to eliminate the need to maintain huge automation staff once the Business Scenario, Component and Utility layers are ready.

With such frameworks BA’s can focus on ensuring that high quality tests are designed and executed thereby mitigating business risks. I wish there is more research and investment to address issues such as these and help improve the overall productivity of both business and IT.

Have you had similar experiences and come across a solution? I will be more than happy to share my research and discuss on the frameworks.

Aparna Katre

Director Strategy | Zen Test Labs