Mistakes Still Prevalent in the 3 Decade Old Testing Industry

‘I make one mistake only once’ –the dream statement of most people, processes and organizations.

The software testing industry being an industry more than 3 decades old is still stuck in the spiral of some mistakes that should have been overcome before they resulted in grave mishaps. Although a lot has been written about the ‘classic mistakes in software testing’, I want to go ahead and highlight the mistakes that are still being overlooked at each phase of testing:

1. Requirements Phase:
We miss on acquiring articulate information. We don’t explore the requirements by using a requirements traceability matrix which is the most ideal system for maintaining the relationship of requirements to design, development, testing and release of software. An RTM also charters out the testable, non-testable and unfrozen requirements- those that have been finalized and signed off.

2. Test Plan
Anybody who has been in software testing knows what importance a test plan holds. It gives a detailed testing effort of the scope of testing, schedule, test deliverables, risks involved and release criteria. However, a number of things that could go wrong are unclear out of scope items and omitting of misplaced assumptions which could result in misdiagnosed estimations. Many a times, under business pressure to release, there is a high tendency to estimate fewer testing cycles which adversely affects the quality of testing.Test design and test data design are also an integral part of the test plan. Faults occurring in them could collapse the test plan to bits, which is possible if the test design is too detailed or too brief, if a wrong template is used or if the expected results are missing. Test data design can cause a major glitch if it is not centralized, rationalized, automatable or hard coded. Ensuring the above pointers makes the process efficient and provides high quality testing.

3. Test Environment
Establishing no separate test environment for testing would result in missing critical defects and the inability to cover business flows. Missing preconditions from the environment will further throttle the process from creating a production like environment which would overlook vital bugs.

4. Test Execution
Mistakes still occurring at the test execution stage would be not optimizing test cases for execution resulting in unnecessary effort which translates to delayed release time, a lack of smoke testing and not prioritizing test cases. Prioritizing test cases guarantees maximum coverage and depth in testing which could be compromised if not done.

These are some of the points that I think are very crucial, my next post will cover more about defect analysis and bug reporting

Zen Test Labs

Communication- A Key Skill to Excel at Testing

Enabling better communication is not a onetime activity. It requires continuous effort across the company. Good internal and external communication is extremely important to a business’s success. In order to work together effectively, there must be clear and coherent communication among all the departments.

Here are a few scenarios wherein communication gaps may arise and lead to poor quality:

1. Continuously Changing Requirements:
At times, requirement changes are implemented directly without updating the specification document. In such cases, there is a chance that the changed requirements remain untested or are tested incorrectly.
Any change in the requirements should be communicated correctly to all stake holders and it is necessary to update the specification document on a timely basis.

2. Configurations:
Lack of clarity from the stakeholders on the configurations to be tested can lead to wasted effort and extra work. Configuration testing can be expensive and time consuming. Investment in hardware and software is required to test the different permutations and combinations. There is also the cost of test execution, test reporting and managing the infrastructure.
Increasing communication between development, QA and stakeholders can help deal with these challenges.

3. Team Size:
When team sizes are large, some members of the team may miss changes in requirements or may not be communicated updates on activities in the project. This could lead to severe problems in the project or project failure.  Each team member should be abreast of the activities in the project through a log or through other means.

4. Changes in  Application Behavior not Communicated:
Continuous changes in the application behavior may lead to requirements being tested incorrectly.  All the functionality implemented in the application should be frozen while testing. If any changes are made to the functionality, they should be communicated to the testing team on a timely basis.

5. Unclear Requirements:
Complex requirements that contain insufficient data may be difficult to understand and therefore, may lead to improper testing. The functional/technical specification documents should be clear and easy to understand; they should contain a glossary, screenshots and examples wherever necessary.

The path to project success is through ensuring that small communication problems are eliminated completely before they build up, so that the message is delivered correctly and completely. Instead of discovering problems, we should figure out how to stop them from appearing in the first place.

Poonam Rathi | Test Consultant | Zen Test Labs

Major Checkpoints for Database Migration Testing

I had the opportunity to work on my very first data migration project a few days ago. The objective of our project was to ensure the correctness and completeness of data migrated from a source system with Oracle as its database to a target system with MS SQL as its database.  I learned a lot while working on this project. I got an opportunity to hone my SQL query writing skills and got well versed with the major checkpoints and activities performed during database migration testing. I wanted to share my learning’s and experience in this post.

23 million data sets had to be migrated from Oracle to SQL. In order to do this, we divided the activities into two phases:

Info1Our job was to test and verify the correctness and completeness of around 23 million data sets at the end of each activity. The real challenge was to perform this job manually without automation. What made it even more interesting was that there’s a chance of up to 5% data loss during data migration.

To achieve our goal, we divided our tasks into 4 major Phases:

Info2

The experience I gained while working on this project can be divided into 3 parts:

  1. Basic skills required for database migration testing
  2. Major checkpoints for database migration testing
  3. Major activities performed during database testing

Basic skills required for database migration testing

Other than testing skills required for all types of testing, one needs following skills for data migration testing

  1. SQL query writing for legacy and target database
  2. Knowledge of the data migration testing tool (if any) being used or advance knowledge of Excel features for data comparison

Major checkpoints for database migration testing, data profile matching

  1. Data profiling
  • Table count
  • Data type matching of columns
  • Identifying different classes of records from business point of view.
  • Performing Check Sum on column holding numerical data
  • Row count matching of legacy and target database
  • Column count matching of legacy and target database
  • Matching of total Null values per column of legacy and target database
  • Matching of Not Null count per column
  • Checking the Distinct count per column
  • Checking the Group By count per column
  • Matching of Summation of numeric fields
  • Checking the Min value per numeric column
  • Checking of Max value per numeric column
  • Matching of Average data value in numeric columns
  • Checking the biggest values in non-numeric columns
  • Checking the shortest values in non-numeric columns

2. Checking  data redundancy
3. Matching  control tables to verify the exact transfer of relationships among tables
4. Functional testing on migrated data
5. Random sampling- Picking and matching data from corresponding tables randomly.

Major activities performed during database testing

  1. Identification and matching of the number of tables in legacy and target databases.
  2. Identification and matching of the data types of columns in corresponding tables.
  3. Identification and matching of the relationship between tables in legacy database and target database.
  4. Identification and matching of business rules in legacy database.
  5. Identification of primary key attribute for each table
  6. Identification of columns with Not Null property
  7. Identification of numeric fields for data profiling
  8. Query writing for both databases to identify all data profiling checkpoints:
    1. Row count
    2. Column count
    3. Null values count per column
    4. Not Null count per column
    5. Distinct values count per column
    6. Group By count per column
    7. Summation of numeric fields
    8. Min value per numeric column
    9. Max value per numeric column
    10. Average value of numeric fields
    11. Biggest values in non-numeric columns
    12. Shortest values in non-numeric columns
  9. Matching the data profiling results of both databases
  10. Query writing for random sampling to ensure that data has been migrated correctly and completely
  11. Using migrated data with the related application to test its functional use

Migrating data is a challenging activity. Users expect it to be fast and efficient, without loss of data.  Thorough data migration testing has to be performed to reduce risk and ensure that data has been migrated as per requirements. I believe that using the points outlined above, you can refine your test approach and ensure a better data migration testing process.

Mayank Raj | Trainee Test Analyst | Zen Test Labs

The Art of Test Automation

Test automation has evolved to become a strategic and integral part of the software development process. Most of us start our test automation careers with record and playback. Over time, some of us move to data-driven test automation, but very few of us move towards the core where in the principles of design and development are applied to test automation. Test automation is like developing a system where test cases are requirements. The depth of thinking and planning that goes into test automation before hitting the record button is similar to developing software

Over the last 10 years, I have seen multiple Fortune clients struggle with automation and some of them eventually getting it right. For some of the projects that failed, we had the best test automation resources and a very stable manual testing practice but in spite of all this there was a huge gap between what was dreamt and what actually got realized. Over the period, we realized that the planning process is a key component for successful test automation. In 65% of the projects that failed the planning process and sequence of steps followed were the reasons.

Based on my experience, the automation process is:

  • Why (Purpose)
  • When (Stable Setup and Manual Process)
  • Which (Tool Selection)
  • What (Test Case Selection)
  • How (Design)

We have written a detailed whitepaper “The Art of Test Automation” based on the test automation process above. Through this white paper, we have attempted to outline how to actually go about automating, planning, prioritizing and using better practices to ensure a lesser risk of complete failure in automation projects.

Some of the important test automation questions that this paper attempts to address:

  • Why automation fails in spite of having technical resources?
  • Is there a standard process to be followed for test automation?
  • When to start and stop automation?
  • Test selection criterion

Download “The Art of Test Automation” to read more about the ideal automation process.

Poonam Rathi | Test Consultant | Zen Test Labs

8 Steps to Improve Your Regression Testing Process

With business and user requirements perpetually in an evolutionary mode, I find that regression testing has become a key component of the software development lifecycle. As testers, we need to keep in mind that a constant change in the functionality of the application lends the system to vulnerabilities in the base functionality too. These vulnerabilities tend to creep in due to an oversight while adding new functionality, poor analysis of impact on interfacing/ integrating applications and many a times due to the fact that customizations are an unknown entity. Poor regression testing can not only result in poor software quality but also impact revenue and cause customer loss.
Based on many years of planning, creating and executing the Quality Assurance programs of multiple Fortune 500 companies, I suggest the following eight step methodology to improve any regression testing process.

Phase 1: Defining
Step 1: Objective Finding (OF) – Challenges and Goal Identification
This step answers one of the most important questions “Why is regression testing not effective in its current state?”

Step 2: Fact Finding (FF) – Data Collation and Analysis
During this stage, teams must trail defects found in the past to conduct a defect root cause analysis. An important part of this step is bug prediction analysis so that defect prone areas in the application can be found.

Step 3: Problem Finding (PF) – Problem Clarification and Statement
Once the results of Steps 1 and 2 are combined, the exact scope of the challenges to address is established. These refined objectives act as the equivalent of a “Requirements Document”.

Phase 2- Scoping
Step 4: Test Cases Finding (TF) –Coverage Gap Analysis
Gaps in test coverage are found based on the current test cases and the application functionality. Techniques to map test cases to requirements and testing techniques are used to identify missing test cases

Step 5: Test Case Centralization (TC) – Test Case Repository Creation
Ensure that all test cases are stored in a centralized repository and in an optimized manner. Each test case must have a clear objective, precondition, steps, expected result and test data.

Step 6 : Test Case Optimization (TO) – Maximum Coverage in Desired Time with Minimum Risk
Statistical techniques such as Classification Tree and Orthogonal Array may be used to run minimum number of test cases in a way where every business process/ function is covered at least once

Phase 3- Executing
Step 7: Reusing Test Components (RT) – A Modular Approach
Create business functions and test data in a way that they can be reused for building manual test cases. Automate the generation of descriptive manual test cases.

Step 8: Test Case Classification (TC) – Test Case Mapping
At this stage, test cases are grouped requirement wise, screen wise, module wise, etc. Small frequently used regression pack/suites are created.

We have written a detailed whitepaper ‘Progress Not Regress’ on improving any regression testing process. We would love to hear your thoughts on it!

Girish Nair | Sr. Consultant | Zen Test Labs

Developer + Tester = 1?

Recently I attended a software testing conference where the main focus of discussion revolved around future of testing and how innovation can be ingrained more in testing!

The speakers included Randy Rice, Michael Bolton and Lee Copeland. When asked about the future of testing, and each one said that it was a too dangerous to predict. They realized that when they were asked this question ten years back, their predictions failed thoroughly J  Quoting Neils Bohr “Prediction is very difficult, especially if it’s about the future”.
A common point said by each was that it would be more complex and besides functional testing, security testing would be a major game.

One of the intriguing points I found was in order to enhance creativity,asking the developer to also play tester. I think it is a recipe for disaster to expect a developer to test his baby.

The reasons why the developer and the tester should be different are:

–          The developer works  on building a particular module but the tester has to think of the integrating that part as well, thus varying  the scope and efficiency expected from the tester and the developer

–          The creative process of a developer works on a structured constructing process, where as a tester proves his creativity by breaking the barriers and rules

–          A developer is entitled only to the code and does not work with the mindset of looking for failures, where as a tester whether access to code or no code works towards digging failures

–          However, when a new Functional Specification Document or a Business Specification Document is acquired, the tester and the developer can start working simultaneously, by adopting Agile more productivity and efficiency is attained over a limited period of time.

If I must combine the role of a tester and a developer, then

A developer can enhance his productivity by unit testing, if there is an error in the code it is easier found by the developer rather than a separate tester who might or might not have the knowledge of that technology/language.He can also build by thinking of end to end business flows.

A tester can wear the developer’s shoes by building automated scripts and applying Oops concepts in his test cases.

I believe this would also add in enhancing the innovation in testing! These are my views acquired over dedicating a period of time in testing, Would be great to know yours?

Poonam Rathi |Test Consultant | ZenTest Labs

Software Testing in 2020

As a CEO of a testing company, a question that plays on my mind constantly is ‘what is the future of Testing?’ In the early 2000s, Ron Radice spoke at a QAI conference in India, where he had predicted that testing will die. His call was that automatic code generators will do the job so efficiently that testing will become obsolete. When he looked at the crystal ball then, he could see that prevention will be the creed and not detection.

Well, when I look at 2020, I believe Ron was right as well as wrong. Yes, code generators are arriving. Yes, there will be automated test case generators. Yes, model based testing will replace rudimentary testing activities. But, the whole boom of software especially in ubiquitous mobile devices means only more testing.

If the future includes automated cars like the Google driverless cars, I cannot imagine such a car with a technology that has not been fully and manually validated. If the future is the “Internet of Things”, I can only imagine that the amount of embedded testing will only explode. If the future is, business operations being handled through apps and app stores that have millions of applications pervading every step of our business and personal life then imagine the amount of mobile testing that will be required. If not anything, as everything gets more interconnected, the consequences of a critical failure will only be catastrophic. Wherever the nexus of cloud, social, mobile and big data takes us, I am thoroughly convinced that the need for testing will only grow.

While there a dime a dozen predictions on how things will look in 2020, my two bits around where testing will find itself as follows:

• Huge business opportunities arising from testing for app stores directly than app manufactures
• Test automation would have evolved from script less automation to automatic test case generators and execution
• The pressure to deploy rapidly in the Mobility and embedded devices space will mean that test automation tools will evolve to provide near and real time support to these areas
• Testing and testers will evolve to become super specialized with domain testers at one end and niche technical testers at the other end.

These are some things that come to mind and as the decade continues to evolve. Would be great to know what the rest of the testing world thinks.

Krishna Iyer|CEO|Zen Test Labs