Top 6 solutions for software testing failures

The cost of software testing is still not valued by its worth. Although it is a critical investment companies avoid spending on testing because they don’t realize the ROI on testing and/or a quantifiable cost of quality. The most common complaints against testing that we repeatedly hear are:

  • It is a necessary evil that stalls a project the closer it gets to a release
  • It is too costly, time consuming without any guaranteed outcome
  • Many a times regression testing is not effective to identify new defects

Having worked on a number of testing projects over the past 12 years, I realize why there is a high tendency to look at testing with such a skeptical eye. I would like to share what we have learnt over time.The top six points in our view to improve the effectiveness of manual testing are:

6. Reducing effort and time in Test Documentation

A lot of teams spend unnecessary time detailing test scenarios during the planning phase which are rarely referred to after 2-3 rounds of testing. This increases maintenance overheads and reduces flexibility and coverage in the long run thus resulting in inefficient testing. Post the initial 6-8 months a large % of test scenarios are outdated and require the same effort in updating. Instead of detailing each and every step for every test scenario, one can cover it with test conditions and the expected results.

5. Focusing on breadth and depth of testing

Many a times when execution is not prioritized the depth of testing takes lead over breadth. By aiming at covering more breadth, we align testing with the business objectives. By doing this the teams aim at being effective first and then efficient. Breadth referring to covering positive  critical cases (across the application) that are frequently used by end user.Depth referring to covering all the test cases for a module.

4. Testing, a continuous activity

Many companies look at testing as a one-time investment. They outsource/ execute in-house once during the start of the development of the product and then rarely test it during the maintenance phases. The primary reason is invariably budget driven and goes onto harm the quality of the product when not tested after newer versions. For every minor release one should ensure all the regression test cases are executed and for every major release all the high and medium priority test cases are executed at least once.

3Remembering the objective of testing

The key objective of testing is to break the system and not to prove that the system works as per the requirements.This has a direct impact and can improve testing effectiveness and the number of defects one will find. It is often observed that many senior testers habitually start proving that system is working as per the requirements which is against the primary objective of testing.

2Strategize Test optimization

Coverage is important but not at cost of redundant test cases. Test optimization is an intelligent way to ensure test coverage in less time. That’s why testing teams need to collaborate more with the development teams. Understanding the high level design and structure of the application makes testing more effective. In development, one of the main principles followed is reuse. So, we can use the same principle while testing the same code which is reused. Why not optimize and test the class/object once and just test the implementation of the class/object on other screens/modules. If the test cases are reusable maintainable and scalable it is an additional advantage to roll out in time and under budget.

1. Focusing on the Business for which you are testing

Testing cannot be done in isolation. Business priorities and challenges are equally and in most of the cases more important than testing needs. One thing I have learnt is that testing cannot drive business decisions, business drives testing most of the times. Aligning testing to the business requirements results in a disciplined and ready to market high quality product.

These are some of the solutions with which I could overcome testing failures. Do share yours if you have new solutions or methods

Mukesh Mulchandani | CTO | ZenTest Labs

Verifying 800 Million data sets in record time!

I recently was fortunate to be a part of a unique project at Zen Test Labs. This was a post-merger scenario wherein the acquirer (bank) had to consolidate the customer information systems of the two banks into a single system. This meant mapping the acquired bank’s product, service and customer portfolio, to a new and modified version of the acquirer’s products and services.

Among many other factors, ensuring seamless service to existing customers of the acquired bank implied that such customers should not expect undue increase in service charges. Processing customer data using enhanced systems required that the service fees were within the threshold that the customer would expect in normal course of business. Testing for “Go Live” was tricky since it required that for each acquired customer, the bank had to compare the results from the “Go Live” with historical data for the customer. With hundreds of thousands of customers and millions of transactions in a month, manual verification was a gigantic task, potentially impossible to accomplish.

Zen Test Labs creatively addressed this situation by leveraging its Data Migration Testing framework and extending it to include customer specific scenario. For example, each data component of the source and target data files were mapped, rules applied and integrated into the testing framework. A utility was then designed to pick each record from the source, apply the logic of migration then check if the corresponding value of the record in the target file is within the tolerance level as per the logic. During execution the selected components from the imported source and target data were compared and flagged if not meeting the tolerance levels. Once all the records were compared the utility reported:

  1. All transactions migrated as per the logic
  2. All transactions which did not meet the tolerance criteria
  3. Transactions in the target database which did not have any relation with the migration process

The framework and utility testing itself adopted an approach with three layers of testing:

  1. Utility testing using dummy data for source, target and the mapping
  2. Sampling of output files and manual verification with real data
  3. Verify against “Thumb Rules”. One of the examples of this was; the total number of Pass records and Fail records should total the count of primary key of source data.

Overall I found this project very challenging and interesting. Leveraging the data migration testing framework we created a comprehensive utility in approximately three weeks. The quality and performance of the utility was so sharp that it compared one data component with 600,000 to 700,000 records in 10 to 12 minutes. The total number of data values verified in this project was over 800 Million in a span of 30 days which is as good as verifying at least one data for the entire population of European Union! With our output files we provided great deal of ‘Data Profiled’ information of migrated customers to the bank which was used to understand behavioral patterns of the migrated customers and the performance of the products after migration.

Ravikiran Indore |Sr Consultant |Zen Test Labs

Automation lessons learnt: Funding automation projects & the role of change management

One of the many reasons test automation is often compromised is in situations where business funds technology projects on a per project basis. Key reason being business benefits of automation, primarily time to market, are realized only during the subsequent releases of the application, never in the release where automation is undertaken. Even when business agrees to fund an automation project, the order of magnitude of benefits is small due to the potentially low levels of automation feasible within the project scope. The benefits accumulate only over a period of time from increasing automation levels and therefore the return on investment is realized over a longer duration. In order to reap benefits from automation business needs to continually invest in it and maintain a long term orientation to ROI. These are typical characteristics of any change initiatives. Test automation initiatives funded by individual business units can therefore learn from the vast expanse of knowledge pertaining to other organizational change initiatives and do not need to reinvent the wheel.

I have captured my experience of change initiatives as applied to automation in the visual below. It shows key components required to not only succeed at a pilot project but also create a cascading positive spiral where the benefits accumulate over time.

 

Like any other change initiative the key components form a chain, where the initiative is just as strong as its weakest link. Successful pilot accompanied by the right communication can act as a feeder to the next project and as long as all key components act in unison incremental benefits from each project can lead to significant cumulative benefits. The problem is that the first cycle tends to be demanding and it needs continuity of the champions until such time that the framework is institutionalized. A failure at early stages can have devastating effects with a stigma associated with it presenting greater roadblocks during subsequent attempts at automation. This is where senior management support from business and IT is crucial. A champion driving each automation cycle to success is central for the overall success of automation!

What do you think of the role of change management in automation projects? Have you had difficulty funding automation projects? Please feel free to share your experiences.

Aparna Katre | Director Strategy | Zen Test Labs

Tug of War between BA’s and QA…Guess who gets stuck?

I have always wondered what the core responsibility of a business analyst is.
Googling it, which is not a great idea, threw up hundreds of responses! The more I analyzed, I realized that they form the bridge between business and technology – translating technical explanations into something that business understands and vice versa. On the one hand, a proficient business analyst performs testing better than those with a strong technology background, and on the other hand they are able to write use cases which can be easily understood by the technical team. They lose credibility if one side is favored against the other, even if it is because they come with one dominant background walking into the BA role. Due to this unique combination of skills, good business analysts are a ‘high in demand’ rare genre.

Since Quality Assurance is a big part of what BAs do, wouldn’t it be wonderful to have some tools which use business vocabulary instead of technical? Most QA related tools like Selenium and QTP demand technical knowledge which business analysts often lack. So much for the rapid progress of technology! What BAs need is a framework that deskills the entire process of test automation! Easier said than done, right?

Well, I have done some research on that front and while I have not come across a dedicated tool for BA’s, I have been exposed to some automation frameworks that use the concept of business scenarios based test case creation using an graphical interface. These frameworks claim automating testing at a click of button by dragging/ dropping business flows. They combine different approaches of automation like functional decomposition and keyword driven to give friendly user interface, are easy to maintain, and allow reuse scaling the entire testing operation. In fact some of them even claim to eliminate the need to maintain huge automation staff once the Business Scenario, Component and Utility layers are ready.

With such frameworks BA’s can focus on ensuring that high quality tests are designed and executed thereby mitigating business risks. I wish there is more research and investment to address issues such as these and help improve the overall productivity of both business and IT.

Have you had similar experiences and come across a solution? I will be more than happy to share my research and discuss on the frameworks.

Aparna Katre

Director Strategy | Zen Test Labs

Common Issues and Solutions in Selenium

Selenium is fast becoming one of the most widely used test automation tools for web based applications. While working on selenium projects we faced some common issues that I thought is best addressed through the solutions I am listing below. In case you have tried other solutions and they have worked for you, please feel free to add your inputs in the comments box.

Issue No. 1. Issue in handling popup windows / Frame
Solution: Pop-ups that are very commonly found on the GUI of web apps are many times not recorded by selenium or cause issues while running selenium scripts. The table below gives a snapshot at how these issues can be handled

Popup Type Description Solution
Alert Popup (Message box) Message box popup on application To handle alert popup you can use getAlert functions e.g. selenium.GetAlert();
This will click on Ok button of alert and return alert message.
Confirmation Popup This type of popup has Ok & Cancel buttons You can select the button you want to click, by using below statements before GetConfirmation
For Ok – chooseOkOnNextConfirmation
For Cancel – chooseCancelOnNextConfirmation
And then use GetConfirmation as follows,
selenium.GetConfirmation();
This will click on selected button & return confirmation message.
Popup Window/ Frame In this type of popup, the Frame/window contains data fields where we have to fill values in fields. this can be done using SelectFrame
Following code will illustrate how this can be done.

selenium.Click(btnSubmit)
// This will open a popup window asking user to enter activation code & click on confirm button
selenium.SelectFrame(frmActivationCode); // this will transfer the control to Activation code frame i.e. popup window
//Now you can process objects from popup frame
selenium.Type(txtactivationCode, stractivecode);
selenium.Click(btnConfirm);
// After this you have to get back the control to parent window again using
selenium.SelectWindow(“”);

Issue No.2. Selenium does not trigger event on changing values
Solution: Selenium does not trigger event on changing values so we have to explicitly trigger the event using fireEvent,

selenium.FireEvent(cmbCategory, “onchange”);
Here, cmbCategory is object on which you want to fire event & onchange is the name of event.

Issue No.3. Synchronization issue leading to timeout
Solution:
For this type of scenario I created a function ‘CheckElementPresent’ which will take object locator as parameter & will return true once object is visible.

CheckElementPresent(btnSubmit)
Here, I have used selenium.IsElementPresent(locator) to verify object is present in for loop with Thread.Sleep

Issue No.4. Handling SSL certificates
Solution: Many times we face issue in SSL certificates for selenium tests. We can handle this issue by using –trustAllSSLCertificates & firefox profiles as shown below,
a. For browser other than firefox
Use following command to start selenium server, this will start selenium server which trust all SSL certificates.
C: > java -jar c:\Selenium\selenium-server2.14.0.jar. –trustAllSSLCertificates
b. For Firefox browser
Create a new firefox profile (steps to create profile – http://support.mozilla.org/en-US/kb/profile-manager-create-and-remove-firefox-profiles)
And accept SSL Certificate using same profile. Now, use the same profile while running selenium server using following command,

C:> java -jar c:\Selenium\selenium-server2.14.0.jar. -firefoxProfileTemplate /C:\Selenium\ Profile

Here were some issues that I faced and the solutions I came up with. Please share any other solutions you know of for the above issues. Also please share other issues that you faced and if possible provide the solutions that you came up with. Happy testing! 

Hemant Jadhav | Senior Consultant | ZenTest Labs

Automating data migration testing

I had the opportunity to be a part of data migration project recently. I was involved in automated data migration testing, which I found it to be a very exciting and challenging form of testing.  I wanted to share my learning’s in this post.

During conversion or migration projects, data from the legacy or source systems is extracted, transformed and loaded into the target system. This is called as the ETL process. The data needs to be transformed as the data model of any two systems is different. As a standard practice the data transformations are managed in the data mapping document, which forms the basis for development as well as testing.

Testing the migrated data is usually a critical and challenging aspect. Testing the migrated data manually is a very time consuming process and so automated data validation is a good way to go ahead.

In my latest project, data from two source systems was migrated to target system. The data from the source systems’ UI was compared with the data from target system’s UI, since we did not have access to the database. Data migration was performed based on incremental loading approach to ensure cent per cent verification. The approach was to load small subsets of data every week for verification. This type of a process was a perfect solution to client’s challenges as in the event of any mismatch only that specific subset of data could be reversed.

I am also listing some of the key challenges we overcame during the course of the project

  1. We had to create scripts that could read source values and the use field level mapping rules to calculate the expected results at the destination. This had to be done because the mappings between the fields of the source system and the target system were different; i.e., both systems had their own structure
  2. We had to verify values at the target system as some extra fields were present in it leading to a mismatch with the source system
  3. We had to read the data on the target system to verify as some amount of data in the source system was in a CSV format with header changes for each customer column
  4. We also created a strong log generation mechanism that generated a result for every iteration. It also went onto ensure that when any mismatch occurs not only field name mismatches are captured but also values get captured
  5. The results also included the time taken to execute each record
  6. To counter the fact that most of the data migration was done in files of XML tab separated formats, we had to generate the input file for automation in excel format

We also went onto to create a customized data migration automation testing framework (illustrated below) to overcome these challenges which lead to a successful project.

Have any of you worked on such projects? Would love to hear some of your experiences.

Anand Gharge | Test Manager | Zen Test Labs

Implementing Object Repository in Selenium

Selenium (http://seleniumhq.org/) has now emerged as one of the top contenders to take on QTP in the test automation tools space. Our teams at Zen Test Labs (www.zentestlabs.com) were one of the early adopters of this automation tool and have built up a decent level of expertise in automating test scripts using selenium. We recently also presented a tutorial on this topic at STARWEST, California in October 2011.

While both tools have their advantages and disadvantages, object repositories are one area that I have found to be important but not supported by Selenium. An object repository is a centralized place for storing properties of objects available in the application under test (AUT) to be used in scripts. QTP comes with its own object repository where one can record and store objects.

Over the course of a recent project I have tried to implement object repository in and it worked really well. I am listing below, how I went about doing this.

1. Create an interface of a name you want to give to your object repository as follows,
Interface ObjectRepository
{
// Now here you can store properties of the object in variable
Static String ddCategory = “id=ctl00_MainContent_ddlCategory”;
Static String btnSave = “”id=ctl00_MainContent_btnSave””;
}

2. Implement the interface into your class and you are good to use objects in your functions
Class TestLibrary implements ObjectRepository
{
public String testFuntion()
{
//here you can your object to perform actions or validations
selenium.select(ddCategory, “Testing”);
if (selenium.isElementPresent(btnSave)
selenium.click(btnSave);
}

}
In the above example, I have implemented it in java and have also tried it in C#. Thus, using OOPS concepts I feel that this can be implemented in any object oriented language supported by Selenium.
What is important to note is that one can also connect to excel or any database and store values. Thus in the event of changes in the application or changes in properties of any object; a simple change in the excel file or DB will reflect across all instances of the object.

Some of the advantages of doing this include:

1. Easily maintain your tests or components when an object in your application changes
2. Manage all the objects for a suite of tests or components in one central location
3. Code becomes more readable & easy to write when user defined objects name are used instead of complex and long property name & value.

These have been some of the approaches that have worked for me and my learning’s. Would be great to get to know other people’s experiences in implementing object repositories within Selenium

Hemant Jadhav | Senior Consultant | ZenTest Labs