Automating data migration testing

I had the opportunity to be a part of data migration project recently. I was involved in automated data migration testing, which I found it to be a very exciting and challenging form of testing.  I wanted to share my learning’s in this post.

During conversion or migration projects, data from the legacy or source systems is extracted, transformed and loaded into the target system. This is called as the ETL process. The data needs to be transformed as the data model of any two systems is different. As a standard practice the data transformations are managed in the data mapping document, which forms the basis for development as well as testing.

Testing the migrated data is usually a critical and challenging aspect. Testing the migrated data manually is a very time consuming process and so automated data validation is a good way to go ahead.

In my latest project, data from two source systems was migrated to target system. The data from the source systems’ UI was compared with the data from target system’s UI, since we did not have access to the database. Data migration was performed based on incremental loading approach to ensure cent per cent verification. The approach was to load small subsets of data every week for verification. This type of a process was a perfect solution to client’s challenges as in the event of any mismatch only that specific subset of data could be reversed.

I am also listing some of the key challenges we overcame during the course of the project

  1. We had to create scripts that could read source values and the use field level mapping rules to calculate the expected results at the destination. This had to be done because the mappings between the fields of the source system and the target system were different; i.e., both systems had their own structure
  2. We had to verify values at the target system as some extra fields were present in it leading to a mismatch with the source system
  3. We had to read the data on the target system to verify as some amount of data in the source system was in a CSV format with header changes for each customer column
  4. We also created a strong log generation mechanism that generated a result for every iteration. It also went onto ensure that when any mismatch occurs not only field name mismatches are captured but also values get captured
  5. The results also included the time taken to execute each record
  6. To counter the fact that most of the data migration was done in files of XML tab separated formats, we had to generate the input file for automation in excel format

We also went onto to create a customized data migration automation testing framework (illustrated below) to overcome these challenges which lead to a successful project.

Have any of you worked on such projects? Would love to hear some of your experiences.

Anand Gharge | Test Manager | Zen Test Labs

Implementing Object Repository in Selenium

Selenium (http://seleniumhq.org/) has now emerged as one of the top contenders to take on QTP in the test automation tools space. Our teams at Zen Test Labs (www.zentestlabs.com) were one of the early adopters of this automation tool and have built up a decent level of expertise in automating test scripts using selenium. We recently also presented a tutorial on this topic at STARWEST, California in October 2011.

While both tools have their advantages and disadvantages, object repositories are one area that I have found to be important but not supported by Selenium. An object repository is a centralized place for storing properties of objects available in the application under test (AUT) to be used in scripts. QTP comes with its own object repository where one can record and store objects.

Over the course of a recent project I have tried to implement object repository in and it worked really well. I am listing below, how I went about doing this.

1. Create an interface of a name you want to give to your object repository as follows,
Interface ObjectRepository
{
// Now here you can store properties of the object in variable
Static String ddCategory = “id=ctl00_MainContent_ddlCategory”;
Static String btnSave = “”id=ctl00_MainContent_btnSave””;
}

2. Implement the interface into your class and you are good to use objects in your functions
Class TestLibrary implements ObjectRepository
{
public String testFuntion()
{
//here you can your object to perform actions or validations
selenium.select(ddCategory, “Testing”);
if (selenium.isElementPresent(btnSave)
selenium.click(btnSave);
}

}
In the above example, I have implemented it in java and have also tried it in C#. Thus, using OOPS concepts I feel that this can be implemented in any object oriented language supported by Selenium.
What is important to note is that one can also connect to excel or any database and store values. Thus in the event of changes in the application or changes in properties of any object; a simple change in the excel file or DB will reflect across all instances of the object.

Some of the advantages of doing this include:

1. Easily maintain your tests or components when an object in your application changes
2. Manage all the objects for a suite of tests or components in one central location
3. Code becomes more readable & easy to write when user defined objects name are used instead of complex and long property name & value.

These have been some of the approaches that have worked for me and my learning’s. Would be great to get to know other people’s experiences in implementing object repositories within Selenium

Hemant Jadhav | Senior Consultant | ZenTest Labs