Testing the Mobile Apps explosion

It won’t be long before it becomes A-android, B-Blackberry, C-Cupcake, D-Donut,-E-Éclair, F-Froyo, G-Gingerbread, if anything, they are words that probably half the planets population (approx. 3.2 billion people) is well versed with. Not only that another 700 million would be over the next 3 years! If you haven’t guessed it by now…I am referring to the explosion of mobile devices into our lives.

At the core of this explosion in mobile devices and here I mean smartphones and tablets; is the innovation in the field of processors. With processing speeds of these mobile devices increasing dramatically, the demand from users to run complex applications has also gone up. Business users want to have the ability to manage their personal and professional lives through a single interface and have apps that allow them to do this. Add the speed at which innovation in devices, processors and OS takes place and it is not a pretty picture for app. manufacturers.

So, what does all of this mean to you if you are an App. manufacturer or an enterprise trying to create mobile apps for your workforce or customer base?

Some of the areas of impact include:

  • A constant need to keep your app. updated with the latest OS upgrades/ devices in the market
  • Build high secure applications that lend peace of mind to users/ administrators
  • Build apps. that are not very heavy on the device resources (for optimum performance)
  • Constantly upgrade/ enhance your application to keep users engaged

Roll out apps at a speed which would put Formula 1 drivers to shame! Well, just joking on that last one there but for the ones that work in this space, you know what I mean!

Over the years of managing the Quality Assurance programs of multiple Fortune 500 companies and having setup a Mobile Testing Lab fairly early on within this space, I want to share the basic methodology that can be used to mitigate risks for you when developing/ deploying your mobile apps.

Mobile Configuration Optimization
Choose an optimum no. of configurations to test your app on using statistical techniques like Classification tree, Orthogonal Arrays, etc.
Mobile Test Automation
Automate as much of the core testing as possible right from the get go. We have experienced a reduction between 50-70% in the testing effort while ensuring complete coverage across devices. Automation built in on the right design principles also leads to high reusablity of scripts.
Mobile Performance Testing
A holistic approach to performance testing should cover areas such as volume testing, endurance testing, performance monitoring, soak testing and testing under real time scenarios.

An in depth whitepaper has also been written on Mobile is changing the face of Software Testing.I would love to hear from readers on their learning’s when developing or testing mobile apps. Please feel free to write to me

Amol Akotkar | Test Consultant | Zen test Labs

Reducing dependence on automation engineers to manage test automation!

I have always wondered what would it be to separate test automation and automation engineers. Considering that Test Automation has always been treated as the holy grail of testing! Enterprises that have managed to achieve high levels of automation in the testing process have enhanced productivity exponentially while improving coverage and thus reducing risk. This has translated into automation engineers holding  design approaches close to their heart and controlling scripting tightly. Given this dynamic, the adoptions of test automation have remained low over the years.

Test Automation today has transitioned from a “Record and Playback” mode to a virtually “Scriptless” mode thus enabling rapid on the go Test Automation

It has resulted in enterprises automating testing to be oblivious to tool specific coding thus making automation suites maintainable and resource independent. However, scriptless automation frameworks still have many missing links. For example, most scriptless automation frameworks  demand extensive Business User involvement particularly to test the technology enablement. There is a possibility it takes longer than acceptable time to market. Among many causes for greater time to market, one cause is extensive manual testing of the solution. It hamstrings the time taken to market since there is heavy dependence on business analysts (from business or IT) in QA (test design and execution). There is a strong dependence on skilled & expensive technical resources for automation. There is a need to manage spikes in demand for QA resources which results in increase of QA costs.

Considering these dynamics, the next stage in the evolution of test automation is driving in the direction of Business Process Model based test automation that aims at synchronizing Operations, Product Management and Quality functions.

At Zen Test Labs we are innovating with multiple products in this space. Our flagship test automation framework, ZenFRAME is one such example. ZenFRAME improves BA and business testers productivity while reducing dependence on technology teams by up to 40%. The GUI enables most non-technical users to create automated test cases faster  thus resulting in close to 33% lesser creation time, read our whitepaper to know how you can implement a business Process model for you QA environment. Would love to hear thoughts from everyone…

Ravikiran Indore |Sr Consultant |Zen Test Labs

 

 

 

Top 6 solutions for software testing failures

The cost of software testing is still not valued by its worth. Although it is a critical investment companies avoid spending on testing because they don’t realize the ROI on testing and/or a quantifiable cost of quality. The most common complaints against testing that we repeatedly hear are:

  • It is a necessary evil that stalls a project the closer it gets to a release
  • It is too costly, time consuming without any guaranteed outcome
  • Many a times regression testing is not effective to identify new defects

Having worked on a number of testing projects over the past 12 years, I realize why there is a high tendency to look at testing with such a skeptical eye. I would like to share what we have learnt over time.The top six points in our view to improve the effectiveness of manual testing are:

6. Reducing effort and time in Test Documentation

A lot of teams spend unnecessary time detailing test scenarios during the planning phase which are rarely referred to after 2-3 rounds of testing. This increases maintenance overheads and reduces flexibility and coverage in the long run thus resulting in inefficient testing. Post the initial 6-8 months a large % of test scenarios are outdated and require the same effort in updating. Instead of detailing each and every step for every test scenario, one can cover it with test conditions and the expected results.

5. Focusing on breadth and depth of testing

Many a times when execution is not prioritized the depth of testing takes lead over breadth. By aiming at covering more breadth, we align testing with the business objectives. By doing this the teams aim at being effective first and then efficient. Breadth referring to covering positive  critical cases (across the application) that are frequently used by end user.Depth referring to covering all the test cases for a module.

4. Testing, a continuous activity

Many companies look at testing as a one-time investment. They outsource/ execute in-house once during the start of the development of the product and then rarely test it during the maintenance phases. The primary reason is invariably budget driven and goes onto harm the quality of the product when not tested after newer versions. For every minor release one should ensure all the regression test cases are executed and for every major release all the high and medium priority test cases are executed at least once.

3Remembering the objective of testing

The key objective of testing is to break the system and not to prove that the system works as per the requirements.This has a direct impact and can improve testing effectiveness and the number of defects one will find. It is often observed that many senior testers habitually start proving that system is working as per the requirements which is against the primary objective of testing.

2Strategize Test optimization

Coverage is important but not at cost of redundant test cases. Test optimization is an intelligent way to ensure test coverage in less time. That’s why testing teams need to collaborate more with the development teams. Understanding the high level design and structure of the application makes testing more effective. In development, one of the main principles followed is reuse. So, we can use the same principle while testing the same code which is reused. Why not optimize and test the class/object once and just test the implementation of the class/object on other screens/modules. If the test cases are reusable maintainable and scalable it is an additional advantage to roll out in time and under budget.

1. Focusing on the Business for which you are testing

Testing cannot be done in isolation. Business priorities and challenges are equally and in most of the cases more important than testing needs. One thing I have learnt is that testing cannot drive business decisions, business drives testing most of the times. Aligning testing to the business requirements results in a disciplined and ready to market high quality product.

These are some of the solutions with which I could overcome testing failures. Do share yours if you have new solutions or methods

Mukesh Mulchandani | CTO | ZenTest Labs

Verifying 800 Million data sets in record time!

I recently was fortunate to be a part of a unique project at Zen Test Labs. This was a post-merger scenario wherein the acquirer (bank) had to consolidate the customer information systems of the two banks into a single system. This meant mapping the acquired bank’s product, service and customer portfolio, to a new and modified version of the acquirer’s products and services.

Among many other factors, ensuring seamless service to existing customers of the acquired bank implied that such customers should not expect undue increase in service charges. Processing customer data using enhanced systems required that the service fees were within the threshold that the customer would expect in normal course of business. Testing for “Go Live” was tricky since it required that for each acquired customer, the bank had to compare the results from the “Go Live” with historical data for the customer. With hundreds of thousands of customers and millions of transactions in a month, manual verification was a gigantic task, potentially impossible to accomplish.

Zen Test Labs creatively addressed this situation by leveraging its Data Migration Testing framework and extending it to include customer specific scenario. For example, each data component of the source and target data files were mapped, rules applied and integrated into the testing framework. A utility was then designed to pick each record from the source, apply the logic of migration then check if the corresponding value of the record in the target file is within the tolerance level as per the logic. During execution the selected components from the imported source and target data were compared and flagged if not meeting the tolerance levels. Once all the records were compared the utility reported:

  1. All transactions migrated as per the logic
  2. All transactions which did not meet the tolerance criteria
  3. Transactions in the target database which did not have any relation with the migration process

The framework and utility testing itself adopted an approach with three layers of testing:

  1. Utility testing using dummy data for source, target and the mapping
  2. Sampling of output files and manual verification with real data
  3. Verify against “Thumb Rules”. One of the examples of this was; the total number of Pass records and Fail records should total the count of primary key of source data.

Overall I found this project very challenging and interesting. Leveraging the data migration testing framework we created a comprehensive utility in approximately three weeks. The quality and performance of the utility was so sharp that it compared one data component with 600,000 to 700,000 records in 10 to 12 minutes. The total number of data values verified in this project was over 800 Million in a span of 30 days which is as good as verifying at least one data for the entire population of European Union! With our output files we provided great deal of ‘Data Profiled’ information of migrated customers to the bank which was used to understand behavioral patterns of the migrated customers and the performance of the products after migration.

Ravikiran Indore |Sr Consultant |Zen Test Labs

Software Testing in 2020

As a CEO of a testing company, a question that plays on my mind constantly is ‘what is the future of Testing?’ In the early 2000s, Ron Radice spoke at a QAI conference in India, where he had predicted that testing will die. His call was that automatic code generators will do the job so efficiently that testing will become obsolete. When he looked at the crystal ball then, he could see that prevention will be the creed and not detection.

Well, when I look at 2020, I believe Ron was right as well as wrong. Yes, code generators are arriving. Yes, there will be automated test case generators. Yes, model based testing will replace rudimentary testing activities. But, the whole boom of software especially in ubiquitous mobile devices means only more testing.

If the future includes automated cars like the Google driverless cars, I cannot imagine such a car with a technology that has not been fully and manually validated. If the future is the “Internet of Things”, I can only imagine that the amount of embedded testing will only explode. If the future is, business operations being handled through apps and app stores that have millions of applications pervading every step of our business and personal life then imagine the amount of mobile testing that will be required. If not anything, as everything gets more interconnected, the consequences of a critical failure will only be catastrophic. Wherever the nexus of cloud, social, mobile and big data takes us, I am thoroughly convinced that the need for testing will only grow.

While there a dime a dozen predictions on how things will look in 2020, my two bits around where testing will find itself as follows:

• Huge business opportunities arising from testing for app stores directly than app manufactures
• Test automation would have evolved from script less automation to automatic test case generators and execution
• The pressure to deploy rapidly in the Mobility and embedded devices space will mean that test automation tools will evolve to provide near and real time support to these areas
• Testing and testers will evolve to become super specialized with domain testers at one end and niche technical testers at the other end.

These are some things that come to mind and as the decade continues to evolve. Would be great to know what the rest of the testing world thinks.

Krishna Iyer|CEO|Zen Test Labs

Top 10 Non Testing skills you never knew you needed!

There is a good amount of discussion on testing skills out there. But what about non testing skills for testers? In 2008, when Mukesh (Zen Test Labs’ CTO) and I had to choose a topic for what we shall present at the STARWEST Conference in the United States, we decided to do a talk on the “Top Ten Non Testing Skills” for testers.
Using our experience with training testers and building a testing company, we got our heads together on what in our opinion are the “Top Ten Non Testing Skills” that testers will find useful. We came up with a laundry list of skills. From that list we deleted any generic skill that any professional ought to have including learning skills, communication and presentation skills, leadership skills, initiative. planning & organizing skills, time control, self confidence, interpersonal skills, self control and even focus on quality. After all our focus was on non testing skills that impact testing!
The list and the premise for why we listed this as important for testers is given below:
1. Collaboration: Good testers not just communicate well, they actively collaborate with everyone from developers to project managers to business analysts.
2. Creativity: Good testers have the ability to think up of test cases that nobody else can think about. They practice lateral thinking in test case design and exploratory testing.
3. Experimentation: Good testers do common things uncommonly and keep experimenting with newer methods, strategies and tools.
4. Passion: Good testers are highly motivated and they never give up. They ask empowering and uplifting questions to themselves and others.
5. Alertness: Good testers have an eye for detail. They use every defect to unearth more defects.
6. Connecting the dots: Good testers have the ability to connect the dots. They understand the requirements, the big picture as well as the details required in the current phase of the project.
7. Introspection: Good testers are honest thinkers. They hold up a mirror to themselves and others.
8. Challenging: Good testers challenge the status quo. They challenge assumptions. They practice critical thinking and challenge claims or bias laden statements.
9. Prioritization: Good testers prioritize their tests, their metrics and their strategies. They have a virtual effort impact matrix going on in their hands constantly.
10. Not carrying testing home: Great testers stop being testers at home. They remind themselves to tune off from work when at home and can enjoy the colors of life, not just the defects!

So, that was the list of non testing skills useful for testers in our opinion. Anything else, you can think of?

Krishna Iyer | CEO | Zen Test Labs

Pizzas, Sodas, Music and Testing…

Couple of weeks back we held our first crowd sourced testing event “the Zentestathon” which turned out to be a memorable event. Testing was on a module of a LMS application. Like all things that happen for the first time there were good results and valuable experiences that we experienced and I wanted to share some things that could help you with your crowd sourcing strategy.

Schedule

To ensure that the focus remains on testing and getting more number of defects, we discovered that maintaining a tight schedule is extremely critical to the success of the program. The schedule break up given below is what we found generating the optimum results

  • Application training-10% of the time
  • Exploring the application by the testers-15% of the time
  • Understanding business rules and requirements- 10% of the time
  • Testing-60% of the time

Other Observations

Some other observations include

  • We realized that, what most testers were skilled at was context driven testing.
  • Testers were able to suit themselves to the application and think of end to end scenarios with minimum amount of guidance
  • They add tremendous value through their experience, mentality and the ability to transcend based on this application.
  • In an extremely short time, they came up with various combinations of valid and invalid scenarios, which ensured test coverage.
  • Percentage of critical defects resulted in 40% by the virtue that there  were so many testers testing different scenarios, optimum test coverage was guaranteed.
  • The more the merrier is something that goes well with this form of testing as scenario coverage improves
  • Simulate end user scenario testing for internet bandwidth and other devices.

Through this initiative, we have validated our hunch and what we heard from a lot of people in the industry; that crowd sourced testing works best with applications that have a large global users base like mobile apps or games. We believe it will be time consuming and strenuous to involve external testers to crowd test for a product that requires a lot of internal communication with cross groups. All in all, this form of testing makes for an interesting venture and definitely has an exciting future ahead. We continue to watch what companies do globally in this space and innovate within this space. What have your experiences in crowd testing been?

Vijeethkumar Chathu |  Test Manager | Zen Test Labs

First dog fooding, now crowd sourcing…next crowd feeding?

When dog fooding was introduced by Microsoft manager Paul Maritz in 1988, it caught on like wild fire in the software space. Conceptually, dogfooding had existed in various forms till this point, but Microsoft was one of the early adopters when it came to incorporating this as a part of their product development cycles. Dog fooding essentially meant that internal users became early adopters of all new technology. Typically used with pre-release or beta versions of products, it gave product teams a bird’s eye view of how their products would work in “real-world” situations.  Forcing those who build products to use them is counter intuitive to the entire process of testing as more often than not they are bling to usability or are too advanced a user of the product. Hence while a lot of companies still conceptually use dog fooding to minimize the risk of critical failure there is an increasing trend to leverage large user bases to test too.

This is where crowdsourced testing has started to kick in. Testing companies are now providing platforms where product companies can test their products at a very low cost; i.e., typically charged a rate per bug detected. In turn, these companies open the platform to a community of testers who register to test voluntarily or as a part of a competition. Testers get paid per bug that they detect. While this kind of testing has opened up an unexplored talent pool (unbiased, cross geographic and large) at a low cost, the need to maintain independent testing either within their environment or outsourced remains. In addition to this, since there is no direct control over this crowd of testers, this source continues to remain an undependable source.

The ideal way forward would be to have a single platform that can integrate in-house testers or dogfooders, outsourced testers and crowd testers into a single platform. I refer to this concept as crowd feeding. Each product should have a crack team of testers from across these three channels that are nurtured over a period of time and have significant understanding of the application they are testing. This is akin to creating an elite panel of testers from these three channels that grow in experience over time.

The reason I mention that all three channels are critical to successful testing is

  1. In-house testers/ dogfooders – Advanced users with in-depth knowledge of product
  2. Outsourced testers- Intermediate/ Advanced independent users with in-depth knowledge of testing
  3. Crowd sourced testers- Cross section of low cost testers with diverse mix of “real-world” situational experience

Would be interesting to see how crowdsourced testing and dog fooding evolve over this year.

Hari Raghunathan | AVP | Zen Test Labs

Why automating regression cycles is a no brainer?

It is very surprising when you talk to customers who find regression cycles either painful or a drain on their productivity and at times a distraction from critical projects. What comes as a surprise is that most of them have an extremely mature and stable application and testing process. Yet, around regression cycles they scamper to get the entire suite executed and face the dilemma of having to deploy before completing the cycle.

Manual regression testing can be a mundane and boring activity for most test teams! Going through screens repeating the same steps and not finding bugs puts a tester’s concentration levels to the test. It also risks critical bugs escaping through due to the way these suites are structured. Over the years of executing numerous regression cycles I have seen that while automating regression is a great way to address the issue, other factors need to be considered to make regression cycles a cakewalk. I am outlining some of the steps that my teams find useful while automating regression suites

  • Find out how many of the existing test cases are automatable
  • Find out the coverage gap to identify ideal number of test cases
  • Run a test optimization drive (based on scientific methods) to arrive at the optimum number of test cases while ensure maximum coverage
  • Do not jump into QTP (explore other low cost options including Selenium)
  • Use a test automation framework that enables business users to develop, manage and execute automated regression suites
  • Train business users to automate on the fly and use the automation engineer only to maintain.

I acknowledge and understand that one can never achieve 100% test automation but the way I look at it, if you can achieve and significant amount (70-80%) you are going to end up achieving a dramatic reduction in the time taken to execute regression cycles. Eliminating pure testers from this process is also truly empowering your test teams to focus on critical projects.

Hari Raghunathan | AVP | Zen Test Labs

Upcoming testing trends?

I was at STARWEST, California earlier this month and it was interesting to hear some of the industry experts talk about current challenges software testers face. Even though, I could not make it to all sessions given the fact that we were presenting the last keynote; the ones I did attend were pretty good.

Most of the talk I heard was around how testing and testers today find themselves in a state of flux.

Development cycles are getting shorter and the ability to test rapidly in an agile mode going forward is what companies want. There was some talk around how testers need to also look at developing programming skills to stay ahead of the curve. I heard a few speakers say that you can’t dedicate time specifically to test, a few mentioned how you need to throw it out to users and let them test while others talked about how crowds could be sourced to find defects. I think this entire line of thinking was coming from one section of the software testing industry; i.e., the ones who are developing apps for mobile, web or cloud.

Take the case of our business, software testing in banks is an extremely critical activity. Not only does it involve large financial data but also invariably will have a lot of elements of regulation and compliance in it.  If I had to tell my customer, I will use a crowd to test your application which you can give your users to validate and all of this will save you 20 days- they’d just go elsewhere…To a bank the risk of losing 20 million is far greater than saving 20 days. Having said that, here’s what I feel. Going forward software testing will evolve into 2 areas; i.e., agile testing and traditional testing.

In the agile mode, most companies will be looking at optimizing, automating and crowd testing in a big way. The ideal scenario here would be to have a system which tells the developer whether he is breaking the code while developing and where he is breaking it. Companies will have a crack team of testers who can test and fix on the fly and the crowd will be used to see if anything got through.

Traditional testing will evolve to be a process wherein regression suites will be automated, business users will manually test releases with an automation engineer automating on the fly. Hybrid test automation frameworks will drive the change in this direction. Areas like performance testing and security will continue to remain specialized areas.

Be it the agile or traditional modes, testers will not be able to survive without one of the three skillsets; i.e., test automation, domain knowledge and the arguable one programming.

What do you think?

Hari Raghunathan | AVP | Zen Test Labs

Follow me on Twitter: hariraghunathan