Software Testing in 2020

As a CEO of a testing company, a question that plays on my mind constantly is ‘what is the future of Testing?’ In the early 2000s, Ron Radice spoke at a QAI conference in India, where he had predicted that testing will die. His call was that automatic code generators will do the job so efficiently that testing will become obsolete. When he looked at the crystal ball then, he could see that prevention will be the creed and not detection.

Well, when I look at 2020, I believe Ron was right as well as wrong. Yes, code generators are arriving. Yes, there will be automated test case generators. Yes, model based testing will replace rudimentary testing activities. But, the whole boom of software especially in ubiquitous mobile devices means only more testing.

If the future includes automated cars like the Google driverless cars, I cannot imagine such a car with a technology that has not been fully and manually validated. If the future is the “Internet of Things”, I can only imagine that the amount of embedded testing will only explode. If the future is, business operations being handled through apps and app stores that have millions of applications pervading every step of our business and personal life then imagine the amount of mobile testing that will be required. If not anything, as everything gets more interconnected, the consequences of a critical failure will only be catastrophic. Wherever the nexus of cloud, social, mobile and big data takes us, I am thoroughly convinced that the need for testing will only grow.

While there a dime a dozen predictions on how things will look in 2020, my two bits around where testing will find itself as follows:

• Huge business opportunities arising from testing for app stores directly than app manufactures
• Test automation would have evolved from script less automation to automatic test case generators and execution
• The pressure to deploy rapidly in the Mobility and embedded devices space will mean that test automation tools will evolve to provide near and real time support to these areas
• Testing and testers will evolve to become super specialized with domain testers at one end and niche technical testers at the other end.

These are some things that come to mind and as the decade continues to evolve. Would be great to know what the rest of the testing world thinks.

Krishna Iyer|CEO|Zen Test Labs

Advertisements

Automation lessons learnt: Funding automation projects & the role of change management

One of the many reasons test automation is often compromised is in situations where business funds technology projects on a per project basis. Key reason being business benefits of automation, primarily time to market, are realized only during the subsequent releases of the application, never in the release where automation is undertaken. Even when business agrees to fund an automation project, the order of magnitude of benefits is small due to the potentially low levels of automation feasible within the project scope. The benefits accumulate only over a period of time from increasing automation levels and therefore the return on investment is realized over a longer duration. In order to reap benefits from automation business needs to continually invest in it and maintain a long term orientation to ROI. These are typical characteristics of any change initiatives. Test automation initiatives funded by individual business units can therefore learn from the vast expanse of knowledge pertaining to other organizational change initiatives and do not need to reinvent the wheel.

I have captured my experience of change initiatives as applied to automation in the visual below. It shows key components required to not only succeed at a pilot project but also create a cascading positive spiral where the benefits accumulate over time.

 

Like any other change initiative the key components form a chain, where the initiative is just as strong as its weakest link. Successful pilot accompanied by the right communication can act as a feeder to the next project and as long as all key components act in unison incremental benefits from each project can lead to significant cumulative benefits. The problem is that the first cycle tends to be demanding and it needs continuity of the champions until such time that the framework is institutionalized. A failure at early stages can have devastating effects with a stigma associated with it presenting greater roadblocks during subsequent attempts at automation. This is where senior management support from business and IT is crucial. A champion driving each automation cycle to success is central for the overall success of automation!

What do you think of the role of change management in automation projects? Have you had difficulty funding automation projects? Please feel free to share your experiences.

Aparna Katre | Director Strategy | Zen Test Labs

Pizzas, Sodas, Music and Testing…

Couple of weeks back we held our first crowd sourced testing event “the Zentestathon” which turned out to be a memorable event. Testing was on a module of a LMS application. Like all things that happen for the first time there were good results and valuable experiences that we experienced and I wanted to share some things that could help you with your crowd sourcing strategy.

Schedule

To ensure that the focus remains on testing and getting more number of defects, we discovered that maintaining a tight schedule is extremely critical to the success of the program. The schedule break up given below is what we found generating the optimum results

  • Application training-10% of the time
  • Exploring the application by the testers-15% of the time
  • Understanding business rules and requirements- 10% of the time
  • Testing-60% of the time

Other Observations

Some other observations include

  • We realized that, what most testers were skilled at was context driven testing.
  • Testers were able to suit themselves to the application and think of end to end scenarios with minimum amount of guidance
  • They add tremendous value through their experience, mentality and the ability to transcend based on this application.
  • In an extremely short time, they came up with various combinations of valid and invalid scenarios, which ensured test coverage.
  • Percentage of critical defects resulted in 40% by the virtue that there  were so many testers testing different scenarios, optimum test coverage was guaranteed.
  • The more the merrier is something that goes well with this form of testing as scenario coverage improves
  • Simulate end user scenario testing for internet bandwidth and other devices.

Through this initiative, we have validated our hunch and what we heard from a lot of people in the industry; that crowd sourced testing works best with applications that have a large global users base like mobile apps or games. We believe it will be time consuming and strenuous to involve external testers to crowd test for a product that requires a lot of internal communication with cross groups. All in all, this form of testing makes for an interesting venture and definitely has an exciting future ahead. We continue to watch what companies do globally in this space and innovate within this space. What have your experiences in crowd testing been?

Vijeethkumar Chathu |  Test Manager | Zen Test Labs

Software Quality, Development and Coding Standards…

The best applications are those that are not only coded properly but also easy to add, debug and maintain. The concept of ‘maintainable code’ is easy to contemplate about but difficult to practice. Developers code in specific and individualistic styles. Their styles of coding become their second nature and they continue to use that in everything they create. Such a style might include the conventions used to name variables and functions ($password, $Password or $pass_word for example). Any style should ensure that the team can read the code easily.

However, what happens when we start to code bigger projects and introduce additional people to help create a large application? Conflicts in the way you write your code will most definitely appear.

This is where the concept of ‘CODING STANDARDS’ comes into play.

Coding standards are very articulate and deeply formulated to be consistent and when developers follow these standards, it makes the end result more uniform, even if different parts of the application are written by different developers. Knowing these standards and the language is always easy, but the catch is in deciding which standard to apply when & where.

I have found that the entire process of testing & quality assurance becomes relatively simpler when developers have followed coding standards. This also goes a long way in improving the performance of the application. When coding standards have been adhered to it results in easy and quick grasping of what the application is supposed to do and what it is not supposed to. During maintenance phase, it undoubtedly, enhances readability which leads to better maintainability. By adhering to these standards testers do not feel disconnected or bowled over when they begin working with the application. I believe it’s even more relevant in the current scenario, when different processes of applications are built by different set of developers (internal teams, vendors, etc).

Some of the secure coding practices, I believe that are high priority:

  • Validate input data
  • Heed compiler warnings
  • Architect and design for security policies
  • Sanitize transferred data
  • Mitigate model threat
  • Checklist

I understand that it is not possible to apply all coding standards at all times, but if applied appropriately, it would enhance performance and reduce scientific misconduct. What are your views on coding standards and its impact on software testing?

Janhavi Hunnur | Marketing | Zen Test Labs

The evolution of test design techniques

Test Design is a constantly evolving area of the software testing life cycle. Surprisingly, it still remains implemented at a relatively smaller scale whenever new application features are defined and this has gone a long way in limiting the effectiveness and efficiency of testing.

The heart of it has always been the test design template, the primary goal of which is to reduce ad-hoc reinvention of test designs. However, most companies/ individuals do not assign as much importance to this as they do to the development process. Ideally, a good test design frees tester to focus on the actual job of testing the application. I have seen that just by using multiple test design techniques, the efficiency gain that we have been able to achieve has been tremendous. Some of them include

  • Specification based techniques like equivalence partitioning, boundary value analysis, state transition testing, use case testing and decision table testing
  • Structure based techniques/ White box testing through statement testing & coverage and decision testing coverage
  • Experience based techniques like error guessing and exploratory testing

Our teams have been using these techniques for quite some time now and have come up with innovative ways of creating classification trees, in order to effectively design and generate test cases, for complicated applications.  One way is, wherein we map the required output so that the classification of inputs and outputs is matched.

We are in the process of publishing a whitepaper on this area early next quarter that details out how one should go about Test Design.

Watch this space for more on the paper…

Sunil Deshmukh | Senior Test Manager | Zen Test Labs

Why automating regression cycles is a no brainer?

It is very surprising when you talk to customers who find regression cycles either painful or a drain on their productivity and at times a distraction from critical projects. What comes as a surprise is that most of them have an extremely mature and stable application and testing process. Yet, around regression cycles they scamper to get the entire suite executed and face the dilemma of having to deploy before completing the cycle.

Manual regression testing can be a mundane and boring activity for most test teams! Going through screens repeating the same steps and not finding bugs puts a tester’s concentration levels to the test. It also risks critical bugs escaping through due to the way these suites are structured. Over the years of executing numerous regression cycles I have seen that while automating regression is a great way to address the issue, other factors need to be considered to make regression cycles a cakewalk. I am outlining some of the steps that my teams find useful while automating regression suites

  • Find out how many of the existing test cases are automatable
  • Find out the coverage gap to identify ideal number of test cases
  • Run a test optimization drive (based on scientific methods) to arrive at the optimum number of test cases while ensure maximum coverage
  • Do not jump into QTP (explore other low cost options including Selenium)
  • Use a test automation framework that enables business users to develop, manage and execute automated regression suites
  • Train business users to automate on the fly and use the automation engineer only to maintain.

I acknowledge and understand that one can never achieve 100% test automation but the way I look at it, if you can achieve and significant amount (70-80%) you are going to end up achieving a dramatic reduction in the time taken to execute regression cycles. Eliminating pure testers from this process is also truly empowering your test teams to focus on critical projects.

Hari Raghunathan | AVP | Zen Test Labs

Upcoming testing trends?

I was at STARWEST, California earlier this month and it was interesting to hear some of the industry experts talk about current challenges software testers face. Even though, I could not make it to all sessions given the fact that we were presenting the last keynote; the ones I did attend were pretty good.

Most of the talk I heard was around how testing and testers today find themselves in a state of flux.

Development cycles are getting shorter and the ability to test rapidly in an agile mode going forward is what companies want. There was some talk around how testers need to also look at developing programming skills to stay ahead of the curve. I heard a few speakers say that you can’t dedicate time specifically to test, a few mentioned how you need to throw it out to users and let them test while others talked about how crowds could be sourced to find defects. I think this entire line of thinking was coming from one section of the software testing industry; i.e., the ones who are developing apps for mobile, web or cloud.

Take the case of our business, software testing in banks is an extremely critical activity. Not only does it involve large financial data but also invariably will have a lot of elements of regulation and compliance in it.  If I had to tell my customer, I will use a crowd to test your application which you can give your users to validate and all of this will save you 20 days- they’d just go elsewhere…To a bank the risk of losing 20 million is far greater than saving 20 days. Having said that, here’s what I feel. Going forward software testing will evolve into 2 areas; i.e., agile testing and traditional testing.

In the agile mode, most companies will be looking at optimizing, automating and crowd testing in a big way. The ideal scenario here would be to have a system which tells the developer whether he is breaking the code while developing and where he is breaking it. Companies will have a crack team of testers who can test and fix on the fly and the crowd will be used to see if anything got through.

Traditional testing will evolve to be a process wherein regression suites will be automated, business users will manually test releases with an automation engineer automating on the fly. Hybrid test automation frameworks will drive the change in this direction. Areas like performance testing and security will continue to remain specialized areas.

Be it the agile or traditional modes, testers will not be able to survive without one of the three skillsets; i.e., test automation, domain knowledge and the arguable one programming.

What do you think?

Hari Raghunathan | AVP | Zen Test Labs

Follow me on Twitter: hariraghunathan