WWW My Homepage


     Articles by category      Recent Articles         All articles   

Measuring Software product Quality:
Introduction of Risk Assessment in Testing Process

Kishore C.S.

 Judging software quality is very difficult task. Experience shows tendency to associate the quality of the software with the number of defects encountered during testing and after release is high. This approach is flawed because:

Any problem in the software gets recorded as a defect ranging from GUI Error to application failure.

If we continue to measure software quality just by the number of defects there will never be able to make any software release. Any software is bound to have defects. The business of the company need not depend on the number of defects in the software. A new perspective of looking at the quality of the software is by Assessing the risks associated with the software release. When a user uses the system, what are the likely risks associated with it and what it means to the user.

Thinking in a practical manner you would not visit a hotel if you are sure that by eating there will spoil your health. Even if the hotel happens to be a 5-star hotel you wouldn't risk it. Right? The name and hype do not matter when it comes to quality. It is the RISK that matters to you.

A list of risks is given in Box below in deploying a computer system [1]:

Computer system strategic risks

•Incorrect results will be produced

•Unauthorized transactions will be accepted by the system

•Computer file integrity will be lost

•Processing cannot be reconstructed

•Continuity of processing will be lost

•Service provided to the user will degrade to an unacceptable level

•Security of the system will be compromised

•Processing will not comply with organizational policy or governmental regulation

•Results of the system will be unreliable

•System will be difficult to use

•Programs will be unmaintainable

•System will not be portable to other hardware and software

•System will not be able to interconnect with other computer systems

•Performance level will be unacceptable

•System will be difficult to operate

Testers must understand that their business is to evaluate the business risk and to report those results to management. Viewed from this perspective, testers must first ensure they understand the business risk, and then develop test strategies focused on those risks. The highest business risk should receive the most test resources.

To effectively identify the business risks it is necessary to evolve risk assessment guidelines. Each defect recorded against software should be associated with a risk factor. An effective way of implementing this would be when a tester records a defect he should associate a risk factor based on the risk assessment guidelines. This risk factor should be reviewed by software development group and they should attach their own risk factor to it. A conflict in associating risk factor by a tester and developer needs careful analysis.

This risk factor can be the basis on which

  1. Prioritization of defects to be fixed should be done.
  2. Decisions to release the software can be made.
  3. Decisions whether to fix the defect/implement a feature can also be made.

 The five key process ideas (KPIs) of good enough software [2]

Systems thinking feeds nicely into risk analysis. Why risk analysis? Here's why. We are engulfed by uncertainty in every second of software development. When was the last time you were on a project that could not fail? If it really couldn't fail, weren't some of the resources or staff for the project more sorely needed elsewhere? In most situations, if we are on a project that can't fail today, tomorrow our project will be expanded, or our resources will be stolen. Efficient software development necessitates risk taking: The real question is whether we take calculated risks or accidental risks.

One way to avoid accidents is through a habit of integrated, structured risk management. By integrated, I mean that every strategy we use to create a positive result (e.g. adding a feature) has an associated risk management strategy (e.g. testing the feature). I've observed this habit in many companies but it was especially apparent in the Borland Languages division, where I worked for several years. We identified risks all through the project planning process, we reviewed them, we put risk reduction strategies in place, and we used postmortems and post-ship metrics to calibrate our judgment. We never quantified our risks, however, because we felt that such quantification would be impractical and misleading.

Let's examine a structured, non-quantitative approach to risk analysis. The structure we use at STL is: context, potential, occurrence, short-term consequences, and long-term consequences. When we analyze risk for our clients, we prepare a grid for each of the top 5-7 risk areas that examines that structure. For each element of the structure, we try to discover factors that increase or decrease the risk, and we suggest ideas for risk reduction.

The risk description is framed in terms of a problem that may happen and consequences that may follow. That risk has a context in which it exists. Eliminate the context, and we eliminate the risk. For example, if we want to eliminate the risk of software failure, we can avoid developing software in the first place. This is no joke: the common ailment of creeping featurism brings with it the attendant problems of testing, maintenance, documentation, and support for those features. We can eliminate those problems if we resist features that aren't essential to the success of the product.

Success of testing team will be the ability to identify high risk defects in software and ensure they are fixed.

 References:

[1]: Effective Methods for Software Testing By William Perry

[2]: The Challenge of "Good Enough" Software by James Bach, Satisfice Inc.