Test Planning

The Purpose of Test Planning

 

As part of a project, testing must be planned to ensure it delivers on its expected outcomes. It must be completed within a reasonable time and budget.

 

But test planning represents a special challenge.

 

The aim of test planning is to decide where the bugs in a product or system will be and then to design tests to locate them. The paradox is of course, that if we knew where the bugs were then we could fix them without having to test for them.

 

Testing is the art of uncovering the unknown and therefore can be difficult to plan.

 

The usual, naïve retort is that you should simply test “all” of the product. Even the simplest program however will defy all efforts to achieve 100% coverage (see the appendix).

 

Even the term coverage itself is misleading since this represents a plethora of possibilities. Do you mean code coverage, branch coverage,or input/output coverage ? Each one is different and each one has different implications for the development and testing effort. The ultimate truth is that complete coverage, of any sort, is simply not possible (nor desirable).

 

So how do you plan your testing?

 

At the start of testing there will be a (relatively) large number of issues and these can be uncovered with little effort. As testing progress more and more effort is required to uncover subsequent issues.


 

The law of diminishing returns applies and at some point the investment to uncover that last 1% of issues is outweighed by the high cost of finding them. The cost of letting the customer or client find them will actually be less than the cost of finding them in testing.

 

The purpose of test planning therefore is to put together a plan which will deliver the right tests, in the right order, to discover as many of the issues with the software as time and budget allow.


 

 

 

 

Defects

 

 

 

 

 

Time


 

 

Risk Based Testing


 

Figure 6: Typical defect discovery rate


 

Risk is based on two factors – the likelihood of the problem occurring and the impact of the problem when it does occur. For example, if a particular piece of code is complex then it will introduce far more errors than a simple module of code. Or a particular module of code could be critical to the success of the project overall. Without it functioning perfectly, the product simply will not deliver its intended result.

 

Both of these areas should receive more attention and more testing than less 'risky' areas.

 

But how to identify those areas of risk?


 

 

20


Software in Many Dimensions

It is useful to think of software as a multi-dimensional entity with many different axes. For example one axis is the code of the program, which will be broken down into modules and units. Another axis will be the input data and all the possible combinations. Still a third axis might be the hardware that the system can run on, or the other software the system will interface with.

 

Testing can then be seen as an attempt to achieve “coverage” of as many of these axes as possible .

Figure 7 : Software dimensions

Remember we are no longer seeking the impossible 100%

coverage but merely 'best' coverage, verifying the function of all the areas of risk.

 

Outlining

 

To start the process of test planning a simple process of ‘outlining’ can be used :

 

1.    List all of the 'axes' or areas of the software on a piece of paper (a list of possible areas can be found below, but there are certainly more than this).

 

2.    Take each axis and break it down into its component elements.

 

For example, with the axis of “code complexity” you would break the program down into the ‘physical' component parts of code that make it up. Taking the axis of “hardware environment” (or platform) it would be broken down into all possible hardware and software combinations that the product will be expected to run on.

 

3.    Repeat the process until you are confident you have covered as much of each axis as you possibly can (you can always add more later).

 

This is a process of deconstructing the software into constituent parts based on different taxonomies. For each axis simply list all of the possible combinations you can think of. Your testing will seek to cover as many elements as possible on as many axes as possible.

 

The most common starting point for test planning is functional decomposition based on a technical specification. This is an excellent starting point but should not be the sole 'axis' which is addressed – otherwise testing will be limited to 'verification' but not 'validation'.

 

Axis / Category

Explanation

Functionality

As derived from the technical specification

Code Structure

The organisation and break down of the source or object code

User interface elements

User interface controls and elements of the program

Internal interfaces

Interface between modules of code (traditionally high risk)

External interfaces

Interfaces between this program and other programs

Input space

All possible inputs

Output space

All possible outputs

Physical components

Physical organisation of software (media, manuals, etc.)

Data

Data storage and elements

Platform and environment

Operating system, hardware platform

Configuration elements

Any modifiable configuration elements and their values

Use case scenarios

Each use case scenario should be considered as an element