Risk Based Testing

Testing – The Conundrum

In some ways testing is a lose/lose proposition. Come in under effort, and it will be inferred that you pad your estimates, do a good job and there will be mutterings about the slipping of the release, and if there is a failure in Production and you did poor testing.

This is all compounded by compression of timeframes after every other milestone has slipped, but as Testing is wedged up against the deployment our slippage or transfer of slippage is externally visible and much harder to mitigate.

So to succeed we need Systems. But we need Systems that exist inside a framework and give us the opportunity to rationalise, justify, clearly articulate the positions we have taken, the underlying decision process and how this impacts the risk profile of a release.

Not all Systems are suitable for all projects, and sometimes a number of Systems could be used to get the testing completed and give the best outcome for a given resource profile.

TestingLinksTo Printer

 

Here are some things that have worked for me over time and I am comfortable recommending.

 

 Risk Based Testing – an Approach

Risk cannot be used in isolation to determine your testing coverage and approach. It needs to be balanced against the importance of the feature[2].

A quick browse of the internet will find many sites and documents about how to quantify risk or prioritise test cases, I struggled to find sites that brought both aspects together in a cohesive framework. I also find it frustrating that most of the work is around gaining testing efficiencies or prioritisation after the scripts are written (ie out of what I have written, what is most important, not what should I have written).

For the purpose of this discussion, I will use the following definitions;

For Risk, I will use Steve Wakeland’s[3] definition which defines risk as ‘the likelihood that a program fault will result in an impact on the business’.

As I am struggling to find a definition of Importance, I will use the following; where a behaviour[4] would have an impact on the financial, operational or reputational position of an organisation. For the purpose of this discussion we are dealing with negative behaviours.

Using the two axes you can break the areas under test into 4 quadrants

4Q png

  • Q1; High Risk, Low Importance
  • Q2; High Risk, High Importance
  • Q3; Low Risk, Low importance
  • Q4; Low Risk, High Importance

 

Q1 High Risk, Low Importance

This is the quadrant where most Test Managers and teams often get distracted and spend time that does not provide a great return on the effort expended.

The important thing to remember is that a negative behaviour here, while carrying more risk has limited impact to an organisation and can be lived with for a period of time.

Any testing that is done here can be quite informal in nature, and should focus on obtaining the most coverage with limited depth.

These will generally be internal systems that do not support an organisation’s core role.

If you have limited Developer resource, you may wish to lower the priority of these defects so they can focus on more important areas

 

Q2 High Risk, High Importance

This is the quadrant where most effort should be expended and negative behaviours found here are going to provide the highest value outcomes to an organisation.

Testing should be more formal and extensive with a focus on both coverage and depth.

These will be systems that are customer facing and support the business’s core role. They also may be systems that feed into other core systems.

 

Q3 Low Risk, Low Importance

This is the quadrant where least effort should be expended.

Testing here can be informal in nature, i.e. guerrilla and rather than test cases I would create test philosophies

These will generally be internal systems that do not support an organisation’s core role.

 

Q4 Low Risk, High Importance

This is the forgotten quadrant, lured by the high risks associated with Q1, resource focus on that at the detriment of this this area.

The key point here is that while features may be rare, when realised they expose a negative impact to an organisation.

Testing here needs to cover both breadth and a reasonable amount of depth.

These tend to be low transaction systems that support the business’s core role.

 

Test Effort Focus

If we depict this diagrammatically, we end up with something that looks like this

TestingDirection

Everything is High Risk and High Importance

This can happen and as long as an organisation accepts that, and funds the project (and the testing) appropriately then you should have no problems. I still believe that you can divide up your features on the same quadrants. Obviously you will need to test more formally across all quadrants and test each quadrant, but rigor and effort should still be more focused as you head towards that top right corner.

IfAllRiskIsHigh

 

 

 

[1] http://www.cs.tut.fi/tapahtumat/testaus04/schaefer.pdf

[2] I use Feature as a generic term to cover off features, functions, applications, nodes etc.

[3] http://www.stickyminds.com/article/strategy-risk-based-testing

[4] I use the term Behaviour as a generic term to cover defects, issues, etc as that is what is observed

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *