I’m a big fan of QA and delivering a good product to the client, and I guess everyone delivering a service is a fan of ‘delivering a good product’. But to me the most important part of a project and ‘delivering a good product’ is getting the QA right. Deliver a product full of bugs and it reduces the confidence in the solution, which in turn you spend your time firefighting. Deliver one that has been tested in a thorough manner and the project is confident in the product and they can concentrate on its usability and be ambassadors for the solution.
The primary purpose of testing is to ensure that defects may be discovered and corrected, and at the completion of testing the stakeholders are confident that the product has met the requirements and is functioning correctly. Testing cannot establish that a product functions properly under all conditions, but it can establish that it does function properly under specific conditions and is acceptable to the target end users.
Part 1: Why is Software testing so hard to get right?
According to a 2002 study by the NIST (National Institute of Standards and Technology) software bugs cost the US economy almost $60BN annually and over a third of this cost could be avoided if better Software testing was performed. To understand how to carry out an effective testing methodology that provides a high level of confidence to the stake holders and end users, we need to look at the key challenges in carrying out good testing.
1.1 Getting the requirements right
The biggest challenge companies’ face in delivering a solution is delivering a solution that meets what the customer requires. Unfortunately what the customer really requires and the customer’s requirements are not often the same. Even with extensive input and analysis of user requirements it’s hard to get away from the fact that users only think they know what they want and this is often based on the system they already know and are comfortable with. Users don’t know what they really want until they see it and begin to use it and understand what the capabilities of the new system is.
1.2 Spending enough time testing
Estimating the amount of time a project takes is a complicated art. I’m using the word ‘art’ because with so many variables it can’t be calculated in a scientific manner. It’s the best approximation based on the experience and knowledge of the individuals creating the estimate. Often the project is planned with the premise that everything will go to plan and hence there is little contingency built into the timelines for things to go wrong. If and when the project overruns due to ‘things going wrong’ in the analysis and the development stages, project managers tend to compress the testing timelines rather than extending the project and therefore forcing the testing into a smaller time frame.
1.3 Involving the right people
A successful project needs the involvement of the right people as early as possible, a lot or projects are management driven and this leads to the design and product satisfying the needs of the management instead of the end users. Traditionally a product is developed and then delivered to the functional team to test, this leads to a certain disconnect between what the user thinks it will do and what the developers think it should do. Developers working on a project can get “heads down” and deliver a product that may work correctly but lacks usability. If a subsequent bug is reported the response is usually ‘it’s working as expected’ especially knowing what the system is supposed to do, it’s easy to miss difficulties for users.
1.4 Testing every line of code or the whole system
How much testing is right? Every line of code is a potential bug and it’s also part of a complicated system full of interdependencies, this coupled with the fact that the system may be developed in one controlled environment and deployed on to many different and changing environments means that there are a lot of variables to consider when carrying out testing. The complexity of software means even if we had infinite time and resources it’s not possible to test every scenario, the best QA process are trying to release the product with the least possible likelihood of there being defects.
One of the aims of software development is to better some existing process or system and therefore a lot of projects are introducing new functionality or wholly new systems and processes and it’s difficult for the testers to know what can and should be tested. When faced with situation like this, testers will revert to what they know and have experience of and this may result in some scenarios never being tested.
1.5 Getting what was expected
The last difficulty applies mainly to customization and extension of ”Off the Shelf” software. Vendors inevitably are always aiming to upsell their products and their capabilities and features. What is offered and what is delivered may have a large chasm of missing or incomplete items key to the business requirements. The customisation and development of this software requires treading a fine line between meeting the requirements and working within the bounds of what the software will allow the developers to do.
Stay tuned for Part 2: How do you do effective Software Testing?…