In the series we will define the basic terms that every developer needs to know about testing. First, we will start with defining what exeactly is testing and discussing the basic terms. The purpose is to give all team members a shared understanding of the fundamental terminology of the quality assurance and all related processes. Later this will improve the communication and reviews quality. It will further increase the testing capabilities of each member.
As part of the professional services we provide at BELLATRIX, we consult companies and help them to improve their QA process and set up an automated testing infrastructure. After the initial review process and giving improvement recommendations for some companies we need to hire new talents that can help the company to scale-up the solutions we provided. This was part of training we did for a company we consulted so that we educate all of their developers.
Definitions
Software quality definition found in IEEE Standard Glossary Of Software Engineering Terminology.
Definition
Software Quality – The degree to which a system, component, or process meets specified requirements
Definition
Software Quality – The degree to which a system, component, or process meets customer or user needs or expectations.
Note
Testing is a process rather than a single activity. It includes all life cycle activities such as static, dynamic testing, planning, preparation, evaluation of software products and related work products.
Main Objectives for Testing
-
Identification
Identify defects in the software
-
Confidence
Enable stakeholders to gain confidence in the system’s level of quality
-
Decision-making
Provide information for decision-making
-
Prevention
Prevent future bugs by improving the development process and standards
-
Help meeting standards
-
Contractual or legal requirements
-
Industry-specific standards
Cost of Failure
To correct a problem at requirements stage may cost $1. To correct the problem post-implementation may cost thousands of dollars. In extreme cases a software or systems failure may cost LIVES.
Terms
Definition
Anomaly – Any condition that deviates from expectation based on requirements specifications, design documents, user documents, standards, etc. or from someone’s perception or experience. Anomalies may be found during, but not limited to, reviewing, testing, analysis, compilation, or use of software products or applicable documentation. [IEEE1044]
Definition
Bug/Defect/Fault/Problem – A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
Definition
Failure – Definition Actual deviation of the component or system from its expected delivery, service or result. Also called ‘external fault’.
Definition
Defect/Fault Masking – An occurrence in which one defect prevents the detection of another.
How Much Testing Is Enough?
Too much testing can delay the product release and increase the product price. Insufficient testing hides risks of errors in the final product. We need a test approach which provides the right amount of testing for the project. We can vary the testing effort based on the level of risk (technical and business risks) in different areas.
Seven Testing Principle
-
1. Testing Shows Presence of Defects
Testing can show that defects are present. Cannot prove that there are no defects. Appropriate testing reduces the probability of undiscovered defects remaining in the software.
-
2. Exhaustive Testing Is Impossible
Testing everything (all combinations of inputs and preconditions) is not feasible except for trivial cases. Risks analysis and priorities should be used to focus testing efforts.
-
3. Early Testing
Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined objectives. The later a bug is found – the more it costs!
-
4. Defect Clustering
Testing effort should be focused proportionally to the expected and later observed defect density of modules. A small number of modules contain most of the defects discovered during pre-release testing or show the most operational failures.
-
5. Pesticide Paradox
Same tests repeated over and over again tend to loose their effectiveness. Previously undetected defects remain undiscovered. The test cases need to be regularly reviewed and revised. New and different tests need to be written.
-
6. Testing Is Context Dependent
Testing is done differently in different contexts.
-
7. Absence-of-errors Fallacy
Finding and fixing defects itself does not help if the system built is unusable and does not fulfill the users’ needs and expectations.
