In the series we will define the basic terms that every developer needs to know about testing. The purpose is to give all team members a shared understanding of the fundamental terminology of the quality assurance and all related processes. Later this will improve the communication and reviews quality. It will further increase the testing capabilities of each member. In this part, we will talk about the different testing types.
As part of the professional services we provide at BELLATRIX, we consult companies and help them to improve their QA process and set up an automated testing infrastructure. After the initial review process and giving improvement recommendations for some companies we need to hire new talents that can help the company to scale-up the solutions we provided. This was part of training we did for a company we consulted so that we educate all of their developers.
Definitions
Definition
System Testing – the process of testing an integrated system to verify that it meets specified requirements.
Definition
Component Testing – the testing of individual software components.
Definition
Component – a minimal software item that can be tested in isolation.
Definition
Integration Testing – testing performed to expose defects in the interfaces and in the interactions between integrated components or systems.
What does the product?
Definition
Functional Testing – testing based on an analysis of the specification of the functionality of a component or system.
How well does the product behave?
Definition
Non-functional Testing – testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability.
Definition
Acceptance Testing – formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.
Definition
Acceptance Criteria – the exit criteria that a component or system must satisfy in order to be accepted by a user, customer, or other authorized entity.
Definition
Off-the-shelf Software – a software product that is developed for the general market, i.e. for a large number of customers, and that is delivered to many customers in identical format.
Acceptance Testing
The focus is on the customer’s perspective and judgment. The customer is actually involved. Performed in a customer’s like environment. Finding defects is not the main focus.
For the integration of a commercial off-the-shelf (COTS) software product into a system, a purchaser may perform only integration testing at the system level and at a later stage acceptance testing.
Operational Testing
-
Disaster recovery
-
Testing of backuh4/restore
-
User management
-
Maintenance tasks
-
Security vulnerabilities
Alpha Testing
Takes place at the developer’s site. A cross-section of potential users and members of the developer’s organization are invited. Developers observe the users and note problems.
Beta Testing
Sends the system to a cross-section of users who install it and use it under real-world working conditions. The users send records of incidents with the system to the development organization.
Functional Testing
The function of a system (or component) is what it does.
-
Referred to as black-box testing
-
May be performed at all levels
-
Can be done from two perspectives:
-
Requirements-based
-
Uses a specification of the functional requirements
-
Business-process-based
-
Uses knowledge of the business processes
-
Use cases are a very useful basis for test cases
Non-Functional Testing
How well or with what quality the system should carry out its function.
-
Performed at all test levels
-
Includes, but is not limited to:
-
Performance testing
-
Load testing
-
Stress testing
-
Usability testing
-
Maintainability testing
-
Reliability testing
-
Portability testing
Non-Functional Quality Characteristics
Definition
Reliability – the ability of the software product to perform its required functions under stated conditions for a specified period of time, or for a specified number of operations.
Definition
Reliability Testing – the process of testing to determine the reliability of a software product.
Definition
Usability – divided into the sub-characteristics understandability, learnability, operability, attractiveness and compliance.
Definition
Usability Testing – testing to determine the extent to which the software product is understood, easy to learn, easy to operate and attractive to the users under specified conditions.
Definition
Efficiency – the capability of the software product to provide appropriate performance, relative to the amount of resources used under stated conditions.
Definition
Efficiency Testing – the process of testing to determine the efficiency of a software product.
Definition
Maintainability – the ease with which a software product can be modified to correct defects, modified to meet new requirements, modified to make future maintenance easier, or adapted to a changed environment.
Definition
Maintainability Testing – the process of testing to determine the maintainability of a software product
Definition
Portability – the ease with which the software product can be transferred from one hardware or software environment to another.
Definition
Portability Testing – the process of testing to determine the portability of a software product.
Re-testing (Confirmation testing)
After a defect is detected and fixed, the software should be re-tested. To confirm that the original defect has been successfully removed. This is called confirmation.
Regression Testing
-
Retest of a previously tested program
-
Needed after modifications of the program
-
Testing for newly introduced faults
-
As a result of the changes made to the system
-
May be performed at all test levels
Maintenance Testing
Definition
Maintenance Testing – testing the changes to an operational system or the impact of a changed environment to an operational system. The process of testing to determine the maintainability of a software product.
Main Тypes
-
Adaptive maintenance (Planned)
-
Product is adapted to new operational conditions
-
Corrective maintenance (Ad-hoc)
-
Defects being eliminated
Testing After Maintenance
-
Anything new or changed should be tested
-
Testing is needed even if only the environment is changed
-
Regression testing is required
-
The rest of the software should be tested for side effects
-
The scope is related to the risk of the change, the size of the existing system and to the size of the change
-
May be done at any or all test levels and for any or all test types
-
Key activity – impact analysis
Definition
Impact Analysis – the assessment of change to the layers of development documentation, test documentation and components, in order to implement a given change to specified requirements.
Triggers for Maintenance Testing
-
Planned modifications
-
Enhancement modifications (adapting software to the user’s wishes, for instance by supplying new functions or enhancing performance)
-
Changes of environment (planned operating system or database upgrades)
-
Corrective and emergency changes
-
Migration (from one platform to another)
-
Should include operational test of the new environment
-
Retirement of a system
-
May include the testing of data migration or archiving if long data-retention periods are required
Static Testing
Definition
Static Testing – testing of a component or system at specification or implementation level without execution of that software, e.g. reviews or static code analysis.
Definition
Static Analysis – analysis of software artifacts, e.g. requirements or code, carried out without execution of these software artifacts.
Early detection of defects prior test execution. Early warning about suspicious aspects of the code or design by the calculation of metrics, such as high complexity measure. Identification of defects not easily found by dynamic testing. Improved maintainability of code and design. Prevention of defects, as lessons learned in development.
Dynamic Тesting
Software is executed using a set of input values and its output is then examined and compared to what is expected.
-
Can only be applied to software code
-
Detect defects and to determine quality attributes of the code
-
Dynamic testing - Testing that involves the execution of the software of a component or system.
Dynamic testing and static testing are complementary methods, as they tend to find different types of defects effectively and efficiently.
