Defining High-Quality Test Attributes for Automated Tests

To be able to write high-quality automated tests, more knowledge is needed than just knowing how to program in a certain language or use a specific framework. To solve these problems, our automated tests should have some core high-quality test attributes. Also, we will talk about related programming principles abbreviated SOLID.

For more detailed overview and usage of many more design patterns and best practices in automated testing, check my book “Design Patterns for High-Quality Automated Tests, C# Edition, High-Quality Tests Attributes, and Best Practices”.  You can read part of three of the chapters:

Benchmarking for Assessing Automated Test Components Performance

Generic Repository Design Pattern- Test Data Preparation

SOLID Principles

Before we can define the high-quality test attributes, we should mention some of the well-known object-oriented programming principles. Throughout the book I will mention some of them. SOLID is a mnemonic acronym for five design principles. The goal of these principles is to help us in writing more understandable, flexible, and maintainable software. 

SOLID stands for:

• SRP – Single Responsibility Principle

• OCP – Open/Closed Principle

• LSP – Liskov Substitution Principle

• ISP – Interface Segregation Principle

• DIP – Dependency Inversion Principle

Note

The principles are a subset of many principles promoted by Robert C. Martin (colloquially known as “Uncle Bob”). He is best known for being one of the authors of the Agile Manifesto. The theory of SOLID principles was introduced by Martin in his 2000th paper “Design Principles and Design Patterns”, although the SOLID acronym itself was introduced later by Michael Feathers.

SRP – Single Responsibility Principle

The principles states – “Every software module should have only one reason to change” which means that every class, method, etc. can do many things, but they should serve a single purpose.

All the methods and variables in the class should support this purpose. Everything else should be removed. It shouldn’t be like a swiss knife providing 20 different utility actions since if one of them is changed, all others need to be updated too.

But if we can have each of those items separated it would be simple, easy to maintain, and one change will not affect the others. The same principle also applies to classes and objects in the software architecture- you can have them as separate simpler classes.

Let me give you an example.

public class CustomerOrder
{
    public void Create()
    {
        try
        {
            // Database code goes here
        }
        catch (Exception ex)
        {
            File.WriteAllText(@"C:exception.txt", ex.ToString());
        }
    }
}

The CustomerOrder class is doing something which it is not supposed to do. It should create purchases and save them in the database, but if you look at the catch block closely, you will see that it also does log activity. It has too many responsibilities.

If we want to follow the Single Responsibility principle SRP, we should divide the class into two separate simple classes. We can move the logging to a separate class.

public class FileLogger
{
    public void CreateLogEntry(string error)
    {
        File.WriteAllText(@"C:error.txt", error);
    }
}

After that the CustomerOrder class can delegate the logging to the FileLogger class and be more focused on the creating purchases.

public class CustomerOrder
{
    private FileLogger _fileLogger = new FileLogger();
    public void Create()
    {
        try
        {
            // Database code goes here
        }
        catch (Exception ex)
        {
            _fileLogger.CreateLogEntry(ex.Message);
        }
    }
}

OCP – Open/Closed Principle

The principle says that “software entities (classes, modules, functions, etc.) should be open for extension but closed for modification“. It means that, if new requirements are written for the already implemented functionality, these can be added in a way so that we don’t need to change the whole structure of the existing code that has been already unit tested.

 Since if we change it and add it directly, we can trigger new regression problems.

How about looking at an example?

public enum OrderType
{
    Normal,
    Silver,
}
public class DiscountCalculator
{
    public double CalculateDiscount(OrderType orderType, double totalPrice)
    {
        if (orderType == OrderType.Silver)
        {
            return totalPrice - 20;
        }
        else
        {
            return totalPrice;
        }
    }
}

Look at the IF condition in the CalculateDiscount method. The problem is that, if we add new order types, we will have to add one more IF condition, meaning that we will have to change the implementation inside the DiscountCalculator class, because of a change that has happened outside of it. If we need to change the class every second week, we need to ensure that the previous requirements are still satisfied, and the new ones integrate well with the old ones. Instead of modifying the existing code for every new condition, we strive to develop a solution that can be extensible.

We can easily refactor this code to follow the Open/Closed principle OCP, so every time a new order type is added, we create a new class, as shown in the example. The existing code stays untouched, and we need to test and check only the new cases.

public class DiscountCalculator
{
    public virtual double CalculateDiscount(double totalPrice)
    {
        return totalPrice;
    }
}
public class SilverDiscountCalculator : DiscountCalculator
{
    public override double CalculateDiscount(double totalPrice)
    {
        return base.CalculateDiscount(totalPrice) - 20;
    }
}
public class GoldDiscountCalculator : DiscountCalculator
{
    public override double CalculateDiscount(double totalPrice)
    {
        return base.CalculateDiscount(totalPrice) - 50;
    }
}

LSP – Liskov Substitution Principle

The principle says that “you should be able to use any derived class instead of a parent class and have it behaved in the same manner without modification“. It means that the child class shouldn’t modify or change how the base class behaves. 

The main point is that the child classes can be used as if they are their parent itself, but if you change behavior in a child class, unexpected issues will occur.

Shall we continue with the same DiscountCalculator example? We want to introduce a new type of discount for bonus points. To calculate it we will add a new method to our base class.

public abstract class DiscountCalculator
{
    public virtual double CalculateRegularDiscount(double totalPrice)
    {
        return totalPrice;
    }
    public virtual double CalculateBonusPointsDiscount(double totalPrice, int points)
    {
        return totalPrice - points * 0.1;
    }
}

As you can see, we have renamed the CalculateDiscount method to CalculateRegularDiscount and we added a new method for calculating the discounted total price based on the bonus points.

public class SilverDiscountCalculator : DiscountCalculator
{
    public override double CalculateRegularDiscount(double totalPrice)
    {
        return base.CalculateRegularDiscount(totalPrice) - 20;
    }
    public override double CalculateBonusPointsDiscount(double totalPrice, int points)
    {
        return totalPrice - points * 0.5;
    }
}
public class GoldDiscountCalculator : DiscountCalculator
{
    public override double CalculateRegularDiscount(double totalPrice)
    {
        return base.CalculateRegularDiscount(totalPrice) - 50;
    }
    public override double CalculateBonusPointsDiscount(double totalPrice, int points)
    {
        return totalPrice - points * 1;
    }
}
public class PlatinumDiscountCalculator : DiscountCalculator
{
    public override double CalculateRegularDiscount(double totalPrice)
    {
        return base.CalculateRegularDiscount(totalPrice) - 100;
    }
    public override double CalculateBonusPointsDiscount(double totalPrice, int points)
    {
        throw new InvalidOperationException("Not applicable for Platinum orders.");
    }
}

What we did here is to add a new type of order- Platinum. We have already applied the maximum allowed discount, so the bonus points discount is not applicable for Platinum orders. In this case, we decide to throw InvalidOperationException to let the calling methods know that this operation is not supported for the Platinum type. As the Polymorphism from the OOP Principles states, we can use any of the child classes calculators derived from the DiscountCalculator, as if they are the actual DiscountCalculator class.

Thanks to this principle, as you can see the code below, I have created a collection of DiscountCalculator objects where I can add Silver, Gold, and Platinum Discount Calculators as if they are instances of the same type - DiscountCalculator. After that, I can go through the list using the parent customer object and invoke the calculation methods.

var _discountCalculators = new List<DiscountCalculator>();
_discountCalculators.Add(new SilverDiscountCalculator());
_discountCalculators.Add(new GoldDiscountCalculator());
_discountCalculators.Add(new PlatinumDiscountCalculator());
foreach (DiscountCalculator discountCalculator in _discountCalculators)
{
    double bonusPointsDiscount = discountCalculator.CalculateBonusPointsDiscount(1250);
}

So far so good, but when the CalculateBonusPointsDiscount of the PlatinumDiscountCalculator is invoked, it leads to InvalidOperationException. The problem is that the PlatinumDiscountCalculator object looks like a DiscountCalculator, but the implementation of the child object has changed the expected behavior of the parent method. So, to follow the Liskov principle, we need to create two interfaces, one for the regular and other for the bonus points discount.

public interface IRegularDiscountCalculator
{
    double CalculateRegularDiscount(double totalPrice);
}
public interface IBonusPointsDiscountCalculator
{
    double CalculateBonusPointsDiscount(double totalPrice, int points);
}

I will spare you all the refactoring that we need to do. Let’s see how we will use the code to fix the problem we faced.

var _discountCalculators = new List<IBonusPointsDiscountCalculator>();
_discountCalculators.Add(new SilverDiscountCalculator());
_discountCalculators.Add(new GoldDiscountCalculator());
// _discountCalculators.Add(new PlatinumDiscountCalculator()); // we cannot add it
foreach (IRegularDiscountCalculator discountCalculator in _discountCalculators)
{
    double bonusPointsDiscount = discountCalculator.CalculateBonusPointsDiscount(1250);
}

Instead of using the base class, we use the IBonusPointsDiscountCalculator interface. This means that all added objects to the collection will have this method implemented otherwise we won’t be able to add it to the list.

ISP – Interface Segregation Principle

The principle states that “clients should not be forced to implement interfaces they don’t use. Instead of one fat interface, many small interfaces are preferred based on groups of methods, each one serving one sub-module“.

 The Single Responsibility principle can be applied not only for classes but for the interfaces as well. Each interface should provide methods that serve a single purpose. In this case the interface gives us functions for more than one goal, this could lead our implementation code to include methods that are not needed in the class.

Why not see an example of why we shouldn’t use one fat interface? We can use our previous example for discount calculators. Instead of creating two separate interfaces we could easily create a single one. To simplify the code, let’s remove the inheritance.

public interface IDiscountCalculator
{
    double CalculateRegularDiscount(double totalPrice);
    double CalculateBonusPointsDiscount(double totalPrice, int points);
}
public class GoldDiscountCalculator : IDiscountCalculator
{
    public double CalculateRegularDiscount(double totalPrice)
    {
        return totalPrice - 50;
    }
    public double CalculateBonusPointsDiscount(double totalPrice, int points)
    {
        return totalPrice - points * 1;
    }
}
public class PlatinumDiscountCalculator : IDiscountCalculator
{
    public double CalculateRegularDiscount(double totalPrice)
    {
        return totalPrice - 100;
    }
    public double CalculateBonusPointsDiscount(double totalPrice, int points)
    {
        throw new NotImplementedException("Not applicable for Platinum orders.");
    }
}

As you can see, we have a similar problem. Since we have one fat interface, we are forced to implement all methods- even when we may not need them. In the future this could become even worse, imagine that we need to add another type of discount, but this time it is not applicable to the Silver orders. This would mean that in the SilverDiscountCalculator we will have a new method that throws NotImplementedException. Better solution would be creating a new interface rather than updating the current interface.

If you decide to follow my suggestion about moving some of OCP examples here, this conclusion section should be revised

DIP – Dependency Inversion Principle

The principle states that “high-level classes should not depend on low-level classes. Both types should be created based on abstractions. These abstractions shouldn’t depend on any low-level details. All details should use the abstractions instead”. 

The high-level classes are usually the ones that contain the business logic, and the low-level ones consist of actions such as CRUD DB operations, reading files or calling web APIs. When we say that a class is tightly coupled it means that it has access to low-level details (that it shouldn’t have) instead of using the abstractions.

How about extending our CustomerOrder example a bit? The requirements have already changed, and we have different types of orders to process- silver, gold and platinum. Also, there are new requirements which state that we need to add two more ways of notifications in case something happens. For example, sending an email and SMS with updates. To do so:

We will create a common interface for all loggers called ILogger.

public interface ILogger
{
    void CreateLogEntry(string errorMessage);
}

After that we will create three separate loggers- one for file logging, one for email and one more for SMS.

public class FileLogger : ILogger
{
    public void CreateLogEntry(string errorMessage)
    {
        File.WriteAllText(@"C:exceptions.txt", errorMessage);
    }
}
public class EmailLogger : ILogger
{
    public void CreateLogEntry(string errorMessage)
    {
        EmailFactory.SendEmail(errorMessage);
    }
}
public class SmsLogger : ILogger
{
    public void CreateLogEntry(string errorMessage)
    {
        SmsFactory.SendSms(errorMessage);
    }
}

The business logic behind the new requirements is that we need to send an email for gold orders and an SMS for platinum orders since they bring our company more money.

public class CustomerOrder
{
    public void Create(OrderType orderType)
    {
        try
        {
            // Database code goes here
        }
        catch (Exception ex)
        {
            switch (orderType)
            {
                case OrderType.Platinum:
                    new SmsLogger().CreateLogEntry(ex.Message);
                    break;
                case OrderType.Gold:
                    new EmailLogger().CreateLogEntry(ex.Message);
                    break;
                default:
                    new FileLogger().CreateLogEntry(ex.Message);
                    break;
            }
        }
    }
}

The code violates the Single Responsibility principle again. It should be focused on creating purchases, but also decides which object to be created, while it is not the work of the CustomerOrder class to determine which instances of the ILogger should be used. The biggest problem here is related to the new keyword. This is an extra responsibility of making the decision which objects to be created, so if we delegate this responsibility to someone other than the CustomerOrder class, that will solve the problem.

We can have different child classes of CustomerOrder for the different types of orders. Also, the logger can be passed as dependency rather than creating it in the method itself.

public class CustomerOrder
{
    private ILogger _logger;
    public CustomerOrder(ILogger logger)
    {
        _logger = logger;
    }
    public void Create()
    {
        try
        {
            // Database code goes here
        }
        catch (Exception ex)
        {
            _logger.CreateLogEntry(ex.Message);
        }
    }
}
public class GoldCustomerOrder : CustomerOrder
{
    public GoldCustomerOrder()
    : base(new EmailLogger())
    {
    }
}
public class PlatinumCustomerOrder : CustomerOrder
{
    public PlatinumCustomerOrder()
    : base(new SmsLogger())
    {
    }
}

Note

Keep in mind that the example was oversimplified. In production code, we usually use special frameworks for the job called inversion of control containers. The container uses the declared injection interfaces to figure out the dependencies and based on them to inject the correct dependencies, instead of passing them as parameters to the methods.

High-Quality Test Attributes

The time has come to define the automated test’s high-quality attributes. You will find that some of these attributes are connected to the SOLID principles and throughout the book, we will continue to talk about these connections. Since the book is also about design patterns, and we haven’t discussed them yet, we will first go through these definitions, and then we will go briefly though each of the high-quality attributes. We will not go into many details now since there will be a dedicated chapter for each of the attributes.

What Is a Design Pattern?

Definition

Design patterns as prescribed solutions to everyday software challenges. They don’t consist of code or any specified algorithm, but instead, they describe how to group your logic smartly, reuse it, or to make it easier to maintain. It is a template for solving design problems, which we can use while we create our software solutions.

Test Maintainability and Reusability

Definition

Maintainability – The ease with which we can customize or change our software solution to accommodate new requirements, fix problems, improve performance.

Imagine there is a problem in your tests. How much time do you need to figure out where the problem is? Is it an automation bug or an issue in the system under test? In the next chapters, we will talk in detail about how to create maintainable tests using various practices and design patterns. But if at the beginning, you haven’t designed your code in such a way, the changes may need to be applied to multiple places which can lead to missing some of them and thus resulting in more bugs. The better the maintainability is, the easier it is for us to support our existing code, accommodate new requirements, or just to fix some bugs.

A closely related principle to this definition is the so-called DRY principle- Don’t Repeat Yourself. The most basic idea behind the DRY principle is to reduce long-term maintenance costs by removing all unnecessary duplication.

Note

As Donald Knuth so eloquently said, “Premature optimization is the root of all evil (or at least most of it) in programming.” We should not remove only the duplicate code and duplicate test implementations, but we also should remove duplicate test goals. David Thomas and Andrew Hunt formulated the DRY principle in their book, The Pragmatic Programmer, by Andrew Hunt and David Thomas, published by Addison-Wesley Professional. The DRY principle is sometimes referred to as Single Source of Truth (SSOT) or Single Point of Truth (SPOT) because it attempts to store every single piece of unique information in a single place only.

Test Readability

By reading the code, you should be able to find out what the code does easily. A code that is not readable usually requires more time to read, maintain, understand and can increase the chance to introduce bugs. Some programmers use huge comments instead of writing more simple readable code. It is much easier to name your variables, methods, classes correctly, instead of relying on these comments. Also, as the time goes by, the comments are rarely updated, and they can mislead the readers.

API Usability

As we mentioned above, the API is the specification of what you can do with a software library. When we use the term usability together with the term API, it means “How easy it is for you as a user to find out what the methods do and how to use them?”. In the case of a Test Library - “How much time a new user needs to create a new test?”

In the programming community, we sometimes use another term for the same thing called syntactic sugar. It describes how easy it is to use or read some expressions. It sweetens the programming languages for humans. The programming statements become more concise and clearer.

Note

The term syntactic sugar was invented by Peter J. Landin in 1964. It described the syntax of a programming language similar to ALGOL which was defined semantically in terms of the applicative expressions of lambda calculus, centered on lexically replacing λ with “where”.

Extensibility

One of the hardest things to develop is to allow these generic frameworks to be extensible and customizable. The whole point of creating a shared library is to be used by multiple teams across the company. However, the different teams work in a different context. So, the library code may not be working out of the box for them. In order to use your library in all these various scenarios, that you cannot (and shouldn’t) consider while developing it, the engineers should be able to customize some parts of it to fit their needs.

In the case of automated tests, imagine that you have a test suite testing a shopping cart. The workflow of the test consists of multiple steps- choosing the product, changing the quantity, adding more products, applying discount coupons, filling billing info, providing payment info and so on. If a new requirement comes - “The billing info should be prefilled for logged users.”, how easy would it be to change the existing tests? Did you write your tests in a way that, if you add this new functionality, it will not affect the majority of your existing tests?

You don’t need to answer these questions yet. We will discuss them in much more detail in the next chapters, and I will explain how to build such solutions.

As you can see this high-quality automated test attribute is closely related to the SOLID principles we already discussed.

Learning Curve

I also like to call this attribute “Easy Knowledge Transfer”. The attribute answers to the question “How easy is it for someone to learn how to add new or maintain the existing tests by himself?”.

The learning curve is tightly coupled to the API usability, but at the same time it means something a bit different. If a new member joins your team, is he able to learn by himself how to use your test automation framework or he needs to read the documentation if it exists? Or you have a mentoring program where you need to teach these new members yourself every time how to use your code? To the end of the book, I will show you how to develop your test automation code in such a way that the new members will be able to learn how to use your solution by themselves.

Summary

I explained with a few examples the different SOLID principles. At the end of the article, you learned what the design patterns are, and which are the five high-quality test attributes.