Assessment System for Tests’ Architecture Design- SpecFlow Based Tests

In my previous article Assessment System for Tests’ Architecture Design, I presented to you eight criteria for system tests architecture design assessment. To understand the system fully, I am going to use it to evaluate a couple of real-world examples, assign ratings to them for each criterion and tell you my reasoning behind them. The third type of tests that we will assess is the SpecFlow based tests. If you are not familiar with SpecFlow you can check my SpecFlow Series.

SpecFlow Based Tests

Feature and Scenario

The SpecFlow uses the Gherkin DSL to describe the behaviour of the system using human-readable syntax. It uses the so-called specifications where the test scenarios are described through Gherkin sentences. On build, the DSL is compiled to MSTest tests.

Feature: Create Purchase in Online Store
In order to receive a book online
As a client
I want to be able to choose it through the browser and pay for it online
@testingFramework
Scenario: Create Successfull Purchase When Billing Country Is United States with American Express Card
When I navigate to "/Selenium-Testing-Cookbook-Gundecha-Unmesh/dp/1849515743"
And I click the 'buy now' button
And then I click the 'proceed to checkout' button
#When the login page loads
And I login with email = "g3984159@trbvm.com" and pass = "ASDFG_12345"
#When the shipping address page loads
And I type full name = "John Smith", country = "United States", Adress = "950 Avenue of the Americas", city = "New Your City", state = "New Your", zip = "10001-2121" and phone = "00164644885569"
And I choose to fill different billing, full name = "John Smith", country = "United States", Adress = "950 Avenue of the Americas", city = "New Your City", state = "New Your", zip = "10001-2121" and phone = "00164644885569"
And click shipping address page 'continue' button
And click shipping payment top 'continue' button
Then assert that order total price = "40.49"

Binding Methods

Another thing you have to do is to define binding methods for every sentence used in your scenarios. Otherwise, the tests will fail. These bindings are defined in standard C# classes marked with Bindings attribute. Each step method is marked with step type attribute containing the step’s regex pattern. Inside the steps’ methods, the pages’ logic is called. This way every step defines and executes a small part of the test’s workflow. With this approach of tests writing the standard MSTest classes don’t exist.

public class CreatePurchaseSteps
{
    [When(@"I navigate to ""([^""]*)""")]
    public void NavigateToItemUrl(string itemUrl)
    {
        var itemPage = UnityContainerFactory.GetContainer().Resolve<ItemPage>();
        itemPage.Navigate(itemUrl);
    }
    [When(@"I click the 'buy now' button")]
    public void ClickBuyNowButtonItemPage()
    {
        var itemPage = UnityContainerFactory.GetContainer().Resolve<ItemPage>();
        itemPage.ClickBuyNowButton();
    }
    [When(@"then I click the 'proceed to checkout' button")]
    public void ClickProceedToCheckoutButtonPreviewShoppingCartPage()
    {
        var previewShoppingCartPage = UnityContainerFactory.GetContainer().Resolve<PreviewShoppingCartPage>();
        previewShoppingCartPage.ClickProceedToCheckoutButton();
    }
    [When(@"the login page loads")]
    public void SignInPageLoads()
    {
        var signInPage = UnityContainerFactory.GetContainer().Resolve<SignInPage>();
        signInPage.WaitForPageToLoad();
    }
    [When(@"I login with email = ""([^""]*)"" and pass = ""([^""]*)""")]
    public void LoginWithEmailAndPass(string email, string password)
    {
        var signInPage = UnityContainerFactory.GetContainer().Resolve<SignInPage>();
        signInPage.Login(email, password);
    }
    [When(@"the shipping address page loads")]
    public void ShippingPageLoads()
    {
        var shippingAddressPage = UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        shippingAddressPage.WaitForPageToLoad();
    }
    [When(@"I type full name = ""([^""]*)"", country = ""([^""]*)"", Adress = ""([^""]*)"", city = ""([^""]*)"", state = ""([^""]*)"", zip = ""([^""]*)"" and phone = ""([^""]*)""")]
    public void FillShippingInfo(string fullName, string country, string address, string state, string city, string zip, string phone)
    {
        var shippingAddressPage = UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        var clientPurchaseInfo = new ClientPurchaseInfo(
        new ClientAddressInfo()
        {
            FullName = fullName,
            Country = country,
            Address1 = address,
            State = state,
            City = city,
            Zip = zip,
            Phone = phone
        });
        shippingAddressPage.FillShippingInfo(clientPurchaseInfo);
    }
    [When(@"I choose to fill different billing, full name = ""([^""]*)"", country = ""([^""]*)"", Adress = ""([^""]*)"", city = ""([^""]*)"", state = ""([^""]*)"", zip = ""([^""]*)"" and phone = ""([^""]*)""")]
    public void FillDifferentBillingInfo(string fullName, string country, string address, string state, string city, string zip, string phone)
    {
        var shippingAddressPage = UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        var shippingPaymentPage = UnityContainerFactory.GetContainer().Resolve<ShippingPaymentPage>();
        var clientPurchaseInfo = new ClientPurchaseInfo(
        new ClientAddressInfo()
        {
            FullName = fullName,
            Country = country,
            Address1 = address,
            State = state,
            City = city,
            Zip = zip,
            Phone = phone
        });
        shippingAddressPage.ClickDifferentBillingCheckBox(clientPurchaseInfo);
        shippingAddressPage.ClickContinueButton();
        shippingPaymentPage.ClickBottomContinueButton();
        shippingAddressPage.FillBillingInfo(clientPurchaseInfo);
    }
    [When(@"click shipping address page 'continue' button")]
    public void ClickContinueButtonShippingAddressPage()
    {
        var shippingAddressPage = UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        shippingAddressPage.ClickContinueButton();
    }
    [When(@"click shipping payment top 'continue' button")]
    public void WhenClickTopPaymentButton()
    {
        var shippingPaymentPage = PerfectSystemTestsDesign.Base.UnityContainerFactory.GetContainer().Resolve<ShippingPaymentPage>();
        shippingPaymentPage.ClickTopContinueButton();
    }
    [Then(@"assert that order total price = ""([^""]*)""")]
    public void AssertOrderTotalPrice(string itemPrice)
    {
        var placeOrderPage = PerfectSystemTestsDesign.Base.UnityContainerFactory.GetContainer().Resolve<PlaceOrderPage>();
        double totalPrice = double.Parse(itemPrice);
        placeOrderPage.AssertOrderTotalPrice(totalPrice);
    }
}

Hook Methods

There are so called hooks classes where a different pre/post execution logic can be defined by a run, feature, step block or step level. For example, here we start a browser before each test and register all needed pages as singletons for the test. After that, we close the browser.

public sealed class SpecflowHooks
{
    
    public static void BeforeTestRun()
    {
        Driver.StartBrowser(BrowserTypes.Chrome);
        UnityContainerFactory.GetContainer().RegisterType<ItemPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<PreviewShoppingCartPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<SignInPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShippingAddressPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShippingPaymentPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<PlaceOrderPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ItemPageBuyBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ItemPageNavigationBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<PlaceOrderPageAssertFinalAmountsBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<PreviewShoppingCartPageProceedBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShippingAddressPageContinueBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShippingAddressPageFillDifferentBillingBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShippingAddressPageFillShippingBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShippingPaymentPageContinueBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<SignInPageLoginBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType<ShoppingCart>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterInstance<IWebDriver>(PerfectSystemTestsDesign.Core.Driver.Browser);
    }
    
    public static void AfterTestRun()
    {
        Driver.StopBrowser();
    }
    
    public static void BeforeScenario()
    {
    }
    
    public static void AfterScenario()
    {
    }
}

Data Driven Tests Examples Table

Specflow supports data driven tests through data tables- a new test is generated for every row in the table. Keep in mind that I formatted the table manually. The Visual Studio support for Gherkin is not on the needed level.

Scenario Outline: Successfully Convert Seconds to Minutes Table
When I navigate to Seconds to Minutes Page
And type seconds for <seconds>
Then assert that <minutes> minutes are displayed as answer
Examples:
| seconds | minutes |
| 1 day, 1 hour, 1 second | 1500 |
| 5 days, 3 minutes | 7203 |
| 4 hours | 240 |
| 180 seconds | 3 |

Pass Multiple Parameters to Step

Readability

This table will pass something like a list of dynamic object to our binding method. By the way it is really implemented using dynamic C# objects.

Scenario: Add Online Store Products with Affiliate Codes
When add products
| Url | AffilicateCode |
| /dp/B00TSUGXKE/ref=ods_gw_d_h1_tab_fd_c3 | affiliate3 |
| /dp/B00KC6I06S/ref=fs_ods_fs_tab_al | affiliate4 |
| /dp/B0189XYY0Q/ref=fs_ods_fs_tab_ts | affiliate5 |
| /dp/B018Y22C2Y/ref=fs_ods_fs_tab_fk | affiliate6 |

Here is how we use the SpecFlow’s parameters table in tests. As I pointed it creates a dynamic list of objects and we can iterate through them.

[When(@"add products")]
public void NavigateToItemUrl(Table productsTable)
{
    var itemPage = UnityContainerFactory.GetContainer().Resolve<ItemPage>();
    IEnumerable<dynamic> products = productsTable.CreateDynamicSet();
    foreach (var product in products)
    {
        itemPage.Navigate(string.Concat(product.Url, "?", product.AffilicateCode));
        itemPage.ClickBuyNowButton();
    }
}

Evaluate SpecFlow Based Tests- Assessment System

Maintainability

Everything mentioned for the behaviours is applicable for this approach too. However, the rating is decreased because additional binding files exist. Moreover, the user cannot use existing tags and constants in the feature files which leads to hard-coded data and copy-paste development.

The main advantage of SpecFlow is the readability. All steps are described in human-readable syntax via the Gherkin DSL. Even the base initialize methods are described with a couple of sentences which makes them more meaningful to the user.

Code Complexity Index

The code complexity index here is not entirely accurate because Visual Studio doesn’t support the calculation of code metrics for Gherkin files. The index for the binding classes is marked as very good. Probably because there are not any base classes. However, they depend on multiple classes such as pages and behaviours or facades. I think you will agree with me that if we could calculate the metrics for the Gherkin files they weren’t going to be very good because of that I decreased the overall rating with one and it is only marked as good.

Usability

The rating for this parameter is calculated as Very Poor because there are a lot of new classes that should be created before the user can use any steps in his test (assuming that he/she is writing a new test from scratch for a new feature). The SpecFlow’s integration with Visual Studio is poor and the suggested steps while writing are not very helpful (IntelliSense). It is a challenge if you need to define a couple of actions with common starting words (you should define different overridden methods in the bindings’ classes + use custom regex patterns). If you use examples’ table to generate tests you should format it manually if you want to be readable.

Flexibility

The rating is only good because in order the SpecFlow’s API to support additional steps you need to create wrapper methods in the bindings’ classes with custom regex expressions.

Learning Curve

I guess it will be harder to write new tests compared to the approach of using only page objects, especially if there isn’t existing tests using the SpecFlow’s sentences’ steps.

Principle of Least Knowledge

You pass only the required parameters to the concrete binding methods. So the rating is marked as excellent.

You can watch my conference talk dedicated to the system or download the [whole slide deck](/whitepapers/heros-journey-to-perfect-system-tests-eight-assessment-c