In the new Build Enterprise Automation Framework Series, we will look into detailed explanations on creating enterprise test automation frameworks. Many people starting a new position have this particular assignment, so I think it is crucial to understand how to follow all high-quality standards and practices properly. I am not a fan of reinventing the wheel, so I will encourage you to leverage other proven open-source solutions if they can be used and fit your needs instead of writing your own. Leverage the knowledge and expertise of proven experts in the field. If you follow this advice, this series will be highly useful to you again because you will understand how the test automation frameworks are designed, written, and maintained + how to work internally. The information about the subject will be invaluable to you choosing the right tool for your tests. The series can be even more useful to you if you decide to do everything from scratch.
Article by article, we will discuss different aspects and features of the enterprise test automation frameworks. At the end of the course, we will have a full-fledged enterprise test automation framework. The first article from the series will talk not so much about code but rather on how to define what we need (our requirements), how to do research properly- finding the right tools that we will base our framework, and lastly, creating a detailed list of desired characteristics for our solution grouping all of the various features that we will build in the next articles. In the course, I will use as a demo/example our open-source BELLATRIX test automation framework, which I believe is one of the most feature richest open-source frameworks out there.
High-level Requirements
Before writing a single line of code, no matter the project, we first need to know what we need to build. To answer this question, we need first to ask what problems are we trying to solve. For example, let’s say that we are working for a big corporation, and we have all sorts of applications- web, Android, iOS, desktop, API, and we don’t have any test automation. Till now, we have done all of the testings manually. However, with time doing all the regression testing repeatedly over and over again resulted in a slower time for release, tedious repetition for the same tests again and again, which resulted in errors due to boredness. We want to build a solution that can help us automate the most critical regression tests and run these tests as often as possible. We want the tests to look similar no matter the platform - web, desktop, mobile, to transition and learn. Lastly, the solution should integrate well with all existing tools and processes like Jira, AzureDevOps, Jenkins, qTest, etc.
High-level requirements:
-
Open Source
-
Test Everything - web, including responsive, iOS, Android, desktop, and REST services
-
Unified Framework API - all features that we provide- use the same syntax. Once you learn how to write tests for the web, you can start immediately writing for mobile, desktop, or REST
-
Integrations - seamlessly integrate the framework with your existing tools and processes. Execute tests in the clouds, distributed and publish test results in reporting solutions
Research Phase
After we have requirements, our next step is again not to start writing code immediately. Instead, we need to check all tools and solutions out there that we can use or modify.
For example, this may include open source frameworks that may meet our requirements. Additionally, we need to check all popular low-level libraries that people use to do automation and test execution - WebDriver, Appium, WinAppDriver, RestSharp, Gauge, MSTest, NUnit, and such. I believe that to be able to write a scalable and maintainable test automation framework- we need to understand in detail all these tools. For example, a month ago, I have started working on a POC (proof of concept) of porting BELLATRIX to Java. To do so, I watched over 100 hours of video training in PluralSight (basically all Java Paths).
Additionally, I read two books. But I haven’t stick only to Java as a programming language. Still, I wanted to understand in as many details as possible how the different build tools (Maven, Ant, Gradle) and helper libraries/frameworks (Spring, JUnit, TestNG) work and how everything is working underneath so that I can create the most appropriate design and architecture. But I also research all other open-source Java test automation frameworks such as JDI, Selenide, and others. Probably you have heard this quote from Pablo Picasso, “Good artists borrow, great artists steal.”. It is the same in our job too. If it is open-source and you satisfy the license agreement, you are free to “steal”, use, and modify.
Ideas for research
-
watch Udemy courses
-
watch PluralSight courses
-
check popular blogs for this subject
-
check GitHub for similar frameworks and tooling
-
review all popular open-source and most-used tools for these areas
-
read technical books (for example, before I built the 1st version of BELLATRIX, I read two books about framework design)
Defining Detailed Requirements
After we did our research we made some decisions and here there are. Here are the core tools and libraries that we will base our framework on.
Selenium WebDriver - the standard for web automation. Open source. Large community. All major cloud platforms support it. Browser vendors support.
Appium - the standard for mobile automation. Open source. Large community. Compatible with major cloud providers. Can run in Selenium Grid.
WinAppDriver - the current standard for Windows desktop automation. Officially supported by Microsoft. Can run in Selenium Grid.
RestSharp - the standard for API automation in C#. Open source. Large community.
For unit testing frameworks we can use MSTest and NUnit we want to support both.
MSTest, NUnit - rich set of assert libraries. Open source. Large community.
Let’s say that we will strive to execute the tests in Azure DevOps CI builds.
To make the test results analyzation easier, we need to publish the results in ReportPortal or Allure. We will research both.
Probably at some point, we will have to add support to BDD technologies to ease some of our team members to write and understand the tests. Probably, we can use SpecFlow for the job or Gauge.
Defining Test Automation Framework Characteristics
1. Test Reliability
Make sure that you don’t have a real problem, and your tests didn’t fail because of test configuration issues or incorrect handling of timeouts.
2. Accelerated Test Creation or API Usability
You can speed up your test creation by providing element snippets, handy elements locators, smart wait for the elements, ready-to-go templates, and much more. Also, the more comfortable and intuitive our API is, the faster we will write new tests.
3. Accelerated Test Execution
The framework can speed up the test execution by enhanced finding and waiting of elements, and providing an improved browser life cycle control and the ability to run tests in headless mode.
4. Human Readable
Test readability can be improved built-in page objects, specific web controls with relevant actions, and properties with proper names. We can also generate test cases that are understandable for non-technical people or integrate BDD tools such as SpecFlow or Gauge.
5. Highly Extensible
Maybe one of the core characteristics of each framework is its capability to be extended and customized to fit different teams’ needs and be used in various contexts.
6. Easy Knowledge Transfer
Everything that your team needs to know, from installation to the most complex features. We can use starter kit demo projects in addition to comprehensive documentation with examples. Also, it is somehow related to the intuitive API design.
7. Troubleshooting Easiness
With the increasing test count and complexity, it will be even more critical that the tests are more maintainable. A significant part of this effort is easier troubleshooting and better support for locating errors.
8. Unified Team Conventions
To unify and force all engineers to write in a unified manner, the framework can integrate static analysis modules to apply coding standards. Also, we can create item templates for generating standardized page objects.
9. Seamlessly Integration with Existing Tools and Processes
The framework needs to integrate with all of your existing tooling - generating test cases and bug reports automatically based on your tests. Publish the test results in all significant test result portals.
10. Advanced Scenarios Support
It needs to support advanced “hard-to-automate” scenarios such as asserting PDFs through image recognition with the help of machine learning, reusing web tests for load testing, automatic machine setup, and the possibility to perform responsive layout and style tests.
