In the new Build Enterprise Automation Framework Series, we will look into detailed explanations on creating custom test automation frameworks. Many people starting a new position have this particular assignment, so I think it is crucial to understand how to follow all high-quality standards and practices properly. I am not a fan of reinventing the wheel, so I will encourage you to leverage other proven open-source solutions if they can be used and fit your needs instead of writing your own. Leverage the knowledge and expertise of proven experts in the field. If you follow this advice, this series will be highly useful to you again because you will understand how the test automation frameworks are designed, written, and maintained + how to work internally. The information about the subject will be invaluable to you choosing the right tool for your tests. The series can be even more useful to you if you decide to do everything from scratch.
Article by article, we will discuss different aspects and features of the custom test automation frameworks. At the end of the course, we will have a full-fledged custom test automation framework. The first article from the series will talk not so much about code but rather on how to define what we need (our requirements), how to do research properly- finding the right tools that we will base our framework, and lastly, creating a detailed list of desired characteristics for our solution grouping all of the various features that we will build in the next articles. In the course, I will use as a demo/example our open-source BELLATRIX test automation framework, which I believe is one of the most feature richest open-source frameworks out there.
In the first article from the series “Enterprise Test Automation Framework: Define Requirements and Characteristics”, we defined our future framework’s requirements and qualities/characteristics. This publication will continue with defining more precisely the various features grouped by the previously described characteristics. They will help us to prioritize and plan our work on the framework. We already looked into a few features in part 1, part 2 and part3. Planning and designing framework modularity is essential for supporting more comfortable usage, maintainability, and extensibility. This is why we created such a plan in the Modularity Planning and Design publication. Next, we incorporated inversion of control containers required for plug-in architecture of the framework. Then we designed another crucial module of our framework responsible for retrieving data from configuration files depending on the build configuration, allowing us to have a separate config file for each execution environment - TEST, UAT, PROD, DEV. We built a module for logging data to console, test output, debug window, files, etc. In this publication, we will start designing modules to execute custom logic from plugins through various points in the test execution lifecycle. We started designing such a system for MSTest. In this publication, we will review the implementation for NUnit. Next, we will continue SpecFlow. I will repeat some explanations for the patterns and their implementation for people who want to read only this article. If you have read the previous article about the MSTest implementation, you can check only the NUnit specifics.
Requirements
As we defined in the first publication of the series, we want our framework to be highly extensible. One of each framework’s core characteristics is its capability to be extended and customized to fit different teams’ needs and be used in various contexts. We want this capability for MSTest, NUnit, and SpecFlow.
Creating Plugin Infrastructure
We will begin with MSTest. However, we will have some common logic for the 3 of the technologies, as you can expect. We will put it in a project called Bellatrix.TestWorkflowPlugins. For SpecFlow, we will have a project named Bellatrix.SpecFlow.TestWorkflowPlugins, we need a separate project because the implementation slightly differs. We will use the Observer Design Pattern internally in these modules, which is ideal for such use cases. We will put core pattern classes in the previously mentioned projects. The actual usage will happen in the base test classes for each test framework. We will put them in a separate projects - Bellatrix.MSTest, Bellatrix.NUnit, Bellatrix.SpecFlow.MSTest and Bellatrix.SpecFlow.NUnit.
Observer Design Pattern
Definition
The Observer design pattern defines a one-to-many relation between objects, so that when one object changes its state, all dependents are notified and updated automatically.
Benefits
-
Strives for loosely coupled designs between objects that interact.
-
Allows you to send data to many other objects in a very efficient manner.
-
No modification needs to be done to the subject in case you need to add new observers.
-
You can add and remove observers at any time.
-
The order of Observer notifications is unpredictable.
Abstract UML Class Diagram
classDiagram
ITestExecutionSubject <|.. MSTestExecutionSubject
ITestBehaviorObserver <|.. OwnerTestBehaviorObserver
MSTestExecutionSubject o-- ITestBehaviorObserver
BaseTest --> MSTestExecutionSubject
class ITestExecutionSubject {
<<interface>>
+Attach(ITestBehaviorObserver observer)
+Detach(ITestBehaviorObserver observer)
+PreTestInit(TestContext ctx, MemberInfo memberInfo)
+PostTestInit(TestContext ctx, MemberInfo memberInfo)
}
class MSTestExecutionSubject {
+Attach(ITestBehaviorObserver observer)
+Detach(ITestBehaviorObserver observer)
+PreTestInit(TestContext ctx, MemberInfo memberInfo)
+PostTestInit(TestContext ctx, MemberInfo memberInfo)
}
class ITestBehaviorObserver {
<<interface>>
+PreTestInit(TestContext ctx, MemberInfo memberInfo)
+PostTestInit(TestContext ctx, MemberInfo memberInfo)
}
class OwnerTestBehaviorObserver {
+PreTestInit(TestContext ctx, MemberInfo memberInfo)
}
class BaseTest {
-ITestExecutionSubject subject
}The participants in this pattern are:
-
ITestExecutionSubject
Objects use this interface to register as observers and to remove themselves from being observers.
-
MSTestExecutionSubject
The concrete subject always implements the ITestExecutionSubject interface. In addition, to attach and detach methods, the specific subject implements different notification methods that are used to update all the subscribed observers whenever the state changes.
-
ITestBehaviorObserver
All potential observers need to implement the observer interface. The methods of this interface are called at different points, when the subject’s state changes.
-
OwnerTestBehaviorObserver
A concrete observer can be any class that implements ITestBehaviorObserver interface. Each observer registers with a specific subject to receive updates.
-
BaseTest
The parent class for all test classes in the framework. Uses the TestExecutionSubject to extend its test execution capabilities via test method/class level defined attributes and concrete observers
Plugins NUnit Implementation
We will provide an easy way for automation engineers to add additional logic to the current test execution via class/test level attributes with our implementation. For example, start and reuse the browser if the previous test hasn’t failed. Also, note that I will use slightly different names from the diagram to be more related to our current context.
Note: Here we will create the so-called classic implementation of the Observer design pattern. In the further reading section, you will find additional resources for implementing the pattern much easier in .NET through events and delegates or via the IObservable and IObserver interfaces. Since these features are present only in .NET, we will develop the classic version which can be implemented in all popular programming languages.
TestBehaviorObserver
Through our custom TestWorkflowPluginEventArgs args class, we will pass some required data to all of the plugins.
public class TestWorkflowPluginEventArgs : EventArgs
{
public TestWorkflowPluginEventArgs() {}
public TestWorkflowPluginEventArgs(TestOutcome testOutcome,
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
string consoleOutputMessage,
string consoleOutputStackTrace,
Exception exception,
List<string>categories,
List<string>authors,
List<string>descriptions) :
this(testOutcome,
testClassType,
consoleOutputMessage,
consoleOutputStackTrace)
{
TestMethodMemberInfo = testMethodMemberInfo;
TestName = testName;
Exception = exception;
Categories = categories;
Authors = authors;
Descriptions = descriptions;
}
public TestWorkflowPluginEventArgs(
TestOutcome testOutcome,
Type testClassType,
string consoleOutputMessage = null,
string consoleOutputStackTrace = null)
{
TestOutcome = testOutcome;
TestClassType = testClassType;
TestClassName = testClassType.FullName;
TestFullName = $"{TestClassName}.{TestName}";
ConsoleOutputMessage = consoleOutputMessage;
ConsoleOutputStackTrace = consoleOutputStackTrace;
Container = ServicesCollection.Current.FindCollection(testClassType.FullName);
ExecutionContext = Container.Resolve<ExecutionContext>();
}
public ExecutionContext ExecutionContext { get; set; }
public IServicesCollection Container { get; set; }
public Exception Exception { get; }
public MemberInfo TestMethodMemberInfo { get; }
public Type TestClassType { get; }
public TestOutcome TestOutcome { get; }
public string TestName { get; }
public string TestClassName { get; }
public string TestFullName { get; }
public string ConsoleOutputMessage { get; }
public string ConsoleOutputStackTrace { get; }
public List <string> Categories { get; }
public List <string> Authors { get; }
public List <string> Descriptions { get; }
}Here is the base class for all observers or I like to call them in BELLATRIX plugins.
public class TestWorkflowPlugin
{
public void Subscribe(ITestWorkflowPluginProvider provider)
{
provider.PreTestInitEvent += PreTestInit;
provider.TestInitFailedEvent += TestInitFailed;
provider.PostTestInitEvent += PostTestInit;
provider.PreTestCleanupEvent += PreTestCleanup;
provider.PostTestCleanupEvent += PostTestCleanup;
provider.TestCleanupFailedEvent += TestCleanupFailed;
provider.PreTestsArrangeEvent += PreTestsArrange;
provider.TestsArrangeFailedEvent += TestsArrangeFailed;
provider.PreTestsActEvent += PreTestsAct;
provider.PostTestsActEvent += PostTestsAct;
provider.PostTestsArrangeEvent += PostTestsArrange;
provider.PreTestsCleanupEvent += PreTestsCleanup;
provider.PostTestsCleanupEvent += PostTestsCleanup;
provider.TestsCleanupFailedEvent += TestsCleanupFailed;
}
public void Unsubscribe(ITestWorkflowPluginProvider provider)
{
provider.PreTestInitEvent -= PreTestInit;
provider.TestInitFailedEvent -= TestInitFailed;
provider.PostTestInitEvent -= PostTestInit;
provider.PreTestCleanupEvent -= PreTestCleanup;
provider.PostTestCleanupEvent -= PostTestCleanup;
provider.TestCleanupFailedEvent -= TestCleanupFailed;
provider.PreTestsArrangeEvent -= PreTestsArrange;
provider.TestsArrangeFailedEvent -= TestsArrangeFailed;
provider.PreTestsActEvent -= PreTestsAct;
provider.PostTestsActEvent -= PostTestsAct;
provider.PostTestsArrangeEvent -= PostTestsArrange;
provider.PreTestsCleanupEvent -= PreTestsCleanup;
provider.PostTestsCleanupEvent -= PostTestsCleanup;
provider.TestsCleanupFailedEvent -= TestsCleanupFailed;
}
protected virtual void TestsCleanupFailed(object sender, Exception ex) { }
protected virtual void PreTestsCleanup(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PostTestsCleanup(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PreTestInit(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void TestInitFailed(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PostTestInit(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PreTestCleanup(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PostTestCleanup(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void TestCleanupFailed(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void TestsArrangeFailed(object sender, Exception e) { }
protected virtual void PreTestsAct(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PreTestsArrange(object sender
TestWorkflowPluginEventArgs e) { }
protected virtual void PostTestsAct(object sender,
TestWorkflowPluginEventArgs e) { }
protected virtual void PostTestsArrange(object sender,
TestWorkflowPluginEventArgs e) { }
protected List<dynamic> GetAllAttributes<TAttribute>(MemberInfo memberInfo)
where TAttribute : Attribute
{
var classAttributes = GetClassAttributes<TAttribute>(memberInfo.DeclaringType);
var methodAttributes = GetMethodAttributes<TAttribute>(memberInfo);
var attributes = classAttributes.ToList();
attributes.AddRange(methodAttributes);
return attributes;
}
protected TAttribute GetOverridenAttribute<TAttribute>(MemberInfo memberInfo)
where TAttribute : Attribute
{
var classAttribute = GetClassAttribute<TAttribute>(memberInfo.DeclaringType);
var methodAttribute = GetMethodAttribute<TAttribute>(memberInfo);
if (methodAttribute != null)
{
return methodAttribute;
}
return classAttribute;
}
protected dynamic GetClassAttribute<TAttribute>(Type currentType)
where TAttribute : Attribute
{
var classAttribute = currentType.GetCustomAttribute<TAttribute>(true);
return classAttribute;
}
protected dynamic GetMethodAttribute<TAttribute>(MemberInfo memberInfo)
where TAttribute : Attribute
{
var methodAttribute = memberInfo?.GetCustomAttribute<TAttribute>(true);
return methodAttribute;
}
protected IEnumerable<dynamic> GetClassAttributes<TAttribute>(Type currentType)
where TAttribute : Attribute
{
var classAttributes = currentType.GetCustomAttributes<TAttribute>(true);
return classAttributes;
}
protected IEnumerable<dynamic> GetMethodAttributes<TAttribute>(MemberInfo memberInfo)
where TAttribute : Attribute
{
var methodAttributes = memberInfo?.GetCustomAttributes<TAttribute>(true);
return methodAttributes;
}
}At the end of the class, you will notice that there are a few protected methods. I added them to ease the work with getting information from attributes put on top of test classes and test methods. We use the .NET Core Reflection API. All plugins that we will develop will use an attribute to set up the needed info.
ExecutionSubject
First is the ITestWorkflowPluginProvider interface that defines all event handlers. This is similar to the test execution subject interface from the diagram.
public interface ITestWorkflowPluginProvider
{
event EventHandler<TestWorkflowPluginEventArgs> PreTestInitEvent;
event EventHandler<TestWorkflowPluginEventArgs> TestInitFailedEvent;
event EventHandler<TestWorkflowPluginEventArgs> PostTestInitEvent;
event EventHandler<TestWorkflowPluginEventArgs> PreTestCleanupEvent;
event EventHandler<TestWorkflowPluginEventArgs> PostTestCleanupEvent;
event EventHandler<TestWorkflowPluginEventArgs> TestCleanupFailedEvent;
event EventHandler<TestWorkflowPluginEventArgs> PreTestsActEvent;
event EventHandler<TestWorkflowPluginEventArgs> PreTestsArrangeEvent;
event EventHandler<TestWorkflowPluginEventArgs> PostTestsActEvent;
event EventHandler<TestWorkflowPluginEventArgs> PostTestsArrangeEvent;
event EventHandler<TestWorkflowPluginEventArgs> PreTestsCleanupEvent;
event EventHandler<TestWorkflowPluginEventArgs> PostTestsCleanupEvent;
event EventHandler<Exception> TestsCleanupFailedEvent;
event EventHandler<Exception> TestsArrangeFailedEvent;
}This is the implementation of the subject class responsible for executing the logic at the right places from all registered observers/plugins. As you can see, we are using the Observer design pattern implementation using event and delegates. The most important part of the code is in the RaiseTestEvent method, where we initialize an instance of the TestWorkflowEventArgs and invoke the specific event handler.
public class TestWorkflowPluginProvider : ITestWorkflowPluginProvider
{
public event EventHandler<TestWorkflowPluginEventArgs> PreTestInitEvent;
public event EventHandler<TestWorkflowPluginEventArgs> TestInitFailedEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PostTestInitEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PreTestCleanupEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PostTestCleanupEvent;
public event EventHandler<TestWorkflowPluginEventArgs> TestCleanupFailedEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PreTestsActEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PreTestsArrangeEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PostTestsActEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PostTestsArrangeEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PreTestsCleanupEvent;
public event EventHandler<TestWorkflowPluginEventArgs> PostTestsCleanupEvent;
public event EventHandler<Exception> TestsCleanupFailedEvent;
public event EventHandler<Exception> TestsArrangeFailedEvent;
public void PreTestsArrange(Type testClassType) =>
RaiseClassTestEvent(PreTestsArrangeEvent, TestOutcome.Unknown, testClassType);
public void TestsArrangeFailed(Exception ex) => TestsArrangeFailedEvent.Invoke(this, ex);
public void PostTestsArrange(Type testClassType) =>
RaiseClassTestEvent(PostTestsArrangeEvent, TestOutcome.Unknown, testClassType);
public void PreTestsAct(Type testClassType) =>
RaiseClassTestEvent(PreTestsActEvent, TestOutcome.Unknown, testClassType);
public void PostTestsAct(Type testClassType) =>
RaiseClassTestEvent(PostTestsActEvent, TestOutcome.Unknown, testClassType);
public void PreTestInit(
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions
) =>
RaiseTestEvent(
PreTestInitEvent,
TestOutcome.Unknown,
testName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
public void TestInitFailed(
Exception ex,
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions
) =>
RaiseTestEvent(
TestInitFailedEvent,
TestOutcome.Failed,
testName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions,
ex.Message,
ex.StackTrace
);
public void PostTestInit(
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions
) =>
RaiseTestEvent(
PostTestInitEvent,
TestOutcome.Unknown,
testName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
public void PreTestCleanup(
TestOutcome testOutcome,
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions,
string message,
string stackTrace,
Exception exception
) =>
RaiseTestEvent(
PreTestCleanupEvent,
testOutcome,
testName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions,
message,
stackTrace,
exception
);
public void TestCleanupFailed(
Exception ex,
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions
) =>
RaiseTestEvent(
TestCleanupFailedEvent,
TestOutcome.Failed,
testName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions,
ex.Message,
ex.StackTrace
);
public void PostTestCleanup(
TestOutcome testOutcome,
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions,
string message,
string stackTrace,
Exception exception
) =>
RaiseTestEvent(
PostTestCleanupEvent,
testOutcome,
testName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions,
message,
stackTrace,
exception
);
public void PreClassCleanup(Type testClassType) =>
RaiseClassTestEvent(PreTestsCleanupEvent, TestOutcome.Unknown, testClassType);
public void TestsCleanupFailed(Exception ex) => TestsCleanupFailedEvent?.Invoke(this, ex);
public void PostClassCleanup(Type testClassType) =>
RaiseClassTestEvent(PostTestsCleanupEvent, TestOutcome.Unknown, testClassType);
private void RaiseClassTestEvent(
EventHandler<TestWorkflowPluginEventArgs> eventHandler,
TestOutcome testOutcome,
Type testClassType
)
{
var args = new TestWorkflowPluginEventArgs(testOutcome, testClassType);
eventHandler?.Invoke(this, args);
}
private void RaiseTestEvent(
EventHandler<TestWorkflowPluginEventArgs> eventHandler,
TestOutcome testOutcome,
string testName,
MemberInfo testMethodMemberInfo,
Type testClassType,
List<string> categories,
List<string> authors,
List<string> descriptions,
string message = null,
string stackTrace = null,
Exception exception = null
)
{
var args = new TestWorkflowPluginEventArgs(
testOutcome,
testName,
testMethodMemberInfo,
testClassType,
message,
stackTrace,
exception,
categories,
authors,
descriptions
);
eventHandler?.Invoke(this, args);
}
}Base Test
We use the subject/provider class to execute all added observers/plugins at the right points. So, in the Bellatrix.NUnit project I added a class called NUnitBaseTest that will be the base class for all NUnit tests. Below you can find the full source code of it, but afterward, we will discuss the essential parts.
public class NUnitBaseTest
{
public ServicesCollection Container;
protected static ThreadLocal<Exception> ThrownException;
private TestWorkflowPluginProvider _currentTestExecutionProvider;
public NUnitBaseTest()
{
Container = ServicesCollection.Current;
AppDomain.CurrentDomain.FirstChanceException += (sender, eventArgs) =>
{
if (eventArgs.Exception.Source != "System.Private.CoreLib")
{
if (ThrownException == null)
{
ThrownException = new ThreadLocal<Exception>(() => eventArgs.Exception);
}
else
{
ThrownException.Value = eventArgs.Exception;
}
}
};
}
public TestContext TestContext => TestContext.CurrentContext;
public void OneTimeArrangeAct()
{
try
{
var testClassType = GetCurrentExecutionTestClassType();
Container = ServicesCollection.Current.CreateChildServicesCollection(
testClassType.FullName
);
Container.RegisterInstance(Container);
_currentTestExecutionProvider = new TestWorkflowPluginProvider();
Initialize();
InitializeTestExecutionBehaviorObservers(_currentTestExecutionProvider);
_currentTestExecutionProvider.PreTestsArrange(testClassType);
TestsArrange();
_currentTestExecutionProvider.PostTestsArrange(testClassType);
_currentTestExecutionProvider.PreTestsAct(testClassType);
TestsAct();
_currentTestExecutionProvider.PostTestsAct(testClassType);
}
catch (Exception ex)
{
_currentTestExecutionProvider.TestsArrangeFailed(ex);
throw ex;
}
}
public void ClassCleanup()
{
try
{
var testClassType = GetCurrentExecutionTestClassType();
Container = ServicesCollection.Current.CreateChildServicesCollection(
testClassType.FullName
);
Container.RegisterInstance(Container);
_currentTestExecutionProvider = new TestWorkflowPluginProvider();
InitializeTestExecutionBehaviorObservers(_currentTestExecutionProvider);
_currentTestExecutionProvider.PreClassCleanup(testClassType);
TestsCleanup();
_currentTestExecutionProvider.PostClassCleanup(testClassType);
}
catch (Exception ex)
{
_currentTestExecutionProvider.TestsCleanupFailed(ex);
throw;
}
}
public void CoreTestInit()
{
if (ThrownException?.Value != null)
{
ThrownException.Value = null;
}
var testClassType = GetCurrentExecutionTestClassType();
Container = ServicesCollection.Current.FindCollection(testClassType.FullName);
var testMethodMemberInfo = GetCurrentExecutionMethodInfo();
var categories = GetAllTestCategories();
var authors = GetAllAuthors();
var descriptions = GetAllDescriptions();
_currentTestExecutionProvider = new TestWorkflowPluginProvider();
InitializeTestExecutionBehaviorObservers(_currentTestExecutionProvider);
try
{
_currentTestExecutionProvider.PreTestInit(
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
TestInit();
_currentTestExecutionProvider.PostTestInit(
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
}
catch (Exception ex)
{
_currentTestExecutionProvider.TestInitFailed(
ex,
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
throw;
}
}
public void CoreTestCleanup()
{
var testClassType = GetCurrentExecutionTestClassType();
Container = ServicesCollection.Current.FindCollection(testClassType.FullName);
var testMethodMemberInfo = GetCurrentExecutionMethodInfo();
var categories = GetAllTestCategories();
var authors = GetAllAuthors();
var descriptions = GetAllDescriptions();
try
{
_currentTestExecutionProvider = new TestWorkflowPluginProvider();
InitializeTestExecutionBehaviorObservers(_currentTestExecutionProvider);
_currentTestExecutionProvider.PreTestCleanup(
(TestOutcome)TestContext.Result.Outcome.Status,
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions,
TestContext.Result.Message,
TestContext.Result.StackTrace,
ThrownException?.Value
);
TestCleanup();
_currentTestExecutionProvider.PostTestCleanup(
(TestOutcome)TestContext.Result.Outcome.Status,
TestContext.Test.FullName,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions,
TestContext.Result.Message,
TestContext.Result.StackTrace,
ThrownException?.Value
);
}
catch (Exception ex)
{
_currentTestExecutionProvider.TestCleanupFailed(
ex,
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
throw;
}
}
public virtual void Initialize() { }
public virtual void TestsArrange() { }
public virtual void TestsAct() { }
public virtual void TestsCleanup() { }
public virtual void TestInit() { }
public virtual void TestCleanup() { }
protected static bool IsDebugRun()
{
#if DEBUG
var isDebug = true;
#else
bool isDebug = false;
#endif
return isDebug;
}
private List<string> GetAllTestCategories()
{
var categories = new List<string>();
foreach (var property in GetTestProperties(PropertyNames.Category))
{
categories.Add(property);
}
return categories;
}
private List<string> GetAllAuthors()
{
var authors = new List<string>();
foreach (var property in GetTestProperties(PropertyNames.Author))
{
authors.Add(property);
}
return authors;
}
private IEnumerable<string> GetTestProperties(string name)
{
var list = new List<string>();
var currentTest = (ITest)TestExecutionContext.CurrentContext?.CurrentTest;
while (
currentTest?.GetType() != typeof(TestSuite)
&& currentTest.ClassName != "NUnit.Framework.Internal.TestExecutionContext+AdhocContext"
)
{
if (currentTest.Properties.ContainsKey(name))
{
if (currentTest.Properties.Count > 0)
{
for (var i = 0; i < currentTest.Properties.Count; i++)
{
list.Add(currentTest.Properties.ToString());
}
}
}
currentTest = currentTest.Parent;
}
return list;
}
private List<string> GetAllDescriptions()
{
var descriptions = new List<string>();
foreach (var property in GetTestProperties(PropertyNames.Description))
{
descriptions.Add(property);
}
return descriptions;
}
private void InitializeTestExecutionBehaviorObservers(
TestWorkflowPluginProvider testExecutionProvider
)
{
var observers = ServicesCollection.Current.ResolveAll<TestWorkflowPlugin>();
foreach (var observer in observers)
{
observer.Subscribe(testExecutionProvider);
}
}
private MethodInfo GetCurrentExecutionMethodInfo()
{
var testMethodMemberInfo = GetType().GetMethod(TestContext.CurrentContext.Test.Name);
return testMethodMemberInfo;
}
private Type GetCurrentExecutionTestClassType()
{
var testClassType = GetType().Assembly.GetType(TestContext.CurrentContext.Test.ClassName);
return testClassType;
}
}The whole point of creating such a complex base test class is that we want to execute an infinite number of plugins before/after clients’ TestInit and before/after the client’s TestCleanup. We can achieve that by using our test workflow plugins provider, which is responsible for executing the logic from all plugins. To allow users to run their logic inside the TestInit and TestCleanup, we added empty virtual methods that they can override in their tests instead of using the NUnit attributes.
public void CoreTestInit()
{
if(ThrownException?.Value != null)
{
ThrownException.Value = null;
}
var testClassType = GetCurrentExecutionTestClassType();
Container = ServicesCollection.Current.FindCollection(testClassType.FullName);
var testMethodMemberInfo = GetCurrentExecutionMethodInfo();
var categories = GetAllTestCategories();
var authors = GetAllAuthors();
var descriptions = GetAllDescriptions();
_currentTestExecutionProvider = new TestWorkflowPluginProvider();
InitializeTestExecutionBehaviorObservers(_currentTestExecutionProvider);
try
{
_currentTestExecutionProvider.PreTestInit(
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
TestInit();
_currentTestExecutionProvider.PostTestInit(
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
}
catch(Exception ex)
{
_currentTestExecutionProvider.TestInitFailed(
ex,
TestContext.Test.Name,
testMethodMemberInfo,
testClassType,
categories,
authors,
descriptions
);
throw;
}
}
public virtual void TestInit() { }
public virtual void TestCleanup() { }Usage Example
Here is an example of using the plugins and the base class. We will implement the used here plugins in the next few articles from the series.
[ScreenshotOnFail(true)]
[Browser(BrowserType.Chrome, BrowserBehavior.ReuseIfStarted)]
public class FullPageScreenshotsOnFailTests : WebTest
{
public override void TestInit()
{
App.NavigationService.Navigate("http://demos.bellatrix.solutions/");
}
public void PromotionsPageOpened_When_PromotionsButtonClicked()
{
var promotionsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Promotions");
promotionsLink.Click();
}
public void BlogsPageOpened_When_BlogsButtonClicked()
{
var blogsLink = App.ElementCreateService.CreateByLinkText<Anchor>("Blogs");
blogsLink.Click();
}
}Here we are configuring two plugins/observers through attributes - controlling/starting/stopping the browser plugin and the one for taking screenshots on failure. Also, the WebTest class derives from the base test class we created. Lastly, we are navigating to a page in the test initialize phase. As you can see, to execute this phase, we are not using NUnit attributes, but we have to override the TestInit method that we defined in the base class.
Summary
In the next article from the series, we will create a similar solution for SpecFlow. In the following publications, we will use this infrastructure to build the plugins from the usage example.
