Some of the must-have features for 5th generation frameworks are related to troubleshooting easiness. With the increasing tests count and complexity, it will be even more critical the tests be more maintainable. A significant part of this effort is the easier troubleshooting and better support for locating errors.
A big part of maintainability is troubleshooting existing tests. Most in-house solutions or open-source ones don’t provide lots of features to make your life easier. This can be one of the most time-consuming tasks. Having 100 failing tests and find out whether there is a problem with the test or a bug in the application. If you use plugins or complicated design patterns the debugging of the tests will be much harder, requiring lots of resources and expertise.
Two of the ways Bellatrix as full-stack test automation framework handles these problems are through full-page screenshots and video recording on test fail.
In this article, I am going to show you how to create a similar cross-platform video recording engine. Moreover, you will be able to configure the engine entirely via attributes. The tests’ integration is implemented through the Observer Design Pattern (previously discussed in the series).
Cross-platform Video Recording Engine
IVideoRecorder Interface
The first thing that we need to do is to create a new interface for our video recorder. It should be as simple as possible. Our interface derives from the IDisposable interface because the recording engines need to free some resources before and after the saving of the recorded video.
public interface IVideoRecorder : IDisposable
{
string Record(string filePath, string fileName);
void Stop();
}FFmpegVideoRecorder
For the implementation of the interface, we are going to use the open source cross-platform CLI video recorder FFmpeg.
public class FFmpegVideoRecorder : IVideoRecorder
{
private Process _recorderProcess;
private bool _isRunning;
public void Dispose()
{
if (_isRunning)
{
// Wait for 500 milliseconds before finishing video
Thread.Sleep(500);
if (!_recorderProcess.HasExited)
{
_recorderProcess?.Kill();
_recorderProcess?.WaitForExit();
}
_isRunning = false;
}
}
public string Record(string filePath, string fileName)
{
string videoPath = $"{Path.Combine(filePath, fileName)}";
string videoFilePathWithExtension = GetFilePathWithExtensionByOS(videoPath);
try
{
if (!Directory.Exists(filePath))
{
Directory.CreateDirectory(filePath);
}
}
catch (Exception ex)
{
throw new ArgumentException($"A problem occurred trying to initialize the create the directory you have specified. - {filePath}", ex);
}
if (!_isRunning)
{
var startInfo = GetProcessStartInfoByOS(videoFilePathWithExtension);
_recorderProcess = Process.Start(startInfo);
_isRunning = true;
}
return videoFilePathWithExtension;
}
public void Stop() => Dispose();
private string GetFFmpegPath() => Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location) ?? throw new InvalidOperationException(), "ffmpeg.exe");
private ProcessStartInfo GetProcessStartInfoByOS(string videoFilePathWithExtension)
{
var startInfo = new ProcessStartInfo
{
FileName = GetFFmpegPath(),
RedirectStandardInput = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
CreateNoWindow = false,
};
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
startInfo.Arguments = $"-f gdigrab -framerate 30 -i desktop {videoFilePathWithExtension}";
}
else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
{
startInfo.Arguments = $"-f avfoundation -framerate 30 -i default {videoFilePathWithExtension}";
}
else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
{
startInfo.Arguments = $"-f x11grab -framerate 30 -i :0.0+100,200 {videoFilePathWithExtension}";
}
else
{
throw new NotSupportedException("The OS is not supported by FFmpeg video recorder. Currently supported OS are Windows, MacOS, Linux.");
}
return startInfo;
}
private string GetFilePathWithExtensionByOS(string videoPathNoExtension)
{
string videoPathWithExtension;
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
videoPathWithExtension = $"{videoPathNoExtension}.mpg";
}
else if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
{
videoPathWithExtension = $"{videoPathNoExtension}.mov";
}
else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
{
videoPathWithExtension = $"{videoPathNoExtension}.mp4";
}
else
{
throw new NotSupportedException("The OS is not supported by FFmpeg video recorder. Currently supported OS are Windows, MacOS, Linux.");
}
return videoPathWithExtension;
}
}There are a few primary parts of the code. When you call the Record method a new C# process is created. Based on the current OS different arguments are passed to the ffmpeg.exe since we use different encoders for the different OS. Also, there is a separate method for choosing the right file extension based again on the OS. Once the process is started, the recorder captures everything in the background. To stop it we call the Dispose method which kills the recorder’s process. When this happens, the results file is saved.
Video Recording Engine- Integration in Tests
VideoRecordingAttribute
We are going to configure the recording engine in the same manner as the execution engine discussed in the previous article from the series (Dynamically Configure Execution Engine). The attribute can be set again on class or method level. This time, it contains a single property of type VideoRecordingMode which specifies when the video recording should be performed.
[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method,
Inherited = true,
AllowMultiple = false)]
public sealed class VideoRecordingAttribute : Attribute
{
public VideoRecordingAttribute(VideoRecordingMode videoRecordingMode)
{
this.VideoRecording = videoRecordingMode;
}
public VideoRecordingMode VideoRecording { get; set; }
}VideoRecordingMode
You can always record the tests’ execution or only for failed/passed tests. Also, you can turn off the recording entirely.
public enum VideoRecordingMode
{
Always,
DoNotRecord,
Ignore,
OnlyPass,
OnlyFail
}VideoBehaviorObserver
I am not going to explain again in much details how the Observer solution works, if you haven’t read my articles about it, I suggest to do so- Advanced Observer Design Pattern via Events and Delegates and Dynamically Configure Execution Engine. In the PreTestInit method, we get the specified VideoRecordingMode. The global app.config configuration ShouldTakesVideosOnExecution if set overrides all attributes. As in the previous examples, the attributes on a method level override those on class level. We get the values of these attributes through reflection. If the mode is not equal to DoNotRecord, we start the video recording.
public class VideoWorkflowPlugin : BaseTestBehaviorObserver
{
private readonly IVideoRecorder _videoRecorder;
private readonly IVideoRecorderOutputProvider _videoRecorderOutputProvider;
private VideoRecordingMode _recordingMode;
private string _videoRecordingPath;
public VideoWorkflowPlugin(IVideoRecorder videoRecorder, IVideoRecorderOutputProvider videoRecorderOutputProvider)
{
_videoRecorder = videoRecorder;
_videoRecorderOutputProvider = videoRecorderOutputProvider;
}
protected override void PostTestInit(object sender, TestExecutionEventArgs e)
{
_recordingMode = ConfigureTestVideoRecordingMode(e.MemberInfo);
if (_recordingMode != VideoRecordingMode.DoNotRecord)
{
var fullTestName = $"{e.MemberInfo.DeclaringType.Name}.{e.TestName}";
var videoRecordingDir = _videoRecorderOutputProvider.GetOutputFolder();
var videoRecordingFileName = _videoRecorderOutputProvider.GetUniqueFileName(fullTestName);
_videoRecordingPath = _videoRecorder.Record(videoRecordingDir, videoRecordingFileName);
}
}
protected override void PostTestCleanup(object sender, TestExecutionEventArgs e)
{
if (_recordingMode != VideoRecordingMode.DoNotRecord)
{
try
{
bool hasTestPassed = e.TestOutcome.Equals(TestOutcome.Passed);
DeleteVideoDependingOnTestOutcome(hasTestPassed);
}
finally
{
_videoRecorder.Dispose();
}
}
}
private void DeleteVideoDependingOnTestOutcome(bool haveTestPassed)
{
if (_recordingMode != VideoRecordingMode.DoNotRecord)
{
bool shouldRecordAlways = _recordingMode == VideoRecordingMode.Always;
bool shouldRecordAllPassedTests = haveTestPassed && _recordingMode.Equals(VideoRecordingMode.OnlyPass);
bool shouldRecordAllFailedTests = !haveTestPassed && _recordingMode.Equals(VideoRecordingMode.OnlyFail);
if (!(shouldRecordAlways || shouldRecordAllPassedTests || shouldRecordAllFailedTests))
{
// Release the video file then delete it.
_videoRecorder.Stop();
if (File.Exists(_videoRecordingPath))
{
File.Delete(_videoRecordingPath);
}
}
}
}
private VideoRecordingMode ConfigureTestVideoRecordingMode(MemberInfo memberInfo)
{
VideoRecordingMode methodRecordingMode = GetVideoRecordingModeByMethodInfo(memberInfo);
VideoRecordingMode classRecordingMode = GetVideoRecordingModeType(memberInfo.DeclaringType);
VideoRecordingMode videoRecordingMode = VideoRecordingMode.DoNotRecord;
var shouldTakeVideos = bool.Parse(ConfigurationManager.AppSettings["shouldTakeVideosOnExecution"]);
if (methodRecordingMode != VideoRecordingMode.Ignore && shouldTakeVideos)
{
videoRecordingMode = methodRecordingMode;
}
else if (classRecordingMode != VideoRecordingMode.Ignore && shouldTakeVideos)
{
videoRecordingMode = classRecordingMode;
}
return videoRecordingMode;
}
private VideoRecordingMode GetVideoRecordingModeByMethodInfo(MemberInfo memberInfo)
{
if (memberInfo == null)
{
throw new ArgumentNullException();
}
var recordingModeMethodAttribute = memberInfo.GetCustomAttribute<VideoRecordingAttribute>(true);
if (recordingModeMethodAttribute != null)
{
return recordingModeMethodAttribute.VideoRecording;
}
return VideoRecordingMode.Ignore;
}
private VideoRecordingMode GetVideoRecordingModeType(Type currentType)
{
if (currentType == null)
{
throw new ArgumentNullException();
}
var recordingModeClassAttribute = currentType.GetCustomAttribute<VideoRecordingAttribute>(true);
if (recordingModeClassAttribute != null)
{
return recordingModeClassAttribute.VideoRecording;
}
return VideoRecordingMode.Ignore;
}
}Most of the work is done in the PostTestCleanup method of the observer. Based on the test’s outcome and the specified video recording mode, we decide whether to save the file or not. All of the code here is surrounded by a try-catch-finally. In the finally block, we call the Dispose method of the video recording engine. To make code more testable we use IVideoRecorderOutputProvider to get the output folder and the video file name.
For more detailed overview and usage of many more design patterns and best practices in automated testing, check my book “Design Patterns for High-Quality Automated Tests, C# Edition, High-Quality Tests Attributes, and Best Practices”. You can read part of three of the chapters:
Defining High-Quality Test Attributes for Automated Tests
Benchmarking for Assessing Automated Test Components Performance
Generic Repository Design Pattern- Test Data Preparation
VideoRecorderOutputProvider
public class VideoRecorderOutputProvider : IVideoRecorderOutputProvider
{
public string GetOutputFolder()
{
var outputDir = ConfigurationManager.AppSettings["videosFolderPath"];
if (!Directory.Exists(outputDir))
{
Directory.CreateDirectory(outputDir);
}
return outputDir;
}
public string GetUniqueFileName(string testName) => string.Concat(testName, Guid.NewGuid().ToString());
}Tests Examples
SearchEngineTests
Additionally to the previously created ExecutionEngineAttribute, we add the new VideoRecordingAttribute. It is configured to save the videos only for the failed tests. Because of that, we fail the tests through the Assert.Fail() method. The videos are saved in the folder specified in the app.config. Also, ffmpeg.exe is copied always to the bin folder.
[TestClass,
ExecutionEngineAttribute(ExecutionEngineType.WebDriver, Browsers.Chrome),
VideoRecordingAttribute(VideoRecordingMode.OnlyFail)]
public class BingTests : BaseTest
{
public void SearchForAutomateThePlanet()
{
var bingMainPage = Container.Resolve<BingMainPage>();
bingMainPage.Navigate();
bingMainPage.Search("Automate The Planet");
bingMainPage.AssertResultsCountIsAsExpected(264);
Assert.Fail();
}
public void SearchForAutomateThePlanet1()
{
var bingMainPage = Container.Resolve<BingMainPage>();
bingMainPage.Navigate();
bingMainPage.Search("Automate The Planet");
bingMainPage.AssertResultsCountIsAsExpected(264);
Assert.Fail();
}
}