Running Parallel Automation Tests Using NUnit v3

With the version 3 release of NUnit the ability to run your automated tests in parallel was introduced (A long running feature request). This brings with it the power to greatly speed up your test execution time by 2-3 times, depending on the average length of your tests. Faster feedback is crucial in keeping tests relevant and useful as part of your software development cycle.

As parallel tests is new to v3, the support is still somewhat limited, but I’ve managed to setup our Automated Test solution at my work, Campaign Monitor, to run tests in parallel and wanted to share my findings.

Technology

We are using the following technologies, so you may have to change some factors to match your setup, but it should provide a good starting point.

  • Visual Studio
  • C#
  • Selenium Webdriver
  • TeamCity

The Setup

Step 1 – Install the latest NUnit package

Within Visual Studio, install or upgrade the NUnit v3 package for your solution via Nuget Package Manager (or your choice of package management tool) using:

Install-Package NUnit
OR Upgrade-Package NUnit

Step 2 – Choose the tests that will be run in parallel

Your Tests can be configured to run in parallel at the TestFixture level only in the current release. So in order to setup your tests to be run in parallel, you choose the fixtures that you want to run in parallel and put a [Parallelizable] attribute at the start of the TestFixture.

Step 3 – Choose the number of parallel threads to run

To specify how many threads will run in parallel to execute your tests, add a ‘LevelOfParallelism’ attribute to your assembly.info file within each project with whatever value you desire. How many threads your tests can handle will be linked to the number of cores your machine running the test has. I recommend either 3 or 4.

Step 4 – Install a test runner to run the tests in parallel

Since this new to NUnit version 3, some test runners do not support it yet. There are 2 methods i’ve found which work.

1 – NUnit Console is a Nuget package that can be installed and runs as a command line. Install it using:

Install-Package NUnit.Console

then open a cmd window and navigate to the location of the nunit-console.exe file installed with the package. Run the tests with this command:

nunit-console.exe <path_to_project_dll> --workers=<number_of_threads>

This will then run the tests in the location specified and using the number of worker threads specified. Outputting the results in the command window.

2 – NUnit Test Runner is an extension for Visual Studio. Search for ‘NUnit3 Test Adapter’ (I used Version 3.0.4, created by Charlie Poole). Once installed, build your solution to populate the Tests into the Test Explorer view. You can filter the tests visible using the search bar to find the subset of tests you want (as decided in Step 2) and then click right click and run tests. This will also run your tests in parallel, using the ‘LevelOfParallelism’ attribute defined in Step 2 to determine the number of worker threads. This gives you nicer output to digest than the console runner, but still feels a bit clunky to use.

Your now setup and running your tests in parallel! Pretty easy right! The tricky part I found was then getting these tests to run in parallel when running through TeamCity, our continuous integration software.

(Optional) Step 5 – Configure TeamCity to run your tests in parallel

We use TeamCity to run our automated tests against our continuous integration builds, and so the biggest benefit to this project was to enable the TeamCity builds to run in parallel. Here’s how I did it

Note: First, I tried using MSBuild to run the NUnit tests as detailed here since this was the way we previously ran our tests before the NUnit v3 beta. However this didn’t work as it requires you to supply the NUnit version in the build script and that doesn’t support NUnit 3 or 3.0.0 or 3.0.0-beta-4 or any other variation i tried. So that was a no-go.

Second, I tried using the NUnit test build step and choosing the v3 type (only available in TeamCity v9 onwards). This lead me through a whole string of errors with conflicting references and unavailable methods and despite my best efforts, would not run the tests. So that was a no-go as well.

The method i decided upon was to use a command line step and run the NUnit console exe directly. So i first setup an MSBuild step that would just copy the NUnit console files and the Test Project files to a local directory on the Build Agent running the tests. Then I setup a command line step with these settings:

Run: Executable with parameters
Command Executable: <path_to_nunit-console.exe>
Command parameters: <path_to_test_dll's> --workers=<thread_count>

And with this, I was able to run parallel tests through team city! I’m sure the setup will get easier once support improves, but for now, it’s a good solution that gives our test suite 3-5 times faster results 🙂

Did you find this helpful, or have any tips you want to share? Please comment below!

Advertisements

2 thoughts on “Running Parallel Automation Tests Using NUnit v3

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s