Thursday, September 22nd, 2016

NUnit-Summary Becoming an “Official” NUnit Application

UPDATE: I’m leaving the post here but the action described has been reversed and the project continues to live at

NUnit-Summary is an “extra” that I’ve maintained personally for some time. It uses built-in or user-supplied transforms to produce summary reports based on the results of NUnit tests.

I have contributed it to the NUnit project and we’re working on updating it to recognize NUnit 3 test results. The program has never had a 1.0 release, but we expect to produce one soon.

This old post talks about the original nunit-summary program.

Thursday, September 22nd, 2016

An Engine Extension for Running Failed Tests – Part 1: Creating the Extension

In a recent online discussion, one of our users talked about needing to re-run the NUnit console runner, executing just the failed tests from the previous run. This isn’t a feature in NUnit but it could be useful to some people. So… can we do this by creating an Engine Extension? Let’s give it a try!

The NUnit Test Engine supports extensions. In this case, we’re talking about a Result Writer extension, one that will take the output of a test run from NUnit and create an output file in a particular format. In this case, we want the output to be a text file with each line holding the full name of a failed test case. Why that format? Because it’s exactly the format that the console runner already recognizes for the --testlist option. We can use the file that is created as input to a subsequent test run.

Information about how to write an extension can be found on the Writing Engine Extensions page of the NUnit documentation. Details of creating a ResultWriter extension can be found on the Result Writers page.

To get started, I created a new class library project called failed-tests-writer. I made sure that it targeted .NET 2.0, because that allows it to be run under the widest range of runtime versions and I added a package reference to the NUnit.Engine.Api package. That package will be published on with the release of NUnit 3.5. Since that’s not out yet, I used the latest pre-release version from the NUnit project MyGet feed by adding to my NuGet package sources.

Next, I created a class to implement the extension. I called it FailedTestsWriter. I added using statements for NUnit.Engine and NUnit.Engine.Extensibility and implemented the IResultWriter interface. I gave my class Extension and ExtensionProperty attributes. Here is what it looked like when I was done.

using System;
using System.IO;
using NUnit.Engine;
using NUnit.Engine.Extensibility

namespace EngineExtensions
    [Extension, ExtensionAttribute("Format", "failedtests")]
    public class FailedTestsWriter : IResultWriter
        public void CheckWritability(string outputPath)
            using (new StreamWriter(outputPath, false, Encoding.UTF8)) { }

        public void WriteResultFile(XmlNode resultNode, string outputPath)
            using (var writer = new StreamWriter(outputPath, false, Encoding.UTF8))
                WriteResultFile(resultNode, writer);

        public void WriteResultFile(XmlNode resultNode, TextWriter writer)
            foreach (XmlNode node in resultNode.SelectNodes("//test-case[@result='Failed']")) // (3)

The ExtensionAttribute marks the class as an extension. In this case as in most cases, it’s not necessary to add any arguments. The Engine can deduce how the extension should be used from the fact that it implements IResultWriter.

As explained on the Result Writers page, this type of extension requires use of the ExtensionPropertyAttribute so that NUnit knows the name of the format it implements. In this case, I chose to use “failedtests” as the format name.

The CheckWriteability method is required to throw an exception if the provided output path is not writeable. We do that very simply by trying to create a StreamWriter. The empty using statement is merely an easy way to ensure that the writer is closed.

The main point of the extension is accomplished in the second WriteResultFile method. A foreach statement selects each failing test, which is then written to the output file.

Testing the Extension

That explains how to write the extension. In Part 2, I’ll explain how to deploy it. Meanwhile, I’ll tell you how I tested my extension in it’s own solution, using nunit3-console.

First, I installed the package NUnit.ConsoleRunner from I used version 3.4.1. Next, I created a fake package subdirectory in my packages folder, so it ended up looking like this:


Note that the new extension “package” directory name must start with “NUnit.Extension.” in order to trick the console-runner and engine into using it.

With this structure in place, I was able to run the console with the --list-extensions option to see that my extension was installed and I could use a command like

nunit3-console mytests.dll --result:FailedTests.lst;format=failedtests

to actually produce the required output.

Saturday, May 2nd, 2009

Using Lambdas as Constraints in NUnit 2.5

Let’s say you have an array of ints representing years, all of which should be leap years.

One way to test this would be to write a custom constraint, LeapYearConstraint. You
could then use it with the Matches syntax element to write your test as

Assert.That( array, Is.All.Matches( new LeapYearConstraint() );

But creating a new constraint for this adhoc problem seems like a bit of overkill.
Instead, assuming you are working with C# version 3, try this:

Assert.That( array, Is.All.Matches( (x) => x%4 == 0 && x%100 != 0 || x%400 == 0 );

If it fails, it will give a generic message: “Expected: matching lambda expression” since NUnit is actually
built with .NET 2.0, but for a quick test it may be just the tool you need.

Wednesday, April 29th, 2009

Ten Reasons to Try NUnit 2,5

NUnit 2.5 has so many new features (see the release notes) that I thought I’d try to come up with my top-ten favorites. It was hard to get down to ten, but here’s what I came up with…

Reason 1: Data-Driven Tests

Users of mbUnit and have enjoyed the flexibility that data-driven (aka parameteried) tests provide for some time. NUnit implements this paradigm in its own way, with it’s own set of attributes. Test methods may have arguments and the data for them may be supplied in a number of ways: inline, from a separate method or class or randomly. This feature gives you a succinct way to express a set of examples to be used in running individual test cases.

Reason 2: Theories

As used in NUnit, a Theory is a generalized statement of how a program should operate, like “For any positive number, the square root is defined as the positive or negative number, which, when multiplied by itself, gives the original number.” Traditional, example-based, testing allows you to select one or more sets of values to use in testing such a program. A Theory, on the other hand, allows you to express the generalization itself, writing a test that will pass for whatever values are passed to it, provided they meet the stated constraints. David Saff has written a number of papers about the use of Theories in testing and has implemented this construct as a part of JUnit. Now you can use the same construct in any .NET language.

Reason 3: Inline Expected Exception Tests

Testing that an expected exception is thrown correctly has always been an issue in NUnit. The ExpectedExceptionAttribute has been available since early releases but has a number of problems. It tests that the exception was thrown somewhere in the test, without specifying the exact place in the code, and it is subject to the syntactic limitations that apply to use of an attribute. With the introduction of the Assert.Throws assertion method and the even more powerful constraint expressions Throws.Exception, Throws.InstanceOf and Throws.TypeOf, exception testing logic can now be moved right into the test along with any other necessary assertions.

Reason 4: Generic Support

NUnit 2.5 provides separate framework assemblies for .NET 1.x and 2.0+. Using .NET 2.0 or higher, a number Up to 2.4, NUnit avoided any use of Generics, in order to maintain backward compatibility. In 2.5, the framework assembly used under .NET 2.0 provides a number of generic Asserts and Constraint expressions for convenience. More significantly, your test methods and classes may now be generic, and NUnit will specialize them using the types you provide.

Reason 5: Lambda Support

If you write your tests using C# 3.0, you may use Lambda expressions in a number of places where NUnit expects a delegate. This is particularly useful in providing a custom definition of equality without explicitly defining an IComparer<T> and can even be used to apply an arbitrary predicate to the members of a collection.

Reason 6: Out-of-Process Execution and Runtime Selection

NUnit 2.4 ran all tests within the same process, using one or more AppDomains for isolation. This works fine for many purposes, but has some limitations. NUnit 2.5 extends this concept to running tests under one or more separate processes. Aside from the isolation it provides, this allows running the tests under a different version of the .NET runtime from the one NUnit is currently using.

Reason 7: PNUnit

PNUnit stands for “parallel NUnit” and is an extension developed by Pablo Santos Luaces and his team at Codice Software and contributed to NUnit. It’s a new way to test applications composed of distributed, communicating components. Tests of each component run in parallel and use memory barriers to synchronize their operation. Currently, pNUnit uses a special executable to launch its tests. In the future, you will be able to run pNUnit tests from the standard NUnit console or gui runner.

Reason 8: Source Code Display

The new stack trace display in the Errors and Failures tab of the Gui is able to display the source code at the location where a problem occured, provided the source is available and the program was compiled with debug information. Currently, syntax coloring for C# is provided and other languages are treated as simple text, but additional syntax definitions will be available in the future.

Reason 9: Timeout and Delayed Constraints

These are two separate features, but they are related. Besides, I’m working hard to keep this down to only ten points! It’s now possible to set a timeout, which will pre-emptively fail a test to fail if it is exceeded. This may be done on a method, fixture, assembly or as a global default. On the other hand, if you need to wait for an action to take place after a delay, you can use the After syntax to delay the application of the constraint. NUnit will subdivide a long delay and apply your test repeatedly until the constraint succeeds or the specified amount of time is up!

Reason 10: Threading Attributes

In past releases, if any test needed to run in the STA, the entire test run had to use the STA. With 2.5, any method, fixture or assembly may be given an attribute that causes it to run on a separate thread in the STA. Other attributes allow requiring an MTA or simply running on a separate thread for isolation. This can eliminate a lot of boilerplate code now required to create a separate thread, launch it and capture the results for NUnit.

This is my own list, of course. Yours may vary. Download the release, try it out and let me know what your own favorites are.

Wednesday, January 7th, 2009

Code Generation in NUnit

The latest code for NUnit 2.5 includes seven generated files, including the Assert class and most of the classes that allow you to write constraint expressions using the NUnit fluent syntax. Some people have asked if generating these files is worth the effort, since the code created is very simple anyway.

There are two reasons for generating this code. The first relates to the syntactic constructs. While it’s relatively straight forward to create a custom constraint and various people have done so, such constraints must be used by invoking their constructors rather than by use of a simple key word. So, for example, if you have written an OddNumberConstraint that tests whether a number is odd and displays an appropriate failure message, you are still not able to write Assert.That(num, Is.Odd) without directly modifying NUnit.

It turns out, based on experience of several people who have tried, that the syntactic modification has a lot of places where you can go wrong. You have to modify at least three additional files, even after you have written the constraint. Using NUnit’s code generation facility, you would simply add a line like this to NUnit’s SyntaxElements.txt:

Gen3: Is.Odd=>new OddConstraint()

Then, after running NUnit’s code generation tool, the files Is.cs, ConstraintFactory.cs and ConstraintExpression.cs would be updated. After rebuilding NUnit – or just the framework – the statement Assert.That(num, Is.Odd) would compile and work correctly. If you wanted a classic assert, you could add the line

Gen: Assert.IsOdd(int num)=>Assert.That(num, Is.Odd)

and Assert.IsOdd would become available for your use, including overloads with an error message and optional arguments.

So, one good reason for generating code is to make it easier to extend NUnit. But an even more important reason is reliability. Take the Assert class as an example. Some of the methods have as many as 24 overloads. In the past, we have seen hidden bugs that affected only one infrequently used overload. By generating the code, we can ensure that the same logic is used in each overload. This doesn’t prevent errors, but it does make it likely that the error will be caught, since it will generally impact many of the overloads in the same way. What’s more, the layout of the SyntaxElements file puts things that need to be updated together right next to one another, so it’s much harder to forget a step.

The NUnit code generation program, GenSyntax.exe, is distributed with the NUnit source, in the tools directory.

Tuesday, August 26th, 2008

NUnit Parameteri[sz]ed Tests Reviewed

Ben Hall has posted a nice summary of the Parameterized Test Features in NUnit 2.5 Alpha-3. Of course, being from the UK and all, he calls them Parameterised Tests.

Some of the new features are similar to approaches taken in mbUnit, about which Ben has written in past blogs. As it happens, those posts were often the best – and sometimes only – source of information I could find on how those features ought to work. So thanks for the cross-pollination you provide, Ben!

Friday, August 1st, 2008

Four Ways to Test Expected Exceptions

Let’s say we are testing a piece of code, using arguments that should cause an exception to be thrown. We want the test to ensure that an exception was thrown, that it was the expected Type of exception and – possibly – that the properties of the exception are what they should be.

In this blog, I’ll show how this might be done:
1. Without any support from the testing framework.
2. With a basic ExpectedException attribute
3. With a handler, as provided by NUnit 2.4
4. With a Throws.Exception constraint, as provided by NUnit 2.5

What I hope to show is that Throws.Exception gives us the level of control we have when we do it all ourselves, while providing a level of ease of use that is comparable to – if not better than – what we get from ExpectedExceptionAttribute.

First, we need an example. Let’s say that we have a method that is supposed to throw an argumentexception if the passed in argument is in an invalid state for the desired operation. Of course, this would be a pretty poor design choice in most situations, but lets just assume this is the way it has to be and that we simply need to test it.

Here’s the test I might write using a try/catch block.

public void TestThatProperExceptionIsThrown()
    Assert.Fail("Expected an exception, but none was thrown");
  catch(ArgumentException ex)
    Assert.AreEqual( ex.ParamName, "myParam" );
    Assert.AreEqual( ex.Message, "My message" );
  catch(Exception ex)
    Assert.Fail( "Expected an ArgumentException but got a "
      + ex.GetType().FullName );

This is somewhat tedious to write, so programmers were happy to get an ExpectedExceptionAttribute to use instead. With that support, you could write

public void TestThatProperExceptionIsThrown()

Or even

[Test,ExpectedException(typeof(ArgumentException), ExpectedMessage="My message")]
public void TestThatProperExceptionIsThrown()

But what about ParamName? NUnit 2.4 introduced the notion of an exception handler, allowing you to write:


[Test,ExpectedException(typeof(ArgumentException), ExpectedMessage=”My message”,Handler=”MyHandler)]
public void TestThatProperExceptionIsThrown()

public void MyHandler(Exception ex)
  Assert.AreEqual(“myParam”, ((ArgumentException)ex).ParamName);

This gives us the functionality, but is somewhat verbose. A more serious problem is that there may be additional code in the test method, before or after the method call. This approach does not guarantee us that the exception came from a particular method.

NUnit 2.5 takes care of this with a new Assert and a corresponding Constraint. Assert.Throws was borrowed from the framework, and is best for simpler cases:

Assert.Throws<ArgumentException>( { delegate MethodUnderTest(anInvalidObject) } );

The corresponding constraint syntax, using Throws.Exception is unique to NUnit and allows us to meet our original requrements in a single statement:

Assert.That( delegate { MethodUnderTest(anInvalidObject },
  Has.Property("ParamName").EqualTo("myParam") &
  Has.Property("Message").EqualTo("My message") );

For new applications, I recommend you set aside ExpectedException and use either Assert.Throws or Throws.Exception. That way, what you are testing is clearly stated right in the code, for everyone to see.

Wednesday, July 2nd, 2008

NUnit’s Generic Test Fixtures

One recent addition to NUnit 2.5 is the ability to define generic test fixtures, allowing the same fixture to be reused for multiple types that implement the same interface or even just having common method signatures. For example, the following code tests multiple implementations of IList.

public class IList_Tests<TList> where TList : IList, new()
  private IList list;

  public void CreateList()
    this.list = new TList();

  public void CanAddToList()
    list.Add(1); list.Add(2); list.Add(3);
    Assert.AreEqual(3, list.Count);

NUnit will create an tree branch containing two fixtures, one that uses ArrayList and another using List. Combined with features like TestCase, this can be quite powerful.

This feature is available in the current source code and will be included in the Alpha-3 release.

Friday, June 27th, 2008

NUnit 2.5 and VS2008

Thanks to a contribution from Microsoft, kindly arranged by Stephen Walther, I now have a copy of Visual Studio 2008 – and NUnit has a VS2008 build! It’s available in CVS and will be part of the Alpha-3 release in a week or two. NUnit operates pretty much on a shoestring these days, now that I’m semi-retired, and contributions like this help a lot.

As with all of NUnit’s Visual Studio builds, the VS2008 build is provided for the convenience of folks who want to build their own copy of NUnit. Our “official” releases are all built with NAnt and anyone who wants to work that way may, of course, do so themselves. Last time I checked, you could also build NUnit with one of the Express editions, so you have lots of options.

Currently, the VS2008 build uses .NET 2.0. I’m not envisioning a separate build for .NET 3.5 until NUnit 3.0. Of course, that could change if we come up with a good reason to change it.

Friday, May 23rd, 2008

Parameterized Tests With NUnit 2.5

It has been possible to write parameterized tests for NUnit for some time, using Andreas Schlapsi’s RowTest Extension, which recognizes the RowTest and Row attributes familiar to MbUnit users.

With the NUnit 2.5 alpha releases, NUnit continues to support that extension – and even bundles a copy of it. But NUnit 2.5 also introduces its own set of attributes for data-based testing, beginning with the TestCaseAttribute, which I will describe here.

Here’s a simple use of this new attribute…

[TestCase(12, 3, 4)]
[TestCase(12, 2, 6)]
[TestCase(12, 4, 3)]
[TestCase(12, 0, 0, ExpectedException = typeof(System.DivideByZeroException),
      TestName = "DivisionByZeroThrowsExceptionType")]
[TestCase(12, 0, 0, ExpectedExceptionName = "System.DivideByZeroException",
      TestName = "DivisionByZeroThrowsNamedException")]
public void IntegerDivisionWithResultPassedToTest(int n, int d, int q)
      Assert.AreEqual(q, n / d);

This creates five different test cases, displayed and reported separately. The two last tests show alternate ways to specify an expected exceptionl. Note the use of the TestName property, to specify a meaningful name for each of these cases.

Since this test consists of a single Assert.AreEqual statement, we can simplify it further by modifying the test method to return a result and indicate the expected result with the result property…

[TestCase(12, 3, Result = 4)]
[TestCase(12, 2, Result = 6)]
[TestCase(12, 4, Result = 3)]
[TestCase(12, 0, ExpectedException = typeof(System.DivideByZeroException),
      TestName = "DivisionByZeroThrowsExceptionType")]
[TestCase(12, 0, ExpectedExceptionName = "System.DivideByZeroException",
      TestName = "DivisionByZeroThrowsException")]
public int IntegerDivisionWithResultCheckedByNUnit(int n, int d)
      return n / d;

For more information on this feature, see the NUnit documentation.