Tuesday, August 26th, 2008

NUnit Parameteri[sz]ed Tests Reviewed

Ben Hall has posted a nice summary of the Parameterized Test Features in NUnit 2.5 Alpha-3. Of course, being from the UK and all, he calls them Parameterised Tests.

Some of the new features are similar to approaches taken in mbUnit, about which Ben has written in past blogs. As it happens, those posts were often the best – and sometimes only – source of information I could find on how those features ought to work. So thanks for the cross-pollination you provide, Ben!

Friday, August 1st, 2008

Four Ways to Test Expected Exceptions

Let’s say we are testing a piece of code, using arguments that should cause an exception to be thrown. We want the test to ensure that an exception was thrown, that it was the expected Type of exception and – possibly – that the properties of the exception are what they should be.

In this blog, I’ll show how this might be done:
1. Without any support from the testing framework.
2. With a basic ExpectedException attribute
3. With a handler, as provided by NUnit 2.4
4. With a Throws.Exception constraint, as provided by NUnit 2.5

What I hope to show is that Throws.Exception gives us the level of control we have when we do it all ourselves, while providing a level of ease of use that is comparable to – if not better than – what we get from ExpectedExceptionAttribute.

First, we need an example. Let’s say that we have a method that is supposed to throw an argumentexception if the passed in argument is in an invalid state for the desired operation. Of course, this would be a pretty poor design choice in most situations, but lets just assume this is the way it has to be and that we simply need to test it.

Here’s the test I might write using a try/catch block.

public void TestThatProperExceptionIsThrown()
    Assert.Fail("Expected an exception, but none was thrown");
  catch(ArgumentException ex)
    Assert.AreEqual( ex.ParamName, "myParam" );
    Assert.AreEqual( ex.Message, "My message" );
  catch(Exception ex)
    Assert.Fail( "Expected an ArgumentException but got a "
      + ex.GetType().FullName );

This is somewhat tedious to write, so programmers were happy to get an ExpectedExceptionAttribute to use instead. With that support, you could write

public void TestThatProperExceptionIsThrown()

Or even

[Test,ExpectedException(typeof(ArgumentException), ExpectedMessage="My message")]
public void TestThatProperExceptionIsThrown()

But what about ParamName? NUnit 2.4 introduced the notion of an exception handler, allowing you to write:


[Test,ExpectedException(typeof(ArgumentException), ExpectedMessage=”My message”,Handler=”MyHandler)]
public void TestThatProperExceptionIsThrown()

public void MyHandler(Exception ex)
  Assert.AreEqual(“myParam”, ((ArgumentException)ex).ParamName);

This gives us the functionality, but is somewhat verbose. A more serious problem is that there may be additional code in the test method, before or after the method call. This approach does not guarantee us that the exception came from a particular method.

NUnit 2.5 takes care of this with a new Assert and a corresponding Constraint. Assert.Throws was borrowed from the xunit.net framework, and is best for simpler cases:

Assert.Throws<ArgumentException>( { delegate MethodUnderTest(anInvalidObject) } );

The corresponding constraint syntax, using Throws.Exception is unique to NUnit and allows us to meet our original requrements in a single statement:

Assert.That( delegate { MethodUnderTest(anInvalidObject },
  Has.Property("ParamName").EqualTo("myParam") &
  Has.Property("Message").EqualTo("My message") );

For new applications, I recommend you set aside ExpectedException and use either Assert.Throws or Throws.Exception. That way, what you are testing is clearly stated right in the code, for everyone to see.

Wednesday, July 2nd, 2008

NUnit’s Generic Test Fixtures

One recent addition to NUnit 2.5 is the ability to define generic test fixtures, allowing the same fixture to be reused for multiple types that implement the same interface or even just having common method signatures. For example, the following code tests multiple implementations of IList.

public class IList_Tests<TList> where TList : IList, new()
  private IList list;

  public void CreateList()
    this.list = new TList();

  public void CanAddToList()
    list.Add(1); list.Add(2); list.Add(3);
    Assert.AreEqual(3, list.Count);

NUnit will create an tree branch containing two fixtures, one that uses ArrayList and another using List. Combined with features like TestCase, this can be quite powerful.

This feature is available in the current source code and will be included in the Alpha-3 release.

Friday, June 27th, 2008

NUnit 2.5 and VS2008

Thanks to a contribution from Microsoft, kindly arranged by Stephen Walther, I now have a copy of Visual Studio 2008 – and NUnit has a VS2008 build! It’s available in CVS and will be part of the Alpha-3 release in a week or two. NUnit operates pretty much on a shoestring these days, now that I’m semi-retired, and contributions like this help a lot.

As with all of NUnit’s Visual Studio builds, the VS2008 build is provided for the convenience of folks who want to build their own copy of NUnit. Our “official” releases are all built with NAnt and anyone who wants to work that way may, of course, do so themselves. Last time I checked, you could also build NUnit with one of the Express editions, so you have lots of options.

Currently, the VS2008 build uses .NET 2.0. I’m not envisioning a separate build for .NET 3.5 until NUnit 3.0. Of course, that could change if we come up with a good reason to change it.

Friday, May 23rd, 2008

Parameterized Tests With NUnit 2.5

It has been possible to write parameterized tests for NUnit for some time, using Andreas Schlapsi’s RowTest Extension, which recognizes the RowTest and Row attributes familiar to MbUnit users.

With the NUnit 2.5 alpha releases, NUnit continues to support that extension – and even bundles a copy of it. But NUnit 2.5 also introduces its own set of attributes for data-based testing, beginning with the TestCaseAttribute, which I will describe here.

Here’s a simple use of this new attribute…

[TestCase(12, 3, 4)]
[TestCase(12, 2, 6)]
[TestCase(12, 4, 3)]
[TestCase(12, 0, 0, ExpectedException = typeof(System.DivideByZeroException),
      TestName = "DivisionByZeroThrowsExceptionType")]
[TestCase(12, 0, 0, ExpectedExceptionName = "System.DivideByZeroException",
      TestName = "DivisionByZeroThrowsNamedException")]
public void IntegerDivisionWithResultPassedToTest(int n, int d, int q)
      Assert.AreEqual(q, n / d);

This creates five different test cases, displayed and reported separately. The two last tests show alternate ways to specify an expected exceptionl. Note the use of the TestName property, to specify a meaningful name for each of these cases.

Since this test consists of a single Assert.AreEqual statement, we can simplify it further by modifying the test method to return a result and indicate the expected result with the result property…

[TestCase(12, 3, Result = 4)]
[TestCase(12, 2, Result = 6)]
[TestCase(12, 4, Result = 3)]
[TestCase(12, 0, ExpectedException = typeof(System.DivideByZeroException),
      TestName = "DivisionByZeroThrowsExceptionType")]
[TestCase(12, 0, ExpectedExceptionName = "System.DivideByZeroException",
      TestName = "DivisionByZeroThrowsException")]
public int IntegerDivisionWithResultCheckedByNUnit(int n, int d)
      return n / d;

For more information on this feature, see the NUnit documentation.

Thursday, May 8th, 2008

NUnit 2.5 Alpha 2 Released

With a European trip about to start, I decided to release a second Alpha so that the new stuff would get some visibility. I won’t be doing another release till late June, so please give this one a try.

As compared to 2.4, NUnit 2.5 has quite a lot:

  • Data-driven tests using [TestCase] and [DataSource]
  • Additional asserts and constraints, including inline exception tests.
  • Parallel and distributed testing using pNUnit.
  • Bundled addins, now including RowTest and CSUnitAddin.

For more info, see the release notes.

In addition, watch the NUnit mailing lists or this blog for info about other extensions as they are released for NUnit 2.5.

Thursday, May 8th, 2008

A Simple NUnit Usage Recipe

Scott White has a blog about NUnit Best Practices. The approach may require adjustment on more complex projects, but it’s a very simple recipe for those starting out with NUnit.

Sunday, May 4th, 2008

Common.Addins.Build : Addin Building Made Simple

Let me start right out by saying: I know some of you won’t find this to be simpler – but it is! If you can’t bring yourself to install NAnt or work from the command line, then this isn’t for you. But if you can get past the initial hump – or if you’re already past it – then this is for you.

The hardest part about building – as opposed to writing – addins is getting the references right. Addin solutions in Visual Studio generally contain from three to five projects: the addin assembly, the assembly referenced from user tests, a unit test assembly, possibly a separate test fixture assembly for use by the tests and one or more samples. All of those have to reference the nunit assemblies they will be used with when they are installed. Getting it right is tedious at best, requiring multiple visits to the Add Reference Dialog. And from time to time, Visual Studio will decide to use a different assembly from the one you expected.

I finally got fed up with all this and decided to write a NAnt script for one of my addins – CSUnitAddin actually. After it was done, I looked closely and realized that most of it was boilerplate – the same code with a few property changes would build other addins. So I started to factor out the common code into an include file, with the result that my CSUnitAddin.build file now looks like this:

<?xml version="1.0"?>
<project name="CSUnitAddin" default="build" basedir=".">

  <!-- Include the common build file -->
  <include buildfile="../common.addin.build"/>    <!-- 1 -->>

  <!-- Define non-default names for sample project -->
  <property name="sample.dll" value="money.dll"/>
  <property name="sample.dir" value="money"/>
  <!-- Define non-default names for fixture project -->
  <property name="fixture.dll" value="CSUnitTestFixtures.dll"/>
  <property name="fixture.dir" value="CSUnitTestFixtures"/>

  <!-- This addin is for the csunit framework -->
  <property name="target.framework.dll" value="csunit.dll"/>

  <!-- Define alt library with proper version of csunit         -->
  <!-- Must be here since lib.dir is defined in the common file -->
  <property name="csunit.version" value="2.0.4"/>
  <property name="alt.lib.dir" value="${lib.dir}/csUnit-${csunit.version}"/>


This example actually has to do more than most, since it uses a “foreign” testing framework. I’ll walk through the steps for you:

  1. First, the script includes the addin.common.build sript, which will do all the work.
  2. Next, it defines non-standard names for the sample assembly and it’s directory. I could have eliminated this by calling them both “sample” but I wanted a non-generic name here.
  3. I do the same thing for the “fixture” assembly – an assembly that contains csunit-based tests, which are loaded and run by my unit test assembly. In this case, I might have stuck with the default (CSUnitAddinFixture) and saved a few lines of xml.
  4. I set the target.framework.dll to csUnit so that my sample and fixture assemblies will be built with a reference to that framework rather than the default nunit framework.
  5. This particular addin has multiple library subdirectories for different versions of csUnit, so I set the alt.lib.dir property to indicate which one I want to use.

So, let’s say that I want to build and test this addin with NUnit 2.5, using both the .NET 1.1 and 2.0 builds. I can do this with two commands:

nant release net-1.1 nunit-2.5 clean build test
nant release net-2.0 nunit-2.5 clean build test

The script will locate my NUnit 2.5 installation, build the addin and it’s associated assemblies (four in all), install it and run the tests using that same installation. NUnit 2.5 has separate net-1.1 and net-2.0 directories, which the script knows about, so the two builds will be installed separately.

This is how I’m building addins now. I use Visual Studio to do editing and syntax checking, leaving the actual builds to the script. As a result, I no longer need to deal with mismatched references or with the tedium of using the gui to switch between NUnit versions.

The Common.Addin.Build script is open source, licensed under the Academic Free Licence 3.0. You can get it here.

Friday, May 2nd, 2008

Using Addins with NUnit-Console

A recent bug pointed out that addins are not recognized when running tests under the console runner. This is due to a missing entry in the nunit-console.exe.config file, which you can easily fix yourself. Follow these steps to have your addins recognized when using the console runner:

  1. Open the nunit-console.exe.config file in any convenient editor – notepad is fine.
  2. Find the <runtime> element
  3. Insert the code below within the <runtime> element. You can copy it from nunit.exe.config if you prefer
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="addins"/>

From this point on, your addins should work equally well in both the Gui and Console runners.

Friday, May 2nd, 2008

NUnit 2.4.7, 2.5, 3.0: What’s With That?

A few folks are confused by the various release numbers being announced or discussed all at one time, so I thought I’d clarify:

NUnit 2.4.7 is the latest production release of NUnit. It’s the one we recommend most people use for your tests. Some fairly critical performance bugs have been fixed in the last few releases, so you should update even if you’re only one or two digits back. See what you’re missing !

NUnit 3.0 is the planned – but not yet released – next generation NUnit. We call it the NUnit Extended Testing Platform, to distinguish it from the current NUnit Framework. It will provide a superset of the functionality of the current framework and is generally described here. I’ll be posting further info on NUnit 3.0 as it progresses.

NUnit 2.5 is release that wasn’t originally planned. The 2.4 series was supposed to be followed by 3.0. However, a number of people asked for a quicker release that included features provided by other test frameworks, which are currently missing from NUnit. So 2.5 is now in alpha, with extensions to support data-driven tests, easier exception testing and a number of other goodies. It’s also being bundled with pNUnit, an extension that supports distributed parallel tests. See the release notes for details. Watch for the second Alpha release sometime next week.

Hopefully, this will clear things up till we generate some more numbers.