Friday, August 1st, 2008

Four Ways to Test Expected Exceptions

Let’s say we are testing a piece of code, using arguments that should cause an exception to be thrown. We want the test to ensure that an exception was thrown, that it was the expected Type of exception and – possibly – that the properties of the exception are what they should be.

In this blog, I’ll show how this might be done:
1. Without any support from the testing framework.
2. With a basic ExpectedException attribute
3. With a handler, as provided by NUnit 2.4
4. With a Throws.Exception constraint, as provided by NUnit 2.5

What I hope to show is that Throws.Exception gives us the level of control we have when we do it all ourselves, while providing a level of ease of use that is comparable to – if not better than – what we get from ExpectedExceptionAttribute.

First, we need an example. Let’s say that we have a method that is supposed to throw an argumentexception if the passed in argument is in an invalid state for the desired operation. Of course, this would be a pretty poor design choice in most situations, but lets just assume this is the way it has to be and that we simply need to test it.

Here’s the test I might write using a try/catch block.

public void TestThatProperExceptionIsThrown()
    Assert.Fail("Expected an exception, but none was thrown");
  catch(ArgumentException ex)
    Assert.AreEqual( ex.ParamName, "myParam" );
    Assert.AreEqual( ex.Message, "My message" );
  catch(Exception ex)
    Assert.Fail( "Expected an ArgumentException but got a "
      + ex.GetType().FullName );

This is somewhat tedious to write, so programmers were happy to get an ExpectedExceptionAttribute to use instead. With that support, you could write

public void TestThatProperExceptionIsThrown()

Or even

[Test,ExpectedException(typeof(ArgumentException), ExpectedMessage="My message")]
public void TestThatProperExceptionIsThrown()

But what about ParamName? NUnit 2.4 introduced the notion of an exception handler, allowing you to write:


[Test,ExpectedException(typeof(ArgumentException), ExpectedMessage=”My message”,Handler=”MyHandler)]
public void TestThatProperExceptionIsThrown()

public void MyHandler(Exception ex)
  Assert.AreEqual(“myParam”, ((ArgumentException)ex).ParamName);

This gives us the functionality, but is somewhat verbose. A more serious problem is that there may be additional code in the test method, before or after the method call. This approach does not guarantee us that the exception came from a particular method.

NUnit 2.5 takes care of this with a new Assert and a corresponding Constraint. Assert.Throws was borrowed from the framework, and is best for simpler cases:

Assert.Throws<ArgumentException>( { delegate MethodUnderTest(anInvalidObject) } );

The corresponding constraint syntax, using Throws.Exception is unique to NUnit and allows us to meet our original requrements in a single statement:

Assert.That( delegate { MethodUnderTest(anInvalidObject },
  Has.Property("ParamName").EqualTo("myParam") &
  Has.Property("Message").EqualTo("My message") );

For new applications, I recommend you set aside ExpectedException and use either Assert.Throws or Throws.Exception. That way, what you are testing is clearly stated right in the code, for everyone to see.

Sunday, March 11th, 2007

NUnit 2.4 Assert Syntax – the Latest Developments

NUnit 2.4 RC2 is out now, correcting a naming conflict with several mock object frameworks that was present in RC1. You can download it here. For a full list of the extensive new features in NUnit 2.4, check out the release notes.

One of those new features is the new constraint-based design for assertions, supported by a new syntax. That syntax, in fact, was the cause of the aforementioned naming conflict, which is now resolved. As a result, RC2 makes more changes than would normally be seen in a release candidate.

I’ve already blogged quite a bit about the new syntax. I originally developed it for NUnitLite, based on design concepts in NMock2. Since there were various experiments and numerous alternatives have been discussed, I’ll describe the whole thing here from scratch, as it is being implemented in NUnit 2.4.

Design Concepts

The “new syntax” is really a second order effect. What this change is really all about is encapsulating individual assertions as objects. We call these objects “constraints” in NUnit, and they encapsulate both a test and the generation of a message.

Readers who have been following the development of NUnit over the past few years will recognize that this is not the first such encapsulation. Back in 2004, we replaced the original procedural implementation of assertions with an object-oriented approach centered around “asserter” objects. Asserters encapsulated everything about an assertion: the test to be made, any test parameters, a message and the actual value being tested.

By eliminating the actual value from the encapsulation, and providing it separately in a method call, we are able to form compound constraints and then apply them to a single actual value. This is the essence of the design, which was arrived at through a series of spikes using various encapsulations.

In addition, the new design provides a MessageWriter, which is used internally by constraints. Most users will not need to use this interface, but those creating their own constraints will want to be familiar with it. There is a clear division of responsibility between the constraint and the writer. A constraint is responsibile for defining message content but the writer is responsible for the appearance of the message. NUnit 2.4 is delivered with one writer, TextMessageWriter, but additional writers supporting formats like html will be available in a future release.

Two Models

NUnit 2.4 supports both the old model of static assert methods and the new syntax. The documentation refers to the older syntax as the “classic model” and the new one as the “constraint-based” model. The classic model has now re-implemented using constraints, so the same underlying code is used no matter which syntax you use.

Some users have expressed concern that the older syntax will eventually fade away. There’s no plan to do that, and I don’t expect it to happen for a long time, if at all. The older syntax maps directly to other frameworks. For example, CollectionAssert uses the same methods and arguments as the class of that name in the Microsoft test framework. Even though the new syntax provides added power, many users will undoubtedly prefer the compatibility that the classic model gives them.

Constraint Objects

The constraint-based model uses the Assert.That method, with an actual value being tested as it’s first argument and a constraint object as the second. The most direct, although not necessarily most convenient, way to express a constraint is to construct the object in-line. The following example uses an instance of the EqualConstraint to perform a test…

Assert.That( 2 + 2, new EqualConstraint( 4 ) );

The same assertion can be written using a helper class to construct the constraint…

Assert.That( 2 + 2, Is.EqualTo( 4 ) );

NUnit 2.4 Supports a wide range of constraints and syntax helpers:

Syntax Helper Constraint Constructor
Is.Null EqualConstraint( null )
Is.True EqualConstraint( true )
Is.False EqualConstraint( false)
Is.NaN EqualConstraint( double.NaN )
Is.Empty EmptyConstraint()
Is.Unique UniqueItemsConstraint()
Is.EqualTo( object expected ) EqualConstraint( object expected )
Is.SameAs( object expected ) SameAsConstraint( object expected )
Is.GreaterThan( IComparable expected ) GreaterThanConstraint( IComparable expected )
Is.GreaterThanOrEqualTo( IComparable expected ) GreaterThanOrEqualConstraint( IComparable expected )
Is.AtLeast( IComparable expected ) GreaterThanOrEqualConstraint( IComparable expected )
Is.LessThan( IComparable expected ) LessThanConstraint( IComparable expected )
Is.LessThanOrEqualTo( IComparable expected ) LessThanOrEqualConstraint( IComparable expected )
Is.AtMost( IComparable expected ) LessThanOrEqualConstraint( IComparable expected )
Is.TypeOf( Type expected ) ExactTypeConstraint( Type expected )
Is.InstanceOfType( Type expected ) InstanceOfypeConstraint( Type expected )
Is.AssignableFrom( Type expected ) AssignableFromConstraint( Type expected )
Is.SubsetOf( ICollection expected ) CollectionSubsetConstraint( ICollection expected )
Is.EquivalentTo( ICollection expected ) CollectionEquivalentTo( ICollection expected )
List.Contains( object expected ) CollectionContainsConstraint( object expected )
Text.Contains( string expected ) SubstringConstraint( string expected )
Text.StartsWith( string expected ) StartsWithConstraint( string expected)
Text.EndsWith( string expected ) EndsWithConstraint( string expected )
Text.Matches( string pattern ) RegexConstraint( string pattern )
Has.Property( string name, object expected ) PropertyConstraint( string name, object expected )
Has.Length( int length ) PropertyConstraint( “Length”, length )
Is.Not.Xxxx, Has.Not.Xxxx, etc. NotConstraint( Xxxx )
operator ! NotConstraint( Xxxx )
Is.All.Xxxx, Has.All.Xxxx, etc. AllItemsConstraint( Xxxx )
operator & AndConstraint( Constraint left, Constraint right )
operator | OrConstraint( Constraint left, Constraint right )

Note that Not and All are used as prefixes to any of the other constraints. The AllItemsConstraint causes the following constraint to be applied to each item in a collection, succeeding only if the constraint succeeds on every item.

Examples of use:

Assert.That( new object[] { 1, 3, 5 }, Is.SubsetOf( new object[] { 5, 4, 3, 2, 1 } );
Assert.That( new string[] { "abc", "bac", "cab" }, Has.All.Length( 3 ) );
Assert.That( 2.0d + 2.0d, Is.EqualTo( 4.0d ).Within( .000005d ) );
Assert.That( "Hello World!", Text.StartsWith( "HELLO" ).IgnoreCase );

The last two examples illustrate the use of constraint modifiers. These are properties or methods of constraints, which are used to modify their operation. They are syntactically valid on all constraints and are ignored by those constraints that do not make use of them. The following constraint modifiers are supported.

  • AsCollection is recognized by EqualConstraint and causes arrays to be compared using the underlying collections, without regard to their respective ranks or dimension lengths.
  • IgnoreCase causes any string comparisons to be case insensitive. It is recognized by EqualConstraint as well as by all the Text constraints.
  • Within( tolerance ) is recognized by EqualConstraint when both values are a floating point type. It causes the comparison to succeed when the difference between the values is less than or equal to the tolerance.

To Inherit or Not

Beginning with version 2.0, NUnit eliminated the need to use inheritance to identify test classes. That policy continues with NUnit 2.4. Test fixtures are identified by use of the TestFixture attribute and users may develop whatever hierarchy of test classes they need. In fact, most large projects develop a set of base classes for use in defining their test fixtures.

With NUnit 2.4, we are introducing a class that is intended to be used as a base for individual fixtures or for entire project test hierarchies. Our choice of a name for this class, AssertionHelper, indicates both what it is and what it is not. It provides a number of methods that are useful in making assertions. It does *not* serve to identify a class as a test fixture in and of itself.

AssertionHelper provides an Expect method, which is identical to the Assert.That method, as well as equivalent methods to those supported by the syntax helpers.
Consequently, assuming the containing class inherits from AssertionHelper, the following assertions are equivalent:

   Assert.AreEqual( 1.5, myObject.calculate(), .00005 );
   Assert.That( myObject.calculate(), Is.EqualTo( 1.5 ).Within( .00005 ) );
   Expect( myObject.calculate(), EqualTo( 1.5 ).Within( .00005 ) );

What Next?

The constraint-based design will continue to be expanded in the final release of NUnit 2.4 and beyond. Where the syntax will go is another question. The underlying constraint model is flexible enough to support a variety of syntactic layers, and the community is just getting started at trying out these ideas.

We expect that many users will want to develop their own constraints and to layer alternate syntax on top of the constraint model. The best of these new ideas will eventually be incorporated either as extensions to NUnit or as part of the framework itself.

So try out the release candidate and then try your hand at writing constraints of your own. See the NUnit documentation or source for a description of the interfaces.

Wednesday, October 4th, 2006

The Tester Tested : Fit and NUnit

NUnit has lots of tests of its own – unit tests that is. It even has some fairly high-level tests that don’t fit well into the normal unit-testing paradigm. But, up to now, the only acceptance tests were manual – a list of things I do before uploading a new release. Since they are manual, they don’t get run all that often and surprises happen.

I’ve been meaning to do some experimenting with writing Fit tests for NUnit, and finally got some time last weekend. You can see the results of my initial experiments here.

For each row, the Fit fixture takes a snippet of code out of the first column, compiles it using the ICodeCompiler interface, loads the resulting assembly into an AppDomain and runs the tests. In addition to verifying that the code compiles, it checks the shape of the test tree that is produced and verifies that the correct number of tests were run, skipped or ignored and if the count of successes and failures are as expected.

This particular fixture verifies that tests written by a programmer using standard NUnit syntax will behave as expected. Of course, we need other sorts of tests, including those that test the various execution options of NUnit. This seems like a good place to start though.


Monday, October 2nd, 2006

More on Assert Syntax

In an earlier post, I presented some ideas about syntax for expressing assertions in tests. I was doing this as a part of the development of NUnitLite, with the idea of eventually putting some of the same concepts back into NUnit.

I received many useful comments. Some of them are still percolating in my brain, but others have already found there way into the implementation of NUnitLite. As a way of paying back, I thought I would tell you here how I ended up resolving some of the issues we discussed.

NUnitLite Syntax by Example

Here are some valid NUnitLite Asserts, assuming proper declaration of variables…

  Assert.That( result, Is.EqualTo( 4 ) );
  Assert.That( result, Is.Not.EqualTo( 4 ) );
  Assert.That( result, !Is.EqualTo( 4 ) );
  Assert.That( obj, Is.Type( typeof(string) ) & Is.EqualTo( "Hello" ) );
  Assert.That( greeting, Contains.Substring( "Hello" ) );
  Assert.That( greeting, Contains.Substring( "HELLO" ).IgnoreCase );
  Assert.That( array, Is.EqualTo( new int[] { 1, 2, 3, 4 } ) );
  Assert.That( matrix, Is.EqualTo( new int[] { 1, 2, 3, 4 } ).AsCollection );
  Assert.That( collection, Contains.Item( obj ) );
  Assert.That( collection, Contains.Item( "HELLO" ).IgnoreCase );
  Assert.That( collection, Is.All.Type( string ) );
  Assert.That( collection, Is.All.Not.Null & Is.All.Type( string ) );
  Assert.That( obj, new UserConstraint(arg) );
  Assert.That( obj, XXX.UserConstraint(arg) );

Roads Taken and Not Taken

I started out trying to make the syntax more terse. However, several people pointed out that modern IDEs provide auto-completion in context, which makes the length of an expression relatively unimportant. In addition, at least one non-English speaker indicated that abbreviations like eq, gt, etc. can be confusing.

I spiked expressions like

  Assert.That( 2+2, Is.GreaterThan(3).And.LessThan(5) );

While this is definitely doable, it’s more complex and I don’t find it very readable. That could be just me, of course. In addition, while it can be implemented as a simple
operator precedence grammar, a simplistic implementation doesn’t provide type safety. That is, you can end up compiling expressions like This.And.And.That or And.That. The current implementation won’t compile such invalid sequences and Visual Studio’s Intellisense only prompts for valid continuations.

The postfix operators like IgnoreCase are what I call Modifiers. They are simply getters that modify the state of the underlying object and return this.

The two final examples illustrate how one might extend the Syntax. UserConstraint is some user-defined test that implements the IConstraint interface. It can be used ‘raw’ as in the first of the examples, or hidden behind some syntactic sugar by defining a helper class similar to NUnitLite’s Is class.

That’s where it is so far… You can download pre-release code from Codeplex. Or watch for an 0.1 release soon.


Friday, September 1st, 2006

More On Syntax: Expected Exceptions

Sometimes you expect an exception to be thrown by a method. So, of course, you want a test for that. NUnit provides the ExpectedExceptionAttribute for that purpose. It has a bit of history…

First, it was a list of exception types. That’s right, in NUnit 1.x, you could specify multiple expected exceptions. At some point, somebody thought “Wait, shouldn’t you know what you’re testing for?” and it was reduced in NUnit 2.0 to a single exception type.

[Test, ExpectedException(typeof( MyException ) )]

Then there was the message. Folks wanted to check for a single exception, so a constructor argument was added that allowed you to specify the message.

[Test, ExpectedException(typeof( MyException ), "My special message" )]

But some messages are pretty long, so you might want to specify a substring…

[Test, ExpectedException(typeof( MyException ), "special", MessageMatch.Contains )]

This doesn’t even get into specifying the exception by name rather than type, using regular expressions, providing a custom user message in case of failure,… but I’m sure you get the idea. The ExpectedExceptionAttribute has been overloaded with options, but people keep asking for more, like checking the value of specific properties on a specific type of exception.

Working on NUnitLite has given me the chance to look at this with fresh eyes. The heart of the problem is that there is only so much you can express between the square brackets when you use an attribute. Properties may only be of certain types and only so much fits on the line in a readable way. On the other hand, it’s reasonably simple to check various properties of an exception in a catch block – or in any other code for that matter.

For NUnitLite, I’m trying a simpler approach. First of all, you can simply specify ExpectedException without any arguments…

[Test, ExpectedException]

That means you expect some sort of exception – but you aren’t saying what. I know that this isn’t a very good testing practice by itself, but there are uses for it, as we’ll soon see. Normally, you’ll want to specify an exception type…

[Test, ExpectedException( typeof(MyException) )]

This works slightly differently from NUnit. The test will succeed if MyException or any exception that inherits from it is thrown – NUnit requires the exact type.

Now suppose you want to verify that MyException is thrown and has an particular message and that a certain property it supports has been set to a certain value. If your test class implements the IExpectException interface, you are able to define a handler to validate the exception. Here’s the interface…

public interface IExpectException
        void HandleException( Exception ex );

Here’s an implementation to deal with the hypothetical example

public void HandleException( Exception ex )
        Assert.That( ex.Message, Is.EqualTo( "my special message" );
        MyException mex = (MyException) ex;
        Assert.That( mex.Param == "xxxxx" );

As I’ve experimented with this, I have found that it was rare for me to need more than one exception handler in a class. I can usually generalize one method sufficiently to make it work for all my needs. But, in some cases, there is a need for a different handler. In that case, you can specify the name of the handler method as an additional property of ExpectedException…

[Test, ExpectedException(Handler="AlternateHandler")]
[Test, ExpectedException(typeof(MyException), Handler="AlternateHandler")]

The method must exist and have the correct signature or a run-time error will be thrown.

This is as far as I’ve taken it. I can think of other ways to expand (read complicate) it, but I’m not doing that. Users can write anything they want in the handler and if something turns out to be needed repeatedly we can think about removing the duplication later.


Wednesday, August 30th, 2006

Thinking About Assert Syntax

I’ve been using the development of NUnitLite (coming soon!) as an excuse to try out alternatives to the standard NUnit syntax for Asserts.

NUnit allows us to write things like…

Assert.AreEqual( expected, actual );
Assert.Greater( x, 5 ); Assert.Less( x, 10 );
Assert.IsTrue( x > 5 && x < 10 );

The last is instructive: if it fails, the message simply tells us that True was expected but the value was actually false. Any more intelligent message must be supplied as an argument...

Assert.IsTrue( x > 5 && x < 10, "Value of x must be < 5 and > 10" );

An alternative idea, borrowed from NMock, is to encapsulate the comparison in some sort of an object. Then you can write things like...

Assert.That( actual, new EqualsMatcher( expected ) );
Assert.That( x, new AndMatcher( 
	new (GreaterThanMatcher( 5 ), new LessThanMatcher( 10 ) ) );

Of course, these aren't too easy to type, but with the addition of some helper methods and the use of operator overloading, we can replace it with

Assert.That( actual, Is.EqualTo( expected ) );
Assert.That( x, Is.GreaterThan( 5 ) & Is.LessThan( 10 ) );

That pretty much covers what I've implemented so far. I'm not fully satisfied with the expressiveness of this syntax, though. It's clear but a bit wordy. I'm thinking of going back to a set of inherited methods to either replace or supplement the methods of the Is class. That might give me something like this...

Assert.That( actual, eq( expected ) );
Assert.That( x, gt( 5 ) & lt( 10 ) );

So what do folks think? Is it worth pursuing this syntactic approach? Or should we just go back to Assert.AreEqual and friends?


Friday, July 28th, 2006

I “Discover” Ruby

This week at OSCON 2006 I finally learned some Ruby. I’ve been meaning to do this and learning by listening to Dave Thomas and Mike Clark seemed like it would be much more fun than simply reading a book. Turns out I was right.

Read the rest of this entry »

Friday, May 19th, 2006

NUnitLite and the GPL

In my last post, about the NUnit teams plans for trying out the Microsoft CodePlex site, I introduced the NUNitLite project. One thing I mentioned was the possibility of its being released under the GPL license. This seems to have gotten more reaction than anything else in the post. I find that a bit disappointing, because I thought there were some other cool things in it. 🙂

Be that as it may, I’ll try to explain here why my next software project might use GPL and what kind of considerations I’m looking at in making a final choice.

Read the rest of this entry »

Tuesday, May 9th, 2006

NUnit Extensibility – A High Level View

NUnit originally identified tests in the time-honored way that is still used by most xUnit frameworks. Test classes inherited from the framework’s TestCase class. Individual test case methods were identified by their naming pattern.

Read the rest of this entry »

Friday, December 30th, 2005

The Death of NUnit Revisited – Will it be Fire?

In my earlier post, The Death Of NUnit – Will it be Ice?, I adopted the notion – promoted in some mailing list posts – that NUnit might be dead or dying. I identified two big worries: death by ice and death by fire. Ice stood for frozen inactivity on the part of the project itself – failure to move ahead. Fire symbolized competitive forces, which might make NUnit obsolete.

Read the rest of this entry »