October, 2006

Wednesday, October 4th, 2006

The Tester Tested : Fit and NUnit

NUnit has lots of tests of its own – unit tests that is. It even has some fairly high-level tests that don’t fit well into the normal unit-testing paradigm. But, up to now, the only acceptance tests were manual – a list of things I do before uploading a new release. Since they are manual, they don’t get run all that often and surprises happen.

I’ve been meaning to do some experimenting with writing Fit tests for NUnit, and finally got some time last weekend. You can see the results of my initial experiments here.

For each row, the Fit fixture takes a snippet of code out of the first column, compiles it using the ICodeCompiler interface, loads the resulting assembly into an AppDomain and runs the tests. In addition to verifying that the code compiles, it checks the shape of the test tree that is produced and verifies that the correct number of tests were run, skipped or ignored and if the count of successes and failures are as expected.

This particular fixture verifies that tests written by a programmer using standard NUnit syntax will behave as expected. Of course, we need other sorts of tests, including those that test the various execution options of NUnit. This seems like a good place to start though.


Monday, October 2nd, 2006

More on Assert Syntax

In an earlier post, I presented some ideas about syntax for expressing assertions in tests. I was doing this as a part of the development of NUnitLite, with the idea of eventually putting some of the same concepts back into NUnit.

I received many useful comments. Some of them are still percolating in my brain, but others have already found there way into the implementation of NUnitLite. As a way of paying back, I thought I would tell you here how I ended up resolving some of the issues we discussed.

NUnitLite Syntax by Example

Here are some valid NUnitLite Asserts, assuming proper declaration of variables…

  Assert.That( result, Is.EqualTo( 4 ) );
  Assert.That( result, Is.Not.EqualTo( 4 ) );
  Assert.That( result, !Is.EqualTo( 4 ) );
  Assert.That( obj, Is.Type( typeof(string) ) & Is.EqualTo( "Hello" ) );
  Assert.That( greeting, Contains.Substring( "Hello" ) );
  Assert.That( greeting, Contains.Substring( "HELLO" ).IgnoreCase );
  Assert.That( array, Is.EqualTo( new int[] { 1, 2, 3, 4 } ) );
  Assert.That( matrix, Is.EqualTo( new int[] { 1, 2, 3, 4 } ).AsCollection );
  Assert.That( collection, Contains.Item( obj ) );
  Assert.That( collection, Contains.Item( "HELLO" ).IgnoreCase );
  Assert.That( collection, Is.All.Type( string ) );
  Assert.That( collection, Is.All.Not.Null & Is.All.Type( string ) );
  Assert.That( obj, new UserConstraint(arg) );
  Assert.That( obj, XXX.UserConstraint(arg) );

Roads Taken and Not Taken

I started out trying to make the syntax more terse. However, several people pointed out that modern IDEs provide auto-completion in context, which makes the length of an expression relatively unimportant. In addition, at least one non-English speaker indicated that abbreviations like eq, gt, etc. can be confusing.

I spiked expressions like

  Assert.That( 2+2, Is.GreaterThan(3).And.LessThan(5) );

While this is definitely doable, it’s more complex and I don’t find it very readable. That could be just me, of course. In addition, while it can be implemented as a simple
operator precedence grammar, a simplistic implementation doesn’t provide type safety. That is, you can end up compiling expressions like This.And.And.That or And.That. The current implementation won’t compile such invalid sequences and Visual Studio’s Intellisense only prompts for valid continuations.

The postfix operators like IgnoreCase are what I call Modifiers. They are simply getters that modify the state of the underlying object and return this.

The two final examples illustrate how one might extend the Syntax. UserConstraint is some user-defined test that implements the IConstraint interface. It can be used ‘raw’ as in the first of the examples, or hidden behind some syntactic sugar by defining a helper class similar to NUnitLite’s Is class.

That’s where it is so far… You can download pre-release code from Codeplex. Or watch for an 0.1 release soon.