Wednesday, December 12th, 2007

Time for a New NUnit

As I’ve worked on NUnit 2.4 and its follow-ups, I have begun to feel that it’s time for a much more significant update to NUnit, a rethinking of what it’s all about and how it should work. Of course, at the same time, I don’t want to lose all the work that has gone into NUnit up to now. Puzzling over this, and helped by a group of dedicated NUnit users and contributors, I think I now have a direction to go.

The new NUnit 3.0 Extended Testing Platform is at the same time a simplification of NUnit and an order of magnitude larger and more complex.

How can that be, you ask? Well, it’s simpler because it’s being divided into pieces and users won’t need to understand everything, just the parts they use. Similarly, contributors will be able to focus on a small part of NUnit, adding their own touches, without rebuilding the whole thing.

On the other hand, it’s more complex because it is moving from being a framework for writing unit tests to a testing platform, one that can work with many frameworks and many runners. The new platform is centered around a testing engine, with an API that can be used by any program needing to load and run tests. You’ll be able to write your own runner, in any suitable lanuage, and get test execution functionality from the NUnit test engine. Want to write it in Ruby? No problem. Using Silverlight or Mono’s Moonlight? Again, you’re free to do so.

At the other end of things, you’ll be able to use our new plugin architecture to enable NUnit to run whatever tests you like. Plugins will be available for common alternative frameworks and to add special functionality like parallel execution to the standard NUnit tests.

I’ll post more about what’s coming in the near future. Meanwhile, for a more detailed overview, read the NUnit 3.0 Vision document.

You can expect early (alpha ) releases in the first part of next year.

Saturday, November 3rd, 2007

NUnit Stable Releases

While NUnit 2.4 was in development, I continued to maintain the NUnit 2.2 series as a stable, bug-fix-only branch. NUnit 2.2.10 was the last in that series.

After NUnit 2.4 came out, I decided to experiment with using a single code line. I would make bug fixes and – at the same time – carefully add new features. Each of the releases since then (2.4.1, 2.4.2, 2.4.3 and 2.4.4) has fixed a number of things, introduced some new features and broken one or two things.

A single code line can work, but it requires discipline. Part of that discipline involves doing many of the same things manually that would be handled automatically by a well-thought-out branching structure. The outcome of my experiment is that the approach didn’t work well for me, so I’m going back to the use of release series branches.

If you use the source code from CVS, the Release-2-4-Branch will get you the latest fixes in the 2.4 series. I’ll be tagging the next releases – 2.4.5, etc. – on that branch. The CVS head will be used for any new development, but I’m not sure how much of that there will be. My best guess is that there will be a 2.5 feature release and that will be it for NUnit 2.x.

NUnit 3.0 is now in the planning stages and I’ll be blogging about it in the coming weeks.

Saturday, November 3rd, 2007

Blogging Again

I haven’t done much blogging for a while due – in part – to the enormous amount of Spam my blog was attracting. But I just updated the software and installed a spam filter. I plan to start posting again, and re-open the blog to comments. Hopefully, the spam won’t overwhelm me this time.

Since I last wrote, NUnit 2.4.3 and 2.4.4 have come out as well as NUnitLite 0.1 and 0.2. I’ll blog separately about the status of each of those.

Tuesday, August 7th, 2007

NUnit Extras: Test Result Summarizer

NUnit is currently built under five different runtimes on Windows – plus two on Linux. Each build is tested under the same runtime on which it was built and on a number of the other runtimes. That’s a lot of TestResult files.

I use nunit-summary.exe to quickly view what happened in a particular run. For example,

nunit-summary TestResult-mono-1.0.xml -noheader -brief

gives me a simple, one-line summary of what happened when I ran my tests under mono-1.0 – the naming convention is built into my build script. The following command gives me a set of files TestResult-net-1.0.txt, TestResult-net-1.1.txt, etc. each containing the summary line as well as any failed or not run tests for the corresponding xml file.

nunit-summary TestResult*.xml -out=*.txt

See the help for the command for other things you can do, including using your own xsl to transform the output.

The nunit-summary program is available on the GitHub.

Tuesday, August 7th, 2007

Speed Up Reloading in NUnit 2.4.2

When running under .NET 2.0, NUnit is rather slow at loading tests these days. Although many folks only noticed the slowdown with the release of NUnit 2.4, loading a large set of tests with NUnit 2.2 also takes more time – about twice as much – under .NET 2.0 as .NET 1.1.

There doesn’t appear to be any easy solution to this problem. Inefficiencies in the Reflection mechanism and/or how we are using it seem to be the root of it and NUnit has been making more and more use of Reflection as new options, attributes and add-in capabilities have come along.

However, with NUnit 2.4.2, there is a simple workaround that may reduce the pain!

Try these steps:

  1. Load your test assembly or NUnit project as usual with the NUnit Gui.
  2. Identify a branch of the tree in which you will be working for the next 30 to 60 minutes. This can be either a TestFixture or a namespace that includes multiple fixtures.
  3. Right-click on the tree node you selected and select Load Fixture from the context menu. NUnit will reload only the selected tree branch.
  4. As you modify your assemblies and reload the tests, NUnit will continue to reload only the tree branch you selected. This can be a substantial time savings. If you wish to select a smaller subset of the tests, you can repeat the above steps. If you need to go back to the full set of tests, right-click in the tree and select Clear Fixture.

At this time, the fixture setting is not remembered between NUnit sessions, so this speed-up is only available to those who keep the Gui running while they work. This is actually how the NUnit Gui is designed to be used.

I’ll be working on speeding up test loading in the long-term, but in the meantime I hope this workaround helps.

Monday, August 6th, 2007

NUnit 2.4.2 Release

NUnit 2.4.2 is out and available here. It includes additions to the new constraint-based Assert syntax and new attributes to facilitate testing under various culture settings. The gui has been enhanced to save the visual state of the tree, including the currently selected test, and to allow loading just the fixture you are working on, which saves a great deal of time on reloads.

For a full description of features, read the release notes..

Thursday, May 3rd, 2007

NUnit 2.4.1 Release

I just released NUnit 2.4. It fixes a few issues that slipped into the 2.4 release an makes some minor improvements. I wanted to get this release out because I’ll be travelling for a few months and won’t be in a position to do further releases until July or August.

You can review the release notes for more info about the contents of this release. You can download it here.

Friday, March 16th, 2007

NUnit 2.2.10 – Final 2.2 Release

Although NUnit 2.4 is almost out the door, there were a few outstanding bug fixes to the NUnit 2.2 series hanging around, so I decided to issue a final 2.2 release. Like all the releases since NUnit 2.2.4, this is primarily a bug fix release.

See the release notes for details of what has changed. You can download it here.

Since NUnit 2.4 is coming out in a matter of days, this release is for those who plan to stay with the 2.2 series for some time. Future work will concentrate on NUnit 2.4 enhancements and on planning for NUnit 3.0.

Sunday, March 11th, 2007

NUnit 2.4 Assert Syntax – the Latest Developments

NUnit 2.4 RC2 is out now, correcting a naming conflict with several mock object frameworks that was present in RC1. You can download it here. For a full list of the extensive new features in NUnit 2.4, check out the release notes.

One of those new features is the new constraint-based design for assertions, supported by a new syntax. That syntax, in fact, was the cause of the aforementioned naming conflict, which is now resolved. As a result, RC2 makes more changes than would normally be seen in a release candidate.

I’ve already blogged quite a bit about the new syntax. I originally developed it for NUnitLite, based on design concepts in NMock2. Since there were various experiments and numerous alternatives have been discussed, I’ll describe the whole thing here from scratch, as it is being implemented in NUnit 2.4.

Design Concepts

The “new syntax” is really a second order effect. What this change is really all about is encapsulating individual assertions as objects. We call these objects “constraints” in NUnit, and they encapsulate both a test and the generation of a message.

Readers who have been following the development of NUnit over the past few years will recognize that this is not the first such encapsulation. Back in 2004, we replaced the original procedural implementation of assertions with an object-oriented approach centered around “asserter” objects. Asserters encapsulated everything about an assertion: the test to be made, any test parameters, a message and the actual value being tested.

By eliminating the actual value from the encapsulation, and providing it separately in a method call, we are able to form compound constraints and then apply them to a single actual value. This is the essence of the design, which was arrived at through a series of spikes using various encapsulations.

In addition, the new design provides a MessageWriter, which is used internally by constraints. Most users will not need to use this interface, but those creating their own constraints will want to be familiar with it. There is a clear division of responsibility between the constraint and the writer. A constraint is responsibile for defining message content but the writer is responsible for the appearance of the message. NUnit 2.4 is delivered with one writer, TextMessageWriter, but additional writers supporting formats like html will be available in a future release.

Two Models

NUnit 2.4 supports both the old model of static assert methods and the new syntax. The documentation refers to the older syntax as the “classic model” and the new one as the “constraint-based” model. The classic model has now re-implemented using constraints, so the same underlying code is used no matter which syntax you use.

Some users have expressed concern that the older syntax will eventually fade away. There’s no plan to do that, and I don’t expect it to happen for a long time, if at all. The older syntax maps directly to other frameworks. For example, CollectionAssert uses the same methods and arguments as the class of that name in the Microsoft test framework. Even though the new syntax provides added power, many users will undoubtedly prefer the compatibility that the classic model gives them.

Constraint Objects

The constraint-based model uses the Assert.That method, with an actual value being tested as it’s first argument and a constraint object as the second. The most direct, although not necessarily most convenient, way to express a constraint is to construct the object in-line. The following example uses an instance of the EqualConstraint to perform a test…

Assert.That( 2 + 2, new EqualConstraint( 4 ) );

The same assertion can be written using a helper class to construct the constraint…

Assert.That( 2 + 2, Is.EqualTo( 4 ) );

NUnit 2.4 Supports a wide range of constraints and syntax helpers:

Syntax Helper Constraint Constructor
Is.Null EqualConstraint( null )
Is.True EqualConstraint( true )
Is.False EqualConstraint( false)
Is.NaN EqualConstraint( double.NaN )
Is.Empty EmptyConstraint()
Is.Unique UniqueItemsConstraint()
Is.EqualTo( object expected ) EqualConstraint( object expected )
Is.SameAs( object expected ) SameAsConstraint( object expected )
Is.GreaterThan( IComparable expected ) GreaterThanConstraint( IComparable expected )
Is.GreaterThanOrEqualTo( IComparable expected ) GreaterThanOrEqualConstraint( IComparable expected )
Is.AtLeast( IComparable expected ) GreaterThanOrEqualConstraint( IComparable expected )
Is.LessThan( IComparable expected ) LessThanConstraint( IComparable expected )
Is.LessThanOrEqualTo( IComparable expected ) LessThanOrEqualConstraint( IComparable expected )
Is.AtMost( IComparable expected ) LessThanOrEqualConstraint( IComparable expected )
Is.TypeOf( Type expected ) ExactTypeConstraint( Type expected )
Is.InstanceOfType( Type expected ) InstanceOfypeConstraint( Type expected )
Is.AssignableFrom( Type expected ) AssignableFromConstraint( Type expected )
Is.SubsetOf( ICollection expected ) CollectionSubsetConstraint( ICollection expected )
Is.EquivalentTo( ICollection expected ) CollectionEquivalentTo( ICollection expected )
List.Contains( object expected ) CollectionContainsConstraint( object expected )
Text.Contains( string expected ) SubstringConstraint( string expected )
Text.StartsWith( string expected ) StartsWithConstraint( string expected)
Text.EndsWith( string expected ) EndsWithConstraint( string expected )
Text.Matches( string pattern ) RegexConstraint( string pattern )
Has.Property( string name, object expected ) PropertyConstraint( string name, object expected )
Has.Length( int length ) PropertyConstraint( “Length”, length )
Is.Not.Xxxx, Has.Not.Xxxx, etc. NotConstraint( Xxxx )
operator ! NotConstraint( Xxxx )
Is.All.Xxxx, Has.All.Xxxx, etc. AllItemsConstraint( Xxxx )
operator & AndConstraint( Constraint left, Constraint right )
operator | OrConstraint( Constraint left, Constraint right )

Note that Not and All are used as prefixes to any of the other constraints. The AllItemsConstraint causes the following constraint to be applied to each item in a collection, succeeding only if the constraint succeeds on every item.

Examples of use:

Assert.That( new object[] { 1, 3, 5 }, Is.SubsetOf( new object[] { 5, 4, 3, 2, 1 } );
Assert.That( new string[] { "abc", "bac", "cab" }, Has.All.Length( 3 ) );
Assert.That( 2.0d + 2.0d, Is.EqualTo( 4.0d ).Within( .000005d ) );
Assert.That( "Hello World!", Text.StartsWith( "HELLO" ).IgnoreCase );

The last two examples illustrate the use of constraint modifiers. These are properties or methods of constraints, which are used to modify their operation. They are syntactically valid on all constraints and are ignored by those constraints that do not make use of them. The following constraint modifiers are supported.

  • AsCollection is recognized by EqualConstraint and causes arrays to be compared using the underlying collections, without regard to their respective ranks or dimension lengths.
  • IgnoreCase causes any string comparisons to be case insensitive. It is recognized by EqualConstraint as well as by all the Text constraints.
  • Within( tolerance ) is recognized by EqualConstraint when both values are a floating point type. It causes the comparison to succeed when the difference between the values is less than or equal to the tolerance.

To Inherit or Not

Beginning with version 2.0, NUnit eliminated the need to use inheritance to identify test classes. That policy continues with NUnit 2.4. Test fixtures are identified by use of the TestFixture attribute and users may develop whatever hierarchy of test classes they need. In fact, most large projects develop a set of base classes for use in defining their test fixtures.

With NUnit 2.4, we are introducing a class that is intended to be used as a base for individual fixtures or for entire project test hierarchies. Our choice of a name for this class, AssertionHelper, indicates both what it is and what it is not. It provides a number of methods that are useful in making assertions. It does *not* serve to identify a class as a test fixture in and of itself.

AssertionHelper provides an Expect method, which is identical to the Assert.That method, as well as equivalent methods to those supported by the syntax helpers.
Consequently, assuming the containing class inherits from AssertionHelper, the following assertions are equivalent:

   Assert.AreEqual( 1.5, myObject.calculate(), .00005 );
   Assert.That( myObject.calculate(), Is.EqualTo( 1.5 ).Within( .00005 ) );
   Expect( myObject.calculate(), EqualTo( 1.5 ).Within( .00005 ) );

What Next?

The constraint-based design will continue to be expanded in the final release of NUnit 2.4 and beyond. Where the syntax will go is another question. The underlying constraint model is flexible enough to support a variety of syntactic layers, and the community is just getting started at trying out these ideas.

We expect that many users will want to develop their own constraints and to layer alternate syntax on top of the constraint model. The best of these new ideas will eventually be incorporated either as extensions to NUnit or as part of the framework itself.

So try out the release candidate and then try your hand at writing constraints of your own. See the NUnit documentation or source for a description of the interfaces.

Tuesday, November 28th, 2006

NUnit 2.2.9

The release process for NUnit 2.4 has taken long enough that a number of bug fixes accumulated on the NUnit 2.2 codebase. I’ve just released NUnit 2.2.9, which addreses a number of bugs.

Read the release notes…