Getting Started with NUnit

Richard Davis, SharpLogic Software

In this article, we will take a look at unit testing, test-driven development, and the NUnit framework. If you are already familiar with these topics and are instead looking for information about how it all ties in with the Software Design Competition, feel free to skip ahead to the NUnit and the Software Design Competition section.

Prerequisite

Read and complete the steps described in the Getting Started With The Software Design Competition article. This will give you a cursory introduction to the process of using the NUnit GUI application to run test code.

Requirements and Setup

One of the great advantages of competing in the Imagine Cup Software Design Competition is that you’re able to use the best tools available, and they’re absolutely free!

1. Download and install a version of Visual Studio 2005 Express from http://msdn.microsoft.com/vstudio/express/. You can use any language to develop your class library, but the baseline source code and tests are provided in C# so you’ll need to have that installed if you want to use them. Install with the default settings.

Note: You may use any version of Visual Studio 2005 that you have available.

2. Download the “NUnit 2.2.8 for .NET 2.0 MSI” installer from http://www.nunit.org/index.php?p=download and install it with the default settings.
3. Download the practice Level 100 Tutorial challenge from http://imaginecup.sharplogic.com/Challenge.aspx?challenge=4265d098-52ac-4eb6-8549-e8ab652d99b8 and unzip it to a convenient location, such as “C:L100MathLib”. At this point you should have a directory structure that looks like the following:

Unzip the challenge package (C:L100MathLibL100Math.zip) to the “C:L100MathLib” directory so that its contents are placed into the “C:L100MathLibL100Math” directory. The resulting directory should look like the following:

Test-Driven Development and Unit Testing

Testing software is an integral phase of the development lifecycle. As software projects grow in size, so too does the complexity. Complexity can refer to the size of the source code base, architecture and design of components, number of code paths, and even algorithmic considerations. The decisions to be made about testing have to do with the “when” and the “how”, not the “why”.

Software testing provides a valuable means to help verify that software was build correctly, but it can also serve as a form of requirements specification up front. Consider the development of a method that computes the division of two integral numbers:

public int Divide(int dividend, int divisor)

{ }

To devise tests for this method we need to think like a tester. This process includes asking questions like the following:

• What are the types of input and what is the set of values they can possess?
• What is the expected return type? Are there any restrictions to be considered upon return to the caller?
• If the input includes reference types, is null (C#) or nothing (VB.NET) ever passed in?

It is important to identify and test for cases where state and input lead to faulty operation. In the case of the Divide method, we can determine that the operation is invalid when the divisor input is zero. Next, we can write a test to exercise the Divide method and check for the desired response. In the case of the Divide method, we expect a System.DivideByZeroException to be thrown when the divisor input is zero.

The test method that we just devised is a type of unit test. Unit tests typically look for a very specific result from an operation. This means that there is often a many to one relationship of tests to one piece of code. Consider the Divide method once again. We already have one test designed, but we also want tests that make sure various valid inputs return the correct results.

As you can see from the Divide example, the development of unit tests can be done before the application code is implemented. This process is known as test-driven development, and it provides many benefits:

• Requirements are self-documented.
• Tests are automatic and repeatable.
• Tests are designed to work with known bad and good input values.

The development of unit tests provides an automatic and repeatable method for verifying the correct functioning of code. Each time code is added or modified in a project, the suite of unit tests can be re-run to make sure that the changes did not break functionality that worked previously. The act of re-running a suite of unit tests is known as regression testing.

Keep in mind that unit testing does not guarantee that software will be free of bugs. Generally speaking, it is not realistic to get full code path coverage that exercises the entire set and combinations of input to all methods in a piece of software. In addition, the quality of testing relies solely on the developer of the tests. There may be cases where a unit test fails to identify problems in code. Nevertheless, test-driven development and unit testing are powerful allies to have throughout the software development lifecycle.

Developing Tests for NUnit

In this section, we will walk through the development of some simple unit tests that target the NUnit testing framework. Make sure that you have downloaded and installed the L100Math example as described in the Requirements and Setup section before continuing.

1. Load the L100Math solution file with Visual Studio 2005:

1. Double-click on the L100Math.sln file that you previously downloaded and extracted to your computer (C:L100MathLibL100MathL100Math.sln).

2. Double-click on the MathHelperTests.cs source file.

2. Examine the MathHelperTests class found within the MathHelperTests.cs source file. Notice that the test class has an attribute named TestFixture attached to it. This attribute is defined as part of the NUnit framework and is used by NUnit to determine which classes contain tests.
3. Examine the test methods found within the MathHelperTests.cs source file:

1. There are two test methods designed to test the functionality of the MathHelper.Add method, TestAdd and TestAdd2.

2. Each test method is marked with the Test attribute so that NUnit can locate designated tests.
3. Each test method is responsible for conducting tests in way that allows the NUnit framework to determine when a test is a success or failure. This is done by using the NUnit Assert class. In the TestAdd test method, we add 3 and 5 together and store the result. Next, the Assert.AreEqual method is given the expected result, the actual result, and a message to be presented to the user in case the provided values are not equal.

4. Add a new test method to the MathHelperTests class which tests for the raising of a DivideByZeroException when the divisor parameter is zero:

1. Copy and paste the following code:

[Test]

[ExpectedException(typeof(DivideByZeroException))]

public void TestDivide2()

{

int quotient = MathHelper.Divide(4, 0);

}

2. Here we used another NUnit framework attribute named ExpectedException to inform NUnit that the test will be considered a success if the specified DivideByZeroException exception type is thrown.
3. Note that we could alternatively wrap the call to the MathHelper.Divide method within a try…catch block. The test will succeed in this case because we prevent the DivideByZeroException from being propagated up the call stack to the NUnit framework.

[Test]

public void TestDivide2()

{

try

{

int quotient = MathHelper.Divide(4, 0);

Assert.Fail("MathHelper.Divide(4, 0) " +

"failed to throw a DivideByZeroException.");

}

catch (DivideByZeroException)

{

}

}

It is important to note the requirements for the most essential NUnit framework attributes, TestFixture and Test. TestFixture requires the following from the class that it is applied to:

• Public scope.
• A public, default constructor without parameters.

If a method that is marked with the TestFixture attribute does not have a default constructor with the correct signature, i.e. you marked it as private instead of public; all tests will fail to run. The reason will be listed under the Tests Not Run tab as shown below.

Note: You should just leave out the constructor unless you need to perform initialization.

The Test attribute requires the following from the method that it is applied to:

• Public scope.
• Void return type.
• No parameters.

If a method that is marked with the Test attribute does not return void, the test will not be run by NUnit and the reason why will be listed under the Tests Not Run tab as shown below.

Assertions are another very important topic to become familiar with as you gain more experience using the NUnit platform. The Assert.AreEqual method, which we just saw in action, is categorized as an equality assertion by NUnit. Another type of equality assertion is the Assert.AreNotEqual method, which does exactly what its name implies. Other categories of assertion include identity, comparison, type, and condition. The following tables describe a number of the different assertion categories and their methods:

Equality Asserts
Method	Description
Assert.AreEqual	Tests whether the expected and actual arguments are equal, asserting when they are not equal.
Assert.AreNotEqual	Opposite of AreEqual method.
Examples
// This example shows a successful AreEqual test. int expected = 5; int actual = 5; Assert.AreEqual(expected, actual, "Expected and actual values are not equal.");

Identity Asserts
Method	Description
Assert.AreSame	Tests whether the expected and actual arguments are the same object reference, asserting when they are not the same.
Assert.AreNotSame	Opposite of AreSame method.
Assert.Contains	Tests whether an object is contained in an array or list.
Examples
// This example shows a successful AreSame test. object objA = new object(); object objB = objA; Assert.AreSame(objA, objB, "Object references are not the same."); // This example shows a successful Contains test. string[] myArray = { "a", "b", "c" }; Assert.Contains("b", myArray, "Array does not contain expected value.");

Comparison Asserts
Method	Description
Assert.Greater	Tests whether one object is greater than another.
Assert.Less	Tests whether one object is less than another.
Examples
// This example shows a successful Less test (arg1 < arg2). int arg1 = 5; int arg2 = 6; Assert.Less(arg1, arg2);

Type Asserts
Method	Description
IsInstanceOfType	Tests whether an object is of a specified type or derived from a specified type, asserting when it is not.
IsNotInstanceOfType	Opposite of IsInstanceOfType method.
IsAssignableFrom	Tests whether it is valid to assign an instance of the specified type to a given object, asserting if it is not.
IsNotAssignableFrom	Opposite of IsAssignableFrom method.
Examples
// This example shows a successful IsInstanceOfType test. Exception myException = new Exception(); Assert.IsInstanceOfType(typeof(Exception), myException); // This example shows a successful IsAssignableFrom test. object myObject = new object(); Assert.IsAssignableFrom(typeof(Exception), myObject); Exception myException = new Exception(); // Assertion guarantees that myObject can be assigned myException (or // any other object that is typeof(Exception)). myObject = myException;

Condition Asserts
Method	Description
Assert.IsTrue	Tests whether a boolean condition is true, asserting when it is not.
Assert.IsFalse	Tests whether a boolean condition is false, asserting when it is.
Assert.IsNull	Tests whether an object is null, asserting when it is not.
Assert.IsNotNull	Opposite of IsNull method.
Assert.IsNaN	Tests whether a double is not a number (NaN), asserting when it is not.
Assert.IsEmpty	One overload tests whether a string is empty (string.Empty or “”), asserting when it is not. Another overload tests whether a collection or array is empty, asserting when it is not.
Assert.IsNotEmpty	Opposite of IsEmpty method.
Examples
// This example shows an unsuccessful IsNotEmpty test. List<int> myList = new List<int>(); Assert.IsNotEmpty(myList, "List is empty.");

Utility Asserts
Method	Description
Assert.Fail	Throws an AssertionException with specified failure message.
Assert.Ignore	Causes the current test to be ignored.
Examples
// This example shows a custom assertion that fails when the // myDouble variable does not fit within the specified value range. double myDouble = 0.94; if (myDouble < 0.95 || myDouble > 1.05) { Assert.Fail("Value is not within the specified range."); }

String Asserts
Method	Description
StringAssert.Contains	Tests whether a string contains an expected substring, asserting when it does not.
StringAssert.StartsWith	Tests whether a string starts with an expected substring, asserting when it does not.
StringAssert.EndsWith	Tests whether a string ends with an expected substring, asserting when it does not.
StringAssert.AreEqualIgnoringCase	Tests whether an expected string is equal to an actual string ignoring case, asserting when they are not.
Examples
// This example shows a successful AreEqualIgnoringCase test. string actual = "test"; StringAssert.AreEqualIgnoringCase("tEsT", actual);

Running Tests Using NUnit

Assuming that you satisfied the prerequisite listed at the beginning of this article, loading and running tests using the NUnit GUI should be a familiar process at this point. In this section, we will take a more detailed look at two ways in which tests can be run and results generated.

NUnit GUI Application

The NUnit GUI application provides a complete, easy to use interface with which you can load, select, and view the results for unit tests. As you are already aware, loading a test assembly is as simple as selecting File | Open. Once an assembly is loaded, the nunit-gui.exe application will continue to monitor the file for changes. If you make changes in Visual Studio and re-compile, the NUnit application will re-load the assembly automatically so that it remains in sync with your development.

1. Build the solution by selecting Build | Build Solution from the main menu.

The NUnit GUI application works with .nunit project files, so any configuration options can be tied to an assembly and saved for future use. If you take a look at the L100MathTests project again, you will notice a file named L100MathTests.nunit. This file stores NUnit project configuration data in an XML format and it is added to the project as a matter of convenience.

Follow these steps to have Visual Studio open the NUnit project file using the NUnit-Gui.exe application by default:

2. Right-click on L100MathTests.nunit and select Open With.
3. Select the NUnit-Gui (NUnit) program followed by the Set as Default button.
4. Select the OK button.

Now you can load the unit tests by simply double-clicking on the .nunit project file from Solution Explorer. If NUnit fails to launch, you may need to repeat steps 2-4, but this time add a new program to the list and navigate to the nunit-gui.exe application explicitly.

You can run all tests by selecting the root node and then selecting the Run button. If tests fail to meet the assertion criteria, informative messages will be placed in the Errors and Failures tab. A red node means that not all tests passed at that node or at its descendent nodes. Green indicates that all tests passed at a node and at its descendent nodes. It is also possible to run individual test classes or even individual tests within test classes. To do so, simply select the test node that you want to run and then select Run.

NUnit test results can be saved for future use by selecting Tools | Save Results as XML from the main menu. If you select the TestAdd method and run the test, the resulting XML document will include the following information:

<test-case name="L100MathTests.MathHelperTests.TestAdd" executed="True" success="False" time="0.047" asserts="0">

<failure>

<message><![CDATA[Tested: MathHelper.Add(3, 5)

expected: <8>

but was: <0>]]></message>

<stack-trace><![CDATA[ at L100MathTests.MathHelperTests.TestAdd() in C:L100MathLibL100MathL100MathTestsMathHelperTests.cs:line 20

]]></stack-trace>

</failure>

</test-case>

This XML tells us that the TestAdd method was executed successfully but failed to meet the conditions of an assertion. A stack trace is also provided to let us know exactly where the assertion that failed is located.

NUnit Console Application

The NUnit console application is an alternative to the GUI application that is capable of performing tests and producing XML test results. As a quick demonstration, let’s use the NUnit console application to test the L100Math library we have been working with.

1. Load a command prompt window by selecting Start | All Programs | Accessories | Command Prompt.
2. Change the current directory to the location where the test DLL is created by Visual Studio when built as shown below.

3. Temporarily modify the Path environment variable to include the path to the nunit-console.exe application. Use the following command:

path=%path%;”c:Program FilesNUnit-Net-2.0 2.2.8bin”

4. Use the NUnit console application to test all test methods contained within the L100MathTests.dll assembly:

nunit-console L100MathTests.dll

Once the NUnit console application has finished running tests, failure messages are output to the console window. If you see errors here, it is an indication that you should check the results XML file for more details. By default, NUnit will create the results XML file in the same directory where you performed the tests and name it TestResult.xml.

5. Open the TestResult.xml file generated from the last test run. You can do this from an Explorer window or by issuing the command:

notepad TestResult.xml

The NUnit console application can also use Visual Studio project files and NUnit project files to run tests, as show in the examples below:

Nunit-console L100MathTests.csproj

Nunit-console L100MathTests.nunit

Many other options are available for use with the NUnit console application. Please refer to the official NUnit documentation for more information.

NUnit and the Software Design Competition

Now that you understand how to write unit tests for the NUnit framework and use the associated tools, let’s take a moment to discuss how the Software Design Competition affects the testing environment and strategy. In this section, we will highlight important points that competitors will need to keep in mind as they develop, test, submit, and debug their entries.

The design and nature of the Software Design Competition has a number of effects upon user-submitted challenge entries:

• Entries must be optimized enough so that they can complete the tests performed on the backend before a set timeout.
• Entries must only use code that can run under the “Execution” permission set.

For some challenges, it may be difficult to know how much optimization is necessary because the backend tests are not available to you. One solution to this problem is to simply not worry about optimization unless you get a timeout message back from the testing server. A timeout message would be something like, “Test has exceeded timeout. 3000 ms.” If you get a timeout message, you will need to examine your solution for areas that need to be optimized.

User-submitted entries will only be given the ability to execute code as defined by the .NET code access security policy. This means that you can not use file I/O, sockets, networking, or other tasks that require more privileges. One way that you can ensure your code meets this requirement during testing is to add an appropriate PermissionSet attribute to the top of your classes. The L100Math library test class, MathHelperTests, shows this in action:

[TestFixture]

[PermissionSet(SecurityAction.PermitOnly, Name = "Execution")]

public class MathHelperTests

Note that you will need to create a unit test in order to exercise all of your code for this attribute to have any affect. A security exception will not be thrown until code is executed that tries to do something that it does not have permission to do.

The design and nature of the Software Design Competition also has a number of effects upon the test suite used on the backend, as well as the test results seen by competitors:

• Backend tests may be more complicated than basic unit tests and use multiple objects containing multiple asserts.
• Backend test results are returned and presented to the competitor with failure messages only – stack traces are not made available.

An ideal unit test is designed to be as simple as possible to help eliminate the possible causes for failure when the test is run. However, the Software Design Competition has a need for a consistent number of unit tests to be performed at each difficulty level. Because of this, unit tests may be more complicated than is ideal. It is important to keep in mind that if a unit test has multiple assertions in it, only the first one will be reported. As a result, a unit test may continue to fail, even as you fix individual bugs.

Competitors will only have access to the failure messages retuned from the testing backend. This will be the only diagnostic information available to you as you develop your solution to challenges. As the challenges become more difficult, the failure messages will become less helpful. For example, Level 100 tests will likely explain the exact method call and parameters that failed, whereas Level 500 tests may only mention what the test was trying to accomplish.

You may run into the following types of failure messages as you progress further into the competition:

• Failure message that indicates a specific method failed to return the expected results.
• Failure message that indicates a method failed with specified input values.
• Failure message that indicates an exception occurred in a specific method.
• Failure message that only indicates an exception occurred (no other helpful information provided).
• Failure message due to incorrect object inheritance.
• Failure message due to missing interface implementation.

If you are having difficulty deciphering a failure message and know which method is causing the problem, try developing your own unit tests for it to see if you can determine the root cause for the problem. You may also want to re-consult the specification provided for the challenge to see if you missed any details pertinent to the problem at hand. Remember, there is no penalty for revising your entries and re-submitting them for scoring. However, only your most recent submission counts, so regression bugs can hurt you if you’re not careful.

Debugging NUnit Tests

Once you begin to develop unit tests you may wonder, “How do I test to make sure the tests are correct?” One tool that you have available is the Visual Studio Debugger. Being able to debug the unit tests is useful since it is one of the easiest ways to exercise our library code. In this section, we will demonstrate how you can take advantage of the debugger to test your unit code.

Let’s return to the L100Math library that we were working with earlier. Make sure that you have the L100Math solution file open in Visual Studio and the MathHelperTests.cs source file open for viewing.

There are some differences between the Visual Studio Express products and the other versions of Visual Studio that necessitate two different sets of steps to demonstrate debugging. Please choose the subsection below that applies to you.

Visual Studio Express

1. Minimize all open programs so that your desktop is visible.
2. Right-click on the desktop and select New | Shortcut.
3. For the location of the shortcut, use the following:

"C:Program FilesMicrosoft Visual Studio 8SDKv2.0GuiDebugdbgclr.exe"

4. Select Next.
5. Use the default name for the shortcut and select Finish.

At this point, you should have a shortcut on the desktop that launches the Microsoft CLR Debugger.

6. Ensure that the current solution is built by selecting Build | Build Solution from the main menu in Visual Studio.
7. Double-click on the L100MathTests.nunit file in Solution Explorer to launch the NUnit GUI application.
8. Launch the debugger from the desktop shortcut.

9. Select File | Open | File from the main menu of the debugger. This will open the Open File dialog box.

10. Navigate to and open the MathHelperTests.cs source file.
11. Place a breakpoint at the first line of the TestAdd method.

12. Select Tools | Attach To Process from the main menu of the debugger.
13. Locate and select the nunit-gui.exe process from the list of available processes.

14. Select Attach to attach to the NUnit GUI process. At this point, the assembly containing the TestAdd method has not been called yet, so you will see a symbol where the breakpoint was set.
15. Switch back to the NUnit GUI application and double-click on the TestAdd method. The breakpoint that you set should be hit and the CLR Debugger application brought back to the foreground.

16. Select Debug | Step Into from the debugger main menu to enter the MathHelper.Add method.

Now that you know how to attach to the NUnit process, the full power of the Microsoft CLR Debugger is at your fingertips.

Visual Studio Standard or higher

1. Place a breakpoint at the first line of the TestAdd method.

2. Right-click on the L100MathTests project in Solution Explorer and select Properties.
3. Select the Debug tab and fill in the following options:

Start External Program text box: C:Program FilesNUnit-Net-2.0 2.2.8binnunit-gui.exe

Command Line Arguments text box: L100MathTests.dll

Working Directory text box: C:L100MathLibL100MathL100MathTestsbinDebug

4. Select Debug | Start Debugging from the main menu or use the F5 shortcut.
5. Switch to the NUnit GUI application and double-click on the TestAdd method. The breakpoint that you previously set should be hit and a yellow cursor and line will be shown.

6. Select Debug | Step Into from the debugger main menu to enter the MathHelper.Add method.

Now that you know how to attach to the NUnit process, the full power of the Microsoft Visual Studio Debugger is at your fingertips.

Resources & References

Here are some addition resources that you may find helpful:

NUnit project - http://nunit.org/index.php?p=documentation

Wikipedia TDD - http://en.wikipedia.org/wiki/Test-driven_development

Wikipedia unit testing - http://en.wikipedia.org/wiki/Unit_test

MSDN unit testing article - http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvs05/html/utfwvs05tmsys.asp