Getting Started

Download sample application

Throughout this guide we'll be using the sample "Money" application included with the NUnit distribution. While not a particular complex project, it is one familiar to anyone who is already familiar with unit testing, regardless of their language of choice. While not absolutely necessary, if you want to follow along with the examples in this guide you can download the source code from the NUnit website.

Configure your solution

If your application already includes unit tests written with NUnit, there's nothing more you'll need to do. TestMatrix does not require any special configuration or enforce any constraints about where your tests are. You can put tests into their own project, their own namespace, or even along side your application code. TestMatrix will find them.

NUnit compatibility

TestMatrix is built on, and is 100% compatible with, NUnit, the de-facto standard framework for unit testing in .NET. NUnit itself is a descendent of JUnit and other xUnit testing frameworks already familiar to most developers. If you are already unit testing, there's nothing more to learn. TestMatrix does not add anything to the NUnit API, and your tests remain compatible with other NUnit test runners such as NAnt and CruiseControl. If you are new to unit testing, then rest assure that the NUnit API is simple to learn and that you'll be picking up a skill that is increasingly expected of any .NET developer.

Adding NUnit references to your projects

It is not necessary to install NUnit in order for TestMatrix to function properly. Since your unit tests are written with NUnit however, you will need to include the proper assembly references for your tests to compile, as you would with any other library:

You can include references to your own copies of NUnit, or use the ones bundled with TestMatrix. To add references to the bundled DLLs, simply select the Fix NUnit References option from the TestMatrix | Unit Testing menu. TestMatrix always ships with the latest version of NUnit. If you are following along with the sample application, you'll need to do this now.

Building your solution

No extra steps are required when building your solution for use with TestMatrix. Each time you build, TestMatrix scans your assemblies for unit tests and updates its internal list. The first time you load your solution therefore, and any time you make changes, they will not be recognized until after you have built. The editor integration is always active however, and will recognize unit tests as soon as you've written them.

Run unit tests

The goal of TestMatrix is to make unit testing easy and convenient, so that you can make it an integral part of your development process. Tight integration with Visual Studio, and a variety of mechanisms for executing tests help make testing a natural, simple process.

Using the Test Explorer

The Test Explorer panel (TestMatrix | Test Explorer) automatically discovers all of the tests in your solution, and organizes them by assembly, namespace, and fixture into a familiar explorer style interface. You can use the Test Explorer to browse, navigate through and execute any of the unit tests in your solution.

From the drop down at the top of the Test Explorer windows you can choose your entire solution, or a single project to load. This allows you to narrow the scope of your testing to particular assembly if you choose. If you are following along with our example, select the Money project from the drop down. The Test Explorer will load all of the tests and display them in a tree organized by namespace. You can open any test in the editor by left clicking on its entry in the tree.

Running tests from the Test Explorer

The Test Explorer allows you to selectively run any combination of unit tests you would like. You can select individual test cases, specific test fixtures, all tests in a given namespace or project, and many any other combinations. Test Explorer lets you select or exclude entire projects or namespaces with one click. Simply check off the tests you want to run and click on the Run Checked Tests button in the toolbar (the one that looks like a pair arrows) to start the testing run.

Running tests from the Solution Explorer

You can run tests from the Solution Explorer. Right click on any solution, project, folder, or class to access a unit testing context menu.

Running tests from within the editor

The Test Explorer is great for running large numbers of tests but your day to day testing will most likely take place in the editor itself. TestMatrix automatically marks all unit tests in the project with a special gutter icon (three colored balls) and uses special highlighting to indicate which tests have passed or failed.

To run a unit test from the editor, right click on the unit test icon in the gutter at the top of the test declaration. From the context menu select Run or Debug to execute the test. If you have not built since the last change, your code will automatically be compiled before running the test. This menu is also available via the normal editor context menu, accessed by right clicking anywhere inside of the test case.

Running tests from the keyboard

By far the most efficient way to run your tests is via a hot key. TestMatrix defines a number of important hot keys that make it possible to run your tests without taking your hands off the keyboard. If you don't like the defaults, you can always change them through the options panel.

Interpreting the test results

When you run your tests the Test Results window automatically appears to give you feedback about the testing process. In addition the progress bars in the Test Explore track the progress of running tests and provide a summary of testing results. The Test Results window can be filtered or sorted as needed. If you would prefer to view the raw test output, it is visible through the regular Visual Studio output window.

The color of the icon beside each test in the Test Explorer and Test Results window indicates the result of the test. You can show or hide different types of results by clicking on the appropriate filter icons in the Test Results window.

Dealing with failing tests

When a test fails, its icon will turn red. Click on the failing test in either the Test Results window or the Test Explorer to navigate to its source it in the editor. As shown in the figure below, the line that caused the failure will be marked with a red stack trace icon. In this case we can see that the assertion on line 242 has failed. Hover your mouse pointer over the stack trace icon to see an explanation of the failure. (I broke the test for illustration purposes of course).

Failing tests are added to their own tab in the Test Explorer. This makes it easy to work down the list, fixing each test one at a time. If you want to step through the tests in the Visual Studio debugger, you can right click on the test in the Test Explorer and select the Debug Test option from the context menu. Be sure to set a breakpoint first!

Track your code coverage

Code coverage is metric which tells you which lines of your source code have been executed when you run your unit tests, and which have not. The ultimate goal is to provide unit tests that cover 100% of your source code in order to ensure that every line is backed by automated tests. Most importantly, code coverage shows you which lines of code were missed by your unit tests, and answers the all important question: How many tests are enough?

Why measuring code coverage is a critical part of the development process

Unit tests improve the quality and reliability of your code, but just having a few tests for each class is not enough. It's easy to write a unit test, but that doesn't mean that it's actually testing all of your code that it should. Code coverage is the only way to know what you're testing and what you aren't. Miss testing a few switch statement values? Forget to test that failing conditional? Coverage profiling will alert you to these holes in your automated testing so that you can get them plugged. As code changes over time, measuring your code coverage helps ensure that each new feature or bug fix is backed by automated tests. Unlike other coverage tools that are run (usually infrequently) through an external application, TestMatrix runs inside of Visual Studio and provides you with automatic, immediate visual feedback in the editor so you are constantly aware of your coverage.

How code coverage is calculated

There are several ways to track code coverage, but they all boil down to keeping a log of each line of code that was executed during the course of a test run and comparing that to your source tree. Each line that was hit during the test run is covered, everything else is not. To do this, your code must be instrumented with special instructions to enable an appropriate level of tracking. Some coverage tools require you to manually add instrumentation statements to your source code, use an alternate compiler, or run your compiled assemblies through a post processing utility. Not only are these extra steps slow and unwieldy, they discourage your from monitoring your coverage as frequently as you should. TestMatrix works differently, dynamically instrumenting your assemblies at the start of each test run. No extra steps are necessary. The execution information collected during the test run is used to build the coverage reports and editor markup that shows you your code coverage.

Recognizing uncovered code in the editor

Uncovered lines of code are indicated by red bars in their gutter and executed statements are marked with a green bar, if you have enabled this option. (Note that some lines such as braces are never executed per say, and will never be marked as either covered or uncovered.

Bring up the Money class from our example. Assuming you have run all the tests in the solution, you can see that the conditional on line 82 has not been never tested with a passing condition, because lines 83 and 84 were never executed. Note that if you have run anything less than your solution's entire test suite the coverage will only reflect the tests that you actually run, giving you more apparently uncovered code than you may have expected. If you wish, you can change the color of the missed statement indicators through the TestMatrix options, or turn them off all together. In case you're wondering, that "thermometer" gutter icon on line 88 is marking the slowest line of code in the method. More on that later...

Finding all of your uncovered code

When you are working on the unit tests for a particular class or method, the immediate feedback provided in the editor is a great way to measure your progress towards complete code coverage. When you wish to review coverage across your entire solution however, it's time to break out the Coverage Profiler panel for a full report. It shows you the coverage percentage of every class in your solution so you you can no what steps are necessary to improve your coverage.

The Coverage Profiler window shows you the coverage you've achieved across the entire solution, broken down by class, method, and lines of code. The percentage scales indicate the proportion of executable statements that were hit during the most recent test run. You can sort by either class name or percentage coverage, making it easy to find the classes most in need over improved unit tests. By default the Coverage Profiler includes information about every class in your solution. You can use the Namespace Filter at the top of the panel to limit the view to selected namespaces and assemblies. The slider bar lets you hide classes that have reached a certain percentage of coverage, basically a tolerance control that determines what percentage of coverage is sufficient. Classes which meet the coverage indicated on the slider will be hidden.

Profile your code

TestMatrix includes an easy to use line level code profiling features that measure both performance and memory utilization simultaneously as part of the unit testing process. This information will help you better understand the implications of your design choices and spot potential performance problems early. The performance profiler tells you how long each line of code took to execute while the memory profiler keeps track of created objects counts and heap usage.

Hot spot indicators

With each run, the worst performing lines of each method are indicated through the use of hot spot icons in their gutter. The icon's length indicates the percentage of time that line contributed to the overall run time of its method. This will help you quickly identify bottle necks and spot potential problem errors. Only statistically interesting lines are indicated. If all lines contributed more or less equally you won't see any hot spots. In the example below, we can see that line 96 contributed to about 65% of the execution time. If you wanted to make this method faster, this is where you'd start.

Viewing profiling information for each line

Even if a line is not indicated as a hot spot, as long as it has been executed you can hover your mouse in the gutter to view its performance information and memory utilization. TestMatrix will report on several key metrics:

Using the Performance Profiler panel

The Performance Profiler is accessed through the TestMatrix menu. This window is used to review profiling information across your entire solution. Information is rolled up to the method level, so that you can see where your code is spending the bulk of its time.

For each method, the Performance Profiler reports the total execution time, the average time per call, the total number of calls, and the percentage of time this method contributed to total execution time of the test run. Basically, the Performance Profiler is the method level roll up of the line level performance metrics visible in the editor. You can perform the following actions within the Performance Profiler:

Using the Memory Profiler panel

The Memory Profiler works like the Performance Profiler window, rolling up line level information up to the method level and summarizing across the entire test run. You can open the Memory Profiler from the TestMatrix menu.

The Memory Profiler tracks object creation and memory utilization by each method. It can be used to help you better understand your code's true impact on memory. You'll be surprised just how many objects get created by seemingly benign operations. You can perform the following actions within the Memory Profiler:

Features of the profiling panel

You can perform the following operations in either of the profiling windows:

Requirements

TestMatrix is installed as an extension to Visual Studio. No external applications or libraries (such as NUnit) need to be installed for it to function properly. If you can run Visual Studio, you can run TestMatrix.

.NET Framework

TestMatrix requires that you install the .NET 3.0 framework. This does not mean that you must compile or build your projects with the .NET 3.0 framework, just that the plugin require this framework to run. You can still run and test in any version of the framework.

While many of the features of TestMatrix are language agnostic, testing and coverage have some language specific aspects that must be considered. The primary languages supported by TestMatrix are C# and Visual Basic.

Visual Studio

Visual Studio is, of course, required to install and run TestMatrix. TestMatrix will run on VS2005, VS2008 and VS2010 that allow support third party plugins. This means it will run fine on Standard, Professional, Team, Ultimate, but not Express.

Operating System and Hardware

TestMatrix supports XP, Vista and Windows 7, in 32-bit or 64-bit editions. Since this is developer tool, we assume that you have a "developer class machine", meaning one with multiple processors and plenty of RAM. We take full advantage of multi-threading and RAM to give you the best possible performance. Please let us know your hardware configuration if you feel you are experiencing slow performance from TestMatrix.

Compatibility with Other Plugins

It's always tricky to coexist with other plugins when you are all competing for the same gutter, keyboard shortcuts, and the like. We have added a number of features to TestMatrix specifically to help cut down on the number of conflicts. You can disable individual plugins for example, and we've eliminated some visual features that were causing issues. We'd love to hear your feedback regarding any conflicts or problems encountered coexisting with other plugins.

Installation and Updates

First Time Install

You can not install TestMatrix along side earlier versions, or the earlier TestRunner or CodeSpell products sold by Mailframe. Uninstall any previous versions before attempting to install TestMatrix. It can run alongside our StudioTools or CodeSpell products just fine of course. You can download TestMatrix from the web site, and once installed you can use the Auto update feature to ensure you have the latest version.

Licensing

No license key is required to evaluate the software. The software includes a time limited trial license. Once you purchase TestMatrix you will receive a license key to unlock the software for permanent use.

Auto Update

TestMatrix has an automatic update feature which will automatically download updates as they become available. When new tools become available you will be able to download them through the auto update facility.

Uninstalling

You can uninstall everything from the Add/Remove Programs panel. If an uninstall leaves your Visual Studio in a funky state, you can run "devenv.exe" and pass the "/resetaddin *" switch to set everything back to its defaults. If that doesn't work, use the "/reset" option, which resets everything.

TestMatrix uses your Local Settings directory to store user specific information, logs, coverage files, and any data that the plugins need to write. If the uninstaller leaves some trash behind (it shouldn't) and you want to really clean up you'll want to get this to. For example...

<USER>\Local Settings\Application Data\SubMain