Debugging the NUnit3TestAdapter – take 2

Background

In version 4.2 of the NUnit3TestAdapter debugging has been made a bit simpler. For the earlier versions, see Debugging the NUnit3TestAdapter.

Introduction

A test adapter sits between a TestHost and the test framework. If you use Visual Studio or dotnet, both starts up a TestHost as a separate process. The testhost is responsible for locating the adapters, and then invoke them to run the test frameworks on the test code. Debugging the adapters is hard, because it sits between these processes, of which you have no control. The way to handle this is to enable launching the adapter in debug mode. This post details how you do that.

First, debugging the adapter require you to compile and consume a debug version of the adapter. The package you debug is the nuget package.

Setting up for debugging

Create a folder to keep the nuget debug packages.
We suggest you use the folder C:\nuget

Clone the adapter repository:

git clone https://github.com/nunit/nunit3-vs-adapter.git

You may choose to create a local branch, e.g. debug, but you don't actually need to that anymore

If you prefer do:

git checkout -b debug

You will debug the adapter in Visual Studio 2019, so start up visual studio:

devenv NUnit3TestAdapter.sln

You will then need a repro project, where you have the code you want to debug. For this guide we will create it from scratch, in a folder called "Whatever", but you will probably have your own project for this. Note however, it can be wise to create a small repro project when you're debugging the adapter, and not use a full blown project.

In the folder for the repro/project to be debugged, create (or modify, if it exist already) a nuget.config file.

The content of the nuget.config should be like:

<?xml version="1.0" encoding="utf-8"?>
<configuration>    
    <packageSources>    
        <add key="local" value="c:\nuget" />
    </packageSources>
</configuration>

(PS 1: Note the value, it should be your chosen nuget folder for debug packages.)

(PS 2: You can also add this key/value set to your global nuget.config, if you need to debug multiple projects. You can do this from inside Visual Studio, or just modify the file itself, which is located at %appdata%\nuget )

Now create the repro project:

dotnet new nunit

You can now start Visual Studio proper :

devenv Whatever.csproj

or use Visual Studio Code

code .

No need anymore to modify the adapter code for debugging

In earlier versions before 4.2, you had to modify the code. You don't need that anymore.

However, if you plan to change something in the adapter, then you can follow the steps below here.

The only file you may want to change is the build.cake file.

In the build.cake file, go to Line 16, and add a useful modifer - it will be the preview version for the package, so something like '-d01' would go fine.
Ensure you have the dash there!

If you do changes in the adapter code, you can just increment this number, for each one.

Building a debug version

Build a debug version is a two-step process, first compile it, then package it.

build -c debug
build -t package -c debug

Notice the version number created for the package, underlined red below:

Given that your nuget folder is in c:\nuget, you can now just run the command 'copynp', replacing the argument with your particular package version.

copynp 4.2.0-dbg

Your debug package is now in the c:\nuget folder.

Using the debug package

Now go to your repro project, and depending on whether you use VS Code or Visual Studio, you have to add this particular package version.

Notice that if you use the old legacy project format, then you better use Visual Studio and do the changes in the Tools/Nuget Package Manager/Manage Nuget packages for Solution.... dialog.

Remember to check the box for "Include prereleases", otherwise you will not see it, AND of course, switch your source to Local or use All. You may have later versions, so it might look like this:

If you use the new SDK format, you can go straight into the csproj file and modify that one, although, if you have multiple files, the method above could be faster.

Starting a debug session

There are two ways to start the session, one from command line and one from Visual Studio itself.

Command line

You need to add a runsettings command to your command line, like :

debug test -- NUnit.DebugExecution=true

If you need to use a runsettings file for other purposes, just add the same setting there.

Visual Studio

Add a runsettings file, or a minimum one like:

<RunSettings>
   <NUnit>
       <DebugExecution>True</DebugExecution>
   </NUnit>
</RunSettings>

Then run your test, and a debug launcher will be started:

Choose the NUnitAdapter (red arrow), and off you go into the adapter code

You are then inside the adapter, and the breakpoint at a Debugger.Launch statement. Step out of this method, and then you're at the top of the Execution process, just after initialization:

You can now set your other breakpoint, single step or whatever you need to do to figure what is going on!.

Some tricks and traps

  • Don't run this with parallell execution, it will just fire up multiple instances of Visual Studio debuggers, and you will drown in them. You can use the runsettings file to disable parallell execution if you have that issue.
  • Limit the number of test cases you're working on. Having too many can get you stuck in the internal loops in the adapter.
  • Remember to use the Dump facility (enable it through runsettings, see Adapter Tips and Tricks. It shows what happens in the interface between Adapter and Framework/Engine.
  • Use the Visual Studio 2019 Tools/Options/Test logging set to Trace to see what happens in the interface between the TestHost and the Adapter.

Happy debugging!

Starting up with C# and .Net Core

Starting up with C# and .Net Core

Get the tools

.Net core

Ensure you have the .net core sdk installed.

Download and install it from here

The run the following command to ensure you got it all right:

dotnet

(then)

dotnet --list-sdks

The first should just show you itself. The second lists the sdks you have installed with their version numbers.

IDEs

You should at least have Visual Studio Code

Start it up and from the Extensions icon (on the left side, 5th down probably) add the following extensions:

  • C# (from Microsoft)
  • GitLens

Optionally you can also install Visual Studio (large IDE) which is superb for debugging. Can come in handy. Select Community Edition, which is free.

Basic c# .net core command line application

Create a basic application, Add a folder Test and then there:

dotnet new nunit

This is a scaffolding operation for a unit test project, the code generated is the csproj project files, and then the UnitTest1.cs file which contains the unit test itself in C# code.

You can build and run test tests simply by:

dotnet test

If you just wanted to build, without running the test, it is just

dotnet build

The start code by writing:

code .

(Notice the dot!)

And it will start up with the project in the editor.

The dotnet new command has multiple templates to choose from. Get a list by using the dotnet new --list

Create a new sibling folder to test (name it cons), and then run dotnet new console which gives you the basic Hello World example program.

You can also add yet another sibling folder MyLib, and then run dotnet new classlib.

Notice, some of the classnames and namespaces are in lower case only. In C# the convention is Pascal convention, first letter Uppercase, the rest lower case.

Notice: Arrange each project you have as a SUBFOLDER under your root solution folder

More info on dotnet new

Now change the name from Class1 to Math

Add a method like:

public double Add (double a, double b)
{
    return a + b;
}

The project is called mylib (as the file mylib.csproj). You rename this to e.g. MyMath.csproj.

Now, go back to the root folder for these three projects and run:

dotnet new sln

It will create a file named after the folder you're in, in my case examples.sln

Assuming you have installed VS Community edition, start it up doing:

devenv examples.sln

It will be slower on startup than VS code, as expected - it is a bigger IDE, but after a while you'll see the Solution explorer to the right side, showing a single node named Solution (which is the last file you created)

Now from the context menu on that node, select Add/Existing projects, and located and add the three csproj files you have recently created.

When finished, it should look like:

Select the top level Build menu, and do Build solution

If you get any problems, open up the 3 csproj files (double click in sln). Ensure the Cons and the Test have TargetFramework set to netcoreapp3.1 and the MyMath have the same set to netstandard2.1

Do another build, and you should be fine..... perhaps a context menu Restore Nuget packages, if it is hard to get going.

Now, click the Dependency node on Cons and select Add Project Reference. and then select the MyMath project. Do the same for the Test project.

Now, take a look inside the csproj files and see that it has now added a ProjectReference node. Now you know how it looks, and you can add these in the editor later, if you prefer that.

<ItemGroup>
    <ProjectReference Include="..\mylib\MyMath.csproj" />
  </ItemGroup>

Now change your program code in Program.cs to look like:

using System;
using System.Linq;

namespace cons
{
    public class Program
    {
        static void Main(string[] args)
        {
            if (!args.Any())
            {
                Console.WriteLine("Math arg1 arg2");
                return;
            }

            if (args.Length != 2)
            {
                Console.WriteLine($"Math needs two arguments, just got {args.Length}");
                return;
            }

            var ok = double.TryParse(args[0], out double a);
            if (!ok)
            {
                Console.WriteLine($"First argument must be a float number, but was {args[0]}");
                return;
            }

            ok = double.TryParse(args[1], out double b);
            if (!ok)
            {
                Console.WriteLine($"Second argument must be a float number, but was {args[1]}");
                return;
            }

            var math = new MyLib.Math();
            var result = math.Add(a, b);
            Console.WriteLine($"Result is {result:F2}");
        }
    }
}

And you can try to run it doing :

dotnet run 45.5  45
Result is 90.50

or without arguments or whatever, to check the error handling.

Notice in the code the use of string interpolation, using the $" {somevar}" syntax.

Notice also the inclusion of the System.Linq, which gave you access to the Any method.

And notice the use of the var keyword, which says that the variable should be anything inferred by what is on the right hand side. So in this case the result will be a double, because the math.Add statement returns a double.

Now, the code is a bit duplicated, and the static Main is a bit big, so let us rearrange it a bit.

using System;
using System.Collections.Generic;
using System.Linq;

namespace cons
{
    public class Program
    {
        static void Main(string[] args)
        {
            var program = new Program(args);
            program.Run();
        }

        private readonly IReadOnlyCollection<string> arguments;

        public Program(IReadOnlyCollection<string> args)
        {
            arguments = args;
        }

        public void Run()
        {
            if (!arguments.Any())
            {
                Console.WriteLine("Math arg1 arg2");
                return;
            }

            if (arguments.Count != 2)
            {
                Console.WriteLine($"Math needs two arguments, just got {arguments.Count}");
                return;
            }

            var result1 = Parse(arguments.First(), "First");
            if (!result1.Ok)
                return;
            var result2 = Parse(arguments.Skip(1).First(), "Second");
            if (!result2.Ok)
                return;
            
            var math = new MyLib.Math();
            var result = math.Add(result1.Result, result2.Result);
            Console.WriteLine($"Result is {result:F2}");
        }

        private (bool Ok, double Result) Parse(string arg, string position)
        {
            var ok = double.TryParse(arguments.First(), out double parsedValue);
            if (!ok)
            {
                Console.WriteLine($"{position} argument must be a float number, but was {arg}");
            }

            return (ok, parsedValue);
        }
    }
}

We have now introduced a few more concepts:

  • The program is split into a simple static Main, and the rest as Instance methods.
  • Constructors with parameters
  • Local member variable
  • Generic list with string
  • Tuples as results from a method
  • Private function
dotnet run 45.5  45
Result is 91.00

Oops - something went wrong somewhere..... That result is not correct.

Unit testing

Now go into the test project.

Return to Visual Studio,a dn look at the left side. You should see a Test Explorer window there. (If not, get it from the top menu, Test/Test Explorer)

Press the Run All button to the left in the top button bar of that hub.

It should go green. Double click the Test1 node, to see the code.

We now want to unit test the program we wrote.

Let us start with checking the Math library we had

        [Test]
        public void TestMath()
        {
            var sut = new MyLib.Math();
            var result = sut.Add(45.5, 45);
            Assert.That(result, Is.EqualTo(90.5).Within(0.001));
        }

Notice the Within constraint added. Doubles are never exact, so we put a range to it.

Now do a Test All again, and it should go green.

Testing the program

The program itself seems to have the error, but it is not that easy to test. So let us make the program itself more testable.

First, add a project reference in the Test project to the cons project.

Then make the Run method return a double, which should be the result from the Add operation. There is one snag here, and that is in case of errors, we can't really see that explicitly. So we will introduce a bool Property.

Add the following to the class:

        public bool Ok { get; private set; } = false;

Now, in the Run method, after the parsing, just before the call the Math.Add, do:

    Ok = true;

This property is now read only from the outside. The private set'er ensures we can set it from inside the class.

Then, make the Parse method public too. We do this so that we can test this by itself. Make a /// comment above the Parse method in Visual Studio, it should expand to a comment block: Add the comment Made public for testing

Go back to the test project, and create a new test class for testing the Program class. The other is now for Math Test. And yes, you should rename it!

Add a new Test method, but now we shall use another type of test:

using System.Collections.Generic;
using cons;
using NUnit.Framework;

namespace examples
{
    public class ProgramTest
    {
        [TestCase("45.5", "45", 90.5)]
        public void TestRunMethod(string a, string b, double expected)
        {
            var sut = new Program(new List<string> {a,b});

            var result = sut.Run();

            Assert.That(result, Is.EqualTo(expected).Within(0.001));
        }
    }
}

This is a parameterized tests, and we can add more cases if we like. Add a few more to check.

Running this test, will show it as red. We sort of knew that already, so we have to dig deeper. We know that the Math test is green, this is sort of integrating the two. Let us see if we can remove the Math class from the equation first.

We start off by adding an interface to the Math class:

    public interface IMath
    {
        double Add (double a, double b);
    }

    public class Math : IMath
    {
        public double Add (double a, double b)
        {
            return a + b;
        }
    }

And then we inject that into the Program instead of new'ing it up internally:

 public class Program
    {
        static void Main(string[] args)
        {
            var program = new Program(args, new MyLib.Math());
            program.Run();
        }

        private readonly IReadOnlyCollection<string> arguments;
        private IMath Math { get; }

        public Program(IReadOnlyCollection<string> args, IMath math)
        {
            arguments = args;
            Math = math;
        }

and using it, we can remove the new'ing further down, and just use the private Math Property. Notice we don't have any setter on it, since it is initialized in the constructor.

        var result = Math.Add(result1.Result, result2.Result);
        Console.WriteLine($"Result is {result:F2}");
        return result;

Now, we can turn to the test again, and let us start with adding in mocking.

In the test project csproj file, add in the NSubstitute package:

 <ItemGroup>
    <PackageReference Include="NUnit" Version="3.12.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.16.1" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
    <PackageReference Include="NSubstitute" Version="4.2.2" />
  </ItemGroup>

First, you need to add in the new MyLib.Math() to the ctor for the Program call.

Then create a new copy of that test, call it TestRunMethodOnly, but now add in a mock for the Math instance.

        [Test]
        public void TestRunMethodOnly()
        {
            var math = Substitute.For<IMath>();
            math.Add(45.5, 45).Returns(90.5);
            var sut = new Program(new List<string> { "45.5", "45" }, math);

            var result = sut.Run();

            Assert.That(result, Is.EqualTo(90.5).Within(0.001));
        }

Try to run these. You will see they both are red, which means the error must be in the Program class, since the last one doesn't have the real Math class, but are using a mock.

Now, let us test the Parse method. Notice we still need to construct the Program class, but we will not be using the arguments, so they can be dummies both of them. We will still use the same arguments though, just since we have it.

        [TestCase("10", 10.0)]
        [TestCase("20.4", 20.4)]
        [TestCase("45.5", 45.5)]
        public void TestParse(string arg, double expected)
        {
            var math = Substitute.For<IMath>();
            var sut = new Program(new List<string> { "45.5", "45" }, math);
            var result = sut.Parse(arg, "whatever");
            Assert.Multiple(() =>
            {
                Assert.That(result.Ok);
                Assert.That(result.Result, Is.EqualTo(expected).Within(0.001));
            });
        }

Running this, we get the following results:

And, we see that regardless of what we give in, the result is equal to 45.5. Which incidentially is the same number as what we have inserted as parameter in the constructor, so the Parse method seems to pick that one up instead of the real parameter.

Looking at the Parse method we see the line:

public (bool Ok, double Result) Parse(string arg, string position)
        {
            var ok = double.TryParse(arguments.First(), out double parsedValue);

and indeed, the argument is arguments.First(), and not the arg parameter to the method. We fix this, and then rerun the tests:

Debugging the NUnit3TestAdapter

A test adapter sits between a TestHost and the test framework. If you use Visual Studio or dotnet, both starts up a TestHost as a seperate process. The testhost is responsible for locating the adapters, and then invoke them to run the test frameworks on the test code. Debugging the adapters is hard, because it sits between these processes, of which you have no control. Continue Reading →