Dude, where is my Kaizen? » Testing

How to: Visual Studio test runner & side by side test code

BjRo — Mon, 18 May 2009 20:34:49 +0000

After spending a lot of time with xUnit.net I recently had to go back to MSTest for my testing again (because it’s the de facto standard at my current client which is not negotiable ;-().

Former colleagues of mine will probably laugh about this, because I’ve had a love-and-hate-relationship with MSTest and especially the VisualStudio runner in the past. I won’t go into details here, but you can read about it here. However since I currently have no other option I use what is there. Better writing tests such an environment, than writing no test at all. At least if you ask me.

This brings me back to the topic of the post. Some prerequisites: I like developing software in a TDD manner and am using Resharper with its ‘Generate’ and ‘Move To’ features a lot. I like the idea of having test-code side by side with the actual tested code. With this a lot of you’re daily TDD life becomes at least a bit easier and painless. I see at least two advantages in such an approach:

You don’t have to copy generated classes from the test-assembly to the regular assembly.
You don’t have to maintain two different folder hierarchies (The one in the code-assembly and the one in the regular assembly) and therefore know exactly where to look for test-code.

This is one of the things I learned from JP Boodhoo in his NBDN bootcamp, credit for the general idea goes to him for this. So how can we achieve this in VisualStudio with MSTest?

DISCLAIMER: The stuff I describe in the rest of the post has to do with modifying the .csproj ‘“ msbuild file of a project. If this is a ‘no go’ for you, you can probably stop reading here . For the rest here is the list of steps I took in order to make it work:

Step 1: Add the ProjectType Guids to your project.

VisualStudio’s test explorer scans only projects which are of a certain project type (aka the test project) for tests. VS identifies such a project based on the value of a particular node in the .csproj file, the ProjectTypeGuid node. Simply copy this one from an actual test project to the .csproj of your regular assembly. Another option of course would be to start with a test project from scratch.



  
    Debug
    AnyCPU
    9.0.30729
    2.0
    
    {3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}
    {890DA3CA-999A-474E-BA87-05A834CAB7B8}
    Library
    Properties
    SpecReport
    SpecReport
    v3.5
    512

Step 2: Apply conditional compilation for not including the test code when compiling in release mode

So, the tests sit right next to the tested code, right? We presumable don’t want our tests to be deployed to production. We can achieve this in MSBuild fairly easy with conditional compilation and wildcards.

Step 3: Apply conditional compilation for not deploying the test-related assemblies when compiling in release mode

In order to get rid of the test-specific assemblies for production deployment you can also apply conditionals to the assembly references of a project.



    
      False
      ..\Externals\RhinoMocks35\Rhino.Mocks.dll
    
    
    
      3.5
    
    
      3.5
    
    
      3.5

Et voila: The approach described in this post seams to work fine. At least I’m not experiencing side effects since I’ve configured it that way …

Beat the It

BjRo — Mon, 09 Feb 2009 09:45:55 +0000

. . . or how one could implement the It-Syntax introduced by MSpec. If you don’t know what the hell I’m talking about take a look at this code. (I don’t know if that’s the current MSpec syntax but I think you’ll get where I’d like to take you . . . )

[Description]
public class Transferring_between_from_account_and_to_account
{
  static Account fromAccount;
  static Account toAccount;

  Context before_each =()=>
  {
    fromAccount = new Account {Balance = 1m};
    toAccount = new Account {Balance = 1m};
  };

  When the_transfer_is_made =()=>
  {
    fromAccount.Transfer(1m, toAccount);
  };

  It should_debit_the_from_account_by_the_amount_transferred =()=>
  {
    fromAccount.Balance.ShouldEqual(0m);
  };

  It should_credit_the_to_account_by_the_amount_transferred =()=>
  {
    toAccount.Balance.ShouldEqual(2m);
  };
}

What you’ll see in this post is part of a spike I did some month ago while I was considering to move the xUnit.BDDExtensions syntax to something more GWT and MSpec like. I finally decided against that, partially because I realized that there is absolutely no need for a MSpec-Clone; partially because I’m still not so happy with the implications / side effects of the syntax (nearly everything in you spec needs to become static).

Anyway, recently JP Boodhoo introduced the same feature to his Jpboodhoo.bdd codebase. A lot of the comments requested a more detailed description of how he implemented that feature. While I can’t answer that question, I can talk about how I tackled that problem. So here we go!

Some prerequisites first

If you wonder how such a sweet syntax is possible in C#, here are some clues:

It uses fields with ommited access modifiers (therefore private fields).
It, When, etc. are delegate types.
Field-initializers are used to specifiy the delegates inline. (and because Field-Initializers are run before the constructor you can only access static and no instance members here. Thats the sideeffect I mentioned earlier).

A closer look at my spike

The following class is the heart of the spike I implemented. It takes a blank object and uses reflection to build an ISpecHandler instance which is ultimately used to run the specification.

    public class SpecFactory : ISpecFactory
    {
    	public ISpecHandler CreateSpecFrom(object candidate)
        {
            var fields = candidate.AllFields();

            return fields.AllFixtureInitializers()
                .Then(fields.AllContextInitializers())
                .Then(fields.AllContextFinalizers())
                .Then(fields.BehaviorUnderTest())
                .Then(fields.AllObservations())
                .ThenFinally(fields.AllFixtureCleaners());
        }
    }

As you might guess a lot of extension methods are used in here, together with the composite pattern. Lets start be inspecting the AllFields() extension method.

        public static IEnumerable AllFields(this object instance)
        {
            var mask = BindingFlags.Instance |
                       BindingFlags.Static |
                       BindingFlags.NonPublic |
                       BindingFlags.FlattenHierarchy;

            return instance
                .GetType()
                .GetFields(mask)
                .Select(x => new FieldInfoAdapter(x, instance));
        }

As its name implies this method reflects all the fields from the supplied object. It returns a collection of IFieldInfo objects. IFieldInfo? Yes I introduced a little adapter here, which helped me to introduce better test isolation in the spike code.

    public class FieldInfoAdapter : IFieldInfo
    {
        private readonly FieldInfo fieldInfo;
        private readonly object host;

        public FieldInfoAdapter(FieldInfo fieldInfo, object host)
        {
            this.fieldInfo = fieldInfo;
            this.host = host;
        }

        public object ReadValue()
        {
            return fieldInfo.GetValue(host);
        }

        public Type FieldType
        {
            get { return fieldInfo.FieldType; }
        }

        public string Name
        {
            get { return fieldInfo.Name; }
        }
    }

And here is how the code that finds all the It delegates was implemented.

public static ISpecHandler AllObservations(this IEnumerable fields)
{
    return fields.AllHandlersOf().AsOne();
}

 private static IEnumerable AllHandlersOf(this IEnumerable fields)
 {
      return fields
           .Where(field => field.FieldType == typeof(HandlerType)  &amp;amp;&amp;amp; !field.Name.StartsWith("CS$"))
           .Select(field => new SpecHandler((Delegate)field.ReadValue()));
 }

  private static ISpecHandler AsOne(this IEnumerable specHandler)
  {
        return new CompositeSpecHandler(specHandler);
  }

The basic stuff happens in the AllHandlersOf method which filters the supplied fields by the fieldtype and also removes all compiler generated fields (which start with “CS$” in their names). The content (aka the delegate) of each field that is left is read and wrapped in a SpecHandler instance.

    public class SpecHandler : ISpecHandler
    {
        private readonly Delegate handler;

        public SpecHandler(Delegate handler)
        {
            this.handler = handler;
        }

        public void Run()
        {
            try
            {
                handler.DynamicInvoke();
            }
            catch (TargetInvocationException e)
            {
                throw e.InnerException.PreserveStackTrace();
            }
        }
    }

Last but not least there is the Then extension method which uses the composite pattern to wrap all found spec handlers into a single instance.

 public static ISpecHandler Then(this ISpecHandler handler, ISpecHandler handlerToRunAfterwards)
        {
            return new CompositeSpecHandler(new[]
            {
                handler,
                handlerToRunAfterwards
            });
        }

Conclusion

It’s not as hard to implement such a syntax in C# as one might initially think. Although the code shown in this post is only spike code and not considered production ready, you can derive all what’s necessary to implement such a syntax. Fields, omitted access modifiers, Field-Initializers and Delegates are the basic parts together with some reflection logic to execute the code.

Poor mans dependency injection

BjRo — Fri, 29 Aug 2008 09:02:08 +0000

While watching the first of JP Boodhoos screencast series I discovered that a concept that I used in order to introduce testing to developers who work mostly in legacy code (I’m using Michael Feathers terminology here) is in fact a well known pattern.

Let me give you an example. You’ve got a class that you want test, but this class somehow depends on an untestable resource like for instance a webservice. You don’t have some kind of dependency inversion to you’re hand and have to solve this problem on your own. A short example:

public class BillingService
{
      public BillingResult DoBilling(BillingData data)
      {
            Bill theBil = MakeBill(data);

            using(BillingClient wc = new BillingClient())
            {
                 wc.DoBilling(theBil);
            }
      }
}

What I mostly recommend in such a situation is

Extract or isolate the ugly (Quote from Jeremy Miller) which is the webservice call here.
Introduce an interface that shields the class from the webservice call.
Make the whole class rely on the abstraction.
Use two constructors. I mostly reffered to them as the opened and the closed constructor.

public class BillingService
{
      private IBillingSystem _BillingSystem;

      public BillingService(IBillingSystem billingSystem)
      {
             _BillingSystem = billingSystem;
      }

      public BillingSystem() : this(new BillingSystem())
      {
      }

      public BillingResult DoBilling(BillingData data)
      {
            Bill theBill = MakeBill(data);

            _BillingSystem.DoBilling();
      }
}

public interface IBillingSystem
{
      void DoBilling(Bill theBill);
}

public class BillingSystem : IBillingSystem
{
      public void DoBilling(Bill theBill)
      {
            using(BillingClient wc = new BillingClient())
            {
                 wc.DoBilling(theBill);
            }
      }
}

With that you can easily replace the untestable code with some kind of testdouble like a stub or a mock.

public class BillingServiceTest
{
     [Fact]
     public void DoBilling_ValidBill_BillIsCorrectlySendToBillingSystem()
     {
           BillingSystemStub billingSystem =  new BillingSystemStub();
           BillingService billingService = new BillingService(billingSystem);
           BillingData billingData = MakeValidBillingData();

           billingService.DoBilling(billingData);

           //Do some assertions
           Assert.NotNull(billingSystem.RecievedBill);
     }
}

Today I learned that this is also called POOR MANS DEPENDENCY INJECTION.

P.S.:
Mabe the example is not the best one but I didn’t want to go into something like Foo and Bar again. Besides that it’s only a simplified example. Consider that before throwing eggs and tomatos at me .

P.S.2:
An alternative to this approach which doesn’t require a new interface is “Extract & Override”.

First steps with BDD

BjRo — Wed, 27 Aug 2008 11:35:06 +0000

One of the positive things of being ill and staying at home is having time to look into stuff I always wanted to look into but never actually had the time to. A good example for this is Behavior Driven Development or in short BDD.

BDD is described by its creator Dan North as an enhancement to classic TDD with an emphasis on behavior specifications. It tries to provide business value by using classes and methods written as human readable sentences which can be reflected in order to generate a behavior specification document for the classes under test. The document can fulfill multiple purposes. It could be used as documentation for the tested system and even more important as an enabler for the ubiquitous language of the project (in terms of Evans DDD).

Before looking at something more concrete about BDD, first a brief history of my personal experience with testing throughout the last 3 years. I think this helps me to clarify, why I liked BDD from the moment I started reading about it.

My first unit tests looked very much like a lot of tests I’ve seen from a lot of people starting with unit tests since. (The examples are using the NUnit syntax but you can project this to any unit testing framework you want.)

Every fresh start is hard.

[TestFixture]
public class FactoryTest
{
    private Factory _Factory;

    [SetUp]
    public void SetUp()
    {
          _Factory = new Factory();
    }

    [Test]
    public void Test1()
    {
           SomeProduct product = _Factory.Create("FooProduct");

           Assert.IsNotNull(product);
           Assert.AreEqual(product.Name, "FooProduct");
           Assert.AreEqual(product.Vendor, "Foo");
    }

    [Test]
    [ExpectedException(typeof(ArgumentNullException))]
    public void Test2()
    {
           _Factory.Create(null);
    }
}

For every class I wanted to test I wrote a corresponding test class, which had the suffix “Test” and contained all tests for that class under test. I discovered later that this is a testing strategy known as TEST-CASE-CLASS-PER-CLASS while reading “XUnit Test Patterns” by Gerard Meszaros. At that time it felt pretty confident with what I did. I mean, I wrote tested code, which was to a particular degree bug free. Yeah right, but let’s have a look at the downside of what I was doing:

When such a test failed I didn’t have a clue what the test was actually verifying. I had to debug the testcode.
When a test like “Test1″ failed I didn’t know what assertion was broken. I had to debug the testcode.
It didn’t scale well. I became quite messy and unclear as the number of tests increased.
I always hat a hard time coming back to the test after having spend days in a different topic.

Incorporating a standard naming scheme for tests

As one result of the observed problems I incorporated a standard naming scheme (inspired by a post of Roy Osherove) to my tests which followed the following scheme:

MethodName_Scenario_Behavior

Applied to the sample code it would look like this:

[TestFixture]
public class FactoryTest
{
    private Factory _Factory;

    [SetUp]
    public void SetUp()
    {
          _Factory = new Factory();
    }

    [Test]
    public void Create_ValidCreationData_CreatesAValidProduct()
    {
           SomeProduct product = _Factory.Create("FooProduct");

           Assert.IsNotNull(product);
           Assert.AreEqual(product.Name, "FooProduct");
           Assert.AreEqual(product.Vendor, "Foo");
    }

    [Test]
    [ExpectedException(typeof(ArgumentNullException))]
    public void Create_NullForArgument_ThrowsArgumentNullException()
    {
           _Factory.Create(null);
    }
}

The naming scheme helped me a lot for maintaining my tests. When a test failed I knew instantly a) which class was tested, b) what the test scenario was and c) what the expected behavior was, that was broken. However, the problem with the multiple assertions still existed. You can fix this issue by providing error messages with every assertion. I must admit that I never really liked that approach. I would rather go down the SINGLE-ASSERTION-PER-TEST-METHOD way. This way you have the behavior of the class under test completely discoverable by a unit test framework which is something I appreciate. (Maybe I’m alone with this. Several members of my team disagree with this.)

Single Assertion per Test

After refactoring the test it would look like this:

[TestFixture]
public class FactoryTest
{
    private Factory _Factory;

    [SetUp]
    public void SetUp()
    {
          _Factory = new Factory();
    }

    [Test]
    public void Create_ValidCreationData_CreatedProductHasName()
    {
           SomeProduct product = _Factory.Create("FooProduct");
           Assert.AreEqual(product.Name, "FooProduct");
    }

    [Test]
    public void Create_ValidCreationData_CreatesProductHasNameOfVendor()
    {
           SomeProduct product = _Factory.Create("FooProduct");
           Assert.AreEqual(product.Vendor, "Foo");
    }

    [Test]
    [ExpectedException(typeof(ArgumentNullException))]
    public void Create_NullForArgument_ThrowsArgumentNullException()
    {
           _Factory.Create(null);
    }
}

One downside of this approach is the sudden code duplication you now have in the code. It isn’t that big in the example but it can be in reality. You should treat test code as you’re treating production code. Don’t repeat yourself! I usually dealt with that situation by switching the test strategy.

Switching to TEST-CLASS-PER-FIXTURE

TEST-CLASS-PER-FIXTURE is again a test strategy described by Gerald Meszaros in this book “XUnit Test Patterns”. The basic idea is that you split your test classes into several test classes based on the common initialization / execution logic the methods share. Applied to the sample code this would result in this:

[TestFixture]
public class FactoryTest_ArgumentTests
{
    private Factory _Factory;

    [SetUp]
    public void SetUp()
    {
          _Factory = new Factory();
    }

    [Test]
    [ExpectedException(typeof(ArgumentNullException))]
    public void Create_NullForArgument_ThrowsArgumentNullException()
    {
           _Factory.Create(null);
    }
}

[TestFixture]
public class FactoryTest_CreatingAValidProduct
{
    private Factory _Factory;
    private SomeProduct _Product;

    [SetUp]
    public void SetUp()
    {
          _Factory = new Factory();
          _Product = _Factory.Create("FooProduct");
    }

    [Test]
    public void Create_ValidCreationData_CreatedProductHasName()
    {
           Assert.AreEqual(_Product.Name, "FooProduct");
    }

    [Test]
    public void Create_ValidCreationData_CreatesProductHasNameOfVendor()
    {
           Assert.AreEqual(_Product.Vendor, "Foo");
    }
}

I’ve worked with that approach for 2 years now. It suited my needs very well. However the tests are often not as expressive as I like them to be in terms of the naming. I’ll guess there’s a better naming scheme when using something like the TEST-CASE-CLASS-PER-FIXTURE testing strategy and that’s where the circle closes for me.

Coming back to BDD

While working through the preparation material for JP Boodhoos Nothing but .NET bootcamp I made first contact with his style of applying BDD. I must admit that I’m pretty amazed, because I’ve seen a lot of things of value for me. Sometimes sourcecode says more than a thousand words, so this is what the sample would look like if his BDD conventions have been applied to it:

[Concern(typeof (Factory))]
public class When_creating_a_valid_product : ContextSpecification
{
     private Factory _Factory;
     private Product _CreatedProduct;

     protected override void establish_context()
     {
            _Factory = new Factory();
     }

     protected override void because()
     {
           _CreatedProduct = _Factory.Create("Foo");
     }

     [Observation]
     public void the_product_should_contain_the_correct_product_name()
     {
          _CreatedProduct.Name.should_be_equal_to("FooProduct");
     }

     [Observation]
     public void the_product_should_contain_the_correct_vendor_name()
     {
          _CreatedProduct.Vendor.should_be_equal_to("Foo");
     }
}

[Concern(typeof (Factory))]
public class When_passing_null_for_product_name : ContextSpecification
{
     private Factory _Factory;
     private Action _Call;

     protected override void establish_context()
     {
            _Factory = new Factory();
     }

     protected override void because()
     {
           _Call = The.Action(() => _Factory.Create(null))
     }

     [Observation]
     public void should_throw_an_ArgumentNullException()
     {
           _Call.should_throw_an();
     }
}

Here is the base class that provides the hooks for the unit test framework (in this case MbUnit).

using MbUnit.Framework;
using Context = MbUnit.Framework.TestFixtureAttribute;

namespace NothinButDotNetPrep.SpecHelpers
{
    [Context]
    public abstract class ContextSpecification
    {
        [SetUp]
        public void setup()
        {
            establish_context();
            because();
        }

        [TearDown]
        public void teardown()
        {
            after_each_specification();
        }

        protected abstract void because();
        protected abstract void establish_context();
        protected virtual void after_each_specification()
        {
        }
    }
}

I like his way to implement the AAA (arrange, act and assert) pattern with the template method pattern and C# 3.0 extension method for the actual assertions very much. It feels like a natural progression to what I’ve been doing previously with a lot better readability. Another interesting thing is that he uses a tool called bddunit to generate a behavior documentation from the tests. A generated documentation would look like this:

Behavior of Factory

When creating a valid product
- the product should contain the correct vendor name.
- the product should contain the correct product name.
When passing null for product name
- should throw an ArgumentNullException

I’ll definitely have to dig more into this. If you’re interested in learning more about BDD here are some links I’ve read this morning.

Dan North: Introducing BDD.
AgileJoe: Attempting to demyistify BDD.
JP: Bodhoo: Getting started with BDD style Context / Specification base naming.
Jimmy Bogard: Converting tests into specs is a bad idea.
David Laribee: Approaching BDD.

Some words regarding MSTest

BjRo — Tue, 19 Aug 2008 19:41:50 +0000

Today I stumbled again about this and feel (once more) a bit frustrated about the current version of MSTest.

In theory Visual Studio VS2008 supports Test-Class-Inheritance. Once you actually start using it, you pretty fast encounter one big limitation:

The test base class must be in the same assembly as the derived test

Mh,Â is that such an uncommon scenario that it’s not supported? Personally I don’t think so. What about BDD extensions for MSTest? Do I need to have a Specification-base class in each test assembly? Oh, c’mon ?!

Just when you think it can’t get worse, you realize that

Tests using Test-Class-Inheritance are NOT EXECUTED when running inside a VS TFS 2008 TeamBuild

Awesome

Besides that we also still have the *.vsmdi-hell (Replace * with the name your Solution and a number between 1 and 100 . . .), but that a different story.

If it was my personal decision I would ditch MSTest right away. It’ll be interesting to see whether I’m able to convince our development leads to go down that road . . .