Possible runtime differences

xUnit.net 1.5 is compiled against the .Net Framework 3.5. If you’re using F# in combination with the VS2010 RC or RTM (like I do) you’ve got at least to options to make them work together.

1. Use multi-targeting  and configure the F# projects to compile for the .NET 3.5 runtime (Properties/Application/Target Framework).
2. Update the app.config files of xunit.console.exe and xunit.gui.exe with a startup section and specify the .NET framework 4.0 version as supported.

   
        <startup>
       	<supportedRuntime version="v4.0.30128" safemode="true"/> <!-- VS2010 RC -->
       	<supportedRuntime version="v4.0.30319" safemode="true"/> <!-- VS2010 RTM -->
        </startup>
   

Pay attention to your parentheses

My choice was to update the xUnit.net configurations. After the update of the configuration files my assembly was loaded, however the test runner failed to detect my unit tests. As it turns out the open parentheses after a test function play an important role.

     [<Fact>]
     let After_converting_a_valid_data_row_the_title_should_have_been_extracted = //This compiles, but the test doesn't show up in the test runner.
        let row = convertDataRow "Test, 1234"
        Assert.Equal(fst(row), "Test")

     [<Fact>]
     let After_converting_a_valid_data_row_the_title_should_have_been_extracted() = //This will work fine
        let row = convertDataRow "Test, 1234"
        Assert.Equal(fst(row), "Test")
  

My first reaction was: WTF? But after reading some more chapters of "Real world functional programming" and a discussion at our local F# book club the behavior makes sense to me. My current understanding is that omitting the parentheses results in a different method signature. You can easily spot this in FSI. The first one is val After_converting_a_valid_data_row_the_title_should_have_been_extracted : unit while the second one results in a val After_converting_a_valid_data_row_the_title_should_have_been_extracted : unit -> unit. What you can see here is that the first function signature doesn’t have a parameter while the second has a parameter of the type unit. One interesting difference between F# and C# is that the F# equivalent to C#’s void is an actual type called unit. The fun part is that () is it’s only value. Parentheses play a completely different role here The xUnit test runner looks for methods with one unit parameter and a return type of unit. That’s why you need the parentheses.

Testing exceptions with xUnit.Net

One little subtlety I came across when testing exceptions is that you have to explicitly "ignore" the return value when you’re using Assert.Throws and pass in a method which doesn’t return unit. Feels a bit strange at first, but explainable. Again a signature mismatch. Assert.Throws expects a method with a unit -> unit signature. You have to do this in order to please the compiler. (If there’s a better way for this, please let me know)

    [<Fact>]
    let Trying_to_convert_an_invalid_format_throws() =
        Assert.Throws(fun () -> convertDataRow "FuBar" |> ignore)
  

The ignore function simply throws away any value it receives, returns a unit and makes the F# compiler happy.

Conclusion

I hope you saw in this post that testing F# with xUnit.net is actually pretty easy. It’s also a wonderful case for language interop on top of the CLR. Go see it for yourself

desc "Merges the assemblies"
task :merge do
  mkdir DEPLOY_DIR unless File.exists?(DEPLOY_DIR)
  cp "xunit.dll".expand_to(:build_dir), "xunit.dll".expand_to(:deploy_dir)
  cp "Rhino.Mocks.dll".expand_to(:build_dir), "Rhino.Mocks.dll".expand_to(:deploy_dir)
  assemblies_to_merge = ["xUnit.BDDExtensions.dll", "StructureMap.dll", "StructureMap.AutoMocking.dll"]
  ilmerge "xunit.bddextensions.dll", assemblies_to_merge
end

Ruby is a fantastic language. The more I learn about it the more I actually like it. Let’s have a look at how this task is implemented. First of all I opened up Ruby’s string class and added new methods to it (this is called MonkeyPatching, and worth a blog post on its own )

class String
  def escape
    "\"#{self.to_s}\""
  end

  def expand_to dir_symbol
    case dir_symbol
      when :build_dir
        path = BUILD_DIR
      when :source_dir
        path = SOURCE_DIR
      when :externals_dir
        path = EXTERNALS_DIR
      when :deploy_dir
        path = DEPLOY_DIR
    end    

    File.join(path, self.to_s)
  end
end

And here is the actual ilmerge – method. It simply gets the complete path to the ILMerge.exe (not shown here), expands the name to all assemblies that have to be merged and escapes them, before the actual call to ILMerge is issued over the console …

def ilmerge(output_name, assemblies)
  ilmerge = get_tool :ILMerge
  expanded_assemblies = assemblies.map do |x|
    x.expand_to(:build_dir).escape
  end

  sh "#{ilmerge.escape} /out:#{output_name.expand_to(:deploy_dir).escape} #{expanded_assemblies.join(" ")}"
end

Currently 3 assemblies pop out of the Rake build. These are xunit.dll, xunit.bddextensions.dll and Rhino.Mocks.dll. While it’s technically no problem to merge them into a single assembly (and of course I would love to have it that way) there are two problems currently not solved. The R# runner dynamically loads the xunit.dll to run the tests and StructureMap.AutoMocking.dll dynamically loads Rhino.Mocks.dll. That’s the reason why I’m currently not able to merge it.

Any suggestions how to solve that?

xUnit.BDDExtensions is a little framework build on top of xUnit which enables a BDD style of testing with xUnit.

Besides an emphasis on a “test case class per fixture” organization scheme for tests, AAA (Arrange, Act and Assert) style of writing tests and left to right assertions, this framework also provides Rhino.Mocks integration out of the box. By hiding the mechanics of Rhino.Mocks (record and replay model, mocks vs. stubs) behind the scenes, xUnit.BDDExtensions also helps on writing cleaner, easier to understand and less brittle tests

xUnit.BDDExtensions is based on JP Boodhoos BDD extensions for MBUnit used in his “Nothing but .NET” bootcamps.

Enough of the talking, let’s look at some code

This is actual code of the xUnit.BDDExtensions trunk. The SpecificationTestCommand class serves as a kind of adapter inside xUnit. It enables the test class to have methods that are run before and after the actual test method is executed.*

    public class SpecificationTestCommand : BeforeAfterCommand
    {
        public SpecificationTestCommand(
            ITestCommand innerCommand,
            MethodInfo testMethod)
            : base(innerCommand, testMethod)
        {
        }

        public override MethodResult Execute(object testClass)
        {
            var specification = testClass as ITestSpecification;

            try
            {

                if (specification == null)
                {
                    throw new InvalidOperationException(
                          "Instance does not implement ITestSpecification");
                }

                specification.InitializeSpecification();
                return base.Execute(testClass);
            }
            finally
            {
                if (specification != null)
                {
                    specification.CleanupSpecification();
                }
            }
        }
    }

Notice several things:

1. It expects the instance to implement ITestSpecification.
2. It calls the InitializeSpecification() method on it
3. Passes the execution to the inner command (in base.Execute(obj))
4. and finally cleans up the specification by calling CleanupSpecification() on it.

So how can a completely isolated happy path test for this look like?

 [Concern(typeof(SpecificationTestCommand))]
 public class when_a_specification_test_command_is_executed_on_a_ITestSpecification_implementer :
        concern_for_specification_test_command
    {
        private MethodResult expectedTestResult;
        private ITestCommand innerCommand;
        private MethodInfo methodInfo;
        private MethodResult testResult;
        private ITestSpecification testSpecification;

        protected override void EstablishContext()
        {
            innerCommand = Dependency();
            testSpecification = Dependency();
            methodInfo = GetTestMethodHandle();
            expectedTestResult = CreateMethodResult(methodInfo);

            innerCommand.WhenToldTo(x => x.Execute(testSpecification))
                     .Return(expectedTestResult);
        }

        protected override SpecificationTestCommand CreateSut()
        {
            return new SpecificationTestCommand(innerCommand, methodInfo);
        }

        protected override void Because()
        {
            testResult = Sut.Execute(testSpecification);
        }

        [Observation]
        public void should_ask_the_test_specification_to_initialize()
        {
            testSpecification.WasToldTo(x => x.InitializeSpecification());
        }

        [Observation]
        public void should_pass_the_test_specification_to_the_inner_test_command()
        {
            innerCommand.WasToldTo(x => x.Execute(testSpecification));
        }

        [Observation]
        public void should_return_the_test_result_from_the_inner_test_command()
        {
            testResult.ShouldBeEqualTo(expectedTestResult);
        }

        [Observation]
        public void should_ask_the_test_specification_to_cleanup()
        {
            testSpecification.WasToldTo(x => x.CleanupSpecification());
        }
    }

Notice several things:

1. In the EstablishContext() method the whole context for the test is created. This includes creating most of the dependencies with Rhino.Mocks through Dependeny() and the necessary configuration of their behavior. This is done in order to quickly get from an interaction based style of testing to a state based style of testing. In AAA terms this method implements the Arrange part.
2. The CreateSut() method is executed after EstablishContext() and creates the actual system under test.
3. The Because() method implements the actual behavior under test. In AAA terms this is the Act part. In the current example this meant executing the test command with an instance implementing the expected interface.
4. Each method marked with the ObservationAttribute contains a single observation which should be fulfilled when the test has been executed. In AAA terms every Observation is an Assert.

That’s basically it. Pretty straight forward imho. If I managed to make you at least a bit curious about this style of testing, you can grab the source with any Subversion client from:

http://xunitbddextensions.googlecode.com/svn/trunk/

Enjoy coding

*
I know xUnit dropped this behavior in favor of the dispose pattern, but having explicit, non technically named methods for that still seems more natural in the context of BDD to me.

So here it is. Any feedback is appreciated . . .

A short description of my problem

I’m using a test-case-class-per-fixture organization and the template method pattern for writing tests in the AAA scheme. Besides that I use a specialized FactAttribute in order to invoke my AAA – hook methods in the correct order. All of this is residing in a separate assembly.

When I use the build-in FactAttribute to mark my tests alll tests are discovered by the Resharper’s TestExplorer, but when I’m using my own marker attributes nothing is discovered.

What seems to be the problem

It took me some time to find out what was actually going on. I’ve never debugged a Resharper-Plugin before. This helped a lot. It was an interesting experience, though. Didn’t knew that there is so much going on behind the scenes. This is what I found out:

• The XUnitTestProvider class is responsible for discovering tests.
• It skips all assemblies which don’t reference xunit directly.

Hm, I had a reference to xunit in my test library project settings. However I was not referencing xUnit stuff from my code directly (Own FactAttribute and xunitext35 extension methods for assertions). That’s when it hit me:

My problem was probably caused by a compiler optimization. It seems that all unused assembly references are removed when building the test assembly. A look into reflector supported my thesis. So my test assembly has no direct reference to the xunit.dll, only an indirect one through the BDD framework.

Patching this is quite easy

This is the original method:

static bool IsTestAssembly(IMetadataAssembly assembly)
{
        foreach (AssemblyReference reference in assembly.ReferencedAssembliesNames)
               if (reference.AssemblyName.Name.ToLowerInvariant() == "xunit")
                     return true;

        return false;
}

When changing this a bit to this, everything runs as expected.

static bool IsTestAssembly(IMetadataAssembly assembly)
{
        foreach (AssemblyReference reference in assembly.ReferencedAssembliesNames)
        {
              string assemblyName = reference.AssemblyName.Name.ToLowerInvariant();

              if (assemblyName.StartsWith("xunit"))
                   return true;
       }

       return false;
}

All tests are discovered and I’m quite happy that I can use BDD, XUnit and Resharper in combination

One thing to be aware of: In order to get the tests not only discovered but also executed in Resharper you’ll have make sure that your test project and the Resharper addin are both using the same, exact version of xUnit. I stumbled over this after I recompiled and installed my patch on my machine. Resharper is not able to execute the tests in case of a version mismatch.

Some of the things I really like about it:

• It has a set of .NET 3.0 Extension methods, which can be used instead of classic assertions (f.e.: “Foo”.ShouldNotBeNull() )
• It runs every every test on a separated instance.
• It uses the constructor of the test class and the Dispose-pattern for fixture initialization and release of resources after a test.
• It’s relatively easy to extend XUnit.

Because of curiosity and my current affection with BDD I decided to port the BDDExtension stuff from  JP Boodhoo to XUnit. So here we go:

Marking specifications with concerns

All specifications for a particular type are marked with the ConcernAttribute.  This is only a marker attribute by which a tool like bddunit is able to correlate specifications and their related type.

[AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = false)]
public class ConcernAttribute : TraitAttribute
{
        private readonly Type _Type;

        public ConcernAttribute(Type type) : base("Concern", type.FullName)
        {
                _Type = type;
        }

        public Type Type
        {
                get { return _Type; }
        }
}

The ConcernAttibute extends the TraitAttribute which is described in the documentation as “an Attribute used to decorate a test method with arbitrary name/value pairs (“traits”).” I changed the AttributeTarget from method to class in order to suit my needs.

Changing the way the test is executed

JP’s original code relied on the SetUp and Teaddown handlers of NUnit / MBUnit which are execute before and after each test method in the same class. Since XUnit doesn’t have that feature any more we’ll have to extend it a bit.

The main extension point for how a test is executed is the FactAttibute used to mark a method as a test. You can extend it and return a different ITestCommand which is then used to execute the test.

public class ObservationAttribute : FactAttribute
{
        protected override IEnumerable EnumerateTestCommands(MethodInfo method)
        {
                var testCommand = base.EnumerateTestCommands(method).First();
                yield return new SpecificationTestCommand(testCommand, method);
        }
}

Do you wonder why the method returns an IEnumerable<ITestCommand>? This is done in order to support something like row tests in MBUnit. The XUnit framework basically enables you to run a test several times. For me this isn’t interesting so I just grabbed the first ITestCommand from it and wrapped it with my own implementation, which looks like this:

public class SpecificationTestCommand : BeforeAfterCommand
{
        public SpecificationTestCommand(
                ITestCommand innerCommand,
                MethodInfo testMethod)
                : base(innerCommand, testMethod)
        {
        }

        public override MethodResult Execute(object testClass)
        {
                var specification = testClass as IContextSpecification;

                if (specification == null)
                {
                       throw new InvalidOperationException(
                          "Instance does not implement IContextSpecification");
                }

                try
                {
                        specification.EstablishContext();
                        specification.Because();
                        return base.Execute(testClass);
                }
                catch(AssertException)
                {
                        throw;
                }
                catch (Exception exception)
                {
                        ExceptionUtility.RethrowWithNoStackTraceLoss(exception.InnerException);
                }

                return null;
        }
}

The code requires the test class to implement the IContextSpecification interface in order to invoke the EstablishContext() and Because() methods for doing AAA based tests. (Do you miss AfterEachSpec? One moment please ) For every interface there is a base class which is in my case the ContextSpecification class.

public abstract class ContextSpecification : IContextSpecification, IDisposable
{
      protected abstract void Because();

      protected abstract void EstablishContext();

      protected virtual void AfterEachSpec()
      {
      }

      #region IContextSpecification Members

      void IContextSpecification.Because()
      {
            Because();
      }

      void IContextSpecification.EstablishContext()
      {
            EstablishContext();
      }

      #endregion

      #region IDisposable Members

      public void Dispose()
      {
            AfterEachSpec();
      }

      #endregion
}

Notice some things. AfterEachSpec() is called from Dispose(), That’s why I didn’t need to include it into the IContextSpecification interface. That interface is implemented explicitly in order to allow the hooks to have different modifiers (protected).

A first test

There’s always an interesting moment when you first run your little experiment. Here we go:

[Concern(typeof(Factory))]
public class When_creating_product_with_a_valid_name : ContextSpecification
{
      private Factory _Factory;
      private Product _CreatedProduct;

      protected override void EstablishContext()
      {
            _Factory = new Factory();
      }

      protected override void Because()
      {
            _CreatedProduct = _Factory.Create("Foo");
      }

      [Observation]
      public void The_product_should_contain_the_correct_name()
      {
            _CreatedProduct.Name.ShouldEqual("Foo");
      }

      [Observation]
      public void The_product_should_contain_the_correct_vendor_name()
      {
            _CreatedProduct.VendorName.ShouldEqual("FooVendor");
      }
}

And the result from TestDriven.NET is:

—— Test started: Assembly: Xunit.BddExtensions.Samples.dll ——
2 passed, 0 failed, 0 skipped, took 0,92 seconds.

Conclusions

It’s in fact very easy to extend XUnit. I like the tool even more after doing this little research. The shown code runs smoothly with TestDriven.NET. However the Resharper – TestRunner seems to have problems with it (TestExplorer stays blank).

I’ll play with that approach in the next few weeks and will blog about my experience with both XUnit and BDD. So if you’re interested in more stay tuned . . .

P.s.: If you’re interested in the code, just drop me a line . . .

Background of that discussion was that my colleague Sergey and I wanted to implement the repository pattern based on Linq 2 NHibernate in a way that exposes NO NHibernate dependency to a surrounding layer. Besides that we didn’t want to dublicate the UnitOfWork pattern that NHibernate implements internally. Because of that we decided to try out ambient transactions (System.Transactions) as our UnitOfWork.

Sergey has already posted about the design we’re currently investigating, so I won’t go into detail about that here. You can read more here:

• RepositoryPatter revised
• UnitOfWork Pattern revised
• Specification Pattern revised

We started with a simple test comparison between the behavior of NHibernates native ITransactions and an NHibernate using TransactionScope for transactions.  What we noticed :

FlushMode.Commit doesn’t work when using ambient Transactions

The following (xUnit) test fails:

[Fact]
public void Session_should_be_clean_after_commited_transaction()
{
    using (var tx = new TransactionScope())
    {
          SaveTwoPatients();
          tx.Complete();
    }

        Session.IsDirty().ShouldBeFalse();
}

But this can be easily fixed like this:

[Fact]
public void Session_should_be_clean_after_commited_transaction_Fixed()
{
    using (var tx = new TransactionScope())
    {
        Transaction.Current.TransactionCompleted += (s, e) =>
        {
            if (e.Transaction.TransactionInformation.Status == TransactionStatus.Committed)
            {
                Session.Flush();
                Session.Clear();
            }
        };

        SaveTwoPatients();
        tx.Complete();
    }

    Session.IsDirty().ShouldBeFalse();
}

For some reason ambient Transaction are FASTER than NHibernates native counter parts

We have also some tests  that perform a transactional insert a 1000 times. The duration of those tests are quite surprising. Have a look:

This has nothing to do with jitting in the CLR or the order how test are executed. I double checked the durations and ran each test on his own.  I didn’t expect that gap, but I’m also not a NHibernate pro. Any thoughts?

I’ll write more about our experiences with that approach soon. Our concept looks good on paper and our first impressions are not disproving either, so stay tuned . . .