sharepoint-deployment-in-enterprise-environment

November 26th, 2010 2 comments

SharePoint Installations: Strategy to aid continuous integration

This document outlines an approach to managing SharePoint installations. Starting from the current deployment approach which is primarily manual, an improved approach is outlined that manages the difference between deployments and migrations and aid the automation of scripting through all environments.

Overview

For continuous integration (CI), we generally want the approach to move a single package for a SharePoint solution through all environments. Secondarily, we want to be scripted and preferably automated. To put the foundations in place we have had to devise a clear strategy. Microsoft does not provide all the tools and techniques for SharePoint continuous integration. In fact, in most cases for new features such as customisation of site page or workflow creation must be performed manually in each environment.

Shortcomings of SharePoint deployments: no real versioning across environments

SharePoint has a well established set of techniques for deployment. Most easily, there are through the GUI approaches that come with Visual Studio or are able to added into Visual Studio. There are other tools that aid deployments through a GUI in a wizard style (on CKS: Development Tools Edition on Codeplex). Alternatively, you can deploy solutions using stsadm and PowerShell cmdlets. Finally, you can create your own custom deployment steps (IDeploymentStep) and deployment configurations (ISharePointProjectExtension) by implementing the core SharePoint library and add these to your solutions and through the deployment build lifecycle. This is all well explained in SharePoint 2010 Development with Visual Studio by Addison Wesley.

All of this allows for custom deployment and retraction of a SharePoint solution with bias toward GUI-based deployment and certainly manual configuration of sites. Manual configuration is particularly problematic in the enterprise environment that wishes to use continuous integration.

There is a missing piece of the puzzle that could aid automation in the moving of solutions through environments. This is the idea of migrations. Migrations are simply a convenient way for you to alter the SharePoint site in a structured and organised manner. So while you could activate features through the GUI by telling operations or administrators you would then be responsible for writing documentation. You’d also need to keep track of changes of which changes need to be run against each environment next time you deploy. Instead, migrations work against the SharePoint API and are versioned. Each SharePoint instance has a record of which version migrations have been run. It applies on new migrations to the site.

Current Approach: without migrations and bias toward GUI

Table 1 outlines the current installation approach is scripted deployments via powershell. Provisioning occurs as part of the SharePoint. However, there is also a lot more configuration required that is currently performed manually through the GUI. This leads to problems in installations across multiple environments. It is also costly in terms of time for training, documentation and problem solving.

Deployment

Provisioning

Configuration

Scripted

GUI

Automated

Manual

Table 1:             Previous State for SharePoint installations

In contrast, Table 2 splits deployment into deployment and migrations and has a bias towards scripting and automation over GUI and manual configuration.


Deployment

Migrations

Provisioning

Configuration

Scripted

GUI

Automated

Manual

Table 2:             New State for SharePoint installations

Approach With Migrations and bias toward scripting

We approach this problem by isolating the installation of sharepoint solutions through provisioning and configuration phases as per Table 3. Provisioning occurs when a new “feature” is deployed. This occurs in the Event Receiver code callbacks.

Configuration is actually an implicit concept in SharePoint. Activation is post-provisioning and can occur on features once provisioned. Configuration on the other hand is the main activity of site administrators in SharePoint – this is the gluing together of the system. For most SharePoint installations, these two activities are usually performed through the GUI by people. As such, this is not conducive to enterprise systems where new code and configurations need to be moved through many environments.


Provisioning

Configuration

Table 3:             Installation process split between provisioning and
configuration

Table 4 outlines the difference between sripted tasks and GUI-based tasks and that there should be a preference toward scripting. Scripted tasks tend to be written in C# using SharePoint API to perform tasks (or as cmdlets). They merely reproduce tasks that would normally be done in the GUI. In most cases, the functionality is the same. However, in a percentage of cases we find that each approach has its own quirks that need to be ironed out. This is very useful for finding issues early on.


Provisioning

Configuration

Scripted

GUI

Table 4:             Tasks should be scripted more than through the GUI

Table 5 outlines the distinction between automated and manual tasks. To aid maintainability through environments tasks must be scripted for continuous integration automation. However, in practice not all tasks should be exclusively automated. Many scripted tasks must also be able to be run within continuous builds and be run manually.


Provisioning

Configuration

Scripted

GUI

Automated

Manual

Table 5:             Automation over Manual – but manual still required

Finally in Table 6, for all of this work we there have split the installation process into two approaches: deployments and migrations. Deployments are scripted in powershell (and msbuild) and have the ability to compile, package and deploy the SharePoint solutions. Deployments trigger provisioning in the SharePoint system by design of SharePoint. What is then missing is the ability to (re)configure the system per deployment. Migrations do this job. They apply a set of changes per installation – they are a little more powerful than this. They can actually apply set of changes in order per installation and keep these versioned. Moreover they can work transactionally and retract changes upon errors (and this can be either automated or done manually). It is based on a technique used for database schema changes – in fact, we have extended an open source library to do the job for us.

Deployments also extend the simplistic notion of only needing a wsp for deployment and include creating zip packages, extracting and deploying solutions via powershell and SharePoint API. The result is not only the standard deployment: wsp uploaded into the solution store; binaries are deployed. But also, as part of the deployment, the powershell scripts may also provision solutions that are usually provisioned through the GUI by the administrator. This will often be a prerequisite for a migration.

Migrations are versioned actions that run after the deployment. Migrations may activate, install/upgrade or configure a feature (or act upon the 14 hive). Many of these tasks have previously been performed through the GUI.

Deployment

Migrations

Provisioning

Configuration

Scripted

GUI

Automated

Manual

Table 6:             Installation is now completed via deployment and migrations

Changes in the code base

Having this strategy means some changes in the code base. Here are some of things we see:

Migration Classes: We now have the configuration code implemented as part of the code base – this includes the ability to make configurations roll forward or backwards

Automation: Configuration can now be performed by build agents

Migration Framework: We have used migratordotnet to perform the migration but did have to extend the library to cater for SharePoint

PowerShell scripts: migrations can be run either automatically as part of installations or run independently by operations – we have also used psake as a build DSL around PowerShell

So far, we haven’t included any code samples. These are to come.

Categories: Uncategorized Tags: ,

SharePoint 2010 books

November 26th, 2010 No comments

I must have trawled through 10 SharePoints 2010 books of late from various publishers Sams, Wrox, Microsoft, Addison Wesley and Apress. I have been asking the question, how do I go about thinking about programming this beast? I want the 50 foot object model and the approach to programming – so testing would be an added bonus. Interestingly, very few do this. In fact, most are made for the person that wants to configure up a system rather than do development. Fair enough, I suppose. Clearly, few people are programming up sites – not. I suppose I can keep going to the SDK or blogs, but again, I haven’t been able to get the right view from these. I have been working with Devs who really know their API. But they too have been struggling to explain to me SharePoint. I have worked with various CMS systems over the years and would have thought that I would have picked it up easier. Oh, well perhaps it is me!

If you are new to doing SharePoint development then I thing that I think that I have learned is that you wanting books with a focus on “Solution” development rather than ones with “user” or even simply “development”. In fact, I haven’t even looked at the ones on Adminstration, Designer or Workflow. Try this search on Amazon

A quick run down.

My pick:

Followed by:

There’s the ones who tell you all about the governance, planning and implementation (rather than development)

Reasonably helpful ones:

Sorry, I haven’t quite got to Real World SharePoint® 2010 by Wrox. I’m hoping its better.

Here’s one that I going to try and get a copy to read:

Categories: Uncategorized Tags:

SharePoint TDD Review

November 26th, 2010 No comments

I’ve been looking around what others have writing about testing SharePoint and before I do a post on how I have gone about it here’s a quick review. It seems there are the usual positions:

  • don’t do it because it doesn’t add value and is difficult given the state of the product, community, developers and product owners
  • don’t do it because it unnecessarily expands the code base
  • do it using mocking frameworks (TypeMock or Pex/Moles)
  • don’t mock because there are too many dependencies and you really need to test in the environment and just testing using units in this way is misleading
  • do it but don’t mock out what are actually integration tests – so appropriately layer your code and tests

So where I got to is that:

  • we need to write SharePoint using layering (SOLID)
  • we need to put an abstraction in front of SharePoint in our code (interfaces or late-bound lambdas – or both)
  • we need to write SharePoint API code in ways that are still accessible to developers who are at home in the SharePoint world – so the code that uses SharePoint is altogether (in practice, this would look like driving the API calls out of WebParts, Event Receivers and into their own classes – think ports and adaptors technique)
  • we need to layer our tests also (unit and integration – see my posts on test automation pyramid)
  • use mocks in unit tests – so no dependencies (which requires abstractions)
  • don’t mock (SharePoint) in integration tests (steel thread/walking skeletons will require this but that around getting code to work that isolates SharePoint)
  • you will be required to use DI (but that doesn’t mean IOC) – this will be easy if you layer your code and follow the rule in hand in dependencies on constructors

So writing good (layered and testable) code is the core issue. Or in the words of Freeman and Price, let’s write maintainable code over code that is easy to write. SharePoint examples are all about the marketing machine showing just how easy it is to use. That, though, isn’t maintainable code from the TDD perspective.

Here’s my most interesting, immediate experience in unit testing SharePoint – we couldn’t use MSTest. I kid you not. It wouldn’t work on 64-bit machines. We found others had had the problem. Go figure.

General case against:

Basic argument that I have a lot of sympathy for (except if I have to be the developer who puts his name against the system and has to do support because this position gives me no protection). This comment shows a really good understanding of the complexities.

So, now lets take a typical SharePoint project. Of course there is a range and gamut of SharePoint projects, but lets pick the average summation of them all. A usual SharePoint project involves many things,

  • Customer communication and prototyping – usually much much more than a typical .NET custom dev project.
  • Plenty of attention to the IT Pro side, where you decide logical and physical layouts of your servers, networks, and your site collections. Custom dev .NET projects are usually simpler to setup and plan.
  • A significant effort in branding – usually greater than your typical .NET project, simply because the master pages and CSS and JS of SharePoint are much more complex.
  • Focus on scalability, when balancing between requirements, and best practices, and experience!
  • Writing some code
  • Establishing roles within your team, which is different from the nature of the SharePoint project. Usually this is more involved in an SP project than a custom .NET dev project, simply because there is a higher overlap between ITPro and Dev in the case of SP.
  • Training required – your business users will need training on the project ~ again, usually more significant than a simple .NET custom dev project.
  • Architecture, and functionality bargaining per RUP for COTS - where there are many ways of achieving a goal, and the way picked is a delicate balance between budget, demands, and technology.

Thus in a typical SharePoint project, the portion where TDD is actually applicable is very small – which is the writing code part. In most, not all, SharePoint projects, we write code as small bandaids across the system, to cover that last mile – we don’t write code to build the entire platform, in fact the emphasis is to write as little code as possible, while meeting the requirements. So already, the applicability of TDD as a total percentage of the project is much smaller. Now, lets look at the code we write for SharePoint. These small bandaids that can be independent of each other, are comprised of some C#/VB.NET code, but a large portion of the code is XML files. These large portion of XML files, especially the most complex parts, define the UI – something TDD is not good at testing anyway. Yes I know attempts have been made, but attempt But in most scenarios, the answer is usually no. So, TDD + SharePoint 2007? Well – screw it! Not worth it IMO.

This is supported in Designing solutions for microsoft® sharepoint® 2010 (p.229)

SharePoint developers have traditionally been slow to adopt modern approaches to soft- ware testing. There are various reasons for this. First, solution development for Microsoft® SharePoint® is often a specialized, fairly solitary process that isn’t automated or streamlined to the same degree as mainstream software development. Second, the ASP. NET programming model that underpins SharePoint development does not naturally lend itself to automated testing. Finally, most of the classes and methods in the SharePoint API do not implement interfaces or override virtual methods, making them difficult to “stub out” using conventional mocking techniques. However, automated, robust testing is an increasingly essential part of enterprise-scale SharePoint development.

Layering:

From Comment 5 at 2/18/2009 6:56 PM

When I’m doing SharePoint development the last thing I think of (or one of the last) is the SharePoint infrastructure. For me, there are two key things that I always do. First, I build an abstraction layer over SP. I treat SP as a service and talk to it on my terms, not theirs. Second, I have a second set of integration tests that really a not that important to the main thrust of the TDD work I do and that’s of testing the SP interactions. There’s two levels of testing here, a) does my code retrieve what I want from SP and give me the results I want and b) does SP do what I think it should. It’s like doing a write-read confirmation of a SQL call vs. making sure SQL is setup correctly and the tables/fields are there.

From Peter at 2/18/2009 9:12 PM

one of the things Bob Martin mentioned was that you can’t test all your code. You have a core set of logic that you test heavily, hopefully getting 100% coverage of this CORE logic, and a volatile set of code, which you shove away in an untested cubbyhole. In my mind, our small SharePoint projects are almost all volatile code, with very little “core logic.”

In Designing solutions for microsoft sharepoint 2010 they recommend:

Testability. Can you test your classes in isolation? If your code is tightly coupled to a user interface, or relies on specific types, this can be challenging.
Flexibility. Can you update or replace dependencies without editing and recompiling your code?
Configuration. How do you manage configuration settings for your solution? Will your approach scale out to an enterprise-scale deployment environment?
Logging and exception handling. How do you log exceptions and trace information in the enterprise environment? Is your approach consistent with that of other developers on the team? Are you providing system administrators with reliable information that they can use to diagnose problems effectively?
Maintainability. How easy is it to maintain your code in a code base that is constantly evolving? Do you have to rewrite your code if a dependent class is updated or replaced?

Different types of tests

Probably the best SharePoint based explanation is in Designing solutions for microsoft sharepoint 2010. Here I will reproduce in full the explanations of unit and integration testing then the final part which backs up my experience that you can’t do integration testing using MSTest (although this is apparently fixed in Visual Studio 2010 SP1 Beta):

Unit Testing
Unit tests are automated procedures that verify whether an isolated piece of code behaves as expected in response to a specific input. Unit tests are usually created by developers and are typically written against public methods and interfaces. Each unit test should focus on testing a single aspect of the code under test; therefore, it should generally not contain any branching logic. In test-driven development scenarios, you create unit tests before you code a particular method. You can run the unit tests repeatedly as you add code to the method. Your task is complete when your code passes all of its unit tests.
A unit test isolates the code under test from all external dependencies, such as external APIs, systems, and services. There are various patterns and tools you can use to ensure that your classes and methods can be isolated in this way, and they are discussed later in this section.
Unit tests should verify that the code under test responds as expected to both normal and exceptional conditions. Unit tests can also provide a way to test responses to error conditions that are hard to generate on demand in real systems, such as hardware failures and out-of-memory exceptions. Because unit tests are isolated from external dependencies, they run very quickly; it is typical for a large suite consisting of hundreds of unit tests to run in a matter of seconds. The speed of execution is critical when you are using an iterative approach to development, because the developer should run the test suite on a regular basis during the development process.
Unit tests make it easier to exercise all code paths in branching logic. They do this by simulating conditions that are difficult to produce on real systems in order to drive all paths through the code. This leads to fewer production bugs, which are often costly to the business in terms of the resulting downtime, instability, and the effort required to create, test, and apply production patches.

Integration Testing
While unit tests verify the functionality of a piece of code in isolation, integration tests verify the functionality of a piece of code against a target system or platform. Just like unit tests, integration tests are automated procedures that run within a testing framework. Although comprehensive unit testing verifies that your code behaves as expected in isolation, you still need to ensure that your code behaves as expected in its target environment, and that the external systems on which your code depends behave as anticipated. That is the role of integration testing.
Unlike a unit test, an integration test executes all code in the call path for each method under test-regardless of whether that code is within the class you are testing or is part of an external API. Because of this, it typically takes longer to set up the test conditions for an integration test. For example, you may need to create users and groups or add lists and list items. Integration tests also take considerably longer to run. However, unlike unit tests, integration tests do not rely on assumptions about the behavior of external systems and services. As a result, integration tests may detect bugs that are missed by unit tests.
Developers often use integration tests to verify that external dependencies, such as Web services, behave as expected, or to test code with a heavy reliance on external dependencies that cannot be factored out. Testers often also develop and use integration tests for more diverse scenarios, such as security testing and stress testing.
In many cases, organizations do not distinguish between integration and unit testing, because both types of tests are typically driven by unit testing frameworks such as nUnit, xUnit, and Microsoft Visual Studio Unit Test. Organizations that employ agile development practices do, however, make this distinction, since the two types of tests have different purposes within the agile process.

Note: In the Visual Studio 2010 release, there is a limitation that prevents you from running integration tests against SharePoint assemblies using Visual Studio Unit Test. Unit tests created for Visual Studio Unit Test must be developed using the Microsoft . NET Framework 4.0 in Visual Studio 2010, whereas SharePoint 2010 assemblies are based on .NET Framework 3.5. In many cases, this is not an issue-.NET Framework 4.0 assemblies are generally compatible with .NET Framework 3.5 assemblies, so you can run a .NET Framework 4.0 test against a .NET Framework 3.5 assembly. However, the way in which SharePoint loads the .NET common language runtime (CLR) prevents the runtime from properly loading and running the tests within Visual Studio Unit Test.
This limitation prevents you from running integration tests with SharePoint within Visual Studio Unit Test. Integration tests execute real SharePoint API logic instead of substituting the logic with a test implementation. Two isolation tools discussed in the following sections, TypeMock and Moles, will continue to work because they intercept calls to the SharePoint API before the actual SharePoint logic is invoked. You can execute integration tests using a third-party framework such as xUnit or nUnit. Coded user interface (UI) tests against SharePoint applications will run without any issues from within Visual Studio 2010.

Against Mocking:

From Eric Shupps at 2/18/2009 7:30 PM

While I agree that mocked objects do have value, I believe that the overall value is much less in an environment like SharePoint. You absolutely cannot duplicate the functionality of the core system, no matter how hard you try – reflection only takes you so far. Most of what Isolator does has no bearing on what actually happens in a live system. Fake objects are fine in tightly controlled environments but they just don’t do the trick in SharePoint. I do, however, encourage you to keep innovating and producing better and better tools for the development community.

For Mocking via IOC:

From Paul at 2/19/2009 12:11 AM

Let me give a real code example. Take this:

private Product[] GetProductsOnSpecialFromList(string url, string listName)
{
       var results = new List<Product>();
       var webProvider = IOC.Resolve<IWebSiteProvider>();
       var list = webProvider.FindWeb(url).FindList(listName);
       foreach (var item in list)
       {
              // Add an item to products list
              // Only include products in the InStock state
              // Only include products which are on special
              // More and more business rules
       }
       return results;
}

What do we have here? Business logic. That’s the kind of stuff I want to do TDD against. But it talks to SharePoint? Well, it talks to an interface. Behind the interface could be SharePoint, or it could be a mock object. I don’t care. It’s called “loose coupling”. Can I test it? Absolutely. And it’s dead easy. I can test all kinds of permutations on my business logic. In fact, I can even tell what happens when a URL is invalid or a site is offline – my mock IWebProvider would thrown an exception, and I’d test that my business logic responded appropriately. Now, eventually, behind that interface, you might want to test SharePoint. For that, you could make a decision – do I write an integration test, or do I not care? Either way, that code should only account for 5% of your code base. You can do TDD for the other 95%. In short, I suspect you find testing hard because you write tightly-coupled, hard to maintain code that mixes business logic/behavior with SharePoint crap. If you wrote loosly coupled code, you’d find testing easy.

References:

Against:

For:

General discussion on TDD around SharePoint

Sample Code:

Categories: Uncategorized Tags: , ,

validating-object-mothers

October 24th, 2010 2 comments

I was in a pairing session and we decided that that this syntax described here is way too noisy:

new Banner { Name="" }.SetupWithDefaultValuesFrom(ValidMO.Banner).IsValid().ShouldBeFalse()

So we rewrote it using anonymous objects which made it nicer to read:

a.Banner.isValid().with(new { Name="john" })

A couple of notes on this style:

  • I am using the style of lowercase for test helpers. As per above a, isValid and with.
  • I am using nested classes to create a DSL feel: eg as@a@._object_ replaces the idea of ValidOM._object_
  • I am wrapping multiple assertions in isValid and isInvalid (they are wrappers that call the domain object IsValid and provide assertions – these assertions can be overwritten for different test frameworks which at this stage is MSTest)

Unit test object builder

This code has two tests: Valid and InvalidHasNoName. The first test checks that I have setup (and maintained) the canonical model correctly and that it behaves well with the validator. The second test demonstrates testing invalid by exception. Here there rule is that the banner is invalid if it has no name.

  using Domain.Model;
  using Microsoft.VisualStudio.TestTools.UnitTesting;

  namespace UnitTests.Domain.Model
  {
      [TestClass]
      public class BannerPropertyValidationTest
      {
          [TestMethod]
          public void Valid()
          {
              a.Banner.isValid();
          }

          [TestMethod]
          public void InvalidHasNoName()
          {
              a.Banner.isInvalid().with(new {Name = ""});
          }
      } 
  }

Here’s the canonical form that gives us the object as a.Banner:

  namespace UnitTests.Domain
  {
      public static class a
      {
          public static Banner Banner
          {
              get  { return new Banner  { Name = "Saver" }; }
          }
      }
  }

At this stage, you should have a reasonable grasp of the object through the tests and what we think is exemplar scenario data. Here’s the model with validations. I won’t show but there is an extension methods

  using Castle.Components.Validator;

  namespace Domain.Model
  {
      public class Banner
      {
        [ValidateNonEmpty]
        public virtual string Name { get; set; }
      }
  }

Builder extensions: with

Now for the extensions that hold this code together. It is a simple piece of code that takes the model and anonymous class and merges them. As a side note I originally thought that I would use LinFu to do this but it can’t duck type against concrete classes, only interfaces. And I don’t have interfaces on a domain model.

	using System;

	namespace Domain
	{
	    public static class BuilderExtensions
	    {
	        public static T with<T>(this T model, object anon) where T : class
	        {
	            foreach (var anonProp in anon.GetType().GetProperties())
	            {
	                var modelProp = model.GetType().GetProperty(anonProp.Name);
	                if (modelProp != null)
	                {
	                    modelProp.SetValue(model, anonProp.GetValue(anon, null), null);
	                }
	            }
	            return model;
	        }
	    }
	}

So if you want to understand how the different scenarios it works on, here’s the tests:

	using Domain;
	using Domain.Model;
	using Microsoft.VisualStudio.TestTools.UnitTesting;

	namespace Tests.Unit
	{
	    [TestClass]
	    public class WithTest
	    {
	        [TestMethod]
	        public void UpdateBannerWithName_ChangesName()
	        {
	            var a = new Banner { Name = "john" };
	            var b = a.with(new { Name = "12345" });
	            Assert.AreEqual(b.Name, "12345");
	        }

	        [TestMethod]
	        public void UpdateBannerWithEmptyName_ChangesName()
	        {
	            var a = new Banner { Name = "john" };
	            var b = a.with(new { Name = "" });
	            Assert.AreEqual(b.Name, "");
	        }

	        [TestMethod]
	        public void UpdateBannerWithEmptyName_ChangesNameAsRef()
	        {
	            var a = new Banner { Name = "john" };
	            a.with(new { Name = "" });
	            Assert.AreEqual(a.Name, "");
	        }

	        [TestMethod]
	        public void UpdateBannerChainedWith_ChangesNameAndDescriptionAsRef()
	        {
	            var a = new Banner { Name = "john" };
	            a.with(new { Name = "" }).with(new { Description = "hi" });
	            Assert.AreEqual(a.Name, "");
	            Assert.AreEqual(a.Description, "hi");
	        }

	        [TestMethod]
	        public void UpdateBannerWithName_ChangesNameOnly()
	        {
	            var a = new Banner { Name = "john", Description = "ab" };
	            var b = a.with(new { Name = "12345" });
	            Assert.AreEqual(b.Name, "12345");
	            Assert.AreEqual(b.Description, "ab");
	        }

	        [TestMethod]
	        public void UpdateBannerWithPropertyDoesntExist_IsIgnored()
	        {
	            var a = new Banner { Name = "john" };
	            var b = a.with(new { John = "12345" });
	            // nothing happens!
	        }
	    }
	}

Builder extensions: validators

Now that we have the builder in place, we want to be able to do assertions on the new values in the object. Here’s what we are looking for a.Banner.isValid().with(new{Name="john"}). The complexity here is that we want to write the isValid or isInvalid before the with. We felt that it read better. This adds a little complexity to the code – but not much.

The general structure below goes something like this:

  1. add an extension method for the isValid on a domain object
  2. that helper returns a ValidationContext in the form of a concrete Validator with a test assertion
  3. we need to create another with on a ValidationContext to run the Validator
  4. Finally, in the with we chain the with with the anonymous class and do the assert
	using Domain;
	using Domain.Model;
	using Microsoft.VisualStudio.TestTools.UnitTesting;

	namespace Tests.Unit
	{
	    public static class ValidationContextExtensions
	    {
	        public static ValidationContext<T> isInvalid<T>(this T model)
	        {
	            return new Validator<T>(model, (a) => Assert.IsFalse(a));
	        }

	        public static ValidationContext<T> isValid<T>(this T model)
	        {
	            return new Validator<T>(model, (a) => Assert.IsTrue(a));
	        }

	        public static T with<T>(this ValidationContext<T> model, object exceptions) where T : class
	        {
	            model.Test.with(exceptions);
	            model.Assert(model.Test.IsValid());
	            return model.Test;
	        }
	    }

	    public interface ValidationContext<T>
	    {
	        T Test { get;  set; }
	        Action<bool> Assertion { get;  set; }
	        void Assert(bool isValid);
	    }

	    public class Validator<T> : ValidationContext<T>
	    {
	        public Validator(T test, Action<bool> assertion) {
	            Test = test;
	            Assertion = assertion;
	        }
	        public T Test { get; set; }
	        public virtual Action<bool> Assertion { get; set; }
	        public void Assert(bool value)
	        {
	            Assertion.Invoke(value);
	        }
	    }
	}

Just a last comment about the interface and implementation. I have gone for naming the interface differently to the interface (ie not IValidator) because I want to avoid using the I@ convention and see where it takes me. In this case, the @with needs to be able to chain itself to something – to me this is the validation context. This context then has a validator in it (eg valid/invalid). In this case the interface isn’t merely a contract it is actually being used concretely itself without an implementation. In fact, we could almost have the interface living without its properties and methods definitions at this stage, or perhaps the interface could have become an abstract class – either solution would entail less code (but slightly weird conventions).

Righto, a bit more showing of tests last:

	using Domain.Model;
	using Microsoft.VisualStudio.TestTools.UnitTesting;

	namespace Tests.Unit
	{

	    [TestClass]
	    public class ValidationContextTest
	    {
	        private const string NameCorrectLength = "johnmore_than_six";
	        private const string NameTooShort = "john";

	        [TestMethod]
	        public void UpdateBannerWithInvalidExtensionOnWith()
	        {
	            var a = new Banner { Name = NameTooShort };
	            a.isInvalid().with(new { Name = "" });
	        }

	        [TestMethod]
	        public void UpdateBannerWithValidExtensionOnWith()
	        {
	            var a = new Banner { Name = NameTooShort };
	            a.isValid().with(new { Name = NameCorrectLength });
	        }

	        [TestMethod]
	        public void NewBannerIsValidWithoutUsingWith()
	        {
	            var a = new Banner { Name = NameTooShort };
	            a.isValid();
	        }

	        [TestMethod]
	        public void NewBannerIsInvalidWithoutUsingWith()
	        {
	            var a = new Banner { Name = NameCorrectLength };
	            a.isInvalid();
	        }
	    }
	}

Just a final bit, if you are wondering between IsValid with caps and isValid. Here’s validator running code on the domain model that we are wrapping in our validators.

	using Castle.Components.Validator;

	namespace Domain.Model
	{
	    public static class ValidationExtension
	    {
	        private static readonly CachedValidationRegistry Registry = new CachedValidationRegistry();

	        public static bool IsValid<T>(this T model) where T : class 
	        {
	            return new ValidatorRunner(Registry).IsValid(model);
	        }
	    }
	}

I hope that helps.

moq-returns-in-order

October 3rd, 2010 No comments

Oh, how I wanted to use this code, but alas as I look at some old code I now need to delete it. Here it is for later use that will never occur. The original site is the remarks below.

using System;
using System.Collections.Generic;
using Moq.Language.Flow;

namespace UnitTests
{
    /// <summary>
    ///  Moq Extensions
    /// </summary>
    public static class MoqExtensions
    {
        /// <summary>
        /// Returns delegates the in order setup.
        /// </summary>
        /// <example>var category = new Mock<ICategory>();
        ///          category.Setup(c => c.Id).ReturnsInOrder(
        ///             () => 1,
        ///             () => 2,
        ///             () => 3,
        ///             () => 4);
        ///
        ///              var expectedIds = new List<int> { 1, 2, 3, 4 };
        ///              foreach (var expectedId in expectedIds) Assert.Equal(expectedId, category.Object.Id);
        /// </example>
        /// <remarks>from http://www.timvw.be/setup-expectation-with-successive-function-calls-using-moq/ </remarks>
        /// <typeparam name="TMock">The type of the mock.</typeparam>
        /// <typeparam name="TResult">The type of the result.</typeparam>
        /// <param name="setup">The setup.</param>
        /// <param name="valueFunctions">The value functions.</param>
        /// <returns></returns>
        public static IReturnsResult<TMock> ReturnsInOrder<TMock, TResult>(this ISetup<TMock, TResult> setup, params Func<TResult>[] valueFunctions) where TMock : class
        {
            var functionQueue = new Queue<Func<TResult>>(valueFunctions);
            return setup.Returns(() => functionQueue.Dequeue()());
        }
    }
}
Categories: Uncategorized Tags:

NoSql: redis and mongodb links

September 29th, 2010 No comments

After an afternoon at Redis and having done the same with MongoDb I’m siding with redis and I suspect it is the sets feel and separating out data from indexing.

MongoDB and Redis: a different interpretation of what’s wrong with Relational DBs

comparing MongoDB and Redis

at a glance

Categories: Uncategorized Tags: , ,

Review: Growing Object-Oriented Software, Guided by Tests

September 29th, 2010 No comments

Review: Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pyrce (2010)

We write code that is easy to maintain over code that is easy to write.

Maintainability is the mantra of the TDDist and I think that this book credibly demonstrates – to the new and experienced – what this looks like in practice. The book is focussed on unit testing in which they make a nice distinction from acceptance and integration testing (p.9). Importantly, unit testing should also be performed within the context of the whole through a test tracer (end-to-end test rig) called a walking skeleton (ch 10). Working between the whole (system) and the specific parts is something that is easy to forget to do and is done at your own peril. By setting up the test harness framework as a walking skeleton and then going into the specifics using mocking strategies I think is an important reminder that interaction testing can be maintainable. Their use of jMock’s allowances vs expectations is an important reminder that we need to use DSLs not only in our domain objects but also in our tests. By doing so, our tests become expressive through a declarative style rather than a procedural one which allows us to understand the intentions of the code and listen to what our tests have to say about our design (p.262-4). I took these reminders back to an old code base and started listening to the tests – I was surprised how I could hear criticisms from non-TDDers about the intrusion of the tests in ways that made think about improvements and yet still remain resolved that test-first makes my code maintainable but not always easy and that I need to make it more expressive and simpler yet.

There is a good introductory fifth of the book dedicated to the fundamentals of why this is important for growing object-oriented software. I particularly liked that they stayed with the Evans’ distinction between objects and values: objects have identity and relationships with each other, whereas values have no identity should be immutable quantities/measurements. The basis though for their later sections are the practical implications of what you can do with objects and values. Different strategies can be used with objects depending on object peer stereotypes: dependencies, notifications and adjustments (p.52). These are the different types of relationships that objects have with each other which help us think through the types of designs we want. Alternatively, values can be refactored via three basic techniques: breaking out, budding off and bundling up (p.59) – basically, ways of reducing complexity through splitting out differences, marking new concepts but deferring nomenclature and grouping similarities. Growing object-oriented software uses examples in Java and more specifically a mocking framework that is not available in C# (or Ruby) where I do most of my work. Yet, I found this perhaps better than if I had examples in my familiar idiom – I will now look at Moq and MSpec to see how I might use jMock’s declarative nomenclature to simplify and make more expressive my tests.

pass behaviours rather than data

This is what I know my tests would tell me if I was listening to them – I know that my style has been pass values around the stack (data style). Instead, I need to pass around more behaviour in the form of callbacks. These are going to give to me precise interfaces with better information hiding (separation of concerns) and clearer abstractions. I am going to go back to my code base and probably get rid of some of the Microsoftisms – interfaces with an “I” prefix and really start to think what is its role and related classes. I’m going to look at constructors to explore how objects communicate with each other and the relationships are between them. In short, the book really reminds me that naming is hard but tests help you get clearer and that that is time well spent to grow you software.

There are only two hard problems in Computer Science: cache invalidation and naming things – Phil Karton

Categories: Uncategorized Tags:

storyq-vs-slim-fitnesse

August 24th, 2010 1 comment

Acceptance testing from user stories via StoryQ and Slim

Recently I was working with a team using user stories for acceptance testing (in .Net). As it turned out I had the opportunity to implement the acceptance tests in both Slim/fitnesse and StoryQ. To me, either framework is fine because they are both the same strategy: they create an abstraction that is understood by the customer that we can run against the system under test. Below is an extract of the user story that was discussed with the customer (and there was also a sketch of the GUI that I haven’t included) and what I’ll show is the StoryQ and Slim implementations up to the point of hooking into the system under test (SUT). The process for each is pretty similar of a two set process: (1) code up the test as the customer representation (2) create (C#) code that wraps hooking into the SUT

    
                  step one           step two
 ------------------------------------------------
  user story  ->  xUnit StoryQ    -> TestUniverse
  user story  ->  story test wiki -> Fixtures

User story

This story was written in plain and developed over three cycles over the course of an hour. There was actually about 7 scenarios in the end.

As a Account Manager
I want to cater for risks terminated in System1 after extraction to System2
So that I offer a renewal to the Broker

    Given I have a policy                                                                                                                                        
      And it has a current risk in System2                                                                                                                           
    When the risk is terminated in System1                                                                                                                        
    Then the risk is deleted in System2                    

    Given I have a policy                                                                                                                                      
    When a deleted risk later than the load date is added                                                                                                    
    Then it should be rerated      
                                                                                                                      
    Given I have a policy                                                                                                                                        
      And it has an current risk in System2                                                                                                                          
    When its status is ready to review                                                                                                                           
      And a risk is deleted in System1                                                                                                                            
    Then the policy should be rated                                                                                                                              
      And the policy should remain in the same status        

Example One: StoryQ

I took this user story and then converted it into StoryQ using the StoryQ converter that you can get with the library. StoryQ has a slight difference: (a) it uses the In order syntax and (b) I had to have a Scenario with each Given/When/Then. It took me in total about half an hour to download, add as a reference and convert the story.

Story to storyq via converter

When doing the conversion, the converter will error until it is in the correct format. Here’s what I had to rewrite to:

Story is Terminated risks after extractions
  In order to offer renewal to the Broker
  As a Account Manager
  I want to cater for risks terminated in System1 after extraction to System2

 With scenario risks become deleted in System2
   Given I have a policy                                                                                                                                        
     And it has a current risk in System2                                                                                                                           
  When the risk is terminated in System1                                                                                                                        
    Then the risk is deleted in System2                    

 With scenario newly deleted risks need to be rerated
   Given I have a policy                                                                                                                                      
   When a deleted risk later than the load date is added                                                                                                    
   Then it should be rerated      

 With scenario status being reviewed
   Given I have a policy                                                                                                                                        
     And it has an current risk in System2                                                                                                                          
   When its status is ready to review                                                                                                                           
     And a risk is deleted in System1                                                                                                                            
   Then the policy should be rated                                                                                                                              
     And the policy should remain in the same status            

The output of the converter in C# is:

new Story("Terminated risks after extractions")
       .InOrderTo("offer renewal to the Broker")
       .AsA("Account Manager")
       .IWant("to cater for risks terminated in System1 after extraction to System2")

       .WithScenario("risks become deleted in System2")
       .Given("I have a policy")
       .And("it has a current risk in System2")
       .When("the risk is terminated in System1")
       .Then("the risk is deleted in System2")

       .WithScenario("newly deleted risks need to be rerated")
       .Given("I have a policy")
       .When("a deleted risk later than the load date is added")
       .Then("it should be rerated")

       .WithScenario("status being reviewed")
       .Given("I have a policy")
       .And("it has an current risk in System2")
       .When("its status is ready to review")
       .And("a risk is deleted in System1")
       .Then("the policy should be rated")
       .And("the policy should remain in the same status")

StoryQ into xUnit

From this output you add it into a xUnit test framework – so far either MSTest or NUnit. We were using MSTest and it looks like this. We need to add

  • Testclass, Testmethod as per MSTest
  • I add indentation myself
  • .ExecuteWithReport(MethodBase.GetCurrentMethod()) at the end
using System.Reflection;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using StoryQ;

namespace Tests.System.Acceptance
{
    [TestClass]
    public class TerminatedRiskAfterExtractionTest
    {
        [TestMethod]
        public void Extraction()
        {
            new Story("Terminated risks after extractions")
                .InOrderTo("offer renewal to the Broker")
                .AsA("Account Manager")
                .IWant("to cater for risks terminated in System1 after extraction to System2")

                .WithScenario("risks become deleted in System2")
                    .Given("I have a policy")
                        .And("it has a current risk in System2")
                    .When("the risk is terminated in System1")
                    .Then("the risk is deleted in System2")

                .WithScenario("newly deleted risks need to be rerated")
                    .Given("I have a policy")
                    .When("a deleted risk later than the load date is added")
                    .Then("it should be rerated")

               .WithScenario("status being reviewed")
                   .Given("I have a policy")
                       .And("it has an current risk in System2")
                   .When("its status is ready to review")
                       .And("a risk is deleted in System1")
                   .Then("the policy should be rated")
                       .And("the policy should remain in the same status")

                .ExecuteWithReport(MethodBase.GetCurrentMethod());
        }
    }
}

Link to the SUT (TestUniverse)

StoryQ needs to link the user story test text to the SUT. One clean approach is to use a TestUniverse – and as you will see it replicates the idea of the fitnesse fixture. A quick summary as you look at the code:

  • add a class TestUniverse, have a reference to it available (_t_) and instatiate it in the Setup()
  • convert string text to methods eg ("I have a policy" to _t.IHaveAPolicy)
  • add methods to the TestUniverse (personally I use resharper and it does the work for me)
  • quite quickly you get reuse patterns eg (_t.IHaveAPolicy)
  • I tend to create methods only as I need them because it gives me a better sense of tests actually implemented
    using System.Reflection;
    using Microsoft.VisualStudio.TestTools.UnitTesting;
    using StoryQ;

    namespace Tests.System.Acceptance
    {
        [TestClass]
        public class TerminatedRiskInSystem1AfterExtractionTest
        {
            private TestUniverse _t;

            [TestInitialize]
            public void Setup()
            {
                _t = new TestUniverse();
            }

            [TestMethod]
            public void Extraction()
            {
                new Story("Terminated risks after extractions")
                    .InOrderTo("offer renewal to the Broker")
                    .AsA("Account Manager")
                    .IWant("to cater for risks terminated in System1 after extraction to System2")

                    .WithScenario("risks become deleted in System2")
                    .Given(_t.IHaveAPolicy)
                        .And("it has a current risk in System2")
                    .When("the risk is terminated in System1")
                    .Then("the risk is deleted in System2")

                    .WithScenario("newly deleted risks need to be rerated")
                    .Given(_t.IHaveAPolicy)
                    .When("a deleted risk later than the load date is added")
                    .Then("the policy should be rerated")

                    .WithScenario("status being reviewed")
                    .Given(_t.IHaveAPolicy)
                        .And("it has an current risk in System2")
                    .When(_t.ItsStatusIsReadyToReview)
                        .And("a risk is deleted in System1")
                    .Then("the policy should be rated")
                        .And("the policy should remain in the same status")

                    .ExecuteWithReport(MethodBase.GetCurrentMethod());
            }

        }

        public class TestUniverse
        {
            // private variables here if needed

            public void IHaveAPolicy()
            {
                // Hook up to SUT
            }

            public void ItsStatusIsReadyToReview()
            {
                // Hook up to SUT
            }

            public void ThePolicyShouldBeReRated()
            {
                // Hook up to SUT
                // Assertions too
            }

        }
    }   

Hopefully, you can see the TestUniverse helps us keep the two parts of the test separated – and of course, you could have them in separate files if you really wanted.

Now when you run this test through your favourite xUnit runner (MSTest, Resharper, TestDriven.Net, Gallio) you will get the results of the test. Or more appropriately, I have the tests ready and waiting. Now we implement the system before return to these after to hook up. Yes, I confess, these system tests are test-last! I explain that somewhere else.

Example Two: Slim

Having got the code in StoryQ to this point, we decided to investigate what the same user story would look like in Slim. This exercise turned out to take somewhat longer. In C#, there are no specific libraries for implementing the Given/When/Then although there is one in java (givenWenZen). So we given our stories we implemented them using a Script table. This turned out to the fruitful.

Slim story test wiki

Here are the wiki tables. In the actual, wiki we also included the original user story around the table which for this example I have included only the scenario.

With scenario risks become deleted in System2

|script|Risk Terminated                  |
|check |I have a policy                 ||
|check |it has a current risk in System2||
|terminate risk                          |
|ensure|risk is deleted                  |

With scenario newly deleted risks need to be rerated

|script|Risk Terminated                                |
|check |I have a policy                               ||
|check |deleted risk later than the load date is added||
|ensure|risk is rerated                                |

With scenario status being reviewed       

|script|Risk Terminated                                        |
|check |I have a policy                        |               |
|check |it has a current risk in System2       |               |
|check |status is in                           |ready to review|
|check |risk is deleted                        |               |
|ensure|policy is rerated                                      |
|check |policy should remain in the same status|ready to review|

My working knowledge of the check, ensure, reject took a while to get up to speed including the mechanism for reporting back values in empty cells (eg |check|I have a policy||). Other than that it took as about an hour to get on top of structure for the table in this case. Script table has worked okay but it did mean that we went from a user story to a story test that in the future we should go straight to. Having said that the user story Given/When/Then is nice for its natural language syntax and helps avoids sticking solely to GUI-type actions.

Slim fixture

The slim fixture is pretty straight forward. We hardcoded in our policy and risk to start with. Later on we would swap this out for the system.

namespace Tests.System.Acceptance
{
    public class RiskTerminated
    {

        public string IHaveAPolicy()
        {
            return "12-4567846-POF";
        }

        public string ItHasACurrentRiskInSystem2()
        {
            return "001";
        }

        public bool terminateRisk()
        {
            return true;
        }

        public bool riskIsDeleted()
        {
            return true;
        }

        public string deletedRiskLaterThanTheLoadDateIsAdded()
        {
            return string.Empty;
        }

        public bool RiskIsRerated()
        {
            return true;
        }

        public bool PolicyIsRerated()
        {
            return true;
        }

        public string statusisin()
        {
            return string.Empty;
        }

        public string policyShouldRemainInTheSameStatus()
        {
            return string.Empty;
        }

        public bool getAListOfCancelledRisksFromSystem1()
        {
            return true;
        }

        public bool theRiskIsNotAvailableInSystem2()
        {
            return true;
        }
    }
}

Now that this is all working we can run the tests in Fitnesse. We already had Fitnesse up and running with Slim build to the correct version. Don’t underestimate the time and expertise needed to get this all up and running – it’s not hard, after the first time that is! We could have also got our wiki running under a Generic Test in Visual Studio but we haven’t gone that far yet.

Some general comments

If you were attentive to the style of tests, you might have noticed that we passed in no values for the tests. In this testing, the setup is limited to fetching existing data and then making changes from there. Because we are interested in knowing which pieces of data were touched the tests do want, in this case, the policies reported back. This is not ideal but pragmatic for the state of the system. Both StoryQ and Slim can report back this type of information – in these examples, Slim has this implemented but not StoryQ.

  • StoryQ has the least barrier to entry and keeps the tests focussed at the developers
  • StoryQ is easy to get onto a build server
  • StoryQ has much better for refactoring because it is in C# in an IDE and is not using (magic) string matching – you could though
  • StoryQ is generally unknown and is harder to introduce particularly to devs that haven’t really grasped Fitnesse
  • Cucumber could just as easily be used (I just thought that I would throw that in from left field)
  • Given/When/Then may be limited as a rubric for testing – but those constraints are also benefits
  • Slim allows realistically allows more comments around tests
  • Wiki reporting is potentially richer and closer to the customer
  • Separate systems of a wiki and code is good separation
  • The wiki and code are nonetheless two different systems – that comes at a cost
  • If the wiki isn’t driving development then I’m not convinced it is cost effective
  • Slim required more knowledge around which table to use and its constraints – that’s (sort of) a one-time cost
  • I wouldn’t mind a Given/When/Then table for Slim – apparently UncleBob might be writing one?

Rock on having customers

A final note in this example. I most interested in the benefits of this type of testing rather than the specific implementation. Anything that allows us to have more conversations with our code base and the people who develop it the better. By writing the user stories down before development means we need to do the “outside in” thing first. By coming back to them, I have found that it flushes assumptions made in development and makes us remember our earlier discussions. It easy to forget that our customers are often good at helping us create abstractions in our code base that we loose sight of in the middle coding the solution. Those abstractions live in the user stories – to hook up the user stories to the SUT we often need to decouple our code in ways that is not required when our only client code of the SUT is the application/UI. So I say rock on having customers that require us to have proxies in the form of code to represent them!

Categories: Uncategorized Tags:

test-automation-pyramid-and-webservices

August 24th, 2010 No comments

Using web services is easy. That is if we listen to vendors. Point to a wsdl and hey presto easy data integration. Rarely is it that easy for professional programmers. We have to be able to get through to endpoints via proxies, use credentials and construct the payloads correctly and all this across different environments. Add the dominance of point-and-click WSDL integration, many developers I talk to do really work at the code level for these types of integration points or if they do it is to the extent of passive code generators. So to suggest TDD on web services is, at best, perplexing. Here I am trying to explain how the test automation pyramid helps how to TDD web services.

Test-first web services?

Can we do test-first development on web services? or is test-first webservices an oxymoron? (ok, assume each is only one word ;-) ) My simple answer is no. But the more complex answer is that you have to accept some conditions to make it worthwhile. These conditions to me are important concepts that lead to do test-first rather than the other way around.

One condition is that my own domain concepts remain clean. To keep it clean this means I keep the domain from the web service from being within my domain. For example, if you have a look at the samples demonstrating how to use a web service the service proxy and the domain is right there in the application – so for web application, you’ll see references in the code behind. This worries me. When the web service changes and the code is auto generated this new code is likely then to be littered throughout that code. Another related condition is that integration points should come through my infrastructure layer because it aids testability. So, if I can get at the service reference directly I am in most cases going to have an ad hoc error handling, logging and domain mapping strategy.

Another condition is that there is an impedance mismatch between our domain the service domain. We should deal with this mismatch as early as possible and as regularly as possible. We also should deal with these issues test-first and in isolation from the rest of the code. In practice, this is a mapper concern and we have vast array of options (eg automapper library, LINQ). These options are likely to depend on the complexity of the mapping. For example, if we use WSE3 bindings then we will be mapping from a XML structure into an object. Here we’ll most likely do the heavy lifting with a XML parser such as System.Linq.Xml. Alternatively, if we are using the ServiceModel bindings then we will be mapping object to object. If these models follow similar conventions we might get away with automapper and if not we are likely to roll our own. I would suggest if you are rolling your own that the interface of automapper might be nice to follow though (eg Mapper.Map<T, T1>(data)).

I think there is a simple rule of thumb with mapping domains: you can either deal with the complexity now or later. But, regardless, you’ll have to deal with the intricacies at some point. Test-first demands that you deal with them one-at-a-time and now. Alternatively, you delay and you’ll have to deal with at integration time and this often means that someone else is finding them – and in its worst case, it is in production! I am always surprised that how many “rules” there are when mapping domains and also how much we can actually do before we try to integrate. I was just working with a developer who has done a lot of this type of integration work but never test-first. As we proceeded test-first, she starting reflecting on how much of this mapping work that she usually did under test conditions could be moved forward into the development. On finishing that piece of work we also we surprised how many tests were required to do the mapping – a rough count was 150 individual tests across 20 test classes. This was for mapping two similar domains each with 5 domain models.

What code do you really need to write?

So let’s say that you accept that you don’t want to have a direct reference to the client proxy, what else it is needed? Of course, the answer is it depends. It depends on:

  • client proxy generated (eg WSE3 vs ServiceModel): when using the client proxy the WSE3 will require a little more inline work around, say, Proxy, SetClientCredential methods whereas the ServiceModel can have it inline or be delegated to the configuration file
  • configuration model (eg xml (app.config) vs fluent configuration (programatic)): you may want to deal with configuration through a fluent configuration regardless of an app.config. This is useful for configuration checking and logging within environments. Personally, the more self checking you now for configuration settings the easier code will be to deploy through the environments. Leaving configuration checking and setting to solely to operations people is good example of throwing code over the fence. Configuration becomes someone else’s problem.
  • reference data passed with each request: Most systems require some form reference data that exists with each and every request. I prefer to avoid handling that at the per request level but rather when service instantiating. This information is less likely to the credential information.
  • security headers: you may need to add security headers to your payload. I forget which WS* standard this relates to but it is a strategy just like proxies and credentials that needs to be catered for. WSE3 and ServiceModel each have their own mechanisms to do this.
  • complexity of domain mappings: you will need to call the mapping concern to do this work but should only be a one-liner because you have designed and tested this somewhere else. It is worth noting the extent of difference though. With simple pass through calls some mappings are almost a simple value. Take for example, a calculation service upon return may be a simple value. However, with domain synching the domain mapping are somewhat a complex set of rules to get the service to accept your data. Actually, I often
  • error handling strategy: we are likely to want to catch exceptions and throw our kind so that we capture further up in the application (eg UI layer). With the use of lambdas this is straightforward and clean to try/catch method calls to the service client.
  • logging strategy particularly for debugging: you are going to need to debug payloads at some stage. Personally, I hate stepping through and that doesn’t help outside of development environments. So, a good set of logging is needed to. I’m still surprised how often code doesn’t have this.

Test automation pyramid

Now that we know what code we need to write, what types of tests are we likely to need? If you are unfamiliar with the test automation pyramid or my specific usage see test automation pyramid review for more details.

System Tests

  • combine methods for workflow
  • timing acceptance tests

Integration Tests

  • different scenarios on each method
  • exception handling (eg bad credentials)

Unit Tests

  • Each method with mock (also see mocking webservices)
  • exception handling on each method
  • mapper tests

More classes!

Without going into implementation details, all of this means that there is a boiler plate of likely classes. Here’s what I might expect to see in one of my solutions. A note on conventions: a slash ‘/’ denotes folder rather than file; <Service>\<Model>\<EachMethod> is specific to your project and indicates that there is likely to be one or more of that type; names with a .xml are xml and all others if not folders are .cs files.

  Core/
    ServiceReference/
      I<Service>Client
      <Service>Exception
    Domain/
      <Model>
      ServiceReference/
        Credential<Model>
  
  Infrastructure/
    Service References/                  <-- auto generated from Visual Studio ServiceModel
    Web Reference/                       <-- auto generated from Visual Studio WSE3
    ServiceReference/
      <Service>Client
      Mapper/
        <Model>Mapper
  
  Tests.System/
    Acceptance/
      ServiceReference/
        <Service>/
          WorkflowTest
          TimingTest
    Manual/
      ServiceReference/
         <Service>-soap-ui.xml
  
  Tests.Integration/
    ServiceReference/
      <Service>/
        <Method>Test
  
  Tests.Unit/
    ServiceReference/
      <Service>/
        ConstructorTests
        <Method>Test
        ExceptionsTest
        Mappers/
          <Model>Test
        Security/
          CredentialTest                  <-- needed if header credential
    Fixtures/
      ServiceReference/
        <Service>/
          Mock/Fake<Service>Client        
          Credentials.xml
          Request.xml                     <-- needed if WSE3
          Response.xml                    <-- needed if WSE3
      Domain/
        <Model>ObjectMother
        ServiceReference/
          Credential<Model>ObjectMother   <-- needed if header credential

That’s a whole lot more code!

Yup, it is. But each concern and test is now separated out and you can work through them independently and systematically. Here’s my point, you can deal with these issues now and have a good test bed so that when changes come through you have change tolerant code and know you’ve been as thorough as you can be with what you presently know. Or, you can deal with it later at integration time when you can ill afford to be the bottle neck in the high visible part of the process.

Potential coding order

Now, that we might have a boilerplate of options, I tend to want to suggest an order. With the test automation pyramid, I suggest to design as a sketch and domain model/services first. Then system test stub writing, then come back through unit and integration tests before completing\implementing the system test stubs. Here’s my rough ordered list:

  1. have a data mapping document – Excel, Word or some form table is excellent often provided by BAs – you still have to have some analysis of differences between your domain and their’s
  2. generate your Service Reference or Web Reference client proxy code – I want to see what the models and endpoints – I may play with them via soapUI – but usually leave that for later if at all needed
  3. write my system acceptance tests stubs – here I need to understand how these services fit into the application and what the end workflow is going to be. For example, I might write these as user story given/when/then scenarios. I do not try and get these implement other than compiling because I will come back to them at the end. I just need a touch point of the big picture.
  4. start writing unit tests for my Service Client – effectively, I am doing test-first creation of my I<Service>Client making sure that I can use each method with an injected Mock/Fake<Service>Client.
  5. unit test out my mappers – by now, I will be thinking about the request/response cycle and will need to creating ObjectMothers to be translated into the service reference domain model to be posted. I might be working in the other direction too – which is usually a straightforward mapping but gets clearer once you start integration tests.
  6. integration test on each method – once I have a good set of mappers, I’ll often head out to the integration point and check out how broken my assumptions about the data mapping are. Usually, as assumptions break down I head back into the unit test to improve the mappers so that the integration tests work – this is where the most work occurs!
  7. at this point, I now need good DEBUG logging and I’ll just ensure that I am not using the step-through debugger but rather good log files at DEBUG level.
  8. write system timing tests because sometimes there is an issue that the customer needs to be aware of
  9. system tests that can be implemented for the unit/integration tests for the method implements thus far
  10. add exception handling unit tests and code
  11. add credential headers (if needed)
  12. back to system tests and finish off and implement the original user stories
  13. finally, sometimes, we need to create a set of record-and-replay tests for other people testing. SoapUI is good for this and we can easily save them in source for later use.

Some problems

Apart from having presented any specific code, here are some problems I see:

  • duplication: your I<Service>Client and the proxy generated once is probably very similar with the difference it that your one returns your domain model objects. I can’t see how to get around this given your I<Service>Client is an anti-corruption class.
  • namespacing/folders: I have suggested ServiceReference/<Service>/ as folder structure. This is a multi-service structure so you could ditch the <Service> folder if you only had one.
  • Fixtures.ServiceReference.<Service>.Mock/Fake<Service>Client: this implementation is up to you. If you are using ServiceModel then you have an interface to implement against. If you are using WSE3 you don’t have an interface – try extending through @partial@s or wrapping with another class.

Test Automation Pyramid – review

August 4th, 2010 No comments

Test Automation Pyramid

This review has turned out to be a little longer than I expected. I have been using my own layering of the test automation pyramid and the goal was to come back and check it against the current models. I think that my usage is still appropriate but because it is quite specific it wouldn’t work for all teams. So I might pick and choose between the others when needed. If you write line of business web applications which tend to have integration aspects then I think this might be useful. The key difference from the other models is my middle layer in the pyramid is perhaps a little more specific as I try and recapture the idea of integration tests within the context of xUnit tests. Read on if you are interested – there are some bullet points at the end if you are familiar with the different models. Just a disclaimer that reference point is .Net libraries and line-of-business web applications because of my work life at the moment. I am suggesting something slightly different than say this approach but doesn’t seem that different to this recent one and you can also find it implicitly in Continuous Delivery by Farley and Humble.

A little history

The test automation pyramid has been credited to Mike Cohn of Mountain Goat Software. It is a rubric concerning three layers that is foremostly concerned with how we think about going about testing – what types of tests we run and how many of each. While various authors, have modified the labels, it has changed very little and I suspect that is because its simplicity captures people’s imagination. In many ways, it makes a elementary point: automating your manual testing approach is not good enough particularly if that testing was primarily through the GUI. Mike explains that this was the starting point for the pyramid, he says,

The first time I drew it was for a team I wanted to educate about UI testing. They were trying to do all testing through the UI and I wanted to show them how they could avoid that. Perhaps since that was my first use it, that’s the use I’ve stuck with most commonly.

Instead, change the nature of the tests – there is a lot of common sense to this. I’ll take for instance a current web application I am working on. I have a test team that spend a lot of time testing the application through the browser UI. It is a complex insurance application that connects to the company’s main system. Testers are constantly looking at policies and risks and checking data and calculations. In using the UI, however, they have tacit knowledge of what they are looking at and why – they will have knowledge about the policy from the original system and the external calculation engine. It is this tacit knowledge and the knowledge about the context – in this case, a specific record (policy) that meets a criteria – that is difficult to automate in its raw form. Yet, each time that do a manual test in this way the company immediately looses its investment in this knowledge but allowing it to be in the head of only one person. Automating this knowledge is difficult however. All this knowledge is wrapped into specific examples found manually. The problem when you automate this knowledge is that it is example-first testing and when you are testing against data like this today’s passing test may fail tomorrow. Or even worse the test may fail the day after when you can’t remember a lot about the example! Therefore, the test automation pyramid is mechanism to get away from a reliance on brittle end-to-end testing and the desire to simply automate your manual testing. It moves towards layering your testing – making multiple types of tests, maintaining boundaries, testing relationships across those boundaries and being attentive to the data which flows.

Here’s the three main test automation pyramids out there in the wild. Take a look at these and then I will compare them below in a table and put them alongside my nomenclature.

Cohn

Cohn

Mezaros

Mezaros

Crispin

Crispin

Test Automation Pyramid: a comparison

	
   Cohn       Mezaros        Crispin            Suggested
   
                            (manual)          (exploratory)
     UI**      System          GUI               System
   Service    Component    Acceptance (api)    Integration
    Unit        Unit        Unit/Component         Unit  

** in a recent blog post and subsequent comments Mike agrees with others that the UI layer may be better called the “End-to-End” tests. Mike points out that when he started teaching this 6-7 years ago the term UI was the best way to explain to people the problem. And that problem being that automating manual tests meaning automating GUI tests and that the result is (see Continuous Testing: Building Quality into Your Projects):

Brittle. A small change in the user interface can break many tests. When this is repeated many times over the course of a project, teams simply give up and stop correcting tests every time the user interface changes.
Expensive to write. A quick capture-and-playback approach to recording user interface tests can work, but tests recorded this way are usually the most brittle. Writing a good user interface test that will remain useful and valid takes time.
Time consuming. Tests run through the user interface often take a long time to run. I’ve seen numerous teams with impressive suites of automated user interface tests that take so long to run they cannot be run every night, much less multiple times per day.

Related models to test automation pyramid

Freeman & Price: while not a pyramid explicitly, they build a hierarchy of tests that correspond to some nested feedback loops:

Acceptance: Dos the whole system work?
Integration: Does our code work against the code we can’t change?
Unit: Do our objects do the right thing, are they convenient to work work?

Stephens & Rosenberg: Design Driven Testing: Test Smarter, Not Harder have four level approach in which the tests across the layers increase in granularity and size as you move down to unit tests – and business requirement tests are seen as manual. Although their process of development is closely aligned to the V model of development (see pp.6-8)

four principal test artifacts: unit tests, controller tests, scenario tests, and business requirement tests. As you can see, unit tests are fundamentally rooted in the design/solution/implementation space. They’re written and “owned” by coders. Above these, controller tests are sandwiched between the analysis and design spaces, and help to provide a bridge between the two. Scenario tests belong in the analysis space, and are manual test specs containing step-by-step instructions for the testers to follow, that expand out all sunny-day/rainy-day permutations of a use case. Once you’re comfortable with the process and the organization is more amenable to the idea, we also highly recommend basing “end-to-end” integration tests on the scenario test specs. Finally, business requirement tests are almost always manual test specs; they facilitate the “human sanity check” before a new version of the product is signed off for release into the wild.

The pyramid gets it shape because:

  • Large numbers of very small unit tests – set a foundation on simple tests
  • Smaller number of functional tests for major components
  • Even fewer tests for the entire application & workflow

With a cloud above because:

  • there are always some tests that need not or should not be automated
  • you could just say that system testing requires manual and automated testing

General rules:

  • need multiple types of tests
  • tests should be complimentary
  • while there looks like overlap, different layers test at a different abstraction
  • it is harder to introduce a layered strategy

So, as Mezaros points out, multiple types of tests aware of boundaries are important:

Mezaros - Why We Need Multiple Kinds of Tests

Compare and Contrast

Nobody disagrees that the foundation are unit tests. Unit tests tell us exactly which class/method is broken. xUnit testing is dominant here as an approach and this is a well documented approach regardless of classical TDD, mocking TDD or fluent-language style BDD (eg should syntax helpers, or given/when/then or given/it/should). Regardless of the specifics, each test tests one thing, is written by the developer, is likely to have a low or no setup/teardown overhead, should run and fail fast and is the closest and most fine grained view of the code under test.

There are some slight differences that may be more than nomenclature. Crispin includes component testing in the base layer whereas Mezaros puts it as the next layer (which he defines the next layer as functional tests of major components). I’m not that sure that in practice they would look different. I suspect Crispin’s component testing would require mocking strategies to inject dependencies and isolate boundaries. Therefore, the unit/component of Crispin suggests to me that the unit test can be on something larger than a “unit” as long as it still meets the boundary condition of being isolated. One question I would have here for Crispin is in the unit tests would you include for instance tests with a database connection? Does this count as a component or an api?

The second layer starts to see some nuances but tends toward telling us which components are at fault. These are tested by explicitly targeting the programatic boundaries of components. Cohn called his services while Crispin, API. Combine that with Mezaros’ component and all are suggesting that there is size of form within the system – a component, a service – is bigger than the unit but smaller than the entire system. For Mezaros, component tests would also test complex business logic directly. For Cohn, the service layer is a the api (or logical layer) in between the user interface and very detailed code. He says, “service-level testing is about testing the services of an application separately from its user interface. So instead of running a dozen or so multiplication test cases through the calculator’s user interface, we instead perform those tests at the service level”. So what I think we see in the second layer is expedient testing in comparison to the UI. Although what I find hard to see is whether or not this level expects dependencies must be running within an environment. For example, Mezaros and Crispin both suggest that we might be running FIT which often require a working environment in place (because of the way they are written). In practice, I find this a source of confusion that is worth considering. (This is what I return to in a clarification of integration testing.)

On to the top layer. This has the least amount of tests and works across the entire application and focuses often on workflow. Cohn and Crispin focus on the UI to try and keep people about keeping these tests small. Mezaros makes a more interesting move to subsume that UI testing into system testing. Inside system tests he also uses acceptance testing (eg FIT) and manual tests. Crispin also account for manual testing in the cloud above the pyramid. Either way manual testing is still an important part of a test automation strategy. For example the use of record-and-replay tools, such as, selenium or WATIR/N have made browser-based testing UI testing easier. You can use these tools to help script browser testing and then you have the choice whether to automate or not. However, UI testing still suffers from entropy and hence is the hardest to maintain over time – particularly if they are dependent data.

Here are some of the issues I find coming up out of these models.

  • the test automation pyramid requires a known solution architecture with good boundary definition that many teams haven’t made explicit (particularly through sketching/diagrams)
  • existing notions of the “unit” test are not as subtle as an “xUnit” unit test
  • people are too willing to accept that some boundaries aren’t stub-able – often pushing unit tests into the service/acceptance layer
  • developers are all too willing to see (system) FIT tests as the testers (someone else’s) responsibility
  • it is very hard to get story-test driven development going and the expense of FIT quickly outweighs benefits
  • we can use xUnit type tests for system layer tests (eg xUnit-based StoryQ or Cucumber)
  • you can combine different test runners for the different layers (eg xUnit, Fitnesse and Selenium and getting a good report)
  • developers new to TDD and unit testing tend to use test-last strategies that are of the style best suited for system layer tests – and get confused why they see little benefit

Modified definition of the layers

Having worked with these models, I want to propose a slight variation. This variation is in line with Freeman and Pryce.

  • Unit – a test that has no dependencies (do our objects do the right thing, are they convenient to work with?)
  • Integration – a test that has only one dependency and tests one interaction (usually, does our code work against code that we can’t change?)
  • System – a test that has one-or-more dependencies and many interactions (does the whole system work?)
  • Exploratory – test that is not part of the regression suite, nor should have traceability requirements

Unit tests

There should be little disagreement about what good xUnit tests: it is well documented although rarely practised in line-of-business applications. As Michael Feathers says,

A test is not a unit test if:
* it talks to the database
* it communicates across the network
* it touches the file system
* it can’t run at the same time as any of your other unit tests
* you have to do special things to your environment (such as editing config files) to run it
Tests that do these things aren’t bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to keep them separate from true unit tests so that we can run the unit tests quickly whenever we make changes.

Yet often unit tests are difficult because it works against the grain of the application framework. I have documented my approach elsewhere for an MVC application that I not only unit test the usual suspects of domain models, services and various helpers such as configurators or data mappers but also my touch points with the framework such as routes, controllers, action redirects (see MvcContrib which I contributed to with the help of a colleague). I would expect unit test of this kind to be test-first because it about design with a touch of verification. If you can’t test-first this type of code base it might be worthwhile spending a little understanding what part of the system you are unable understand (sorry mock out!). Interestingly, I have found that I can test-first BDD-style parts of my GUI when in the browser and using jQuery and JsSpec (or QUnit) (here’s an example) – you have to treat javascript as first-class citizen at this point so here’s a helper for generating the scaffolding for the jquery plugin.

So I have a check list unit tests I expect to find in a line-of-business unit test project:

  • model validations
  • repository validation checking
  • routes
  • action redirects
  • mappers
  • any helpers
  • services (in the DDD sense)
  • service references (web services for non-Visual Studio people)
  • jquery

Integration tests

Integration tests may have a bad name if you agree with JB Rainsberger that integration tests are a scam because you would see integration tests as exhaustive testing. I agree with him on that count so I have attempted to reclaim integration testing and use the term quite specifically in the test automation pyramid. I use it to mean to test only one dependency and one interaction at a time. For example, I find this approach helpful with testing repositories (in the DDD sense). I do integration test repositories because I am interested in my ability to manage an object lifecycle through the database as the integration point. I therefore need tests that prove CRUD-type functions through an identity – Eric Evans (DDD, p.123) has a diagram of an object’s lifecycle that is very useful to show the links between an object lifecycle and the repository.

Title

Using this diagram, we are interested in the identity upon saving, retrieval, archiving and deletion because these touch the database. For integration tests, we are likely to less interested in creation and updates because they wouldn’t tend to need the dependency of the database. These then should be pushed down to the unit tests – such as validation checking on update.

On the web services front, I am surprised how often I don’t have integration tests for them because the main job of my code is (a) test mappings between the web service object model and my domain model which is a unit concern or (b) the web services use of data which I would tend to make a system concern.

My checklist in the integration tests is somewhat less than unit tests:

  • repositories
  • service references (very lightly done here just trying to think about how identity is managed)
  • document creation (eg PDF)
  • emails

System tests

System testing is the end-to-end tests and it covers any number of dependencies to satisfy the test across scenarios. We then are often asking various questions in the system testing, is it still working (smoke)? do we have problems when we combine things and they interact (acceptance)? I might even have a set of tests are scripts for manual tests.

Freeman and Pryce (p.10) write:

There’s been a lot of discussion in the TDD world over the terminology for what we’re calling acceptance tests: “functional tests”, “system tests.” Worse, our definitions are often not the same as those used by professional software testers. The important thing is to be clear about hour intentions. We use “acceptance tests” to help us, with the domain experts, understand and agree on what we are going to build next. We also use them to make sure that we haven’t broken any existing features as we continue developing. Our preferred implementation of the “role” of acceptance testing is to write end-to-end tests which, as we just noted, should be as end-to-end as possible; our bias often leads us to use these interchangeably although, in some cases, acceptance tests might not be end-to-end.

I find smoke tests invaluable. To keep them cost effective, I also keep them slim and often throw some away once I have found that the system has stabilised and the cost of maintaining is greater than the benefit. Because I tend to write web applications my smoke tests are through the UI and I use record-and-replay tools such as selenium. My goal with these tests is to target parts of the application that are known to be problematic so that I get as earlier a warning as possible. These types of system tests must be run often – and often here means starting in the development environment and then moving through the build and test environments. But running in all these environment comes with a significant cost as each tends to need its own configuration of tools. Let me explain because the options all start getting complicated but may serve as a lesson for tool choice. In one project, we used selenium. So we using selenium IDE in Firefox for record and replay. Developers use this for their development/testing (and devs have visibility of these tests in the smoke test folder). These same scripts were deployed through onto the website so that testing in testing environments could be done through the browser (uses selenium core and located in the web\tests folder – same tests but available for different runners). On the build server, the tests through seleniumHQ (because we ran Hudson build server) – although in earlier phases we had actually used selenium-RC with NUnit but I found that in practice hard to maintain [and we could have used Fitnesse alternatively and selenese!]. As an aside, we found that we also used the smoke tests as our demo back to client so we had them deploy on the test web server so that run through the browser (using the selenium core). To avoid maintenance costs, the trick was to write them specific enough to test something real and general enough not to break over time. Or if they do break it doesn’t take too long to diagnose and remedy. As an aside, selenium tests when broken into smaller units use a lot of block-copy inheritance (ie copy and paste). I have found that this is just fine as long as you use find-and-replace strategies for updating data. For example, we returned to this set of tests two years later and they broke because of date specific data that had expired. After 5 minutes I had worked out that the tests with a find-and-replace for a date would fix the problem. I was surprised to tell the truth!

Then there are the acceptance tests. These are the hardest to sustain. I see them as having two main functions: they have a validation function because they link customer abstractions of workflow with code (validation). The other is that they ensure good design of the code base. A couple of widely used approaches to acceptance type tests are FIT/Fitnesse/Slim/FitLibraryWeb (or StoryTeller) and BDD-style user stories (eg StoryQ, Cucumber). Common to both is to create an additional layer of abstraction for the customer view of the system and another layer than wires this up to the system under test. Both are well documented on the web and in books so I won’t go labour the details here (just a wee note that I you can see a .net bias simply because of my work environment rather than personal preference). I wrote about acceptance tests and the need for a fluent inteface level in a earlier entry:

I find that to run maintainable acceptance tests you need to create yet another abstraction. Rarely can you just hook up the SUT api and it works. You need setup/teardown data and various helper methods. To do this, I explicitly create “profiles” in code for the setup of data and exercising of the system. For example, when I wrote a Banner delivery tool for a client (think OpenX or GoogleAds) I needed to create a “Configurator” and an “Actionator” profile. The Configurator was able to create a number banner ads into the system (eg html banner on this site, a text banner on that site) and the Actionator then invoked 10,000 users on this page on that site. In both cases, I wrote C# code to do the job (think an internal DSL as a fluent interface) rather than say in fitnesse.

I have written a series of blogs on building configurators through fluent interfaces here.

System tests may or may not have control over setup and teardown. If you do, all the better. If you don’t you’ll have to work a lot harder in the setup phase. I am currently working on a project which a system test works across three systems. The first is the main (legacy – no tests) system, the second a synching/transforming system and the third which is the target system. (Interestingly, the third only exists because no one will tackle extending the first and the second only because the third needs data from the first – it’s not that simple but you get the gist.) The system tests in this case become one less of testing that the code is correct. Rather it is more one of does the code interact with the data from the first system in ways that we would expect/accept? As a friend pointed out this becomes an issue akin to process control. Because we can’t setup data in the xUnit fashion, we need to query the first system directly and cross reference against third system. In practice, the acceptance test helps us refine our understanding of the data coming across – we find the implicit business rules and in fact data inconsistencies, or straight out errors.

Finally manual tests. When using Selenium my smoke tests are my manual tests. But in the case of web services or content gateways, there are times that I want to just be able to make one-off (but repeatable) tests that require tweaking each time. I may then be using these to inspect results too. These make little sense to invest in automation but they do make sense to have checked into source leaving it there for future reference. Some SoapUI tests would fit this category.

A final point I want to make about the nature of system tests. They are both written first and last. Like story-test driven development, I want acceptance (story tests) tests written into the code base to begin with and then set to pending (rather than failing). I then want them to be hooked up to the code as the final phase and set to passing (eg Fitnesse fixtures). By doing them last, we already have our unit and integration tests in place. In all likelihood by the time we get to the acceptance tests we are now dealing with issues of design validation in two senses. One, that we don’t understand the essential complexity of the problem space – there are things in there that we didn’t understand or know about that are now coming into light and this suggests that we will be rewriting parts of the code. Two, that the design of code isn’t quite as SOLID as we thought. For example, I was just working on an acceptance test where I had do a search for some data – the search mechanism I found had constraints living the SQL code that I couldn’t pass in and I had to have other data in the database.

For me, tests are the ability for us as coders to have conversations with our code. By the time we have had the three conversations of unit, integration and acceptance, I find that three is generally enough to have a sense that our design is good enough. The acceptance is important because helps good layering and enforce that crucial logic doesn’t live in the UI layer but rather down in the application layer (in the DDD sense). The acceptance tests, like the UI layer, are both clients of the application layer. I often find that the interface created in the application layer not surprisingly services the UI layer that will require refactoring for a second client. Very rarely is this unhelpful and often it finds that next depth of bugs.

My system tests checklist:

  • acceptance
  • smoke
  • manual

Exploratory

The top level system and exploratory testing is all about putting it together so that people across different roles can play with the application and use complex heuristics to check its coding. But I don’t think the top level is really about the user interface per se. It only looks that way because the GUI is most generalised abstraction that we believe that customers and testers believe that they understand the workings of the software. Working software and the GUI should not be conflated. Complexity at the top-most level is that of many dependencies interacting with each other – context is everything. Complexity here is not linear. We need automated system testing to follow critical paths that create combinations or interactions that we can prove do not have negative side effects. We also need exploratory testing which is deep, calculative yet ad hoc that attempts to create negative side effects that we can then automate. Neither strategy aspires for illusive, exhaustive testing – as JB Rainsberger argues – which is the scam of integration testing.

General ideas

Under this layering, I find a (dotnet) solution structure of three projects (.csproj) is easier to maintain and importantly the separation helps with the build server. Unit tests are built easily, run in place and fail fast (seconds not minutes). The integration tests require more work because you need the scripting to migrate up the database and seed data. These obviously take longer. I find the length of these tests and their stability are a good health indicator. If they don’t exist or are mixed up with unit tests – I worry a lot. For example, I had a project that these were mixed up and the tests took 45 minutes to run so tests were rarely run as a whole (so I found out). As it turned out many of the tests were unstable. After splitting them, we got unit tests to under 30 seconds and then started the long process of reducing integration test time down which we got down to a couple of minutes on our development machines. Doing this required splitting out another group of tests that were mixed up into the integration tests. These were in fact system tests that tested the whole of the system – that is, there are workflow tests that require all of the system to be in place – databases, data, web services, etc. These are the hardest to setup and maintain in a build environment. So we tend to do the least possible. For example, we might have selenium tests to smoke test the application through the GUI but it is usually the happy path as fast as possible that exercises that all are ducks are lined up.

Still some problems

Calling the middle layer integration testing is still problematic and can cause confusion. I introduced integration because it is about the longer running tests which required the dependency of an integration point. When explaining to people, they quickly picked up on the idea. Of course, they also bring their own understanding of integration testing which is closer to the point that Rainsberger is making.

Some general points

  • a unit test should be test first
  • an integration test with a database tests is usually the test of the repository to manage identity through the lifecycle of a domain object
  • a system test must cleanly deal with duplication (use other libraries)
  • a system test assertion is likely to be test-last
  • unit, integration and system tests should each have their own project
  • test project referencing should cascade down from system to integration through to the standalone unit
  • the number and complexity of tests across the projects should reflect the shape of the pyramid
  • the complexity of the test should be inversely proportional to the shape of the pyramid
  • unit and integration test object mothers are subtly different because of identity
  • a system test is best for testing legacy data because it requires all components in place
  • some classes/concerns might have tests split across test projects, e a repository class should have both unit and integration tests
  • framework wiring code should be pushed down to unit tests, such as, in mvc routes, actions and redirects,
  • GUI can sometime be unit tested, eg jQuery makes unit testing of browser widgets realistic
  • success of pyramid is an increase in exploratory testing (not an increase in manual regressions)
  • the development team skill level and commitment is the greatest barrier closely followed by continuous integration

Some smells

  • time taken to run tests is not inversely proportional to the shape of the pyramid
  • folder structure in the unit tests that in no way represents the layering of the application
  • only one test project in a solution
  • acceptance tests written without an abstraction layer/library

References

  1. The Forgotten Layer of the Test Automation Pyramid
  2. From backlog to concept
  3. Agile testing overview
  4. Standard implementation of test automation pyramid
  5. Freeman and Price, 2010, Growing Object-Oriented Software, Guided by Tests
  6. Stephens & Rosenberg, 2010, Design Driven Testing: Test Smarter, Not Harder