Skip to content

Fluent Configurator – thoughts on getting a design

In a previous post series I wrote about Configuration DSLs – I’m now just looking at this from another angle of going back to some reading about the microsoft mechanism itself.

A quick note about the previous approach. The configuration is centralised into one Configuration object and then the application has this available. This is similar to Microsoft approach because we have ConfigurationManager available everywhere. The problem for me is that with ConfigurationManager the central configuration point is the app.config and that is how developers expect the application to work – working against such conventions is often creates more friction than is necessary.

Therefore, if you can’t use a (replace the) central configuration object as above, you need to have multiple xml settings. But how? Microsoft has a design time changer – the app designer. AFAIK this works when you deploy from your design environment. That is far from ideal and means that you create a build per environment. I often see this often or something that approximates this approach. But it does not lend itself to a build server. Here’s a well documented approach that can work in both design and deploy time with some adjustments as this one is design time only and we would need to implement the same code in the deploy.proj for msbuild

So the next approach which I have used is to update through xml replacements. This is easiest done replacing entire xml files which are includes into the main xml file. This can be done easily at deploy time. The problem is that all environment settings are shipped. In the situation where the package is for the one client – this isn’t actually a problem. The question is the switching mechanism to swap over xml files. There is the other issue of reconciling differences between environment settings.

This leads to a solution of having all settings in the same file. This approach is outlined in Creating Custom Configuration Section which I mentioned below. This approach could work well for the settings in your application. For example, using a proxy, I hook this up myself with my WebRequest. However, this approach wouldn’t work for log4net settings. So, we still need a better approach for specifying settings in different environments.

Where I want to head is to write a wrapper that manages writing out the xml from C# configuration. This would allow a management strategy that works with the current approach, but could also work in the first mode too of providing its own configuration references. Because settings are in C# they are hidden unless needed. Using an approach like migratordotnet we interrogate the dll for the settings and make updates at deploy time. This interrogation can take the form of dry-run, diffs, updates, backups.

Configuration is a complex and has many views. All those view combined often leads to mistakes so we need it to be as simple as possible. Very simple approaches have often resulted in cut-and-paste management – I think this is simplistic and error prone. As a developer, I understand that for my application I need a way to extend my reach into the production environment and yet give operations enough control for customisation. Because I certainly don’t want to do overnight support, nor do I want to know the credentials for production systems.

There is a final issue. How do we know what environment we are in? (and what are these called? – local, build, test, prod) Given windows conventions and mixed in with a little *nix I would go with this:

  • System environment variable – in C# accessed through a registry key and batch (eg set APP_ENV=Prod)
  • Process environment variable – eg passed in via commandline and used in say msbuild
  • machine.config
  • app/web.config – I would use this other than it has been update from one of the above (if you are using this it is likely that it was manually set)

Note: I think that for enterprise type solutions (only one production deployment) we do know about every environment that we are deploying to. If not, these ideas are not for you because you have multiple deployments (customers). To me, this is consumer-type software and requires that installers can manage themselves.

References

[1] – General about ConfigurationManager

If you want to learn about configuration then here is the key series by Jon Rista on The Code Project

[2] – Creating custom configuration sections for environment

  • example to provide environment specific settings (dev through to prod)
  • implementation is for runtime switching
  • use a custom implementation of AppSettings
  • uses the machine.config for global settings
  • an approach that requires a specific implementation and xsd updates. Good example that the current implementation is limited and customisations are complex.
  • design time checking through xsd (rather than intellisense)

[3] – using compile/deploy time switching

[4] – design time deltas with deploy time switching

[5] – poking files

*
  • “XmlMassUpdate MSBuildCommunity Tasks”:
  • “XMLpoke in nant”
  • “XMLConfig”
2010 03 14 todd Uncategorized Comments (0) Permalink

Fluent Controller MvcContrib – Part II – Coding the Controller Actions and Redirects

In the last entry, we created designed our controller through a test:

[Test]
public void SuccessfulIndex()
{
    GivenController.As<UserController>()
        .ShouldRenderItself(RestfulAction.Index)
        .WhenCalling(x => x.Index());
}

Now, we need to code this:

A Fluent Controlller in action

In fact, to make test run above, you need to do nothing else than inherit from the AbstractRestfulFluentController. 

using MvcContrib.FluentController;

public class UserController : AbstractRestfulFluentController
{
    public ActionResult Index()
    {
        return View();
    }
}

Let’s now look at how to create a skinny controller. Let’s take a redirection design:

GivenController.As<UserController>()
    .ShouldRedirectTo(RestfulAction.Index)
    .IfCallSucceeds()
    .WhenCalling(x => x.Create(null));

Here’s the controller code we need:

public ActionResult Create(object model)
{
    return CheckValidCall()
        .Valid(x => RedirectToAction(RestfulAction.Index))
        .Invalid(() => View("New", model));
}

A more complex example of Create is where it calls a repository. Here if the Repository throws an exception it will execute .Invalid otherwise it will execute .Valid.

public ActionResult Create(object model)
{
    return CheckValidCall(() => UserRepository.Create(model))
        .Valid(x => View(RestfulAction.New, x))
        .Invalid(() => RedirectToAction(RestfulAction.Index));
}

So far you have seen two fluent controller actions available, however, also available are:

  • InvalidNoNewErrors
  • Other

And within the context of CheckValidCall you also get access:

  • controller
  • model
  • isValid
  • newErrors

Another feature to be aware of is that you can create own extensions. The current CheckValidCall returns a FluentControllerAction<T> so all you have to do is create an extension on this as you would with an extension method. You can even rewrite the CheckValidCall and create your own one.

2010 03 07 todd Uncategorized Comments (0) Permalink

Fluent Controller MvcContrib – Part I – Designing Controller Actions and Redirects

In an earlier post/03/test-automation-pyramid-asp-net-mvc/ I said that I unit test my controllers. What I didn’t say was that I (with most of the work done by my colleague Mark and field tested by Gurpreet) had write code to make this possible. To unit test our controllers we put a layer of code that was an abstraction for “redirections”. This has turned out to be very successful for the line of business applications we write. Mark coined this a fluent controller. Out fluent controller has these benefits:

  • test-first design of controller redirections
  • also isolate that the controller makes the correct repository/service calls
  • lower the barrier to entry by developer new to MVC
  • avoids a fat controller antipattern
  • standardises flow within actions

I also tend toward a REST design in the application so we also wanted it to live on top of the SimlyRestful contrib. You’ll find there’s an abstract class both with and without simplyRestful. I will make a quick, unsubstantiated comment. The fluent controller does not actually help you create a controller for a REST application – it is Restful but really not a full implementation of REST.

Designing Controller Actions

In a simplyRestful design, the flow is standardised per resource. We have out 7 actions that we decide what is going to be implemented.

Fluent Controller Action Redirects/Renders

How to write it test first?

In words, I have a User Controller that displays a user and I can do the usual CRUD functions. These examples are taken from the MvcContrib source unit tests

This test, I enter in on Index and it should just render itself and I don’t need to pass anything in:

using MvcContrib.TestHelper.FluentController;
	
[Test]
public void SuccessfulIndex()
{
    GivenController.As&lt;UserController>()
        .ShouldRenderItself(RestfulAction.Index)
        .WhenCalling(x => x.Index());
}

Got the hang of that one? Here’s some ones that redirect based on whether or not the repository/service call was successful or not. This example, imagine, you have just said Create me the user:

[Test]
public void SuccessfulCreateRedirectsToIndex_UsingRestfulAction()
{

    GivenController.As&lt;UserController>()
        .ShouldRedirectTo(RestfulAction.Index)
        .IfCallSucceeds()
        .WhenCalling(x => x.Create(null));
}

[Test]
public void UnsuccessfulCreateDisplaysNew_UsingString()
{
    GivenController.As&lt;UserController>()
        .ShouldRenderView("New")
        .IfCallFails()
        .WhenCalling(x => x.Create(null));
}

Here’s a test that ensures that the correct status code is returned. In this case, I will have done a GET /user and I would expect a 200 (OK) result. This is very useful if you want to play nicely with the browser – MVC doesn’t also return the status code you expect.

[Test]
public void ShowReturns200()
{
    GivenController.As&lt;UserController>()
        .ShouldReturnHead(HttpStatusCode.OK)
        .WhenCalling(x => x.Show());
}

The .ShouldReturnHead was a helper method, it actual delegated by to .Should which you too can use to write your own custom test helpers:

[Test]
public void GenericShould()
{
    GivenController.As&lt;UserController>()
        .Should(x => x.AssertResultIs&lt;HeadResult>().StatusCode.ShouldBe(HttpStatusCode.OK))
        .WhenCalling(x => x.Show());
}

Now we can start combining some tests. In this case, we want to create a new customer and if it does, render the New view and ensure that ViewData.Model has something in it (and we could check that it is customer).

[Test]
public void ModelIsPassedIntoIfSuccess()
{
    var customer = new Customer { FirstName = "Bob" };

    GivenController.As&lt;UserController>()
        .ShouldRenderView(RestfulAction.New)
        .Should(x => x.AssertResultIs&lt;ViewResult>().ViewData.Model.ShouldNotBeNull())
        .IfCallSucceeds(customer)
        .WhenCalling(x => x.Create(customer));

}

Sometimes, we only want return actions based on header location, so we can set this up first .WithLocation.

[Test]
public void HeaderSetForLocation()
{
    GivenController.As&lt;UserController>()
        .WithLocation("http://localhost")
        .ShouldReturnEmpty()
        .WhenCalling(x => x.NullAction());
}

There is also access to the Request through Rhino Mocks, try it like this.

[Test]
public void GenericHeaderSet()
{
    GivenController.As&lt;UserController>()
        .WithRequest(x => x.Stub(location => location.Url).Return(new Uri("http://localhost")))
        .ShouldReturnEmpty()
        .WhenCalling(x => x.CheckHeaderLocation());
}
2010 03 07 todd Uncategorized Comments (0) Permalink

test-automation-pyramid-asp-net-mvc

Test Automation Pyramid in ASP.NET MVC

This is a reposting of my comments from Mike Cohn’s Test Automation Pyramid

I often use Mike’s Test Automation Pyramid to explain to clients’ testers and developers how to structure a test strategy. It has proved the most effective rubric (say compared with the Brian Marick’s Quadrant’s model:http://www.exampler.com/old-blog/2003/08/21/ – as further evolved from Crispin and Gregory:http://www.amazon.com/Agile-Testing-Practical-Guide-Testers/dp/0321534468) to get people thinking about what going on in testing the actual application and its stress points. I want to add that JB Rainsberger’s talk:http://www.infoq.com/news/2009/04/jbrains-integration-test-scam mentioned above is crucial to understanding why that top level set of tests can’t prove integrity of the product by itself.

It has got me thinking that perhaps that we need to rethink some assumptions behind these labels. The difference of opinion in this blogs also suggests this. So I thought I would spend some time talking about how I use the pyramid and then come back to rethinking its underlying assumptions.

I have renamed some parts of the pyramid so that at a first glance it is easily recognisable by clients. This particularly renaming is in the context of writing MVC web applications. I get teams to what their pyramid looks like for their project – or what they might want it to be because it is often upside down.

My layers:

– System (smoke, acceptance)
– Integration
– Unit

I also add a cloud on top (I think from Crispin and Gregory) for exploratory testing. This is important for two reasons: (1) I want automated testing so that I can allow more time for manual testing and to emphasise that (2) there should be no manual regression tests. This supports Rainsberger’s argument not to use the top-level testing as proof of the systems integrity – to me the proof is in the use of the system. Put alternatively, automated tests are neither automating your tester’s testing nor are they a silver bullet. So if I don’t have a cloud people forget that manual testing is part of the automated test strategy (plus with a cloud when the pyramid is inverted it makes a good picture of ice cream in a cone and you can have the image of a person licking the ice cream and it falling off ;-) .)

In the context of an MVC application, this type pyramid has lead me to some interesting findings at the code base level. Like everyone is saying, we want to drive testing down towards the Unit tests because they are foundational, discrete and cheapest. To do this, it means that I need to create units that can be tested without boundary crossing. For an asp.net MVC (just like Rails), this means that I can unit test (with the aid of isolation frameworks):

– models and validations (particularly using MotherObjects)
– routes coming in
– controller rendering of actions/views
– controller redirection to actions/views
– validation handling (from errors from models/repositories)
– all my jQuery plugin code for UI-based rendering
– any HTML generation from HtmlHelpers (although I find this of little value and brittle)
– any of course all my business “services”

I am always surprised at how many dependencies I can break throughout my application to make unit tests – in all of these cases I don’t not need my application to be running in a webserver (IIS or Cassini). They are quick to write, quick to fail. They also require additional code to be written or libraries to be provided (eg MvcContrib Test Helpers).

For integration tests, I now find that the only piece of the application that I still requires a dependency is the connection to the database. Put more technically, I need to check that my repository pattern correctly manages my object’s lifecycle and its identity; it is also ensuring that I correctly code the impedance mismatch between the object layer of my domain and relational layer of the database. In practice, this is ensuring a whole load of housekeeping rather than business logic: eg my migrations scripts are in place (eg schema changes, stored procs); my mapping code (eg ORM) and that the code links all this up correctly. Interestingly, I now find that this layer in terms of lines of code is less than the pyramid suggests because there is a lot of code in a repository service that can be unit tested – it is really only the code that checks identity that requires a real database. The integration tests left tend then to map linearly to the CRUD functions. I follow the rule, one test per dependency. If my integration tests get more complicated it is often time to go looking for domain smells – in the domain driven design sense I haven’t got that bounded context right for the current state/size of the application.

For the top layer, like others I see it as the end-to-end tests and it covers any number of dependencies to satisfy the test across scenarios.

I have also found that there are actually different types of tests inside this layer. Because it is web application, there is the smoke test – some critical path routes that show that all the ducks are lined up – selenium, watir/n and even Steve Sanderson’s MVCIntegationTest are all fine. I might use these tests to target parts of the application that are known to be problematic so that I get as earlier a warning as possible.

Then there are the acceptance tests. This is where I find the most value not only because it links customer abstractions of workflow with code but also as importantly because it makes me attend to code design. I find that to run maintainable acceptance tests you need to create yet another abstraction. Rarely can you just hook up the SUT api and it works. You need setup/teardown data and various helper methods. To do this, I explicitly create “profiles” in code for the setup of data and exercising of the system. For example, when I wrote a Banner delivery tool for a client (think OpenX or GoogleAds) I needed to create a “Configurator” and an “Actionator” profile. The Configurator was able to create a number banner ads into the system (eg html banner on this site, a text banner on that site) and the Actionator then invoked 10,000 users on this page on that site. In both cases, I wrote C# code to do the job (think an internal DSL as a fluent interface) rather than say in fitnesse.

Why are these distinctions important? A few reasons. The first is that the acceptance tests in this form are a test of the design of the code rather than the function. I always have to rewrite parts of my code so that the acceptance tests can hook in. It has only ever improved my design such as separation of concerns and it often has given my greater insight into my domain model and its bounded contexts. For me, these acceptance tests are yet another conversation with my code – but by the time I have had unit, integration and acceptance test conversations about the problem the consensus decision isn’t a bad point to be at.

Second is that I can easily leverage my DSL for performance testing. This is going help me in the non-functional testing (or the fourth quarter of the Test Quadrants model).

Third is that this is precisely the setup you need for a client demo. So at any point, I can crank up the demo data for the demo or exploratory testing. I think it is at this point that we have a closed loop: desired function specified, code to run, and data to run against.

Hopefully, that all makes some sense. Now back to thinking about the underlying assumptions of what is going on at each layer. I think we are still not clear on what we really testing at each layer in the pyramid: most tend to be around the physical layers, the logical layers or the roles within the team. For example, some are mapping it to the MVC particularly because the V maps closely to the UI. Others are staying in a traditional unit, functional and integration partly because the separation of roles within a team.

I want to suggest that complexity is a better underlying organisation. Happy to leave the nomenclature alone: the bottom is where there are no dependencies (unit), the second has one dependency (integration) and top have as many as you need to make it work (system). It seems to me that the bottom two layers require you to have a very clear understanding of your physical and logical architecture expressed in terms of boxes and directed lines ensure that you test each line for every boundary.

If you look back to my unit tests it identified logical parts of the application and tested at boundaries. Here’s one you might not expect. The UI is often seen as a low value place to test. Yet, frameworks like jQuery suggest otherwise and breakdown our layering: I can unit test a lot of the browser code which is traditionally seens as UI layer. I can widgetize any significant interactions or isolate any specific logic and unit test this outside the context of the application running (StoryQ has done this).

The integration tests tested across a logical and often physical boundary. It has really only one dependency. Because there is one dependency the nature of complexity here is still linear. One dependency equals no interaction with other contexts.

The top level is all about putting it together so that people across different roles can play with the application and use complex heuristics to check its coding. But I don’t think the top level is really about the user interface per se. It only looks that way because the GUI is most generalised abstraction that we believe that customers and testers believe that they understand the workings of the software. Working software and the GUI should not be conflated. Complexity at the top-most level is that of many dependencies interacting with each other – context is everything. Complexity here is not linear. We need automated system testing to follow critical paths that create combinations or interactions that we can prove do not have negative side effects. We also need exploratory testing which is deep, calculative yet ad hoc that attempts to create negative side effects that we can then automate. Neither strategy aspires for illusive, exhaustive testing – or as JB Rainsberger argues – which is the scam of integration testing.

There’s a drawback when you interpret the pyramid along these lines. Test automation requires a high level of understanding of your solution architecture, its boundaries and interfaces, the impedance mismatches in the movement between them, and a variety of toolsets required to solve each of these problems. And I find requires a team with a code focus. Many teams and managers I work with find the hump of learning and its associated costs too high. I like the pyramid because I can slowly introduce more subtle understandings of the pyramid as the team gets more experience.

2010 03 06 todd Uncategorized Comments (1) Permalink

validation-specification-testing

Validation specification testing for c# domain entities

I have just been revisiting fluent configuration in the context of writing a fluent tester for validation of my domain entities. In that post, I wrote my test assertions as:

   ValidMO.ImageBanner.ShouldNotBeNull();
   ValidMO.ImageBanner.IsValid().ShouldBeTrue();
   new ImageBanner().SetupWithDefaultValuesFrom(ValidMO.ImageBanner).IsValid().ShouldBeTrue();
   new ImageBanner { Name = "" }.SetupWithDefaultValuesFrom(ValidMO.ImageBanner).IsValid().ShouldBeFalse();

This is okay and I have had luck teaching it. But, it feels long-winded for what I really want to do:

  • check that my fake is valid
  • check validations for each property (valid and invalid)

Today, I was looking at Fluent Nhibernate and noticed their persistence specification testing. This looks much more expressive:

	[Test]
	public void CanCorrectlyMapEmployee()
	{
	    new PersistenceSpecification<Employee>(session)
	        .CheckProperty(c => c.Id, 1)
	        .CheckProperty(c => c.FirstName, "John")
	        .CheckProperty(c => c.LastName, "Doe")
	        .VerifyTheMappings();
	}

How about this then for a base test:

	Test.ValidationSpecification<ImageBanner>(ValidMO.ImageBanner)
		.CheckPropertyInvalid(c => c.Name, "")
		.Verify();

This test makes some assumptions:

  • It will call IsValid() method on the entity
  • It will need to make assertion with your current test framework (fine, we do that in storyq)

You can see that I prefer the static constructor rather than using new.

There are obviously, a range of syntax changes I could make here that would mimic the validation attributes. For example:

	Test.ValidationSpecification<ImageBanner>(ValidMO.ImageBanner)
		.CheckPropertyNotEmpty(c => c.Name)
		.CheckPropertyMandatory(c => c.Name)
		.Verify();

Because the .CheckProperty would be extension method, you could easily add then as you go for your validations. Let’s start with that because that’s all I need for now – we will want to be able to change the callable IsValid method. Fluent nhibernate also passes in a IEqualityComparer that makes me wonder if a mechanism like this could be useful – it certainly looks cool!

2010 03 03 todd Uncategorized Comments (0) Permalink

MsBuild execution of SoapUI testrunner

I have just been writing an msbuild runner to wrap testrunner for soapUI. Here it is. There are a couple of techniques to note:

  • stacking the args for the commandline in an ItemGroup (see thanks below)
  • don’t forget to Html Escape quots when invoking command line ie &amp;quot
  • I also dependency check for each
  • And the usual that the default target is Help and it tells about the targets and dependencies
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="HelpTest"  ToolsVersion="3.5" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <PropertyGroup>
    <SoapUiProject Condition="'$(TestProject)'==''">agent-soapui-project.xml</SoapUiProject>
    <SoapUITestResultsFolder Condition="'$(SoapUITestResultsFolder)'==''">$(MSBuildProjectDirectory)</SoapUITestResultsFolder>
  </PropertyGroup>

  <ItemGroup>
    <Args Include="-a"/>
    <Args Include="-e$(Endpoint)" Condition="'$(Endpoint)'!=''"/>
    <Args Include="-s$(TestSuite)" Condition="'$(TestSuite)'!=''"/>
    <Args Include="-c$(TestCase)" Condition="'$(TestCase)'!=''"/>
    <Args Include="-f$(SoapUITestResultsFolder)" Condition="'$(SoapUITestResultsFolder)'!=''"/>
  </ItemGroup>

  <PropertyGroup>
    <SoapUiPath>$(ProgramFiles)\eviware\soapUI-3.0.1\bin</SoapUiPath>
    <JAVA_HOME Condition="'$(JAVA_HOME)'==''">$(SoapUiPath)\..\jre</JAVA_HOME>
    <TestProject>$(MSBuildProjectDirectory)\src\AcceptanceTest\$(SoapUiProject)</TestProject>
    <SoapUiRunner>&quot;$(SoapUiPath)\testrunner.bat&quot; $(TestProject) @(Args,' ')</SoapUiRunner>
  </PropertyGroup>

  <Target Name="TestAll" DependsOnTargets="AcceptanceTests"/>

  <Target Name="CheckDependencies">
    <Error Text="SoapUI testrunner is not installed at: $(SoapUiPath)" Condition="!(Exists('$(SoapUiPath)'))"/>
     <Error Text="JAVA_HOME environment variable not set correctly to point to an JRE" Condition="!Exists('$(JAVA_HOME)')"/>
  </Target>
  
  <Target Name="AcceptanceTests" DependsOnTargets="CheckDependencies">
    <Message Text="Building Test Command: $(SoapUiRunner)"/>
    <Exec Command="$(SoapUiRunner)"/>
  </Target>

  <Target Name="HelpTest">
    <Message Text="
    
    msbuild /t:TestAll
    
    Variables that can be overridden:
      SoapUiProject=soapui-project.xml
      SoapUITestResultsFolder=$(MSBuildProjectDirectory)
      Endpoint=http://12.0.0.7:8090/SERVICE
      TestSuite=WorkSuite
      TestCase=UpdateCase

    Targets:
     - TestAll
     - AcceptanceTests
	
    Dependencies: 
    - soapUI must be installed to $(SoapUiPath)
    - JAVA_HOME - currently set to: $(JAVA_HOME)
             
             " />
  </Target>

</Project>

Reference:

The technique for stacking args in the ItemGroup is fro:
  • http://weblogs.asp.net/lorenh/archive/2008/12/11/msbuild-trick-for-making-lt-exec-gt-calls-more-maintainable.aspx
Tagged , ,

2010 01 20 todd Deployment Comments (2) Permalink

One deployment for all environments

Deploy the package

This set of instructions should be reasonably straightforward. Always have the same deployment technique throughout all environments. In this case, I am explaining a strategy that which uses MSBuild to trigger installations and upgrades. This simple tenet that can be hard to follow.

Often we also need manual intervention, or simply need to break up the process. To make a process manual we should wrap a commandline with a runner process (eg a bat or cmd file). This means that manual processes are still scripted rather than GUI-based. Moving from isolated scripts to automation is simply the process of chaining scripts together.

The basic checkout-and-compile for packaging and then download-and-deploy for installation sequence should be used across the local, build, test and production environments.

A basic deploy/install process

From commmandline:

msbuild /t:Install

This command will invoke the following lifecycle (this is found in build.tasks):

<Target Name="Install" DependsOnTargets="Package;Extract;Deploy"/>

We have already seen that packaging creates a zip file with <Target Name="Package" DependsOnTargets="Version;Clean;Publish;Zip"/>. The install then takes that package unzips it and then deploys. The idea here is that all it does it replicate what the user would do manually. The deploying is often quite straightforward but timeconsuming. In the sample, the main deployment is to an IIS. All of the third-party libraries have tasks for integration points and it worthwhile looking at them all. Personally, I am using extension pack at the moment.

So I am just said, the script works as though it is a person. Here’s where some complexity lies in the scripts. There are two part of the deployment scripts.

  1. the Deploy target is the worker for doing a deployment (that lives in the build.tasks)
  2. the deployment scripts that are bundles in the package and do the work (eg deploy.proj)

Deploy target as worker

Let’s have a look at the targets for the worker task. Before you look at it, let’s just remember that if I did this manually I would simply get copy of the package, unzip and then double click on the bat file that invokes msbuild. This is the download-and-deploy philosophy.

Here are two targets: Extract and Deploy. A quick explanation that I hope really isn’t needed. Extract first creates a directory to put the files in it doesn’t exist and then does the extraction via an extensionpack Zip task. Deploy goes and runs the deploy.bat file that invokes msbuild.

<Target Name="Extract">
   <MakeDir Directories="$(ExtractPath)" Condition = "!Exists('$(ExtractPath)')"/>
   <MSBuild.ExtensionPack.Compression.Zip TaskAction="Extract" ExtractPath="$(ExtractPath)" ZipFileName="$(ReleaseZipFile)"/>
 </Target>

 <Target Name="Deploy">
   <Exec Command="$(ExtractPath)\deploy.bat $(ExtractPath)" ContinueOnError="false" />
 </Target>

Deployment that actually installs

Now we’ll look at the deployment process once a package has been extracted more fully. Again, it doesn’t get any more complex in practice; there is just more of it. Let’s pickup from the worker process. It has just “clicked” on deploy.bat which effectively is the same as doing an msbuild command (for now let’s stay simple). It has now invoked deploy.proj which does the work. It now goes through the motions to install the site with the DeployWebsite target. If you’ve installed on IIS this process is familiar. If not, it’s damn fine documentation! You can see that you need to create Application Pools, Http Bindings and add Applications. 

  <Target Name="DeployWebsite" DependsOnTargets="CopyWebSite">
    <Message Text="Deploying site version: $(Version) for environment $(Environment)"/>
    <CallTarget Targets="CreateAppPool"/>
    <CallTarget Targets="CreateWebsite"/>
    <CallTarget Targets="CreateBindingHttp"/>
    <CallTarget Targets="CreateBindingHttps" Condition="'$(Environment)'!='Test'"/>
    <CallTarget Targets="AddApplication"/>
    <CallTarget Targets="ModifyWebsite" Condition="'$(Environment)'!='Test'"/>
    <CallTarget Targets="SetupDirectoryBrowsing" Condition="'$(Environment)'=='Test'"/>
    <CallTarget Targets="StartWebsite"/>
  </Target>

While we have this deployment in mind, I’ll add a couple of notes on this deployment. It has two modes of deployment: test and non-test. The test deployment doesn’t run https and it does have a multi-site structure which we use directory browsing for moving between sites. This is a bigger topic of hot-swapping sites that I might get back to.

A quick summary is that it is each to deploy a package in your own environment by scripting tasks you might normally do with a point-and-click, or drag-until-you-drop, approach. This is an important start. Now you need do the same process on the build server (or build agent).

Build Server deploy: allowing it to push out a deployment

The build server should be no different. But it often is. It would ideally not be running as a interactive user and this can cause problems. In windows, there aren’t many good solutions. We don’t have ssh generally available that would make life easy. Running everything under Network Service or Local System is a poor solution that we often resort to.

Having read this entry you might asking yourself, how do we deal with the database? I’ll deal with that next because it cuts across the important issue of first-time installs and redeployments/upgrades.

Tagged
2010 01 18 todd Deployment Comments (0) Permalink

Understanding the packaging

Build the package

So what we’ll do is actually build a package by using the command msbuild /t:PackageBeta

Below is the output that I have shorted so that you can see major targets that create the package as a zip file in the location: CodeToDeploy\Releases\Sample-Beta-0.1.0.0.zip

  • Version
  • Clean
  • Publish
  • Zip
C:\src\deployment>msbuild /t:PackageBeta
Microsoft (R) Build Engine Version 3.5.30729.1
[Microsoft .NET Framework, Version 2.0.50727.3053]
Copyright (C) Microsoft Corporation 2007. All rights reserved.

Build started 1/15/2010 4:53:21 PM.
Project "C:\src\deployment\build.proj" on node 0 (PackageBeta target(s)).
Project "C:\src\deployment\build.proj" (1) is building "C:\src\deployment\build
.proj" (1:2) on node 0 (Package target(s)).
  Version: 0.1.0.0
  Updating File "C:\src\deployment\src\Infrastructure\Properties\AssemblyInfo.c
  s".
    Replaced matches with "[assembly: AssemblyVersion("0.1.0.0")]".
  Updating File "C:\src\deployment\src\Test\Properties\AssemblyInfo.cs".
    Replaced matches with "[assembly: AssemblyVersion("0.1.0.0")]".
  Updating File "C:\src\deployment\src\UI\Properties\AssemblyInfo.cs".
    Replaced matches with "[assembly: AssemblyVersion("0.1.0.0")]".

  ...

Clean:
  Removing directory "C:\src\deployment\CodeToDeploy\Publish".
Publish:
  Creating directory "C:\src\deployment\CodeToDeploy\Publish\site".
  Creating directory "C:\src\deployment\CodeToDeploy\Publish\site\bin".
  Copying file from "C:\src\deployment\Src\UI\bin\UI.dll" to "C:\src\deployment
  \CodeToDeploy\Publish\site\bin\UI.dll".

  ...

Zip:
  Creating ZipFile: C:\src\deployment\CodeToDeploy\Publish\..\Releases\Sample-B
  eta-0.1.0.0.zip
Done Building Project "C:\src\deployment\build.proj" (Package target(s)).

Done Building Project "C:\src\deployment\build.proj" (PackageBeta target(s)).


Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.70

Sorry, where was that release package?

The current scripts output the release file into a folder called CodeToDeploy (yes, this conventions was grabbed from CodeCampServer for asp.net mvc). In this folder, we build the raw output files to Publish (yes, this is a microsoft convention) and the packages to Release. The release file also then has the version number in the file. Here’s what it looks like below:

  CodeToDeploy/
    Publish/
      ... [all the files here]
    Releases/
      Sample-Beta-0.1.0.0.zip

Unzip and explore

Now, we can unzip the archive as below (of course, it is the same of the publish folder!)

    lib/
        extensionpack
        migratordotnet
    scripts/
        db/
            create.sql
            domainuser-for-db.sql
            drop.sql
            user-for-db.sql
        db-setup.tasks
    site/
      ... [all the application site stuff here]
    0.1.0.0
    Deploy.bat
    deploy.proj
    deploy.rsp
    directorybrowsing.xml
    Migrate.bat
    _Beta_ENVIRONMENT

Let’s work through this:


lib

This contains any libraries that are required for deployment process (rather than actually running the site). In this case we will be migrating a database and deploying to IIS.

scripts

These are all the deployment scripts needed. This includes scripts for first-time install and as well subsequent deployments (reinstalls/upgrades). Here you will se that all the creation scripts for the sql database are included.

site

This is the actual code that gets deployed to IIS

0.1.0.0

This is an empty file that simply has the version number as the name for quick identification.

Deploy.bat, .proj, .rsp

These are the workhorse files for each installation. The bat file is the GUI runner to invoke msbuild so that double click installs can be performed. The proj file has the targets (in this case there are no child tasks). The rsp file is there because it can be used to set specific environment variables that we want picked up regards of commandline or gui base bootstrapping

directorybrowsing.xml

Sorry, this is an odd fellow. It is copied to certain IIS environments to allow for directory browsing. At this stage, this approach is a path of least resistance.

Migrate.bat

This file is GUI runner just like deploy.bat except it is focussed exlusively on migrations. If you look at its contents you can see that it is msbuild.exe /t:MigrateUp /v:d /fl which is targetting Migrate up and outputting a verbose log to the default log file msbuild.log

_Beta_ENVIRONMENT

This file tells us that it was build specifically for the Beta environment. I use this technique if I have create builds specific to an environment. I avoid this approach as discussed earlier.

Understand the versioning

A last comment in this packaging process. Versioning of libraries and packages is always done. There are two main files that control the versioning:

 VERSION.xml
 scripts/
    version.tasks


VERSION.xml

This has already been mentioned before and can be found . It is the major.minor.build number. You’ll update this infrequently

  <PropertyGroup>
    <Major>0</Major>
    <Minor>1</Minor>
    <Build>0</Build>
  </PropertyGroup>


version.tasks

It does two things. It gets the revision number from the source control – the sample uses SVN but it can be TFS, git, Peforce. It creates a version of the format: major.minor.build.revision. It updates all the AssemblyInfo.cs files based on the version. Now that I have told you that, let’s see what Help tells us!

    >msbuild /t:HelpVersion
    
    Microsoft (R) Build Engine Version 3.5.30729.1
    [Microsoft .NET Framework, Version 2.0.50727.3053]
    Copyright (C) Microsoft Corporation 2007. All rights reserved.

    Build started 1/15/2010 5:45:27 PM.
    Project "C:\src\deployment\build.proj" on node 0 (HelpVersion target(s)).


          Updates all the AssemblyInfo.cs files with VERSION (major, minor, build)
          and revision looked up from svn

          msbuild /t:Version

          Targets:
           - Version
           - SvnVersion

          Dependencies: svn commandline client (svnversion.exe - installed to 
                                            C:\Program Files\Subversion\bin)


Points to watch out for:

  • revision with 0 means that it was probably a local build without access to source control
  • AssemblyInfo.cs files get updated but you don’t want to check this in and you don’t want these to come up as changed so I suggest that you check them into source and then put an ignore or lock on them.
  • Build server should really be the only place that creates this version number
  • release packages have version numbers in the filename for ease and archiving
Tagged
2010 01 15 todd Deployment Comments (0) Permalink

Commandline msbuild in action

Commandline msbuild

So you’ve checked out the code. Now commandline it to the root folder of the project. Run msbuild.exe. Did it work? Probably not. That is because you don’t have msbuild on your path environment variable. There’s a couple of ways. When you installed Visual Studio, on the Start Menu it provides you with starting a console with msbuild on path.

Starting console from Start Menu

Alternative one, go and update your path environment variable for all time.

[pict - Changing environment variable]

Alternative two, you can double click on build.bat. (But you are only delaying the inevitable getting msbuild on path)

Double click on bat file in folder

Here’s the resulting console. Msbuild automatically uses build.proj because (a) it is a proj file and (b) it is the only one in the folder.

msbuild with Help (1 of 2)

msbuild with Help (2 of 2)

Help as default target

What you see in the screen above is that our build file by default makes no action other than to tell you what you can do. In other words, by default it won’t do bad. I therefore always have a Help target in every proj and task file and that that is the Default target. Here is an example from build.tasks that demonstrates how the default target is Help and what the structure of the help information.

<Project DefaultTargets="HelpBuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="HelpBuild">
    <Message Text="

    msbuild /t:Package

    Examples:
      msbuild /t:Install
      msbuild /t:Install /p:ReleaseEnvironment=Beta
      msbuild @build.properties /t:Package /v:d 

    Variables that can be overridden:
      DropLocation=C:\Binaries
      ReleaseEnvironment=Dev|[Test]|Beta|Prod
      BuildCmd=Build|[Rebuild] 

    Targets:
     - Compile
     - Clean
     - CleanReleases
     - Publish
     - Zip
     - Extract
     - Deploy
     - TODO

     - Package (Compile;Clean;Publish;Zip)
     - Install (Compile;Clean;Publish;Zip;Extract;Deploy) - will require IIS
                                                            setup because it 
                                                            runs the 
                                                            deploy.bat 
                                                            set of tasks

    Log output: msbuild.log

    Dependencies: Zip/Extract must have J# Redist installed

http://www.msbuildextensionpack.com/help/3.5.4.0/index.html

             " />
  </Target>

</Project>

The general structure for help is as follows:

  • what your likely (default) call is going be
  • examples of likely calls including variables
  • variables that can be overridden
  • targets
  • additional information

Tip: Don’t make the text in the help wider than 64 characters so that it looks pretty in the console

Build.proj as manifest and linking helps

When you ran msbuild, it also showed a number of Help targets. This is because the primary job of the build.proj is to link all the targets together. It looks like this and the points to note are:

  • Import includes each tasks
  • Help is the default target that calls the help targets in of the child tasks

Note: we need to suffix each Help (eg Build = HelpBuild, Package = HelpPackage) to avoid warnings in msbuild.

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Help"  ToolsVersion="3.5" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildProjectDirectory)\scripts\build.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\package.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\install.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\db-setup.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\migrations.tasks" />

  <Target Name="Help" DependsOnTargets="HelpBuild;HelpPackage;HelpInstall;HelpDbSetup;HelpMigrations"/>
</Project>

Summary

  • Setup msbuild on path
  • Always have Help as a default target
  • Tasks files need the Help target to have the task suffix to avoid warnings [bla microsoft]
Tagged
2010 01 15 todd Deployment Comments (0) Permalink

MSBuild deployment sample for web applications

Check out the code sample

Download the Deployment Code Sample from github and select to download the zip archive of the source

Overview of the code

    VERSION.xml
    build.bat
    build.proj

    lib/
        MSBuildCommunityTasks/
        extensionpack/
        migratordotnet/
        sdc/
        teambuild/
    
    scripts/
        build.tasks
        
        db/
            create.sql
            domainuser-for-db.sql
            drop.sql
            user-for-db.sql
        
        db-setup.tasks
        deploy.proj
        directorybrowsing.xml
        install.tasks
        migrations.tasks
        package.tasks
        version.tasks
    
    src/
        Infrastructure/
            Database/
                Migrations/
                    Migr_001_CreateBanner_Table.cs
                Sql/
                    migration-latest.sql
        Sample.sln
        UI/

    tools/
        [these are needed to be installed on machines]

Explanation/Overview of files and folders


VERSION.XML

This holds the major, minor and revision number for the product. This is effectively the marketing name of the product and is changed rarely. This “version” combined with the revision number from the source control is what is used to version DLLs.

build.bat and build.proj

Build.bat is the merely a GUI-based, double-click runner for invoking msbuild file build.proj. Build.proj is a manifest file allowing access to the real worker tasks in the scripts/ folder.

lib/

As you would expect holds all the dependent libraries. I always bundle up my script files ensuring that they are available in every environment. This project demonstrates the use of three msbuild libraries. Each have there strengths so I just use as needed and don’t get too hung up about which one. I often also bundle up the Visual Studio teambuild targets because this allows me to avoid having to install the correct version of Visual Studio on the build server (this problem has got better over time and haven’t reviewed problems here in a whiles).

scripts/

This is the focus of the sample. I keep my build scripts in their own folder as to not clutter up root and I keep them out of src/ so that I get better reuse and a cleaner folder. I also keep my initial database creation scripts here too. These are the first-time creation scripts rather than the migration scripts which are in src/Infrastructure/Database/

src/

Here’s the home of the application code. A couple of conventions, the demo is using a domain-driven-design naming conventions (UI, infrastructure) and java too (src cf Source). Importantly, I see that database work such as schema changes and data transformations are tightly coupled with the application code. Hence they are part of the application code base. I think this type of versioning is far simpler. There is a good blog entry that I can’t find that compares the “migration approach” versus the database compare. Both work compared with the free for all I see – I just prefer migrations and have never found its simplicity to cause problems. Plus it allows me to go up and down in migrations. The UI code is there just for illustrative purposes so that there is something to deploy.

tools/

All this binary packages that need to be installed on the target machine to make the system work but can’t be (or shouldn’t be bundled). If my packaging strategy is to “download-and-deploy” my source code strategy is to “checkout-and-compile”. So I want as much as possible available with source so that in two year’s time I don’t have to go looking. It is surprising how this pushes some people’s buttons.

MSBuild files: proj vs tasks vs xml

My assumption is that you know how msbuild works and the structure of the xml files. I use three extensions to mark out different types. Yet, they all follow the xml schema definition for msbuild.


proj:

a file to be run by msbuild – hopefully the only one in the folder so that you don’t have to specify it

tasks:

a specific set of tasks that is included/imported into a proj file and allows for clarity of purpose and reuse

xml:

a set of properties that are imported into a task or project file

Tagged , ,
2010 01 15 todd Uncategorized Comments (0) Permalink