Archive for the ‘Deployment’ Category

MsBuild execution of SoapUI testrunner

January 20th, 2010 5 comments

I have just been writing an msbuild runner to wrap testrunner for soapUI. Here it is. There are a couple of techniques to note:

  • stacking the args for the commandline in an ItemGroup (see thanks below)
  • don’t forget to Html Escape quots when invoking command line ie &quot
  • I also dependency check for each
  • And the usual that the default target is Help and it tells about the targets and dependencies
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="HelpTest"  ToolsVersion="3.5" xmlns="">

    <SoapUiProject Condition="'$(TestProject)'==''">agent-soapui-project.xml</SoapUiProject>
    <SoapUITestResultsFolder Condition="'$(SoapUITestResultsFolder)'==''">$(MSBuildProjectDirectory)</SoapUITestResultsFolder>

    <Args Include="-a"/>
    <Args Include="-e$(Endpoint)" Condition="'$(Endpoint)'!=''"/>
    <Args Include="-s$(TestSuite)" Condition="'$(TestSuite)'!=''"/>
    <Args Include="-c$(TestCase)" Condition="'$(TestCase)'!=''"/>
    <Args Include="-f$(SoapUITestResultsFolder)" Condition="'$(SoapUITestResultsFolder)'!=''"/>

    <JAVA_HOME Condition="'$(JAVA_HOME)'==''">$(SoapUiPath)\..\jre</JAVA_HOME>
    <SoapUiRunner>&quot;$(SoapUiPath)\testrunner.bat&quot; $(TestProject) @(Args,' ')</SoapUiRunner>

  <Target Name="TestAll" DependsOnTargets="AcceptanceTests"/>

  <Target Name="CheckDependencies">
    <Error Text="SoapUI testrunner is not installed at: $(SoapUiPath)" Condition="!(Exists('$(SoapUiPath)'))"/>
     <Error Text="JAVA_HOME environment variable not set correctly to point to an JRE" Condition="!Exists('$(JAVA_HOME)')"/>
  <Target Name="AcceptanceTests" DependsOnTargets="CheckDependencies">
    <Message Text="Building Test Command: $(SoapUiRunner)"/>
    <Exec Command="$(SoapUiRunner)"/>

  <Target Name="HelpTest">
    <Message Text="
    msbuild /t:TestAll
    Variables that can be overridden:

     - TestAll
     - AcceptanceTests
    - soapUI must be installed to $(SoapUiPath)
    - JAVA_HOME - currently set to: $(JAVA_HOME)
             " />



The technique for stacking args in the ItemGroup is fro:

Categories: Deployment Tags: , ,

One deployment for all environments

January 18th, 2010 No comments

Deploy the package

This set of instructions should be reasonably straightforward. Always have the same deployment technique throughout all environments. In this case, I am explaining a strategy that which uses MSBuild to trigger installations and upgrades. This simple tenet that can be hard to follow.

Often we also need manual intervention, or simply need to break up the process. To make a process manual we should wrap a commandline with a runner process (eg a bat or cmd file). This means that manual processes are still scripted rather than GUI-based. Moving from isolated scripts to automation is simply the process of chaining scripts together.

The basic checkout-and-compile for packaging and then download-and-deploy for installation sequence should be used across the local, build, test and production environments.

A basic deploy/install process

From commmandline:

msbuild /t:Install

This command will invoke the following lifecycle (this is found in build.tasks):

<Target Name="Install" DependsOnTargets="Package;Extract;Deploy"/>

We have already seen that packaging creates a zip file with <Target Name="Package" DependsOnTargets="Version;Clean;Publish;Zip"/>. The install then takes that package unzips it and then deploys. The idea here is that all it does it replicate what the user would do manually. The deploying is often quite straightforward but timeconsuming. In the sample, the main deployment is to an IIS. All of the third-party libraries have tasks for integration points and it worthwhile looking at them all. Personally, I am using extension pack at the moment.

So I am just said, the script works as though it is a person. Here’s where some complexity lies in the scripts. There are two part of the deployment scripts.

  1. the Deploy target is the worker for doing a deployment (that lives in the build.tasks)
  2. the deployment scripts that are bundles in the package and do the work (eg deploy.proj)

Deploy target as worker

Let’s have a look at the targets for the worker task. Before you look at it, let’s just remember that if I did this manually I would simply get copy of the package, unzip and then double click on the bat file that invokes msbuild. This is the download-and-deploy philosophy.

Here are two targets: Extract and Deploy. A quick explanation that I hope really isn’t needed. Extract first creates a directory to put the files in it doesn’t exist and then does the extraction via an extensionpack Zip task. Deploy goes and runs the deploy.bat file that invokes msbuild.

<Target Name="Extract">
   <MakeDir Directories="$(ExtractPath)" Condition = "!Exists('$(ExtractPath)')"/>
   <MSBuild.ExtensionPack.Compression.Zip TaskAction="Extract" ExtractPath="$(ExtractPath)" ZipFileName="$(ReleaseZipFile)"/>

 <Target Name="Deploy">
   <Exec Command="$(ExtractPath)\deploy.bat $(ExtractPath)" ContinueOnError="false" />

Deployment that actually installs

Now we’ll look at the deployment process once a package has been extracted more fully. Again, it doesn’t get any more complex in practice; there is just more of it. Let’s pickup from the worker process. It has just “clicked” on deploy.bat which effectively is the same as doing an msbuild command (for now let’s stay simple). It has now invoked deploy.proj which does the work. It now goes through the motions to install the site with the DeployWebsite target. If you’ve installed on IIS this process is familiar. If not, it’s damn fine documentation! You can see that you need to create Application Pools, Http Bindings and add Applications.

  <Target Name="DeployWebsite" DependsOnTargets="CopyWebSite">
    <Message Text="Deploying site version: $(Version) for environment $(Environment)"/>
    <CallTarget Targets="CreateAppPool"/>
    <CallTarget Targets="CreateWebsite"/>
    <CallTarget Targets="CreateBindingHttp"/>
    <CallTarget Targets="CreateBindingHttps" Condition="'$(Environment)'!='Test'"/>
    <CallTarget Targets="AddApplication"/>
    <CallTarget Targets="ModifyWebsite" Condition="'$(Environment)'!='Test'"/>
    <CallTarget Targets="SetupDirectoryBrowsing" Condition="'$(Environment)'=='Test'"/>
    <CallTarget Targets="StartWebsite"/>

While we have this deployment in mind, I’ll add a couple of notes on this deployment. It has two modes of deployment: test and non-test. The test deployment doesn’t run https and it does have a multi-site structure which we use directory browsing for moving between sites. This is a bigger topic of hot-swapping sites that I might get back to.

A quick summary is that it is each to deploy a package in your own environment by scripting tasks you might normally do with a point-and-click, or drag-until-you-drop, approach. This is an important start. Now you need do the same process on the build server (or build agent).

Build Server deploy: allowing it to push out a deployment

The build server should be no different. But it often is. It would ideally not be running as a interactive user and this can cause problems. In windows, there aren’t many good solutions. We don’t have ssh generally available that would make life easy. Running everything under Network Service or Local System is a poor solution that we often resort to.

Having read this entry you might asking yourself, how do we deal with the database? I’ll deal with that next because it cuts across the important issue of first-time installs and redeployments/upgrades.

Categories: Deployment Tags:

Understanding the packaging

January 15th, 2010 No comments

Build the package

So what we’ll do is actually build a package by using the command msbuild /t:PackageBeta

Below is the output that I have shorted so that you can see major targets that create the package as a zip file in the location: CodeToDeploy\Releases\

  • Version
  • Clean
  • Publish
  • Zip
C:\src\deployment>msbuild /t:PackageBeta
Microsoft (R) Build Engine Version 3.5.30729.1
[Microsoft .NET Framework, Version 2.0.50727.3053]
Copyright (C) Microsoft Corporation 2007. All rights reserved.

Build started 1/15/2010 4:53:21 PM.
Project "C:\src\deployment\build.proj" on node 0 (PackageBeta target(s)).
Project "C:\src\deployment\build.proj" (1) is building "C:\src\deployment\build
.proj" (1:2) on node 0 (Package target(s)).
  Updating File "C:\src\deployment\src\Infrastructure\Properties\AssemblyInfo.c
    Replaced matches with "[assembly: AssemblyVersion("")]".
  Updating File "C:\src\deployment\src\Test\Properties\AssemblyInfo.cs".
    Replaced matches with "[assembly: AssemblyVersion("")]".
  Updating File "C:\src\deployment\src\UI\Properties\AssemblyInfo.cs".
    Replaced matches with "[assembly: AssemblyVersion("")]".


  Removing directory "C:\src\deployment\CodeToDeploy\Publish".
  Creating directory "C:\src\deployment\CodeToDeploy\Publish\site".
  Creating directory "C:\src\deployment\CodeToDeploy\Publish\site\bin".
  Copying file from "C:\src\deployment\Src\UI\bin\UI.dll" to "C:\src\deployment


  Creating ZipFile: C:\src\deployment\CodeToDeploy\Publish\..\Releases\Sample-B
Done Building Project "C:\src\deployment\build.proj" (Package target(s)).

Done Building Project "C:\src\deployment\build.proj" (PackageBeta target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.70

Sorry, where was that release package?

The current scripts output the release file into a folder called CodeToDeploy (yes, this conventions was grabbed from CodeCampServer for mvc). In this folder, we build the raw output files to Publish (yes, this is a microsoft convention) and the packages to Release. The release file also then has the version number in the file. Here’s what it looks like below:

      ... [all the files here]

Unzip and explore

Now, we can unzip the archive as below (of course, it is the same of the publish folder!)

      ... [all the application site stuff here]

Let’s work through this:


This contains any libraries that are required for deployment process (rather than actually running the site). In this case we will be migrating a database and deploying to IIS.


These are all the deployment scripts needed. This includes scripts for first-time install and as well subsequent deployments (reinstalls/upgrades). Here you will se that all the creation scripts for the sql database are included.


This is the actual code that gets deployed to IIS

This is an empty file that simply has the version number as the name for quick identification.

Deploy.bat, .proj, .rsp

These are the workhorse files for each installation. The bat file is the GUI runner to invoke msbuild so that double click installs can be performed. The proj file has the targets (in this case there are no child tasks). The rsp file is there because it can be used to set specific environment variables that we want picked up regards of commandline or gui base bootstrapping


Sorry, this is an odd fellow. It is copied to certain IIS environments to allow for directory browsing. At this stage, this approach is a path of least resistance.


This file is GUI runner just like deploy.bat except it is focussed exlusively on migrations. If you look at its contents you can see that it is msbuild.exe /t:MigrateUp /v:d /fl which is targetting Migrate up and outputting a verbose log to the default log file msbuild.log


This file tells us that it was build specifically for the Beta environment. I use this technique if I have create builds specific to an environment. I avoid this approach as discussed earlier.

Understand the versioning

A last comment in this packaging process. Versioning of libraries and packages is always done. There are two main files that control the versioning:



This has already been mentioned before and can be found . It is the number. You’ll update this infrequently



It does two things. It gets the revision number from the source control – the sample uses SVN but it can be TFS, git, Peforce. It creates a version of the format: It updates all the AssemblyInfo.cs files based on the version. Now that I have told you that, let’s see what Help tells us!

    >msbuild /t:HelpVersion
    Microsoft (R) Build Engine Version 3.5.30729.1
    [Microsoft .NET Framework, Version 2.0.50727.3053]
    Copyright (C) Microsoft Corporation 2007. All rights reserved.

    Build started 1/15/2010 5:45:27 PM.
    Project "C:\src\deployment\build.proj" on node 0 (HelpVersion target(s)).

          Updates all the AssemblyInfo.cs files with VERSION (major, minor, build)
          and revision looked up from svn

          msbuild /t:Version

           - Version
           - SvnVersion

          Dependencies: svn commandline client (svnversion.exe - installed to 
                                            C:\Program Files\Subversion\bin)

Points to watch out for:

  • revision with 0 means that it was probably a local build without access to source control
  • AssemblyInfo.cs files get updated but you don’t want to check this in and you don’t want these to come up as changed so I suggest that you check them into source and then put an ignore or lock on them.
  • Build server should really be the only place that creates this version number
  • release packages have version numbers in the filename for ease and archiving
Categories: Deployment Tags:

Commandline msbuild in action

January 15th, 2010 No comments

Commandline msbuild

So you’ve checked out the code. Now commandline it to the root folder of the project. Run msbuild.exe. Did it work? Probably not. That is because you don’t have msbuild on your path environment variable. There’s a couple of ways. When you installed Visual Studio, on the Start Menu it provides you with starting a console with msbuild on path.

Starting console from Start Menu

Alternative one, go and update your path environment variable for all time.

[pict - Changing environment variable]

Alternative two, you can double click on build.bat. (But you are only delaying the inevitable getting msbuild on path)

Double click on bat file in folder

Here’s the resulting console. Msbuild automatically uses build.proj because (a) it is a proj file and (b) it is the only one in the folder.

msbuild with Help (1 of 2)

msbuild with Help (2 of 2)

Help as default target

What you see in the screen above is that our build file by default makes no action other than to tell you what you can do. In other words, by default it won’t do bad. I therefore always have a Help target in every proj and task file and that that is the Default target. Here is an example from build.tasks that demonstrates how the default target is Help and what the structure of the help information.

<Project DefaultTargets="HelpBuild" xmlns="">

  <Target Name="HelpBuild">
    <Message Text="

    msbuild /t:Package

      msbuild /t:Install
      msbuild /t:Install /p:ReleaseEnvironment=Beta
      msbuild /t:Package /v:d 

    Variables that can be overridden:

     - Compile
     - Clean
     - CleanReleases
     - Publish
     - Zip
     - Extract
     - Deploy
     - TODO

     - Package (Compile;Clean;Publish;Zip)
     - Install (Compile;Clean;Publish;Zip;Extract;Deploy) - will require IIS
                                                            setup because it 
                                                            runs the 
                                                            set of tasks

    Log output: msbuild.log

    Dependencies: Zip/Extract must have J# Redist installed

             " />


The general structure for help is as follows:

  • what your likely (default) call is going be
  • examples of likely calls including variables
  • variables that can be overridden
  • targets
  • additional information

Tip: Don’t make the text in the help wider than 64 characters so that it looks pretty in the console

Build.proj as manifest and linking helps

When you ran msbuild, it also showed a number of Help targets. This is because the primary job of the build.proj is to link all the targets together. It looks like this and the points to note are:

  • Import includes each tasks
  • Help is the default target that calls the help targets in of the child tasks

Note: we need to suffix each Help (eg Build = HelpBuild, Package = HelpPackage) to avoid warnings in msbuild.

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Help"  ToolsVersion="3.5" xmlns="">
  <Import Project="$(MSBuildProjectDirectory)\scripts\build.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\package.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\install.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\db-setup.tasks" />
  <Import Project="$(MSBuildProjectDirectory)\scripts\migrations.tasks" />

  <Target Name="Help" DependsOnTargets="HelpBuild;HelpPackage;HelpInstall;HelpDbSetup;HelpMigrations"/>


  • Setup msbuild on path
  • Always have Help as a default target
  • Tasks files need the Help target to have the task suffix to avoid warnings [bla microsoft]
Categories: Deployment Tags:

Introduction to using msbuild to manage ( mvc) application lifecycles

January 15th, 2010 No comments

Introduction: what’s there to really manage?
Part 1: Code Overview
Part 2: Commandline msbuild in action
Part 3: Understanding the packaging
Part 4: Build & Test on your local machine
Part 5: First-time installs and redeployments
Parts 5-7 to come
Summary and References


Ever since … well since ever … I have needed to manage the lifecycle of my applications. Through-the-GUI approaches hit a limit of manageability quickly so like most others I use scripting. In mvc, I use msbuild. It is already part of the .Net framework and generally accepted. I say generally accepted because it is not used a lot where I work. Only some seniors in the dotnet space actually use this toolset to manage their application lifecycle. This is a pity because many of these seniors have actually had a life before dotnet and often microsoft. They know about commandline tools. One of the problems for me is that this also means that we don’t have emerging developers who can actually think through the lifecycle of their applications, let alone script its phases. So let me get this little rant out of the way … in microsoft’s GUI-based application-lifecycle-management approach they attempt to make development accessible to a wider audience and speed up development; the wider goal is to commoditise development which in effect increases the uptake of microsoft products (ie revenues). The effect is that developers are deskilled, the application lifecycle is fragmented and unreliable across environments. Msbuild for all its limitations helps us to counter these problems: the build scripts help increase the transparency of the lifecycle and allow us to reproduce each step through each environment.

I personally find msbuild quirky and many features counter intuitive. I would really like to add “at first” after the counter intuitive but actually I still find its design that way. Output params, task items, default values for parameters are examples. But it is powerful and can do the job. If I had my way I would head down the Rake road like many others already have. It is cleaner and clearer: simply put, its build for purpose. Roll on iron ruby!

So this series of blogs spell out how I go about deploying web apps. There are lots of variations within each step that I will attempt to avoid distracting you with. So my goals is the illustrate the general approach with specific examples. These examples are taken over the last four projects that I have worked on.

Application lifecycle

The application lifecycle are the phases through environments that your code goes through. Here I mean that code are your binaries, data, configuration, documentation and setup scripts. Take a look at my provisional/simplified list below:

  • Environments: Local(Dev), Build, Test, Production (perhaps even pre-production as a clone of production)
  • Configurations: Dev, Test, Production (production might be seen as the no-name environment)
  • Phases: Checkout/Update, Build (Compile/ReBuild), Test, Package, Deploy

Looking at the list, you’ll see that I already have added synonyms suggesting immediate difficulties in agreeing what things are called. For example, the “local” environment is often called “dev” – by local, I am meaning that you have an environment which tends to allow you to notionally pull out your LAN cable such that you can setup your environment for development and testing. For many, dev is something where the codebase for the dev lives but the data that they test against lives somewhere else. For me, when pursuing a layered test automation strategy, not having data under source control is a code smell: it works but it isn’t ideal.

Around configuration

Another complexity is matching configurations with environments. No one needs to be preached to on the importance of configuration management through the phases through the environments. Yet, this is difficult to get agreement on in dotnet because in practice microsoft’s out-of-the-box approaches are inadequate and people put up with it. My cynical view is that making providing inadequate tools creates the need for the tools in the first place because we are always seeking a solution to the fundamental problem. I think that this is also fuelled by the pleasures people in experiencing this pain. I take for instance that it is difficult to securely do a push deploy to a remote machine. By the time I exhaust all my secure options and I open it up to the Network Service which goes against all my beliefs that I experience relief and pleasure at simply getting the job done. In fact, is some weird way, I am grateful when the job is done regardless. So configuration management is just about always the bone of contention. Below are three approaches. The first two are acceptable approaches to automation. The third is still a dominant practice that I wish to discount.

Option one: save each environment configuration in source control
Option two: save each environment’s configuration in their environment and pass through at deployment
Option three: manually update environment configuration in an ad hoc way

I prefer option two and often concede with option one. Option one is currently in the sample code I will provide: there is a separate web.config for each environment. The big problem with this approach is that to automate well you really should provide a package for each environment. You do this because as a rule the configuration settings from one environment should never be available to other environments. Conversely put, production settings should never be known outside the production environment.

So option two is preferable because I can save the configuration of each environment in each environment. I have range of options of how I store those settings. I can use environments variables, hand into commandline msbuild or file-based settings (as response files or project files). I don’t use registry settings. Ideally, these environment settings are centrally managed but are separate from source control.

Around phases

You would think that the phases are generally agreed upon. I don’t think so. At best I get agreement on the get source code (checkout/update), do something and build (compile and test in some form), and then send it somewhere to do something (deploy). the GUI-based tools in Visual Studio do little to help the situation for the local environment. Source control is often managed in ways that the inexperienced developer is in control. In web applications, they can then go publish which compiles, publishes and runs a local version of an application/web server. Next you can then publish from your local environment to say the test environment. That is, after you have manually updated settings in the local for the test. I want better separation and knowing what phases occur in what order and in which environment. Here’s a stab at it based on the scripts I will explain:

- Environment: Local
  - Configuration: Dev
    - Phases: Checkout, Update, Compile, UnitTest, Deploy, Migrations, IntegrationTest, AcceptanceTest
- Environment: Build
  - Configuration: Test
    - Phases: Checkout, Update, Compile, UnitTest, Package, Deploy, IntegrationTest, Notify
- Environment: Test
  - Configuration: Test
    - Phases: GetPackage, Deploy, Migrations, AcceptanceTests
- Environment: Production
  - Configuration: Producton
    - Phases: GetPackage, Deploy, Migrations

Let me compare and contrast the phases to see how each is subtly different but together they make up a coherent the application lifecyle.

  • the local environment tends to have all phases except that in practice you don’t create packages for deployment
  • the build server in contrast is all about creating the package that will be moved through all of the subsequent environments (so it watches the source code respository and works out whether the package should be released and hence people notified)
  • the test environment is the place the confirms that the build server made the correct decisions to release the package and is also the place for verification at the system level. This environment is also the place where we try out the application and its migrations against the production data. So while the package moves forward through environments, data comes backwards. It is the place where the two get a change to meet and greet (integrate). We are looking for stability over time in the this environment. This environment may have acceptance tests that also get run to look for the “non-functional” tests.
  • the production environment should be no different to the prior (test) environment.

First-time cycle of phases versus subsequent cycles

What I have actually just described are the phases went the application is actually underway. What we forget is that each environment has to be setup cycle in the first place. The classic is the new developer who takes days to get everything going. There are phases in this part of the cycle that too can be scripted.

- Environment: Local
  - Configuration: Dev
    - Phases: Setup Repository, Create Database with user perms, Create Aliases, Create IIS
- Environment: Build
  - Configuration: Test
    - Phases: Create Users, Create Database with user perms, Create Aliases, Create IIS
- Environment: Test
  - Configuration: Test
    - Phases: Create Users, Create Database with user perms, Create Aliases, Create IIS
- Environment: Production
  - Configuration: Producton
    - Phases: Create Users, Create Database with user perms, Create Aliases, Create IIS

The goal of this approach is

* the local environment for developers is separated for all other areas of the system
* create packages on the build server that are potentially releasable to production
* deploying a package should aim for “download-and-deploy”
* configurations and dependencies should be setup only once prior to the first deploy
* deployments to test should lag development as little as possible
* the trend of success to test is the indicator of readiness for deployment to production
* changes must flow through environments in order, every time

Understanding msbuild sample deployment project: an lifecycle overview

  • Check out the code sample
  • Overview of the code
  • Commandline msbuild
  • Help as default target
  • Build.proj as manifest and linking helps
  • build the package
  • unzip and explore
  • understand the versioning
  • Build in dev
  • Run the migrations in dev
  • rebuild
  • setup the db
  • deploy to iis
  • back in dev
  • update migrations
  • build
  • package
  • unzip
  • Script your initial setups
  • Scripted redeployments
  • up migrations
  • up iis
  • view