SharePoint Installations: Strategy to aid continuous integration

This document outlines an approach to managing SharePoint installations. Starting from the current deployment approach which is primarily manual, an improved approach is outlined that manages the difference between deployments and migrations and aid the automation of scripting through all environments.

Overview

For continuous integration (CI), we generally want the approach to move a single package for a SharePoint solution through all environments. Secondarily, we want to be scripted and preferably automated. To put the foundations in place we have had to devise a clear strategy. Microsoft does not provide all the tools and techniques for SharePoint continuous integration. In fact, in most cases for new features such as customisation of site page or workflow creation must be performed manually in each environment.

Shortcomings of SharePoint deployments: no real versioning across environments

SharePoint has a well established set of techniques for deployment. Most easily, there are through the GUI approaches that come with Visual Studio or are able to added into Visual Studio. There are other tools that aid deployments through a GUI in a wizard style (on CKS: Development Tools Edition on Codeplex). Alternatively, you can deploy solutions using stsadm and PowerShell cmdlets. Finally, you can create your own custom deployment steps (IDeploymentStep) and deployment configurations (ISharePointProjectExtension) by implementing the core SharePoint library and add these to your solutions and through the deployment build lifecycle. This is all well explained in SharePoint 2010 Development with Visual Studio by Addison Wesley.

All of this allows for custom deployment and retraction of a SharePoint solution with bias toward GUI-based deployment and certainly manual configuration of sites. Manual configuration is particularly problematic in the enterprise environment that wishes to use continuous integration.

There is a missing piece of the puzzle that could aid automation in the moving of solutions through environments. This is the idea of migrations. Migrations are simply a convenient way for you to alter the SharePoint site in a structured and organised manner. So while you could activate features through the GUI by telling operations or administrators you would then be responsible for writing documentation. You’d also need to keep track of changes of which changes need to be run against each environment next time you deploy. Instead, migrations work against the SharePoint API and are versioned. Each SharePoint instance has a record of which version migrations have been run. It applies on new migrations to the site.

Current Approach: without migrations and bias toward GUI

Table 1 outlines the current installation approach is scripted deployments via powershell. Provisioning occurs as part of the SharePoint. However, there is also a lot more configuration required that is currently performed manually through the GUI. This leads to problems in installations across multiple environments. It is also costly in terms of time for training, documentation and problem solving.

Deployment
Provisioning		Configuration
Scripted	GUI
Automated	Manual

Table 1:             Previous State for SharePoint installations

In contrast, Table 2 splits deployment into deployment and migrations and has a bias towards scripting and automation over GUI and manual configuration.

Deployment	Migrations
Provisioning		Configuration
Scripted				GUI
Automated			Manual

Table 2:             New State for SharePoint installations

Approach With Migrations and bias toward scripting

We approach this problem by isolating the installation of sharepoint solutions through provisioning and configuration phases as per Table 3. Provisioning occurs when a new “feature” is deployed. This occurs in the Event Receiver code callbacks.

Configuration is actually an implicit concept in SharePoint. Activation is post-provisioning and can occur on features once provisioned. Configuration on the other hand is the main activity of site administrators in SharePoint – this is the gluing together of the system. For most SharePoint installations, these two activities are usually performed through the GUI by people. As such, this is not conducive to enterprise systems where new code and configurations need to be moved through many environments.

Provisioning

Configuration

Table 3:             Installation process split between provisioning and
configuration

Table 4 outlines the difference between sripted tasks and GUI-based tasks and that there should be a preference toward scripting. Scripted tasks tend to be written in C# using SharePoint API to perform tasks (or as cmdlets). They merely reproduce tasks that would normally be done in the GUI. In most cases, the functionality is the same. However, in a percentage of cases we find that each approach has its own quirks that need to be ironed out. This is very useful for finding issues early on.

Provisioning	Configuration
Scripted		GUI

Table 4:             Tasks should be scripted more than through the GUI

Table 5 outlines the distinction between automated and manual tasks. To aid maintainability through environments tasks must be scripted for continuous integration automation. However, in practice not all tasks should be exclusively automated. Many scripted tasks must also be able to be run within continuous builds and be run manually.

Provisioning	Configuration
Scripted			GUI
Automated		Manual

Table 5:             Automation over Manual – but manual still required

Finally in Table 6, for all of this work we there have split the installation process into two approaches: deployments and migrations. Deployments are scripted in powershell (and msbuild) and have the ability to compile, package and deploy the SharePoint solutions. Deployments trigger provisioning in the SharePoint system by design of SharePoint. What is then missing is the ability to (re)configure the system per deployment. Migrations do this job. They apply a set of changes per installation – they are a little more powerful than this. They can actually apply set of changes in order per installation and keep these versioned. Moreover they can work transactionally and retract changes upon errors (and this can be either automated or done manually). It is based on a technique used for database schema changes – in fact, we have extended an open source library to do the job for us.

Deployments also extend the simplistic notion of only needing a wsp for deployment and include creating zip packages, extracting and deploying solutions via powershell and SharePoint API. The result is not only the standard deployment: wsp uploaded into the solution store; binaries are deployed. But also, as part of the deployment, the powershell scripts may also provision solutions that are usually provisioned through the GUI by the administrator. This will often be a prerequisite for a migration.

Migrations are versioned actions that run after the deployment. Migrations may activate, install/upgrade or configure a feature (or act upon the 14 hive). Many of these tasks have previously been performed through the GUI.

Deployment	Migrations
Provisioning		Configuration
Scripted				GUI
Automated			Manual

Table 6:             Installation is now completed via deployment and migrations

Changes in the code base

Having this strategy means some changes in the code base. Here are some of things we see:

Migration Classes: We now have the configuration code implemented as part of the code base – this includes the ability to make configurations roll forward or backwards

Automation: Configuration can now be performed by build agents

Migration Framework: We have used migratordotnet to perform the migration but did have to extend the library to cater for SharePoint

PowerShell scripts: migrations can be run either automatically as part of installations or run independently by operations – we have also used psake as a build DSL around PowerShell

So far, we haven’t included any code samples. These are to come.