Chris Surfleet's MentalReboot

Code insights tinged with a spot of insanity

Over the last couple of years, we’ve been working slowly through the Continuous Deployment evolution; we started continuously integrating, and then began continuously deploying to test, and then to staging. It’s been hugely liberating but the inevitable next step was pretty daunting: can we get any closer to continuous deployments to live? Putting aside the ideal of a deployment on each checkin for a now I decided the first step should be getting a TFS build which can do the deployment.

This is the first in a 3-part series in which I will go through a complete FTP deployment solution, which will be able to push any configuration of your web application to a remote server and will be fully configurable from a TFS Build Definition. For this first part I will go over how to build the FTP component, while the later posts will cover first a custom TFS workflow and then some custom configuration management.

Why a Team Build Activity?

There are FTP tasks available in MsBuild, so it would be totally possible to get your application FTP’d from your build file. Why not? Well I think that we should have a separation of concerns here, your csproj file handles building and constructing your product, while Team Build handles deployment. This then allows you to control all your deployment setup in each Build Definition.

(A side note here, this leaves the question of where you handle configuration such as web.config transformation, personally I do this from Team Build but it’s a grey area.)

If you can’t be bothered working through the tutorial (shame on you!) you can get the finished product here: FtpDeployment.zip (57.53 kb)

WinSCP

This solution makes use of the WinSCP utility, so you will need to install it first on your build server and dev boxes. This install includes a COM object we can send our FTP commands to.

Project Setup

OK, create yourself a new class library project and add references to the following assemblies, and ensure each has Copy Local set to false:

  • Microsoft.Build.Framework
  • Microsoft.Build.Utilities.v4.0
  • Microsoft.TeamFoundation
  • Microsoft.TeamFoundation.Build.Client
  • Microsoft.TeamFoundation.Build.Workflow (In C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\)
  • Microsoft.TeamFoundation.TestImpact.BuildIntegration (Also in PrivateAssemblies)
  • Microsoft.TeamFoundation.TestImpact.Client (In C:\Windows\assembly\GAC_MSIL\
    Microsoft.TeamFoundation.TestImpact.Client\10.0.0.0__b03f5f7f11d50a3a\)
  • Microsoft.TeamFoundation.VersionControl.Client
  • Microsoft.TeamFoundation.VersionControl.Common
  • Microsoft.TeamFoundation.WorkItemTracking.Client
  • System.Activities
  • System.Activities.Presentation

FTP Code

Now create a new class called FtpSynchronizer to hold the code we will use to perform the FTP. Add the following method to it:

public bool PublishWebsite(string ftpSite, string ftpUserName,
            string ftpPassword, string ftpFolderPath, string localFolder,
            string winScpLocation = @"C:\Program Files\WinSCP\winscp.com")
{
  using (Process winScp = new Process())
  {
    winScp.StartInfo.FileName = winScpLocation;
    winScp.StartInfo.RedirectStandardInput = true;
    winScp.StartInfo.UseShellExecute = false;
    winScp.StartInfo.CreateNoWindow = true;
    winScp.Start();

    winScp.StandardInput.WriteLine("option batch abort");
    winScp.StandardInput.WriteLine("option confirm off");
    winScp.StandardInput.WriteLine("open ftp://" + ftpUserName + ":" +
                  ftpPassword + "@" + ftpSite);
    winScp.StandardInput.WriteLine("synchronize remote " + localFolder +
                  " /" + ftpFolderPath);
    winScp.StandardInput.WriteLine("exit");

    winScp.StandardInput.Close();

    winScp.WaitForExit();

    return (winScp.ExitCode == 0);
  }
}

The first block is firing up WinSCP in the background and preparing it for commands, it’s the second block we are more interested in. This simply opens up a batch-capable connection to an FTP site and sends a “synchronize remote” command. This will copy any newer files from the local folder to the FTP server. It will not delete files on the server which are not on the client, this is the safer behaviour but could be a gotcha in certain circumstances.

After firing the commands we wait for WinSCP to do its stuff, and then return true if everything was successful, or false if not. This isn’t ideal, and I’ll post soon on how to add better error handling to this component, but it is workable in a basic scenario.

At this point, test your class. Go set yourself up a unit test project and create some tests against your FTP site. Done? Brill!

App_offline.htm

This code works, but it would be nice if users of the application weren’t shown an ugly 404 page if they try to access the site while its being updated. Lets use an app_offline file!

In the FtpSynchronize class add the following at the beginning of the PublishWebsite method:

string appOfflinePath = Path.Combine(Path.GetTempPath(), "app_offline.htm");

using (StreamWriter stream = File.CreateText(appOfflinePath))
{
  stream.Write("<html><body><h1>Application currently undergoing maintenance, " +
      "check back in a few minutes.</h1></body></html>");
  stream.Close();
}

 

Obviously you could make the page contents nicer if you like. Then you simply need to upload it before your synchronization, and remove it after, so surround the “synchronize remote” line like this:

winScp.StandardInput.WriteLine("put " + appOfflinePath);
winScp.StandardInput.WriteLine("synchronize remote " + localFolder +
                     " /" + ftpFolderPath);
winScp.StandardInput.WriteLine("rm app_offline.htm");

Done!

Creating the TFS CodeActivity

We’ve got a nice little FTP synchronizer now, but we need to be able to access it from Team Build. For this we’ll create a new class called FtpSynchronizeActivity:

 

[BuildActivity(HostEnvironmentOption.All)]
[BuildExtension(HostEnvironmentOption.All)]
public sealed class FtpSynchronizeActivity : CodeActivity<bool>
{
  protected override bool Execute(CodeActivityContext context)
  {
    throw new NotImplementedException();
  }
}

This will be an activity we can use in a TFS workflow which will return success or failure. (If that made no sense don’t worry, we’ll get to it shortly) We’re going to need some inputs to pass the required info to the FtpSynchronizer so add these above the Execute method:

public InArgument<string> WinScpComLocation { get; set; }

[RequiredArgument]
public InArgument<string> LocalFolder { get; set; }

[RequiredArgument]
public InArgument<string> FtpSite { get; set; }

[RequiredArgument]
public InArgument<string> FtpUserName { get; set; }

[RequiredArgument]
public InArgument<string> FtpPassword { get; set; }

public InArgument<string> FtpFolderPath { get; set; }

These values will be available as properties of the activity in our workflow. All we need to do now is link up the Execute method to the FtpSynchronizer:

protected override bool Execute(CodeActivityContext context)
{
  string winScpLocaton = context.GetValue(WinScpComLocation);
  string localFolder = context.GetValue(LocalFolder);
  string ftpSite = context.GetValue(FtpSite);
  string ftpUserName = context.GetValue(FtpUserName);
  string ftpPassword = context.GetValue(FtpPassword);
  string ftpFolderPath = context.GetValue(FtpFolderPath);

  if (ftpFolderPath == null)
    ftpFolderPath = String.Empty;

  FtpSynchronizer synchronizer = new FtpSynchronizer();
  return String.IsNullOrWhiteSpace(winScpLocaton) ?
    synchronizer.PublishWebsite(ftpSite, ftpUserName, ftpPassword,
                    ftpFolderPath, localFolder) :
    synchronizer.PublishWebsite(ftpSite, ftpUserName, ftpPassword,
                    ftpFolderPath, localFolder, winScpLocaton);
}

Next Steps

All that remains to be done is include the activity in a Team Build workflow. If you've not done this before I'll be covering it in part 2 coming up in a few days. In part 3 we will overcome a couple of problems with configuration and add a bit of extra functionality to the deployment workflow. Watch this space!

Full code for this example is available here: FtpDeployment.zip (57.53 kb)

Did you find this post useful or interesting? I'd be really grateful if you could return the favour by clicking this google link:



Hey, it’s been a while! The usual #lifegetsintheway excuses all apply, but you knew that right? Fear not though, treasured reader, for today we are going to delve into the adrenaline-filled world of MsBuild/TFS interaction!

But Why?

There are plenty of reasons for wanting to interact with your TFS server from MsBuild. Maybe you want to ensure that you always get the latest version of a referenced DLL on each build? You could need to update a source-controlled log file when a certain configuration is built on your continuous integration machine? All of these sorts of things are possible and I’ll be going over some of the more complex scenarios in future articles. For now though I want to fix a bit of an annoyance with the web or service installers I went through in previous posts. If you haven’t worked through these don’t worry, you can get the finished source for the service installer here: WixServiceInstallerExample.zip (184.90kb). (You will need WiX installed, but that is a good thing)

Lets Go

Lets start by recreating the problem; step one is to add the service installer solution to source control and check everything in. Once done, try to build the installer project. Oh no!

Unable to execute transformation. Access to the path ‘……..’ is denied.

What is happening here? The installer build process needs write access to the file WixServiceInstallerExample.wxs, which is denied because it is currently checked in. You can verify this by checking the file out and building again, which will work. So all we need to do is make this automatic.

Open up the project file for the installer (right click > Unload project, right click > Edit WixServiceInstaller.wixproj). Add the following three properties to the top PropertyGroup element:

<TfsExeLocation>$(Vs110ComnTools)..\IDE\tf.exe</TfsExeLocation>
<CheckinComment>Checkin for installer build.</CheckinComment>
<CheckinOverrideReason>Checkin for new installer build.</CheckinOverrideReason>

The first item is the path to the ft.exe file on your computer. Unfortunately we have to do all of this stuff using the command line! The nice thing about defining it here though is that if you have a wacky install on your build box like I had to deal with for one client, causing the TFS build to break, you can override this property in your TFS build definition. If you are using VS2010 then use Vs100ComnTools instead.

The other two items are there for when we check the file back in again, ensuring we don’t get any policy blocks on checkins.

Now, right at the beginning of the BeforeBuild target at the bottom of the file, add this line:

<Exec Command=""$(TfsExeLocation)" checkout %(ProjectReference.Filename).wxs"
               Condition="'%(ProjectReference.ContentProject)'=='True'" />

The Exec command is deliciously powerful; suddenly the power of any command-line tool you can find or build is at your beck and call. There really isn’t a huge amount to this; we pass a ‘checkout’ command and the path to the wxs file to the ef.exe command. Next add the following right at the end of the BeforeBuild target:

<Exec Command=""$(TfsExeLocation)" checkin %(ProjectReference.Filename).wxs
                 /comment:"$(CheckinComment)" /noprompt
                 /override:"$(CheckinOverrideReason)""
        Condition="'%(ProjectReference.ContentProject)'=='True'" />

This time we’re checking the file in. It’s a pretty similar command to the last one but we are just passing a few more arguments specifying comments and policy override reasons to be stored next to the checkin. The noprompt option tells tf.exe not to pop up any dialogs etc.

Check out the show!

That was easy huh? You can go ahead and reload the project now. Make sure all files are checked in and run the build. Keep watching the solution explorer window and you should see the wxs file’s icon changing to checked out, and back to checked in as the build progresses. Happy days.

Limitations

Just bear in mind that this code is always run when you build. That means that even if the wxs file is checked out before you build, it will end up checked in afterwards.

Hopefully this post will have been of use to someone, but I’m off to get started on the Christmas celebrations. Merry Christmas to those of you who celebrate it!

Did you find this post useful or interesting? I'd be really grateful if you could return the favour by clicking this google link:



The web.config XDT transformations used in Visual Studio’s web projects are great for ensuring you don’t end up releasing say, a release version of your code pointing to the test database (yeah, that one with all the amusing names and phrases in it). Unfortunately this functionality wasn’t extended to the app.config. Luckily though we can add it in with minimal hassle, read on for the knowledge.

A quick pre-warning

The visual studio team are working on putting this functionality into the next version as standard, and the hacks we are putting into place here could make the upgrade a bit trickier. Personally I think that undoing these changes later is a small price to pay for the benefits you’ll get now. Just don’t come hunting me down when visual studio 2012/13/whatever gets released!

Setting your project up for transformations

This section is all about setting up the project as a whole. I’d advise doing this on a clean project and then using this as your project template so that it’s all ready and waiting for you.

Firstly, you’ll need to create a new Windows Service project. Now right click on the project in Solution Explorer and select Unload Project. This will grey out the project and mark it as unavailable. Now right click again and select Edit [projectname]:

This will load up the msbuild file which defines your project. All of the funky stuff we’re about to do will happen in here. Don’t worry if you don’t know much (or anything) about msbuild though, you’ll pick it up quickly.

Take a look at the first PropertyGroup node at the top of the file. This is defining some top-level information for the compiler to use. Add the following at the end of it. Right after the FileAlignment node:

<ProjectConfigFileName>App.config</ProjectConfigFileName>

This will be used by the existing transformations functionality to know which files to transform. Next, scroll right to the bottom of the file. You should see an Import node pointing to Microsoft.CSharp.Targets. These import nodes work just like a using statement in c# - we’re importing some existing functionality. So add a new Import node below to bring in the web publishing stuff, which includes the transformation targets:

<Import Project="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.targets" />

This is actually all we need to do to get transforms happening when you build against the TransformWebConfig target, but currently they won’t be written to your output directory. Instead they will appear under obj\x86\Debug\TransformWebConfig. We just need to do a little bit of msbuild code to copy them to the output directory for us.

Right at the bottom of the file, just before the closing </Project> tag enter the following:

<Target Name="TransformAppConfig">
  <CallTarget Targets="TransformWebConfig" />
  <Copy SourceFiles="$(TransformWebConfigIntermediateLocation)\transformed\App.config"
   DestinationFiles="$(OutDir)\$(AssemblyName).exe.config"
   Condition="Exists('$(TransformWebConfigIntermediateLocation)\transformed\App.config')" />
</Target>

This creates our own target, which runs the hacked TransformWebConfig target and then copies any created transformation over the version in our build output path – easy!

Gotcha – OutputPath vs OutDir (MsBuild junkies only!)

It is important that you use the OutDir value to copy the transformed config to here. OutputPath will work fine for local builds but if you try and automate it through TFS Team Build it will mysteriously disappear. Using OutDir will work fine though.

Adding config transformation files

We’re ready to actually add our config files now so exit the msbuild file and reload the project. Create two config files called App.config and App.Debug.config. In the properties for each ensure their Build Action is set to Content so that they get picked up by the TransformWebConfig target. Next open the App.Debug.config and modify the Configuration with a new namespace declaration:

http://schemas.microsoft.com/XML-Document-Transform

This will tell the TransformWebConfig target that this is an XDT transformation rather than a normal config file. Add a transform to the file too so that we can test that everything is working (I’m going to assume you know how to use the transforms if you’re reading this!).

Building the project

Now all we need to do is run the build. For this you’ll need to open the visual studio command prompt and navigate to the folder containing you project file. Enter the following (assuming your project file is called FakingWebConfigService.csproj):

msbuild FakingWebConfigService.csproj /t:TransformAppConfig

With any luck you should see a load of output messages followed by a green Build Succeeded message. Take a look at your project output and your transformed config file will be sat there – magic!

Tidying up solution explorer

We’ve got a nicely working solution now, but it would be nice if we could get the transforms to be a step down the ‘tree’ from the main config file like they do in web apps. This only requires a little bit of hacking so let’s do it now. Unload the project and open its msbuild file. Locate the Content node for each of your transform files and tell it to depend on the main App.config file:

<Content Include="App.Debug.config">
  <DependentUpon>App.config</DependentUpon>
</Content>

That’s it! Reload the project and your transforms should show up in the right place.

How to fire off the transformation process

There are a few options for when to fire off the transform process which I’ll go over briefly:

Keep running it from the command line

Perfectly reasonable, but it means each member of the team needs to know the target name and how to use the command line msbuild. It also means specifying configurations manually – you’d have to add a /p:Configuration=Release argument to the msbuild call to do a release version for example. But it works and it’s better than what we had before.

Run it on every build

We can make it run on every build simply by adding an AfterTargets attribute to our Target node in the msbuild file:

<Target Name="TransformAppConfig" AfterTargets="Build">

Again, this works fine and is more foolproof than running from the command line. My issue with it is that web projects don’t do this on build – only deploy. I think the inconsistency is asking for trouble, but I’m sure it works fine for some teams.

Run it as part of a TFS Build

You can get Team Build to run the TransformAppConfig target for you as part of a central build. This is much better than the previous two options (in my opinion anyway) and is easy to set up. I’ll assume that you know how to set up a new build definition (if not check out the first part of my post on Continuous Deployment). All you then need to do is go into the Process tab, drop down the advanced node and enter /t:TransformAppConfig /p:Configuration=YOURCONFIGURATION into the MsBuild Arguments box:

When you run the build now it will transform your configs for you.

Include it in your WiX installer

This is my favoured solution. It is perfectly possible to cause the config transformations to happen as part of the build cycle of a WiX installer. I’ll went through this technique in my next post on creating Windows Service installers in WiX which should be posted in the next week or so. If you’re new to WiX then get with the program and check out my introductory series on WiX and deployment.

Thank you ladies and gentlemen, and goodbye!

Hopefully this post has been of help to those of you frustrated by the partial config transformation implementation in Visual Studio 2010. If you’d like to be informed when the next step posts on this topic are released you could always subscribe to my RSS feed.

Source Code

Full source code is available here: FakingWebConfigService.zip (29.68 kb)

Did you find this post useful or interesting? I'd be really grateful if you could return the favour by clicking this google link:



Continuous Integration, its sooooo last week. If you’re thinking about getting into CI, you’re behind the curve. Seriously. Fear not though, it’s a simple matter to not only continuously integrate, but to continuously deploy your application (which, by the way, is a form of continuous improvement (But I’ll stop spouting now. (Honest))).

What is Continuous Integration

Continuous integration is a well established software development technique used by teams all over the world. Martin Fowler describes it better than I can:

“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”

Great! Solves problems of team members not running tests and helps to (though does not ensure) promote regular and small checkins to source control. It means that when I get latest from TFS I’m much more likely to get code that works. But it doesn’t solve all your problems.

What Continuous Integration Doesn’t Do

In our SCRUM team we have developers working on local copies of the system who benefit hugely from CI. But we also have product owners who need the latest copy of the application – NOW.

Consider Joanne, a non-fictional analyst on our team. She’s logged 15 bugs in the last couple of days (we were having a sloppy week, what of it?) of which I’ve fixed 10 – resolving them on checkin. These are all sat happily in our latest CI build so get propagated to all the other developers PCs, but when Joanne goes to check them on the test server everything is still broken! So she comes over to me complaining that she can’t get on with her work until I put the latest build on the test server. I drop what I’m doing to deploy it and we both end up wasting time and effort, not to mention getting frustrated at each other.

Continuous Deployment

Enter continual deployment. In this fancy new version of CI, each integrated build is deployed somewhere. In the example above we will always deploy to the test server. This has the same benefits for all the non-dev team members as CI had for the devs:

  • The version of the system they see is the same as the developers see;
  • Fixes get to them for testing faster;
  • If the work item in TFS says it's fixed, it will be on the test server.

Now, that somewhere doesn’t have to be the test server. The holy grail of CD in fact is to be continually deploying to the live environment. Yeah, right, I can hear you saying. You remember the days before unit testing (you don’t? You lucky, lucky thing) and you can remember the frantic dash to fix issues deployed to live (or even the fixes done on live back in the days of classic asp) before anyone important notices. But these are not those days. These days we have automated unit tests, automated integration tests, automated UI tests! Most importantly we have well defined acceptance criteria for your product backlog items, allowing you to create those tests in confidence.

I’m sure that you looked at that and shook your head; it’s a rare team that pulls all those things together well enough that they can be confident enough to deploy to live on each checkin. It can be done though, flickr have been continually deploying to live for a long time now, and their site isn’t exactly a piece of crap. Generally though, it’s an ideal to work towards which can streamline and speed up your development process.

I’m Convinced Already, Just Show Me How To Do It!

OK, OK, here we go then. Let’s start by learning to crawl.

Setting Up Continuous Integration

I’m assuming a few prerequisites here:

  1. You are running on TFS 2010;
  2. You have permission to create build definitions;
  3. Your code is in source control!
  4. You have a series of unit tests in a test project.

First, we set up the build definition. In team explorer, open up your team project and its Builds node. Right click and select New Build Definition:

The dialog that pops up has 6 panes, the first of which asks for a name and description. Ignore the next pane and click into Workspace which, if your environment is anything like ours, will contain quite a few working folders. Generally you will only need one workspace here for the project you are building, so delete the rest:

Next up is the Build Defaults pane; in here select a build controller (don’t have one? Take a look here) and specify a network path to copy the output of the build to. Even if you don’t intend to save every build it’s a good idea to do this at first while you test the process.

Now move onto the Process pane, this is where you customise your build when you start doing funky stuff like deployment. For now though just change the Projects To Build item to point to the project file of your web project and add a configuration to build. Finally, change the Automated Tests item so that the wildcard search will pick up your test project:

The final pane contains settings for how many previous builds to keep. In general I turn this number right down, as you can always build a previous changeset anyway, but its personal preference.

Save your new build definition and it will appear in team explorer. Right click it and select Queue New Build. This will fire up the Build Explorer window and will display your build in it. After a while it should show up as finished. Click into it now.

The screen in front of you is truly a wonderful creation. At the top is information about how long the build took, followed by information on any warnings or errors found during the build (there were errors? Go fix your build now, we’ll wait!).  If you check the output folder you specified you should see your built project.

The last step is to have this happen on every checkin. Edit your build definition and on the Trigger tab select Continuous Integration. That’s it! Now do a checkin of your code against a work item and check the Build Explorer – you should see a new build kicking off. Once this is done have a look at the results. At the bottom you should see something like this:

Cool right? You’ve got a changeset associated with your build and an associated work item. You might also have some impacted tests.

Awesome, we’re continually integrating. Next up, let’s appease Joanne.

Setting Up Continuous Deployment

The technology we’ll be using for our deployment is MsDeploy. This is a service which will integrate with IIS 6 or 7 and allow you to automatically upload your application to a virtual directory (or even create one). The first step is to get it installed and started on your test server. Take a look at the installation walkthrough on the Microsoft IIS site for help with that.

All installed? Great, let’s get deploying! Right click on your web project in solution explorer and select ‘Publish’. You should see a screen similar to this (you may have to select ‘New…’ in the top combo):

You’ll need to use your own settings here obviously! The account you use will need to have permissions to edit the virtual directory (or local admin for the entire box if you’re on IIS6). Click ‘Publish’ and with a little luck you’ll get a Publish Succeeded’ message in the status bar.

Take a look at IIS now and you should see a virtual directory for TestSite. You can set up all the settings for this such as framework version, default pages etc and they will be remembered on subsequent publishes (it is actually possible to package these settings up on IIS7 but its beyond the scope of this post). Visit the site in the browser and ensure it works.

All that remains is to get this deploy happening as part of the CI build. For this we will need a few command arguments using the same values as in the dialog:

/p:DeployOnBuild=True
/p:DeployTarget=MsDeployPublish
/p:MSDeployPublishMethod=RemoteAgent
/p:CreatePackageOnPublish=True
/p:DeployIisAppPath="WebRoot/TestSite"
/p:MsDeployServiceUrl=MyTestServer
/p:username=zg\c6029126
/p:password=MyPassword

If you don’t want a user and password including in this string you can just give the account used by the TFS build server permissions and leave those items out of the command string.

Now edit the build definition for the build we created earlier. In the Process tab expand the Advanced section and drop your command string in the MsBuild Arguments field:

We’re done! Save the build definition and check in some code. When the build finishes check your test server – the new version should have been deployed.

What Next?

What we’ve achieved here is great but there is plenty more we can do to streamline our processes. The obvious miss here is the database side of the equation – you might well be developing the database in line with the application, but currently it will need to be deployed manually each time. I’ll be writing soon about how you can deploy databases in this way, as well as how to customise your build process to deploy multiple webs to different locations on every checkin. If you’d like to be notified when I add a new post, you could always subscribe to my feed.

Did you find this post useful or interesting? I'd be really grateful if you could return the favour by clicking this google link:



Month List

Sign in

Top Programming Sites Vote for me at Fuelmyblog