DRY your CruiseControl.NET Configuration

Don’t Repeat Yourself (DRY) is one of the principles of good software development. The idea is that there should ideally be one and only one “source of knowledge” for a particular fact or calculation in a system. Basically it comes down to not copying-and-pasting code around or duplicating code if at all possible. The advantages of this are many.

Advantages of DRY

  • There will be less code to maintain
  • If a bug is found, it should only have to be fixed in one place
  • If an algorithm or process is changed, it only needs to be changed in one place
  • More of the code should become reusable because as you do this you will parameterize methods to make them flexible for more cases

If it’s good for code isn’t it good for other things like configuration? Why yes it is.

Using CruiseControl.NET Configuration Builder

The Configuration Preprocessor allows you to define string properties and full blocks of XML to use for substitution and replacement. To start using the Configuration Preprocessor, you add xmlns:cb=”urn:ccnet.config.builder”, an xml namespace, to your document to tell the config parser that you plan to do this.

From there you can define a simple property like:

<cb:define client="xxx"/>

Or you can make it a full block of XML:

<cb:define name="svn-block">
    <sourcecontrol type="svn">
        <trunkUrl>http://svn.example.com/svn/$(client)/$(project)/trunk</trunkUrl>
        <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
        <executable>svn.exe</executable>
        <autoGetSource>true</autoGetSource>
    </sourcecontrol>
</cb:define>

Defining Reusable Blocks

Using these ideas I wanted to come up with a templated approach that would allow me to share configuration among multiple projects. That way, if I added new statistics or change the layout of my build server, I would only have to change it in a single place. Thus keeping things DRY. It also encourages more consistency across multiple projects making things easier to understand.

So, I started defining some reusable blocks in the main ccnet.config file which you can see below. The exact details will depend on your configuration of course.

Full Example of config.xml

<!--
How to add a new project:
Step 1. Create a config file named "<config>-project.xml"
Step 2. Add the project reference below
-->
 
<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
 
    <!-- cb defines to compose reusable blocks of configuration -->
    <!-- use <cb:define client="xxx"/> and <cb:define project="yyy"/> to use these -->
 
    <cb:define name="svn-block">
        <sourcecontrol type="svn">
            <trunkUrl>http://svn.example.com/svn/$(client)/$(project)/trunk</trunkUrl>
            <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
            <executable>svn.exe</executable>
            <autoGetSource>true</autoGetSource>
        </sourcecontrol>
    </cb:define>
 
    <cb:define name="msbuild-20-block">
        <msbuild>
            <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
            <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
            <projectFile>build.proj</projectFile>
            <buildArgs>$(build-args) </buildArgs>
            <targets>$(build-targets)</targets>
            <timeout>600</timeout>
            <logger>D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll</logger>
        </msbuild>
    </cb:define>
 
    <cb:define name="msbuild-35-block">
        <msbuild>
            <executable>C:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.exe</executable>
            <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
            <projectFile>build.proj</projectFile>
            <buildArgs>$(build-args) </buildArgs>
            <targets>$(build-targets)</targets>
            <timeout>600</timeout>
            <logger>D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll</logger>
        </msbuild>
    </cb:define>
 
    <cb:define name="merge-block">
        <!-- Merge the output of tests, code coverage and fxcop -->
        <merge>
            <files>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.Test.xml</file>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageMerge.xml</file>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageSummary.xml</file>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.FxCop.xml</file>
            </files>
        </merge>
    </cb:define>
 
    <cb:define name="loggers-block">
        <xmllogger>
            <logDir>D:\Builds-Net\projects\$(client)\$(project)\logs</logDir>
        </xmllogger>
        <rss/>
        <modificationHistory  onlyLogWhenChangesFound="true" />
    </cb:define>
 
    <cb:define name="stats-block">
        <statistics>
            <statisticList>
                <firstMatch name="Svn Revision" xpath="//modifications/modification/changeNumber" />
                <firstMatch name="Coverage" xpath="//coverageReport/project/@coverage" generateGraph="true"/>
                <firstMatch name="Warnings" xpath="//msbuild/@warning_count" generateGraph="true"/>
                <firstMatch name="Errors" xpath="//msbuild/@error_count" generateGraph="true"/>
 
                <!-- NDepend -->
                <!--
                <firstMatch name="ILInstructions" xpath="//ApplicationMetrics/@NILInstruction" />
                <firstMatch name="Avgerage Complexity" xpath="//ApplicationMetrics/MethodCC/@Avg" />
                <firstMatch name="Max Complexity" xpath="//ApplicationMetrics/MethodCC/@MaxVal" />
                <firstMatch name="LinesOfCode" xpath="//ApplicationMetrics/@NbLinesOfCode" generateGraph="true"/>
                <firstMatch name="LinesOfComment" xpath="//ApplicationMetrics/@NbLinesOfComment" generateGraph="true"/>
                -->
            </statisticList>
        </statistics>
    </cb:define>
 
    <cb:include href="config-client-project.xml"/>
    <cb:include href="config-client2-project-trunk.xml"/>
</cruisecontrol>

At the end of the file you can see the cb:include references. Those are one-line includes to include the configuration of each project. This makes things easier to manage, I think, because you only have to look at the individual project configuration.

Using Reusable Blocks in Individual Configuration Files

From there I need to make use of those defined blocks in in individual file. The first thing I needed to do was to set the parameters that I had defined as simple string replacements in the reusable blocks. Normally you would do that with cb:define as I showed above. But the trick is that you can only have one property with a given name defined. If you include multiple project configurations that doesn’t work. What does work is using cb:scope definitions. This allows for a value to be defined only within a specific scope.

<cb:scope 
         client="ExampleClient" 
         project="SpecialProject"
         build-args="/p:Configuration=Debug"
         build-targets="Clean;Test">
 ...
</cb:scope>

From there you just need to start including the blocks that you defined in the main ccnet.confg within the scope block.

Full Example of Project Configuration

 
<!-- CruiseControl.NET configuration -->
<project name="ExampleClient SpecialProject" xmlns:cb="urn:ccnet.config.builder">
 
    <cb:scope 
         client="ExampleClient" 
         project="SpecialProject"
         build-args="/p:Configuration=Debug"
         build-targets="Clean;Test">
 
        <cb:svn-block/>
 
        <tasks>
            <cb:msbuild-35-block/>
        </tasks>
 
        <publishers>
 
            <cb:merge-block/>
 
            <!-- Enable collection of project statistics -->
            <cb:stats-block/>
 
            <cb:loggers-block/>
 
            <email mailhost="smtp.example.com" from="ccnet@example.com" includeDetails="true">
                <users>
                    <user name="Developer One" group="buildmaster" address="dev1@example.com" />
                    <user name="Developer Two" group="developers" address="dev2@example.com" />
                </users>
                <groups>
                    <group name="developers" notification="change" />
                    <group name="buildmaster" notification="change" />
                </groups>
            </email>
        </publishers>
    </cb:scope>
</project>

As you can see, the only one I didn’t template out was the email block because that depends on the developers working on each project.

Have fun bringing simplicity and consistency to your Cruise Control.NET configuration!

For the full details see the CruiseControl.NET Configuration Preprocessor documentation.

CruiseControl With a Specific Version of Grails

Continuous Integration is a good practice in software development. It helps catch problems early to prevent them from becoming bigger problems later. It helps to reinforce other practices like frequent checkins and unit testing as well. I’m using CruiseControl (CC) for Continuous Integration at the moment.

One of the things about Grails is that it is really run through a series of scripts and classes that set up the environment. The Ant scripts really just delegate the work to those grails scripts. To run properly, the GRAILS_HOME environment needs to be set so that it can find the proper classes, etc. This is not a problem if you are running a single Grails application in Continuous Integration. The issue arises when you want to run multiple against different version of Grails. A project I’m working on uncovered a bug in the 1.0.2 release of Grails. The code worked fine on 1.0.1 so I wanted to run against that specific version of Grails.

It ends up this is not to hard with a few small changes to your Ant build.xml file.

First you can declares some properties that have the paths to the Grails directory and the grails executable (the .bat version if your CC server is on Windows).

<property name="cc-grails.home" value="C:\grails-1.0.1" />
<property name="cc-grails" value="${cc-grails.home}\bin\grails.bat" />

Next you can declare a custom target to execute on the CC server. You reference the ‘cc-grails’ property declared. The key is that you must override the GRAILS_HOME when you execute the grails script.

<target name="cc-test" description="--> Run a Grails applications unit tests">
    <exec executable="${cc-grails}" failonerror="true">
        <env key="GRAILS_HOME" value="${cc-grails.home}"/>
	<arg value="test-app"/>
    </exec>                               
</target>

Now the Continuous Integration of your Grails app runs against a specific version of Grails.

The Full build.xml

<project name="project" default="test">
 
    <condition property="grails" value="grails.bat">
        <os family="windows"/>
    </condition>
    <property name="grails" value="grails" />
    <property name="cc-grails.home" value="C:\grails-1.0.1" />
    <property name="cc-grails" value="${cc-grails.home}\bin\grails.bat" />
 
	<!-- ================================= 
          target: clean              
         ================================= -->
    <target name="clean" description="--> Cleans a Grails application">
		<exec executable="${grails}" failonerror="true">
			<arg value="clean"/>
		</exec>                               
    </target>
 
	<!-- ================================= 
          target: war              
         ================================= -->
    <target name="war" description="--> Creates a WAR of a Grails application">
		<exec executable="${grails}" failonerror="true">
			<arg value="war"/>
		</exec>                               
    </target>
 
	<!-- ================================= 
          target: test              
         ================================= -->
    <target name="test" description="--> Run a Grails applications unit tests">
		<exec executable="${grails}" failonerror="true">
			<arg value="test-app"/>
		</exec>                               
    </target>
 
    <!-- ================================= 
      target: cc-test              
     ================================= -->
    <target name="cc-test" description="--> Run a Grails applications unit tests in Continuous Integration mode">
		<exec executable="${cc-grails}" failonerror="true">
            <env key="GRAILS_HOME" value="${cc-grails.home}"/>
			<arg value="test-app"/>
		</exec>                               
    </target>
 
	<!-- ================================= 
          target: deploy              
         ================================= -->
    <target name="deploy" depends="war" description="--> The deploy target (initially empty)">
        <!-- TODO -->
    </target>
</project>

Pragmatic Project Automation

In a previous post, I wrote a review of Pragmatic Version Control Using Subversion which covers effectively using version control systems on software development projects. This entry covers another book by the Pragmatic Programmers in their Pragmatic Starter Series.

Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps

Pragmatic Project Automation by Mike Clark

Index:

  1. Introduction
  2. One-Step Builds
  3. Scheduled Builds
  4. Push-Button Releases
  5. Installation and Deployment
  6. Monitoring

Pragmatic Project Automation shares a lot with the rest of the Pragmatic Starter Series. As the name implies these books are intended to get you started down the path without necessarily showing you the entire way. The book is relatively short coming in at a slim 152 pages. This makes it a quick read and allows it to cover the topic from a high level. It sells the idea of automation and shows how to do automation in all aspects of software development. It shows examples everything from the simplest scripts using Unix shell and Windows tools all the way up to purpose-build programs that will make automation very easy for larger and more complex projects.

Chapter 1 – Introduction

The introduction lays the groundwork for automation. It categorizes automation into 3 types:

  • Scheduled Automation
  • Triggered Automation
  • Commanded Automation

Commanded Automation is probably the most familiar to most Java developers. This is the idea that you run a small script or an Ant build. This kind of automation is user driven and run when they want to do a certain task. Scheduled Automation, as it’s name implies, is something that is run based on time while Triggered Automation is automation that happens in response to a change in environment such as checking in code to a version control system. This groudwork is built upon during the rest of the book.

Chapter 2 – One-Step Builds

This is the most common kind of automation from what I’ve seen. This chapter covers building source code using a script. Being a Pragmatic book, it starts with a simple example using a shell script. It then does a good job of building a good template for an Ant build. One of the interesting things in this chapter is that it discusses using Groovy, a Java based scripting language, that has built-in support for Ant scripting. In many ways this Groovy example is very interesting because it’s a real scripting language so you can easily create your own custom code without the overhead of creating an Ant task. Maybe JRuby will catch on here (hint, hint JRuby folks) as I like the Ruby syntax a lot more than Groovy.

Chapter 3 – Scheduled Builds

Chapter 3 starts with the easiest way of doing a scheduled build, using cron on Unix or at on Windows to periodically run a build script. I really like the fact that the authors mentioned this because I really think it lowers the barrier to people getting into automation. There’s no need to over engineer a solution if you have simple needs. Of course this simple technique is fairly limited so they don’t stop there.

This chapter also covers using CruiseControl for doing both Scheduled and Triggered builds. It walks through the setup of your first project in CruiseControl. I’m a huge fan of Continuous Integration because it’s an always on backup for the processes that your team wants to enforce.

Chapter 4 – Push-Button Releases

Doing a build which you do everyday, hopefully multiple times a day, is one thing, but how do you move code from source control to QA, a staging server or a production server? Well you automate it of course! This chapter covers creating a script that you can use to turn your code into a form you can send to a customer. The simple examples are just creating the proper artifacts and creating archives like Zip or Tar files. It also talks about a number of processes including using automation to manage branching and tagging in your version control system. The branching and tagging automation not only makes the “release manager’s” life easier, it really helps to create a consistent process that means you’ll always be able to find and recreate any version in the future.

Chapter 5 – Installation and Deployment

This chapter covers 2 slightly unrelated (to me) but interesting ideas. The most obvious based on the name is creating an installer. It discusses everything from a simple Zip file “install” to using a full-blown installation system. The book uses the Open Source Nullsoft Scripted Installer (NSIS) as an example of building a Windows install. I’ve never used NSIS before so I can’t speak for its strengths or weaknesses, but it is used in a lot of Open Source projects. The introduction is very brief, so plan on doing some other reading if you want to use Nullsoft.

Chapter 6 – Monitoring

To wrap the book up, we get the Make chapter of the book. Now that you have triggered and scheduled builds running and applications in production, how do you know when there is a problem? This chapter talks about a number of good ideas starting with the easy ones of email and cell phones but taking some unanticipated turns into things like RSS feeds as well.

Some of the more interesting ideas where using custom applications to drive visual feedback monitors. Two of the more interesting examples where using Lava Lamps and Ambient Orbs to signal when a build is broken. The Lava Lamps would sit in the office and turn on/off red or green Lava Lamps based on the build status. Even more interesting is the Ambient Orb. It can glow in multiple colors. It communicates over a national wireless network like the pager network. So now you can take your magic orb with you and it will notify you when there’s a problem with the build.

The only problem with this chapter is that it really doesn’t have any implementations for these cool ideas. It leaves a lot up to the imagination.

Conclusion

The book is 152 pages, as I said above, which means that it is by no means a comprehensive reference to any of the topics covered. If you are already sold on the idea of automation, this would be good ammunition to convince other people on a team or in management to support the idea. If you are completely unfamiliar with the topic, then this book will offer you a good basis for getting started. If you are already familiar with the topic and are looking to expand on what you know or get ideas for more ways to automate then you probably won’t get a lot out of this book and should probably look elsewhere.

Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps

Other Related

Ant
Cruise Control
Java Development with Ant

Continuous Integration Revisited

I had a chance to install and play with JetBrain’s new Team City beta today. Team City at its most basic is a Continuous Integration server. Continuous Integration (CI) systems are used to help manage a team’s software development process. Every time a developer checks in code, the CI server will check out the code from the source control system, compile the code, run unit test and perform static analysis and code coverage. This is a great way to back up your development process to ensure that you always have a working source tree and development is never held up because of mistakes.

There are other CI systems available, for example, in the past, I’ve used and liked CruiseControl. It is a very capable system that uses Ant build files to compile and test your code. It’s also Open Source and free.

I want to look at a couple of things that I really liked about Team City that make it shine.

Team City Configuration

The first part is the installation and configuration. It’s dead simple.

Team City is distributed as a WAR file, so if you already have a Servlet container like Tomcat up and running it’s as easy as dropping the WAR file into the webapps directory and deploying it through the management interface.

After the installation all of the configuration of a project can be done directly through the web interface, which is very nice compared to hand editing XML files that is often the case in other CI systems like CruiseControl.

The only gotcha is that in addition to the Team City itself, you have to install one or more Build Agents (this is really a feature, more on that later). I skipped this step at first and of course nothing would compile. Directly from the web application you can run either a Java Web Start or a Windows Installer application that will install a Build Agent on the computer. In my case I used the same computer where I was installing Team City – in a more intensive environment, it might make sense to install it on a different machine.

Multiple Build Agents

While it’s an extra step at the beginning, having multiple Build Agents is a really great idea. You can use many machines to build your projects spreading the load around to different machines. This is especially great if you have big project with lots of unit tests (you do have lots of unit tests right?) or just a bunch of projects and configurations that you would like to build. As software developers know, compiling code and running tests is a pretty intensive activity. Spreading the load around means that you can do more concurrent builds. This is a great way to decouple where things get compiled and tested from the centralized management application. This way you don’t have to have multiple instances of the CI system running on different machines if you want to load balance them.

Multi-Platform

Not only is the server itself multi-platform, thanks to it being written in Java, but Team City can compile and test both Java and .NET code. It can use a number of different configurations for compiling and testing. In Java, it can currently use Ant, Maven and Idea build files. On the .NET side of things, it can use MSBuild files and Visual Studio 2005 Solution files.

In addition to compiling code for these platforms, it can use either JUnit or NUnit for running Unit Tests as well. I really like this aspect of it because it offers shops that do more than one language a way to use the same systems for all of their development. At work we do both .NET development and Java development for our clients, so this makes a lot of sense for us. At least in the consulting world, this is a big win.

Delayed Commits

Team City has IDE integration built in. Currently it only supports Idea (why would you need anything else?) but they have already announced that for the 1.5 release they will support Eclipse, Netbeans and Visual Studio. There are a number of features that you need to have the Plugin installed to use, so for the short term Team City will be an OK solution, but not a great solution for non-Idea users.

Delayed Commits is quite possibly the best idea in Team City. It basically allows you to “commit” your changes to the server, but only have them checked into the source control repository if they compile and test correctly. No longer do you have to wait around for the CI server to report back that your checkin was successful before going to lunch or going home for the night. You can now Delay Commit your code and if everything works out it will be checked in for all the other developers to pull down, if not no one will be the wiser and you will fix any issues when you get back to your workstation.

Buttery

The UI is also very nice. They effectively use AJAX techniques to have a really nice user experience.

While this product is in Beta still and I haven’t had a chance to use it in anger yet, I think it’s a very promising tool. JetBrains has a great track record in delivering excellent products. I’m a huge fan of Idea and I can’t do .NET development without ReSharper. I think they have another hit on their hands with Team City. I haven’t heard anything about pricing yet, but the JetBrains stuff seems to usually be affordable.

As an aside, I have no intention of taking anything away from CruiseControl. It is still a great application as well (did I mention it was free).

(Jet Brains people, Yes, I’m willing to take paid endorsements :))