Using Quartz.NET, Spring.NET and NHibernate to run Scheduled Tasks in ASP.NET

Running scheduled tasks in web applications is not normally a straightforward thing to do. Web applications are built to respond to requests from users and respond to that request. This request/response lifecycle doesn’t always match well to a long running thread that wakes up to run a task every 10 minutes or at 2 AM every day.

ASP.NET Scheduled Task Options

Using ASP.NET running on Windows, there are a number of different options that you could choose to implement this. Windows built in Scheduled Tasks can be run to periodically perform execute a program. A Windows Service could be constructed that used a Timer or a Thread to periodically do the work. Scheduled Tasks and Windows Service require you to write a standalone program. You can share DLLs from your Web application but in the end it is a separate app that needs to be maintained. Another option if you go this route is to turn the Scheduled Task or Service being run into a simple Web Service or REST client that can call your Web application but doesn’t need any knowledge of the jobs themselves.

Another option is an Open Source tool called Quartz.NET. Quartz.NET is based on the popular Java scheduled task runner called (not surprisingly) Quartz. Quartz.NET is a full-featured system that manages Jobs that do the work and Triggers that allow you to specify when you want those jobs run. It can run in your web application itself or as an external service.

The simplest approach to get started is to run directly in your Web application as a process in IIS. The downside to this is that IIS will periodically recycle it’s processes and won’t necessarily start a new one until a new web request is made. Assuming you can deal with this indeterministic behavior then in an IIS process will be fine. It also creates a relatively easy path that will allow you to migrate to the external service process at a later point if need be.

I’m an ALT.NET kind of .NET developer, so I like to use tools like NHibernate for ORM and Spring.NET for Dependency Injection, AOP and generally wiring everything together. The good news is that Spring.NET supports Quartz.NET through its Scheduling API. Start with that for some basic information on using Quartz.NET with Spring. The bad news is that the documentation is a bit thin and the examples basic. I attempt to remedy that in part here.

Using Quartz.NET, NHibernate and Spring.NET to run Scheduled Tasks

The goal is to integrate an existing Spring managed object like a Service or a DAL that uses NHibernate with a Quartz Job that will run on a periodic basis.

To start with you need to create an interface for your service and then implement that interface. The implementation I’ll leave to you and your problem, but the example below you can image uses one or more NHibernate DALs to lookup Users, find their email preferences, etc.

Implementing Services and Jobs

public interface IEmailService
{
    void SendEveryoneEmails();
}

When implementing your Job you need to know a few details about how Quartz works:

  1. The first thing to understand is that if you are going to use the AdoJobScheduler to store your Jobs and triggers in the database the Job needs to be Serializable. Generally speaking your DAL classes and NHibernate sessions and the like are not going to be serializable. To get around that, we make the properties set-only so that they will not be serialized when they are stored in the database.
  2. The second thing to understand is that your Job will not be running in the context of the Web application or a request so anything you have to set up connections (such as an OpenSessionInView filter) will not apply to Jobs run by Quartz. This means that you will need to setup your own NHibernate session for all of the dependent objects to use. Luckily Spring provides some help with this in the SessionScope class. This is the same base class as is used by the OpenSessionInView filter.

Using the Service interface you created, you then create a Job that Quartz.NET can run. Quartz.NET provides the IJob interface that you can implement. Spring.NET provides a base class that implements that interface called QuartzJobObject helps deal with injecting dependencies.

using NHibernate;
using Quartz;
using Spring.Data.NHibernate.Support;
using Spring.Scheduling.Quartz;
 
public class CustomJob : QuartzJobObject
{
    private ISessionFactory sessionFactory;
    private IEmailService emailService;
 
    // Set only so they don't get serialized
    public ISessionFactory SessionFactory { set { sessionFactory = value; } }
    public IEmailService EmailService { set { emailService = value; } }
 
    protected override void ExecuteInternal(JobExecutionContext ctx)
    {
        // Session scope is the same thing as used by OpenSessionInView
        using (var ss = new SessionScope(sessionFactory, true))
        {
            emailService.SendEveryoneEmails();
            ss.Close();
        }
    }
}

Wiring Services and Jobs Together with Spring

Now that you have your classes created you need to wire everything together using Spring.

First we have our DALs and Services wired in to Spring with something like the following:

<object id="UserDAL" type="MyApp.DAL.UserDAL, MyApp.Data">
  <property name="SessionFactory" ref="NHibernateSessionFactory" />
</object>
<object id="EmailService" type="MyApp.Service.EmailService, MyApp.Service">
  <property name="UserDAL" ref="UserDAL" />
</object>

Next you create a Job that references the Type of the Job that you just created. The type is referenced instead of the instance because the lifecycle of the Job is managed by Quartz itself. It deals with instantiation, serialization and deserialization of the object itself. This is a bit different than what you might expect from a Spring service normally.

<object id="CustomJob" type="Spring.Scheduling.Quartz.JobDetailObject, Spring.Scheduling.Quartz">
    <property name="JobType" value="MyApp.Jobs.CustomJob, MyApp.Jobs" />
</object>

Once your Job is created, you create a Trigger that will run the Job based on your rules. Quartz (and Spring) offer two types of Jobs SimpleTriggers and CronTriggers. SimpleTriggers allow you to specify things like “Run this task every 30 minutes”. CronTriggers follow a crontab format for specifying when Jobs should run. The CronTrigger is very flexible but could be a little confusing if you aren’t familiar with cron. It’s worth getting to know for that flexibility though.

<object id="CustomJobTrigger" type="Spring.Scheduling.Quartz.CronTriggerObject, Spring.Scheduling.Quartz">
    <property name="JobDetail" ref="CustomJob"/>
    <property name="CronExpressionString" value="0 0 2 * * ?" /> <!-- run every morning at 2 AM -->
    <property name="MisfireInstructionName" value="FireOnceNow" />
</object>

The last piece that needs to be done is the integration of the SchedulerFactory. The SchedulerFactory brings together Jobs and Triggers with all of the other configuration needed to run Quartz.NET jobs.

A couple of things to understand about configuring the SchedulerFactory:

  1. Specifying <property name=”DbProvider” ref=”DbProvider”/> (where DbProvider is the db:provider setup used by your Nhibernate configuration) tells the SchedulerFactory to use the AdoJobProvider and store the Jobs and Trigger information in the database. The tables will need to exist already and Quartz provides a script for this task.
  2. Running on SQL Server requires a slight change to Quartz. It uses a locking mechanism to prevent Jobs from running concurrently. For some reason the default configuration uses a FOR UPDATE query that is not supported by SQL Server. (I don’t understand exactly why a .NET utility wouldn’t work with SQL Server out of the box?)
    To fix the locking a QuartzProperty needs to be set:
    <entry key=”quartz.jobStore.selectWithLockSQL” value=”SELECT * FROM {0}LOCKS WHERE LOCK_NAME=@lockName”/>
  3. The JobFactory is set to the SpringObjectJobFactory because it handles the injection of dependencies into QuartzJobObject like the one we created above.
  4. SchedulerContextAsMap is a property on the SchedulerFactory that allows you to set properties that will be passed to your Jobs when they are created by the SpringObjectJobFactory. This is where you set all of the Property names and the corresponding instance references to Spring configured objects. Those objects will be set into your Job instances whenever they are deserialized and run by Quartz.

Here’s the whole ScheduleFactory configuration put together:

<object id="SchedulerFactory" type="Spring.Scheduling.Quartz.SchedulerFactoryObject, Spring.Scheduling.Quartz">
    <property name="JobFactory">
        <object type="Spring.Scheduling.Quartz.SpringObjectJobFactory, Spring.Scheduling.Quartz"/>
    </property>
    <property name="SchedulerContextAsMap">
        <dictionary>
            <entry key="EmailService" value-ref="EmailService" />
            <entry key="SessionFactory" value-ref="NHibernateSessionFactory" />
        </dictionary>
    </property>
    <property name="DbProvider" ref="DbProvider"/>
    <property name="QuartzProperties">
        <dictionary>
            <entry key="quartz.jobStore.selectWithLockSQL" value="SELECT * FROM {0}LOCKS WHERE LOCK_NAME=@lockName"/>
        </dictionary>
    </property>
    <property name="triggers">
        <list>
            <ref object="CustomJobTrigger" />
        </list>
    </property>
</object>

Conclusion

Scheduled tasks in ASP.NET applications shouldn’t be too much trouble anymore. Reusing existing Service and DAL classes allows you to easily create scheduled tasks using existing, tested code. Quartz.NET looks to be a good solution for these situations.

Posted in .NET, Web | Tagged , , , , | 6 Comments

Using Ruby Subversion Bindings to Create Repositories

Subversion has a default Web UI that is served up by Apache if you run Subversion that way. It is pretty boring and read-only. Then there are things like WebSVN that make it less boring, but still read-only. I got curious about what it would take to make something even less boring and NOT read-only. Why not allow me to create a new repository on the server? Why not allow me to create a new default directory structure for a new project in a repository all through a web interface?

Parent Path Setup

All of my repositories use the idea of the SVNParentPath. This makes Apache assume that every directory under a given path is an SVN Repository. That structure makes it easy to deal with multiple repositories and secure them with a single security scheme. Using that assumption then it is easier to write some code that will list existing repositories and create new ones in a known location.

SvnAdmin With Ruby Subversion Bindings

Subversion provides language bindings for a number of different languages (Java, Python, Perl, PHP and Ruby) in addition to the native C libraries. Using these bindings it becomes fairly easy to deal with Subversion. The only hiccup will be dealing with the apparent lack of documentation for the code. So be prepared to do some exploration, digging and reading of the code.

I chose to try this using Ruby because it was quick and easy and it was a language I was already familiar with.

First you need to know how to create a new repository and open an existing repository. Fortunately those are simple, one-line operations:

Svn::Repos.create(repos_path, {})
repos = Svn::Repos.open(repos_path)

There was nothing special (from what I could tell) that would allow you to determine if a repository already existed, so I just created a simple function using the Ruby File operations to determine if a directory already existed. This code would allow me to determine if I needed to create new repository or open an existing one:

def repository_exists?(repos_path)
   File.directory?(repos_path)
end

Now I have a repository open so I wanted to build a project structure using the default conventions I use for Subversions projects. My convention is to have a repository named after a client, the top-level directories are named for the client’s project and then each project has the standard trunk, branches and tags within that. Depending on the kind of work you do that convention may or may not make sense for you.

With that decided, I created the code to write that structure in a repository. The one thing I found is that interacting with the Subversion repository allowed you to do things within a transaction that would force all of the changes to be recorded as a single commit. I thought this was a good thing, so performed these operations as a transaction:

txn = repos.fs.transaction
 
# create the top-level, project based directory
txn.root.make_dir(project_name)
 
# create the trunk, tags and branches for the new project
%w(trunk tags branches).each do |dir|
  txn.root.make_dir("#{project_name}/#{dir}")
end
 
repos.commit(txn)

Finally I put all of those things together into a class. The class had the concept of being initialized to a base Parent Path so all of the operations would know to start from that location:

require "svn/repos"
 
class SvnAdmin
   def initialize(parent_path)
     @base_path = parent_path
   end
 
   # Provides a list of directory entries. path must be a directory.
   def ls(client_name, path="/", revision=nil)
     repos_path = File.join(@base_path, client_name)
     repos = ensure_repository(repos_path)
 
     entries = repos.fs.root(revision).dir_entries(path)
     entries.keys
   end
 
   def create(client_name, project_name)
     repos_path = File.join(@base_path, client_name)
     repos = ensure_repository(repos_path)
 
     txn = repos.fs.transaction
     txn.root.make_dir(project_name)
     %w(trunk tags branches).each do |dir|
       txn.root.make_dir("#{project_name}/#{dir}")
     end
 
     repos.commit(txn)
   end
 
   private
   def ensure_repository(repos_path)
     if ! repository_exists?(repos_path)
       Svn::Repos.create(repos_path, {})
     end
     repos = Svn::Repos.open(repos_path)
   end
 
   def repository_exists?(repos_path)
     File.directory?(repos_path)
   end
 
end

SvnAdmin from Rails

Now that I had some simple code to create new repositories or add a new project to an existing repository I decided to wrap it in a simple Rails application that would allow me to create repositories using a web-based interface.

To start with, I’m not going to use a database or any ActiveRecord classes in this project (which you might do if you wanted authentication or something else) so I disabled ActiveRecord in the config/environment.rb

config.frameworks -= [ :active_record ]

Then I created an ReposController to manage the Subversion repositories. This controller contains a couple of simple actions:

  1. An index action to list the existing repositories (directories)
  2. A new action to display a form to enter the client and project names
  3. A create action to use the SvnAdmin class to create a new repository and/or project
require "svnadmin"
 
class ReposController < ApplicationController
  layout 'default'
 
  def index
    @dirs = Dir.new(SVN_PARENT_PATH).entries.sort.reject {|p| p == "." or p == ".."}
  end
 
  def new
  end
 
  def create
    svn = SvnAdmin.new(SVN_PARENT_PATH)
    repos = params[:repos]
 
    respond_to do |format|
 
      begin
        svn.create(repos[:client_name], repos[:project_name])
        flash[:notice] = "Successfully created."
        format.html { redirect_to :action => "index" }
        format.xml  { head :ok }
      rescue
        flash[:error] = "Failed to create structure."
        format.html { redirect_to :action => "index" }
        format.xml  { render :xml => object.errors, :status => :unprocessable_entity }
      end
    end
  end
end

You can also easily create a route and a ProjectsController that allows you to see all of the projects within a repository.

The route in config/routes.rb is simply:

  map.connect ':repos/projects/',
      :controller => 'projects',
      :action => 'index'

And the ProjectsController looks up the :repos param to open the proper repository and list the top-level directories with it:

require "svnadmin"
 
class ProjectsController < ApplicationController
  layout 'default'
 
  def index
    repos_path = params[:repos]
    svn = SvnAdmin.new(SVN_PARENT_PATH)
    @projects = svn.ls(repos_path)
  end
end

Hopefully that will help you handle your Subversion administration. It should let you code up your conventions so that they are followed whenever a new repository is created or a new project is started.

Posted in Code, Ruby | Tagged , , | 1 Comment

DRY your CruiseControl.NET Configuration

Don’t Repeat Yourself (DRY) is one of the principles of good software development. The idea is that there should ideally be one and only one “source of knowledge” for a particular fact or calculation in a system. Basically it comes down to not copying-and-pasting code around or duplicating code if at all possible. The advantages of this are many.

Advantages of DRY

  • There will be less code to maintain
  • If a bug is found, it should only have to be fixed in one place
  • If an algorithm or process is changed, it only needs to be changed in one place
  • More of the code should become reusable because as you do this you will parameterize methods to make them flexible for more cases

If it’s good for code isn’t it good for other things like configuration? Why yes it is.

Using CruiseControl.NET Configuration Builder

The Configuration Preprocessor allows you to define string properties and full blocks of XML to use for substitution and replacement. To start using the Configuration Preprocessor, you add xmlns:cb=”urn:ccnet.config.builder”, an xml namespace, to your document to tell the config parser that you plan to do this.

From there you can define a simple property like:

<cb:define client="xxx"/>

Or you can make it a full block of XML:

<cb:define name="svn-block">
    <sourcecontrol type="svn">
        <trunkUrl>http://svn.example.com/svn/$(client)/$(project)/trunk</trunkUrl>
        <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
        <executable>svn.exe</executable>
        <autoGetSource>true</autoGetSource>
    </sourcecontrol>
</cb:define>

Defining Reusable Blocks

Using these ideas I wanted to come up with a templated approach that would allow me to share configuration among multiple projects. That way, if I added new statistics or change the layout of my build server, I would only have to change it in a single place. Thus keeping things DRY. It also encourages more consistency across multiple projects making things easier to understand.

So, I started defining some reusable blocks in the main ccnet.config file which you can see below. The exact details will depend on your configuration of course.

Full Example of config.xml

<!--
How to add a new project:
Step 1. Create a config file named "<config>-project.xml"
Step 2. Add the project reference below
-->
 
<cruisecontrol xmlns:cb="urn:ccnet.config.builder">
 
    <!-- cb defines to compose reusable blocks of configuration -->
    <!-- use <cb:define client="xxx"/> and <cb:define project="yyy"/> to use these -->
 
    <cb:define name="svn-block">
        <sourcecontrol type="svn">
            <trunkUrl>http://svn.example.com/svn/$(client)/$(project)/trunk</trunkUrl>
            <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
            <executable>svn.exe</executable>
            <autoGetSource>true</autoGetSource>
        </sourcecontrol>
    </cb:define>
 
    <cb:define name="msbuild-20-block">
        <msbuild>
            <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable>
            <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
            <projectFile>build.proj</projectFile>
            <buildArgs>$(build-args) </buildArgs>
            <targets>$(build-targets)</targets>
            <timeout>600</timeout>
            <logger>D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll</logger>
        </msbuild>
    </cb:define>
 
    <cb:define name="msbuild-35-block">
        <msbuild>
            <executable>C:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.exe</executable>
            <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory>
            <projectFile>build.proj</projectFile>
            <buildArgs>$(build-args) </buildArgs>
            <targets>$(build-targets)</targets>
            <timeout>600</timeout>
            <logger>D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll</logger>
        </msbuild>
    </cb:define>
 
    <cb:define name="merge-block">
        <!-- Merge the output of tests, code coverage and fxcop -->
        <merge>
            <files>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.Test.xml</file>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageMerge.xml</file>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageSummary.xml</file>
                <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.FxCop.xml</file>
            </files>
        </merge>
    </cb:define>
 
    <cb:define name="loggers-block">
        <xmllogger>
            <logDir>D:\Builds-Net\projects\$(client)\$(project)\logs</logDir>
        </xmllogger>
        <rss/>
        <modificationHistory  onlyLogWhenChangesFound="true" />
    </cb:define>
 
    <cb:define name="stats-block">
        <statistics>
            <statisticList>
                <firstMatch name="Svn Revision" xpath="//modifications/modification/changeNumber" />
                <firstMatch name="Coverage" xpath="//coverageReport/project/@coverage" generateGraph="true"/>
                <firstMatch name="Warnings" xpath="//msbuild/@warning_count" generateGraph="true"/>
                <firstMatch name="Errors" xpath="//msbuild/@error_count" generateGraph="true"/>
 
                <!-- NDepend -->
                <!--
                <firstMatch name="ILInstructions" xpath="//ApplicationMetrics/@NILInstruction" />
                <firstMatch name="Avgerage Complexity" xpath="//ApplicationMetrics/MethodCC/@Avg" />
                <firstMatch name="Max Complexity" xpath="//ApplicationMetrics/MethodCC/@MaxVal" />
                <firstMatch name="LinesOfCode" xpath="//ApplicationMetrics/@NbLinesOfCode" generateGraph="true"/>
                <firstMatch name="LinesOfComment" xpath="//ApplicationMetrics/@NbLinesOfComment" generateGraph="true"/>
                -->
            </statisticList>
        </statistics>
    </cb:define>
 
    <cb:include href="config-client-project.xml"/>
    <cb:include href="config-client2-project-trunk.xml"/>
</cruisecontrol>

At the end of the file you can see the cb:include references. Those are one-line includes to include the configuration of each project. This makes things easier to manage, I think, because you only have to look at the individual project configuration.

Using Reusable Blocks in Individual Configuration Files

From there I need to make use of those defined blocks in in individual file. The first thing I needed to do was to set the parameters that I had defined as simple string replacements in the reusable blocks. Normally you would do that with cb:define as I showed above. But the trick is that you can only have one property with a given name defined. If you include multiple project configurations that doesn’t work. What does work is using cb:scope definitions. This allows for a value to be defined only within a specific scope.

<cb:scope 
         client="ExampleClient" 
         project="SpecialProject"
         build-args="/p:Configuration=Debug"
         build-targets="Clean;Test">
 ...
</cb:scope>

From there you just need to start including the blocks that you defined in the main ccnet.confg within the scope block.

Full Example of Project Configuration

 
<!-- CruiseControl.NET configuration -->
<project name="ExampleClient SpecialProject" xmlns:cb="urn:ccnet.config.builder">
 
    <cb:scope 
         client="ExampleClient" 
         project="SpecialProject"
         build-args="/p:Configuration=Debug"
         build-targets="Clean;Test">
 
        <cb:svn-block/>
 
        <tasks>
            <cb:msbuild-35-block/>
        </tasks>
 
        <publishers>
 
            <cb:merge-block/>
 
            <!-- Enable collection of project statistics -->
            <cb:stats-block/>
 
            <cb:loggers-block/>
 
            <email mailhost="smtp.example.com" from="ccnet@example.com" includeDetails="true">
                <users>
                    <user name="Developer One" group="buildmaster" address="dev1@example.com" />
                    <user name="Developer Two" group="developers" address="dev2@example.com" />
                </users>
                <groups>
                    <group name="developers" notification="change" />
                    <group name="buildmaster" notification="change" />
                </groups>
            </email>
        </publishers>
    </cb:scope>
</project>

As you can see, the only one I didn’t template out was the email block because that depends on the developers working on each project.

Have fun bringing simplicity and consistency to your Cruise Control.NET configuration!

For the full details see the CruiseControl.NET Configuration Preprocessor documentation.

Posted in .NET, Automation | Tagged , , | 1 Comment

StringBuilder and my Biggest Pet Peeve

What You Should Know About Strings

In both Java and .NET (and other languages) String objects are immutable. They don’t change. Period. If you “change” a String, it creates a new String. That includes String concatenation using a +

// One string created "foo"
String foo = "foo";
// foo exists, "bar" is created and the combination of foo and "bar" is a third string
String bar = foo + " bar";

Ok, if you don’t know this, fine. But if you don’t know this, why would you EVER use a StringBuilder?

Why Does StringBuilder Exist?

We know that Strings are immutable. If you need to do a bunch of string modification, concatenation, replacement – you will create a bunch of strings. Ok, great…why do I care? We care because we are told that creating a lot of Objects (and then later having to Garbage Collect them) is inefficient. To start with, I will guarantee right now that concatenating strings WILL NOT be the thing that prevents your application from performing. Guaranteed. Period.

Ok fine, it’s not going to be a problem. But you want to be a responsible coder and not do things that are intentionally inefficient if you can help it.

So you use a StringBuilder. StringBuilder is implemented internally as an array of characters. The code manages the allocation and copying of data to new arrays if the buffer gets filled. It sometimes over allocates the new buffer so that it has to perform allocations less often. You sacrifice a bit of memory overhead to avoid some Object creation and Garbage Collection later.

My Biggest Pet Peeve

Your use of StringBuilder is a premature optimization but probably a forgivable one.

So, WHY, OH WHY do you do this:

// One string created "foo"
StringBuilder sb = new StringBuilder();
sb.append("Foo: " + fooValue + " \n");
sb.append("Bar: " + barValue + "\n");

It makes me have violent thoughts. Please stop.

Update:
For some of the reasons why Strings are immutable, see this post on immutability and its positive qualities.

Posted in Code | Tagged , | 5 Comments

MSBuild Task for PartCover

<![Rant[

I continue to lament the dearth of option for Test Coverage in the .NET world.

In the Java world you have open source tools like Emma and Cobertura that are widely used and supported (and many more) as well as proprietary tools like Clover available.

.NET we have an open source NCover SF that requires you to do odd code instrumentation and is essentially dead it seams, another NCover which is proprietary and costs money and PartCover which is open source, but doesn't seem real active.

Don't get me wrong, NCover.org is a good option if you are willing to spend the money for it. But with a team of 30+ and a CI server, I'm not sure if I want to drop $10k on it. (NCover up until version 1.5.8 was Free Software (GPL) before it was closed. Who has the source and why haven't you forked it yet?)

]]>

If you’re not willing to pay that basically leaves PartCover. But of course you want to integrate your code coverage with your automated build. There was no support for MSBuild out of the box, so I decided to build it.

Creating an MSBuild Task

I need to do 2 things to run PartCover:

  1. Register the native PartCover.CorDriver.dll
  2. Execute the PartCover.exe with the proper options

Register Native DLL

To see the details on how to register a native DLL using .NET code so my earlier post Register and Unregister COM DLL from .NET Code.

Execute PartCover

The MSBuild framework provides a ToolTask base class whose whole purpose is for executing external command line tools. I used this as the base of the task.

1. ToolName

First you override the ToolName property to return the name of the EXE to run. Nothing special here, it’s just the executable name.

protected override string ToolName
{
    get { return "PartCover.exe"; }
}
2. Properties

Next to start build the task you go about defining all of the settings that a user will need to set to execute the task. You then create those as Properties on the class and they will be set by MSBuild. Start with the simple things that someone will need to pass to get the tool to execute properly. You can build from there for other properties. If possible give the properties sane defaults so that people don’t have to override them in their build file.

// ...
 
/// <summary>
/// The application to execute to get the coverage results.
/// Generally this will be your unit testing exe.
/// </summary>
public string Target
{
    get { return _target; }
    set { _target = value; }
}
 
/// <summary>
/// The arguments to pass to the <see cref="Target"/> executable
/// </summary>
public string TargetArgs
{
    get { return _targetArgs; }
    set { _targetArgs = value; }
}
 
public string WorkingDirectory
{
    get { return _workingDirectory; }
    set { _workingDirectory = value; }
}
 
// ...
3. Command Arguments

Then you need to override string GenerateCommandLineCommands() method. The whole purpose of this method is to construct any command line parameters that need to be passed to the ToolName command using the Properties defined in the task.

protected override string GenerateCommandLineCommands()
{
    StringBuilder builder = new StringBuilder();
    AppendIfPresent(builder, "--target", Target);
    AppendIfPresent(builder, "--target-work-dir", WorkingDirectory);
    AppendIfPresent(builder, "--target-args", QuoteIfNeeded(TargetArgs));
    AppendIfPresent(builder, "--output", Output);
 
    AppendMultipleItemsTo(builder, "--include", Include);
    AppendMultipleItemsTo(builder, "--exclude", Exclude);
 
    Log.LogCommandLine(builder.ToString());
 
    return builder.ToString();
}
5. Execute

Finally, if you have anything special to do, you can override the Execute(). In this case, I wanted to handle the registering and de-registering of the Core.dll. Make sure that you call the base.Execute() method so that the TaskTarget can do the work that it needs to do.

public override bool Execute()
{
    string corDriverPath = Path.Combine(ToolPath, CorDriverName);
    Log.LogMessage("CoreDriver: {0}", corDriverPath);
    using (Registrar registrar = new Registrar(corDriverPath))
    {
        registrar.RegisterComDLL();
        return base.Execute();
    }
}

To see the whole thing, download the files at the bottom of this post.

How to Use PartCover with MSBuild

Now that you have a Custom task you need to create a Target in your MSBuild file to execute the task.

 
<!-- Register the PartCover.MSBuild.dll so the PartCover task is available -->
<UsingTask TaskName="PartCover.MSBuild.PartCover" AssemblyFile="$(LibDirectory)/PartCover/PartCover.MSBuild.dll" />
 
<!-- Setup a property so you can use it in your task -->
<ItemGroup>
    <TestAssemblies Include="src/ZorchedProj/bin/$(Configuration)/ZorchedProj.Tests.dll" />
</ItemGroup>
 
<!-- Create a Target to call the PartCover task -->
<Target Name="Test" DependsOnTargets="CoreBuild">
 
     <!-- Configure the task to execute -->
    <PartCover ToolPath="$(LibDirectory)/PartCover"
                    Target="$(LibDirectory)NUnitnunit-console.exe"
                    TargetArgs="%(TestAssemblies.FullPath) /xml=%(TestAssemblies.Filename).xml /labels /nologo /noshadow"
		    WorkingDirectory="$(MSBuildProjectDirectory)"
		    Output="partcover.xml"
                    Include="[Zorched.*]*"
                    Exclude="[*.Test]*"
    />
</Target>

Download the code:
PartCover MSbuild.zip

Good luck and I hope someone else finds this useful.

Posted in .NET, Unit Testing | Tagged , , , | 15 Comments

Tracking Project Metrics

How do you track the health of your software projects? Ideally you could come up with few, easy to collect metrics and have your Continuous Integration system generate the information and maybe even graph it over time. What we track is going to be based on a set of beliefs and assumptions, so I think we should make that clear.

My Software Metrics Beliefs and Assumptions

  • The simplest code that solves the problem is the best. Simple does not mean rote or repetitive. It means well designed, well abstracted, well factored.
  • Unit Testing improves the quality of code.
  • Overly complex code, code that is not well factored, “big” code is hard to unit test.
  • The metrics need to be easy to interpret and easy to gather or I won’t do it.

Based on those beliefs and assumptions we have defined the kinds of things we care about. We want simple, small classes and methods. We want classes that fit the Single Responsibility Principle. We want unit test coverage. And we want to know when we deviate from those things.

Inevitably this won’t tell you the whole picture of a project. Some deviation is inevitable as well (we’re not perfect). But this is giving us a picture into a project that would let us look at “hot spots” and determine if they are things we want to deal with. It will never tell you if the system does what a user really wants. It will never fully tell you if a project will be successful.

The Metrics I Came Up With

  • Unit Test Coverage – How many code paths are being exercised in our tests.
  • Cyclomatic Complexity – Number of methods over 10, 20, 40
  • Lines of Code – General size information
  • Methods over a certain size – Number of methods over 15, 30, 45 lines
  • Classes over a certain size – Number of classes over 150, 300, 600 lines
  • Afferent/Efferent Coupling – Dead code and code that does too much

Tools

I’m currently thinking mostly about .NET projects because at work we do a lot of small to mid-size projects using .NET. Many of the tools already exist and are Open Source in the Java world and many Java Continuous Integration servers support calculating some of those metrics for you already. So you’re probably in pretty good shape if you’re doing Java development.

FxCop

Out-of-the-box FxCop doesn’t really give you many of these things. But it does provide you a way that you can fairly easily your own rules using their API.

For Example to calculate Cyclomatic Complexity you can implement a class that extends BaseIntrospectionRule and overrides the VistBranch and VisitSwitchInstruction.

public override void VisitBranch(Branch branch)
{
    if (branch.Condition != null)
    {
         Level ++;
     }
     base.VisitBranch(branch);
 }
 
public override void VisitSwitchInstruction(SwitchInstruction switchInstruction)
{
    Level += switchInstruction.Targets.Count;
    base.VisitSwitchInstruction(switchInstruction);
}

Coverage

PartCover is an open source code coverage tool for .NET. It does the job but does not have what I’d call any “wow” factor.

NCover is a proprietary coverage tool. High on wow factor, but expensive. NCover was once free and open source and gained a good following, but they switched and closed it and made the decision to charge $150 – $300 a license for it depending on the version.

(I lament the state of the .NET ecosystem with regard to tooling. Either an OK open source version or an expensive commercial version is not a real choice. There are so many good options in the Unit Testing space, but not in the coverage space.)

NDepend Mini-Review

Disclosure: I received a free copy of NDepend from the creator. I would like to think that didn’t influence this at all, but wanted to make that known.

One of the tools that I’ve been looking at for .NET projects is NDepend. It seems to cover all of the cases that I mentioned above except for the Code Coverage case (although it integrates with NCover, but I haven’t looked at that). It has a really cool query languages, that looks a lot like SQL, that lets you customize any of the existing metrics that it tracks or write your own (it comes with so many that it seems like I just customize). It comes with so many metrics that in practice it can be seem quite overwhelming. I think the right thing to do for most projects is to pick the handful that you care about and limit it to that.

NDepend comes with NAnt and MSBuild tasks that will let you integrate it into your build automation. It also comes with an XSL stylesheet to integrate the NDepend output into CruiseControl.NET for reporting purposes.

Some things you might run into:

  • C# support is the primary language (e.g. No support for code level Cyclomatic Complexity for non-C# language, but still supports IL level CC). Only a problem if you are forced to do VB.NET of course, or use another CLR language.
  • A single license is $400 and 20 licenses or more are over $200 each. So price could be a concern to many people. I’m thinking it might make the most sense to run this on a CI server, so in that case you would probably only need a small number of licenses.
  • It integrates with code coverage tools, but currently only NCover. See the previous comments about the cost of NCover combined with the cost of NDepend if price is an issue.

What Else?

What does everyone else think? What do you care about and what are the right metrics for keeping track of the health of a software project?

Posted in .NET | Tagged , , , | 1 Comment

Register and Unregister COM DLL from .NET Code

Background

The command regsvr32 is used to register a native, unmanaged code DLL so that it is available via COM. With a registered DLL you can use COM Interop to call that code from .NET Managed code.

Regsvr32 is unmanaged code and as such makes use of some existing functions that are defined in the kernel32.dll. Fortunately .NET makes available a pretty easy to use foreign function interface (FFI) in the form of P/Invoke.

In general to call an unmanaged function you just need to use the DllImport annotation on an extern function to tell the CLR how to access the function.

e.g.:

[DllImport("shell32.dll")]
static extern int DllGetVersion(ref DLLVERSIONINFO pdvi);

Registering an Unmanaged DLL with C# Code

regsvr32 actually calls functions defined within the DLL itself in what is known as a self-registering DLL. So assuming your DLL is self-registering then you should be able to use this approach as well. The only thing we need to do is figure out what functions to call.

It ends up there are 2 basic functions: LoadLibrary and GetProcAddress.

LoadLibrary

LoadLibrary returns a handle the module (a pointer to a structure).

After you are done with you library you can clean up by calling FreeLibrary passing it the handle that was returned from LoadLibrary.

GetProcAddress

GetProcAddress finds a function defined in a module and returns a pointer to that function. A function pointer allows you to call a method in a dynamic way. It is functionally equivalent to a delegate in managed code.

In C e.g.:

(*some_func)();

Put it All Together

Now we have a basic algorithm to register a DLL:

  1. LoadLibrary to get a handle to the library
  2. GetProcAddress to get a function pointer to the proper function to register the DLL
  3. Call the function returned from GetProcAddress
  4. Cleanup

Mix that in with some error checking code and I got the following:

public class Registrar : IDisposable
{
    private IntPtr hLib;
 
    [DllImport("kernel32.dll", CharSet = CharSet.Ansi, ExactSpelling = true, SetLastError = true)]
    internal static extern IntPtr GetProcAddress(IntPtr hModule, string procName);
 
    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern IntPtr LoadLibrary(string lpFileName);
 
    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern bool FreeLibrary(IntPtr hModule);
 
    internal delegate int PointerToMethodInvoker();
 
    public Registrar(string filePath) 
    {
        hLib = LoadLibrary(filePath);
        if (IntPtr.Zero == hLib)
        {
            int errno = Marshal.GetLastWin32Error();
            throw new Win32Exception(errno, "Failed to load library.");
        }
    }
 
    public void RegisterComDLL()
    {
        CallPointerMethod("DllRegisterServer");
    }
 
    public void UnRegisterComDLL()
    {
        CallPointerMethod("DllUnregisterServer");
    }
 
    private void CallPointerMethod(string methodName)
    {
        IntPtr dllEntryPoint = GetProcAddress(hLib, methodName);
        if (IntPtr.Zero == dllEntryPoint)
        {
            throw new Win32Exception(Marshal.GetLastWin32Error());
        }
        PointerToMethodInvoker drs = 
               (PointerToMethodInvoker) Marshal.GetDelegateForFunctionPointer(dllEntryPoint, 
                           typeof(PointerToMethodInvoker));
        drs();
    }
 
    public void Dispose()
    {
      if (IntPtr.Zero != hLib)
      {
          UnRegisterComDLL();
          FreeLibrary(hLib);
          hLib = IntPtr.Zero;
      }
    }
}

Note:
The requirement I was dealing with was a build script so I wanted to register the unmanaged DLL, use it, and then unregister it so the computer would be in its previous state. If you want to leave the DLL registered, such as for an install program, you would need to modify the above example.

To call this code you just need to pass it a path to the dll that needs to be registered.

using (Registrar registrar = new Registrar("path\\to\\com.dll"))
{
    registrar.RegisterComDLL();
    return base.Execute();
}

Resources:
Check out pinvoke.net for a lot of good documentation and example of how to call native methods from managed code.

Posted in .NET, Code | Tagged , | 4 Comments

Windows Subversion Maintenance Scripts

Previously I wrote about the Subversion Maintenance Scripts that I use for doing things like backups and upgrades. They were bash shell scripts that automated dealing with multiple Subversion repositories. The secret was that those guys were being run using Cygwin on Windows. Recently we got a new, more powerful server to run our Virtuals on. I decided to rewrite the scripts using native Windows Batch files so that I wouldn’t have to install Cygwin again. Hopefully someone else will find this useful too.

Windows Batch Script for Creating a Default Repository Structure

I work for a custom software development consulting firm so we have a lot of different clients. We don’t want to mix different clients’ code so we create a repository per client. We often get multiple projects per client though (yeah, they like us) so we go ahead and create a project under the client. We also like to use the trunk, tags and branches directories by default.

With all of these rules: Automate!

create_repo.bat

@echo off
 
REM Check command line args to make sure a customer and project name were passed
IF (%1)==() GOTO CMDLINE
IF (%2)==() GOTO CMDLINE
 
REM If svnadmin is not in your path, uncomment the following
REM PATH=%PATH%;"D:\Program Files\Subversion\bin"
 
REM Set up variables with the directories
SET CUSTOMER=%1
SET PROJECT=%2
SET CUR_DIR=%~dp0
SET CUR_DIR=%CUR_DIR:\=/%
SET SVN_DIR=%CUR_DIR%/repos/%CUSTOMER%
SET PROJ_DIR=file:///%SVN_DIR%/%PROJECT%
 
echo Creating directory in %SVN_DIR%
 
svnadmin create "%SVN_DIR%"
svn -m "Create default structure." mkdir %PROJ_DIR% %PROJ_DIR%/trunk %PROJ_DIR%/tags %PROJ_DIR%/branches
GOTO END
 
:CMDLINE
echo Usage: %0 ^<customer^> ^<project^>
 
:END
echo Done

Some of the interesting pieces:
%~dp0 gives you the directory that the current script lives in.
%CUR_DIR:\=/% replaces the backslash (\) with forward slashes (/) so that the SVN file:// URL will work correctly.

Windows Batch Script to Dump Your Subversion Repositories

The next thing you might want to do is to dump all of your repositories so that you can back them up or upgrade SVN or move to a new server. With just one repository it’s easy to do by hand. With 20+ repositories it’s much nicer to run a script and then go do something else.

dump_all.bat

@echo off
 
SET DUMP_DIR=dumps
SET REPOS_DIR=repos
 
IF NOT EXIST %DUMP_DIR% (
echo Creating directory %DUMP_DIR%
md %DUMP_DIR%
)
 
echo Dumping...
 
FOR /f %%i IN ('dir /B /A:D %REPOS_DIR%\*') DO (
    echo Dumping %REPOS_DIR%\%%i to %DUMP_DIR%\%%i.dump
    svnadmin dump %REPOS_DIR%\%%i > %DUMP_DIR%\%%i.dump
)

Some of the interesting pieces:
FOR /f %%i IN (‘dir /B /A:D %REPOS_DIR%\*’) DO is the part that finds each directory (in this case an SVN repository). Then for each repository you run the svnadmin dump command and send the dump file to a different directory.

Windows Batch Script to Load Your Subversion Repositories

After you’ve dumped all of your repositories and you need to restore them, because you’ve upgraded or moved them to a new server we need to do the reverse of the dump script.

restore_all.bat

@echo off
 
SET DUMP_DIR=dumps
SET REPOS_DIR=repos
 
echo Restoring....
 
REM %%~ni becomes the dump name without .dump
 
FOR /f %%i IN ('dir /B %DUMP_DIR%\*.dump') DO (
    echo Restoring %DUMP_DIR%\%%i to %REPOS_DIR%\%%~ni
    svnadmin create %REPOS_DIR%\%%~ni
    svnadmin load %REPOS_DIR%\%%~ni < %DUMP_DIR%\%%i
)

Some of the interesting pieces:
FOR in this case is the same as the dump, except it’s looking for all the dump files.
%%i holds the name of the dump file (e.g. repo.dump). %%~ni strips off the extension to give you just the name of the file (e.g. repo).

Now you can manage your repositories with ease if you’re stuck on a Windows machine.

Posted in Automation | Tagged , , | 3 Comments

Swapping Values Without a Temp Variable

It’s been a long time since I’ve done any C programming (the only C I ever really did was in school 10+ years ago) and I’ve never done it outside of school. I figure since I look at and study languages that are pretty esoteric, it’s only fair to throw in a language that is really widely used. It should make my Objective-C better (which probably isn’t saying much). It also gives me a chance to go “a little closer to the metal” I guess. Hey look, a Bus Error!

Anyway, one of the things that I love/hate about C are the little tricks that people often use. Sometimes they’re really fun but often they take a bit of thinking to understand if you’re not familiar with them. One of those that I ran across again was swapping values without a temp variable.

How Normal People Swap Values

The simplest way of swapping two values is ususally accomplished using a temp variable to hold one of the values for a moment.

int a = 2;
int b = 5;
int temp = 0;
 
temp = a;
a = b;
b = temp;

And Another Way

But you can accomplish the same thing using an Exclusive Or (XOR) operation.

To review bitwise operations:

& AND
  0101 (5)
  1100 (12) 
  ----
& 0100 (4)

| OR
  0101 (5)
  1100 (12) 
  ----
| 1101 (13)

^ XOR
  0101 (5)
  1100 (12) 
  ----
| 1001 (9)

The ^= just combines the XOR with an Assignment.

int a = 2;
int b = 5;
 
a ^= b;
b ^= a;
a ^= b;

Now isn’t that totally intuitive? Yeah, not to me either. Technically once you know this piece of trivia, you don’t really need to think about it. You’ll just recognize the pattern and understand what’s going on. But understanding why it works is probably a good thing.

So, I just came up with an example that shows what’s happening:

int a = 2;  // 0010
int b = 5;  // 0101
 
a ^= b;     // 0111 = 0010 ^ 0101
b ^= a;     // 0010 = 0101 ^ 0111
a ^= b;     // 0101 = 0111 ^ 0010

What are some of the principles in why this works?:

a ^ b == b ^ a

XOR is a reversible binary operation.

a = a ^ b  // now a is some other value
b = a ^ b  // now b is the original value of a
a = a ^ b  // now a is the original value of b

This isn’t a proof of any sort, but hopefully it will help you understand why it actually works. If you want even more of an explanation, and a proof, check out the article on Wikipedia: XOR Swap Algorithm

Posted in C | 1 Comment

Scheme/HtDP Unit Testing Functions

How to Design Programs (HtDP) provides a series of unit testing functions that allow you to test the output of any kind of function. The current version talks about testing your code, but doesn’t offer a lot of guidance into how to do that. Everything you need is there though, so in case you are doing this on your own and don’t have a teacher to tell you to do this, here’s some explanation that I hope helps.

check-expect

check-expect takes the results of a function and the expected value. This is the simplest of the testing constructs which answers the question “Does the expected result of a function call equal the result returned by the function?”. It handles all of the details about types of values returned, so it can work with a simple value like a number, a complex value like a structure or a list, or even a picture.


;; add: number number -> number
;; add two numbers together
(define (add n m)
  (+ n m))

;; Tests
(check-expect (add 1 1) 2)

check-within

check-within is similar to check-expect, but it allows for a third argument to tell wether or not the result is within a certain difference of the actual answer. This is good for certain functions that might produce random results, but the most common case is when dealing with floating point numbers. Floating point numbers are not always represented precisely, so there needs to be some “room for error” so to speak.


(define PI 3.14)
(check-within (/ 22 7) PI .003)

(check-within (random 10) 0 9)

check-error

Finally, you can use check-error to test cases where your function throws an error. Those error conditions are an important part of a function contract, so testing them should be done just like any other possible conditions.

One of the things that threw me at first was what to use as the expected value of a check-error test.

An error is HtDP is “throw” by a function like:


(error 'function-name "some message")

After a bit of trial and error, I found out the expected result is a string like:


; "function-name: some message"
(check-error (error 'list-pick "list too short")  "list-pick: list too short")

More Realistic Example


;; list-pick : list-of-symbols N[>= 0]  ->  symbol
;; to determine the nth symbol from alos, counting from 0;
;; signals an error if there is no nth item
(define (list-pick alos n)
  (cond
    [(empty? alos) (error 'list-pick "list too short")]
    [(= n 0) (first alos)]
    [(> n 0) (list-pick (rest alos) (sub1 n))]))

;; Tests
(check-error (list-pick empty 1)  "list-pick: list too short")
(check-expect (list-pick (list 'a 'b 'c 'd) 3) 'd)
(check-error (list-pick (list 'a 'b 'c 'd) 4) "list-pick: list too short")

Posted in Scheme | Tagged , , | Leave a comment