Scala and Adding New Syntax

One interesting thing about some languages is their support for adding new syntax. While all languages have the ability to add new functions or types some have specific properties that make it easy to add what looks like new built-in syntax.

Scala is an Object Oriented language. You can declare classes and objects, do inheritance and composition, and all the other things you might expect from an OO language. Scala is also a Functional language because functions are first-class citizens (also called a functor). And when I say Scala is an OO language I really mean it: everything is an Object. Even functions are Objects. (Chew on that one for a bit.)

Scala also supports the idea of optional parenthesis for method calls that only take a single argument (Note: This applies to method calls on object only. Not to functions.). This ends up being for a very practical reason. Take the following example:

1 + 2

This is a very nice way to write an addition operation. In reality what’s happening is:

1.+(2)

1 is an object and + is a method on that object that takes a single parameter. Applying the previous rule we get to remove the dot and the the parenthesis. Which allows us to write our previous example 1 + 2.

The good news is they bring this consistency to the language as a whole, so any method call can optionally use the dot. Any call to a method that only takes a single parameter can exclude the parenthesis around its arguments. These features make it pretty easy to emulate the built-in syntax of a language.

Your Own While Loop

Let’s say I want to write my own while loop:

def mywhile(condition: => Boolean)(command: => Unit) {
if (condition) {
command
mywhile(condition)(command)
}
}

var x = 1
mywhile(x < 100000) {
println(x)
x += 1
}

As you can see, I end up calling mywhile the same as I would call a built-in while. This is implemented as a tail-recursive function. If the condition is met, the command is executed. The function then recurses, calling itself to continue. x < 100000 is an anonymous function that returns a boolean expression.

Your Own Do...While Loop

A while loop can be built using just a single function. What if you want to create a do...while loop instead? In this case you can make use of the OO/functional hybrid.


class Repeater(command: => Unit){
final def aslongas(condition: => Boolean) {
command
if (condition) aslongas(condition)
}
}

def mydo(command: => Unit): Repeater = {
new Repeater(command)
}

var x = 0
mydo {
x += 1
println(x)
} aslongas (x < 100000)

In this case I use recursion again to do the looping. But I use an Object to bind the command to and an aslongas method to run that command and check the looping condition. I use a function mydo to bootstrap an instance of the Repeater class. Scala gives us the ability to use functions and objects when they make sense.

Why Should You Care?

Ok, so you're not going to write your own while loops. The language has them built-in already. But what this allows you to see is how you can add new "syntax". That ability makes it quite convenient and easy to write higher-order syntax to solve application specific problems or to create DSLs.

Update: Changed the until name to 'aslongas' since it really wasn't until the condition was met.

Grails Embedded Classes ClassCastException

Using ORM tools allow you to map the data to a database independently of how your object model looks. Grails supports one-to-many and one-to-one relationships if you want to have the data in different table. But what about when you want to map a single table to multiple objects? In Grails a has a relationship where all the data is stored in a single table is defined by using the embedded syntax. (This creates a component in the Hibernate mapping world.)

An example of using the embedded syntax to create a compositional relationship:

class Person {
static embedded = ['address']

String name
int age
Address address = new Address()
}

class Address {
String street
String street2
String city
String state
String zip

static constraints = {
street2(nullable:true)
}
}

This is all great, but…
When you attempt to databind against the Person model including Address properties, you end up with an exception:

2009-04-17 20:36:10,058 [13260127@qtp2-0] ERROR errors.GrailsExceptionResolver – java.lang.ClassCastException: Address$__clinit__closure1
org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.ClassCastException: Address$__clinit__closure1
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:92)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1061)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:910)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:892)
at groovy.lang.Closure.call(Closure.java:279)
at groovy.lang.Closure.call(Closure.java:274)

It ends up there is a bug in Grails versions 1.0.4, 1.0.5 and 1.1 that is causing this. That bug is related to the constraints on the embedded class, in this case the Address class. Removing the constraints allows the object to be properly databound, with the obvious downside of lacking constraints.

I have filed a bug about this: GRAILS-4446. So hopefully it will be fixed soon. Hopefully if other people are having this problem they will help them out.

Update:
There is another workaround which is to declare the embedded class in its own file which makes it a first-class Domain object. When that’s done the constraints work as expected. The downside is that this means that a table will be created for the embedded domain class (and never used).

Update Table Data in Grails using Ajax Calls

Using Ajax for simple forms can offer users a very clean, simple and fast way to input data. I came across a situation recently where I was looking into replacing a document based workflow with an application. The documents themselves contained a series of different kinds of transactions that could have occurred. There were not a set number of any of the types and the user needed a simple way to enter many rows of data.

Grails offers some very easy to use simple Ajax controls. The formRemote tag has an update parameter that can be used to specify a DOM element to update after the form is submitted. This is Ajax in it’s simplest form. Submit some data and replace the contents of a DOM element with a partial-page update. This works well if you want to update a div for example. But what if you want to add a row to a table? In the standard update method, you would have to re-render the entire table contents. In many cases that will be fine, but I figured I could do better than that and just render the new row and add it to the table.

Defining Domain Classes

For this example we’ll start with some simple classes. An Account can have many Checks posted against it. There are no set number, so we define a hasMany relationship between the Account and the Checks.


class Account {
static transients = ['checksTotal']
static hasMany = [checks:Check]

Date date = new Date()

Double getChecksTotal() {
if (! checks)
return 0
return checks.inject(0) { current, check-> current + check.amount }
}

static mapping = {
checks joinTable: false
}
}

class Check {
String name
Double amount
String checkNumber
String reason
}

Creating a Controller

Next we need to create a Controller to mediate between our Domain and our Views. The Controller is setup to generate the default scaffold. We’ll end up overriding the show view to add a form on the page to easily post Checks to that Account. The action is the postCheck action. This is our Ajax action that will post the Check to the Account.


class AccountController {

def scaffold = true

def postCheck = {
def account = Account.get(params.id)
if(account) {
def check = new Check(params)
account.addToChecks(check)
if(! account.hasErrors() && account.save()) {
render template:'check', bean:check, var:'check'
}
}
}
}

The postCheck method calls the render method to render a template and return it as the Ajax response. This template contains the markup for a single Check row.

You can see the check template here:

${check?.name} ${check?.checkNumber} ${check?.reason}

Creating the Ajax Form

Now we need to add the form to our account/show.gsp so that we can call the AccountController.postCheck action. The example form is below. The thing to notice is that we’re not using the update parameter of the formRemote, but rather the onSuccess parameter. The update method tell the Ajax callback what element to replace but we can’t do that with a table. onSuccess on the other hand allows us to handle the returned values in our own way. Here’ we’re going to call the appendTableRow Javascript function to handle the response.

In this case I definitely don’t want to replace the entire node because the first row of my table is actually the form itself. This gives it a very nice look where the data is entered under the appropriate header and then immediately shows up as the following row when the user hits enter.

Checks

onSuccess="appendTableRow('checks', e)">


Name Check # Amount Reason


Luckily the appendTableRow function is made pretty easy by using the prototype APIs. The event returned to us is an HTML snippet, so in this case we just want to insert that snippet directly into the DOM which is accomplished with the insert prototype function.

function appendTableRow(tableId, e) {
$(tableId).down('tbody').down('tr').insert({after: e.responseText});
}

In the end you have a data table that allows for easy, fast updates with almost no hassle.

Using Quartz.NET, Spring.NET and NHibernate to run Scheduled Tasks in ASP.NET

Running scheduled tasks in web applications is not normally a straightforward thing to do. Web applications are built to respond to requests from users and respond to that request. This request/response lifecycle doesn’t always match well to a long running thread that wakes up to run a task every 10 minutes or at 2 AM every day.

ASP.NET Scheduled Task Options

Using ASP.NET running on Windows, there are a number of different options that you could choose to implement this. Windows built in Scheduled Tasks can be run to periodically perform execute a program. A Windows Service could be constructed that used a Timer or a Thread to periodically do the work. Scheduled Tasks and Windows Service require you to write a standalone program. You can share DLLs from your Web application but in the end it is a separate app that needs to be maintained. Another option if you go this route is to turn the Scheduled Task or Service being run into a simple Web Service or REST client that can call your Web application but doesn’t need any knowledge of the jobs themselves.

Another option is an Open Source tool called Quartz.NET. Quartz.NET is based on the popular Java scheduled task runner called (not surprisingly) Quartz. Quartz.NET is a full-featured system that manages Jobs that do the work and Triggers that allow you to specify when you want those jobs run. It can run in your web application itself or as an external service.

The simplest approach to get started is to run directly in your Web application as a process in IIS. The downside to this is that IIS will periodically recycle it’s processes and won’t necessarily start a new one until a new web request is made. Assuming you can deal with this indeterministic behavior then in an IIS process will be fine. It also creates a relatively easy path that will allow you to migrate to the external service process at a later point if need be.

I’m an ALT.NET kind of .NET developer, so I like to use tools like NHibernate for ORM and Spring.NET for Dependency Injection, AOP and generally wiring everything together. The good news is that Spring.NET supports Quartz.NET through its Scheduling API. Start with that for some basic information on using Quartz.NET with Spring. The bad news is that the documentation is a bit thin and the examples basic. I attempt to remedy that in part here.

Using Quartz.NET, NHibernate and Spring.NET to run Scheduled Tasks

The goal is to integrate an existing Spring managed object like a Service or a DAL that uses NHibernate with a Quartz Job that will run on a periodic basis.

To start with you need to create an interface for your service and then implement that interface. The implementation I’ll leave to you and your problem, but the example below you can image uses one or more NHibernate DALs to lookup Users, find their email preferences, etc.

Implementing Services and Jobs


public interface IEmailService
{
void SendEveryoneEmails();
}

When implementing your Job you need to know a few details about how Quartz works:

  1. The first thing to understand is that if you are going to use the AdoJobScheduler to store your Jobs and triggers in the database the Job needs to be Serializable. Generally speaking your DAL classes and NHibernate sessions and the like are not going to be serializable. To get around that, we make the properties set-only so that they will not be serialized when they are stored in the database.
  2. The second thing to understand is that your Job will not be running in the context of the Web application or a request so anything you have to set up connections (such as an OpenSessionInView filter) will not apply to Jobs run by Quartz. This means that you will need to setup your own NHibernate session for all of the dependent objects to use. Luckily Spring provides some help with this in the SessionScope class. This is the same base class as is used by the OpenSessionInView filter.

Using the Service interface you created, you then create a Job that Quartz.NET can run. Quartz.NET provides the IJob interface that you can implement. Spring.NET provides a base class that implements that interface called QuartzJobObject helps deal with injecting dependencies.


using NHibernate;
using Quartz;
using Spring.Data.NHibernate.Support;
using Spring.Scheduling.Quartz;

public class CustomJob : QuartzJobObject
{
private ISessionFactory sessionFactory;
private IEmailService emailService;

// Set only so they don't get serialized
public ISessionFactory SessionFactory { set { sessionFactory = value; } }
public IEmailService EmailService { set { emailService = value; } }

protected override void ExecuteInternal(JobExecutionContext ctx)
{
// Session scope is the same thing as used by OpenSessionInView
using (var ss = new SessionScope(sessionFactory, true))
{
emailService.SendEveryoneEmails();
ss.Close();
}
}
}

Wiring Services and Jobs Together with Spring

Now that you have your classes created you need to wire everything together using Spring.

First we have our DALs and Services wired in to Spring with something like the following:



Next you create a Job that references the Type of the Job that you just created. The type is referenced instead of the instance because the lifecycle of the Job is managed by Quartz itself. It deals with instantiation, serialization and deserialization of the object itself. This is a bit different than what you might expect from a Spring service normally.


Once your Job is created, you create a Trigger that will run the Job based on your rules. Quartz (and Spring) offer two types of Jobs SimpleTriggers and CronTriggers. SimpleTriggers allow you to specify things like “Run this task every 30 minutes”. CronTriggers follow a crontab format for specifying when Jobs should run. The CronTrigger is very flexible but could be a little confusing if you aren’t familiar with cron. It’s worth getting to know for that flexibility though.



The last piece that needs to be done is the integration of the SchedulerFactory. The SchedulerFactory brings together Jobs and Triggers with all of the other configuration needed to run Quartz.NET jobs.

A couple of things to understand about configuring the SchedulerFactory:

  1. Specifying <property name=”DbProvider” ref=”DbProvider”/> (where DbProvider is the db:provider setup used by your Nhibernate configuration) tells the SchedulerFactory to use the AdoJobProvider and store the Jobs and Trigger information in the database. The tables will need to exist already and Quartz provides a script for this task.
  2. Running on SQL Server requires a slight change to Quartz. It uses a locking mechanism to prevent Jobs from running concurrently. For some reason the default configuration uses a FOR UPDATE query that is not supported by SQL Server. (I don’t understand exactly why a .NET utility wouldn’t work with SQL Server out of the box?)
    To fix the locking a QuartzProperty needs to be set:
    <entry key=”quartz.jobStore.selectWithLockSQL” value=”SELECT * FROM {0}LOCKS WHERE LOCK_NAME=@lockName”/>
  3. The JobFactory is set to the SpringObjectJobFactory because it handles the injection of dependencies into QuartzJobObject like the one we created above.
  4. SchedulerContextAsMap is a property on the SchedulerFactory that allows you to set properties that will be passed to your Jobs when they are created by the SpringObjectJobFactory. This is where you set all of the Property names and the corresponding instance references to Spring configured objects. Those objects will be set into your Job instances whenever they are deserialized and run by Quartz.

Here’s the whole ScheduleFactory configuration put together:







Conclusion

Scheduled tasks in ASP.NET applications shouldn’t be too much trouble anymore. Reusing existing Service and DAL classes allows you to easily create scheduled tasks using existing, tested code. Quartz.NET looks to be a good solution for these situations.

Using Ruby Subversion Bindings to Create Repositories

Subversion has a default Web UI that is served up by Apache if you run Subversion that way. It is pretty boring and read-only. Then there are things like WebSVN that make it less boring, but still read-only. I got curious about what it would take to make something even less boring and NOT read-only. Why not allow me to create a new repository on the server? Why not allow me to create a new default directory structure for a new project in a repository all through a web interface?

Parent Path Setup

All of my repositories use the idea of the SVNParentPath. This makes Apache assume that every directory under a given path is an SVN Repository. That structure makes it easy to deal with multiple repositories and secure them with a single security scheme. Using that assumption then it is easier to write some code that will list existing repositories and create new ones in a known location.

SvnAdmin With Ruby Subversion Bindings

Subversion provides language bindings for a number of different languages (Java, Python, Perl, PHP and Ruby) in addition to the native C libraries. Using these bindings it becomes fairly easy to deal with Subversion. The only hiccup will be dealing with the apparent lack of documentation for the code. So be prepared to do some exploration, digging and reading of the code.

I chose to try this using Ruby because it was quick and easy and it was a language I was already familiar with.

First you need to know how to create a new repository and open an existing repository. Fortunately those are simple, one-line operations:

Svn::Repos.create(repos_path, {})
repos = Svn::Repos.open(repos_path)

There was nothing special (from what I could tell) that would allow you to determine if a repository already existed, so I just created a simple function using the Ruby File operations to determine if a directory already existed. This code would allow me to determine if I needed to create new repository or open an existing one:

def repository_exists?(repos_path)
File.directory?(repos_path)
end

Now I have a repository open so I wanted to build a project structure using the default conventions I use for Subversions projects. My convention is to have a repository named after a client, the top-level directories are named for the client’s project and then each project has the standard trunk, branches and tags within that. Depending on the kind of work you do that convention may or may not make sense for you.

With that decided, I created the code to write that structure in a repository. The one thing I found is that interacting with the Subversion repository allowed you to do things within a transaction that would force all of the changes to be recorded as a single commit. I thought this was a good thing, so performed these operations as a transaction:


txn = repos.fs.transaction

# create the top-level, project based directory
txn.root.make_dir(project_name)

# create the trunk, tags and branches for the new project
%w(trunk tags branches).each do |dir|
txn.root.make_dir("#{project_name}/#{dir}")
end

repos.commit(txn)

Finally I put all of those things together into a class. The class had the concept of being initialized to a base Parent Path so all of the operations would know to start from that location:

require "svn/repos"

class SvnAdmin
def initialize(parent_path)
@base_path = parent_path
end

# Provides a list of directory entries. path must be a directory.
def ls(client_name, path="/", revision=nil)
repos_path = File.join(@base_path, client_name)
repos = ensure_repository(repos_path)

entries = repos.fs.root(revision).dir_entries(path)
entries.keys
end

def create(client_name, project_name)
repos_path = File.join(@base_path, client_name)
repos = ensure_repository(repos_path)

txn = repos.fs.transaction
txn.root.make_dir(project_name)
%w(trunk tags branches).each do |dir|
txn.root.make_dir("#{project_name}/#{dir}")
end

repos.commit(txn)
end

private
def ensure_repository(repos_path)
if ! repository_exists?(repos_path)
Svn::Repos.create(repos_path, {})
end
repos = Svn::Repos.open(repos_path)
end

def repository_exists?(repos_path)
File.directory?(repos_path)
end

end

SvnAdmin from Rails

Now that I had some simple code to create new repositories or add a new project to an existing repository I decided to wrap it in a simple Rails application that would allow me to create repositories using a web-based interface.

To start with, I’m not going to use a database or any ActiveRecord classes in this project (which you might do if you wanted authentication or something else) so I disabled ActiveRecord in the config/environment.rb
config.frameworks -= [ :active_record ]

Then I created an ReposController to manage the Subversion repositories. This controller contains a couple of simple actions:

  1. An index action to list the existing repositories (directories)
  2. A new action to display a form to enter the client and project names
  3. A create action to use the SvnAdmin class to create a new repository and/or project


require "svnadmin"

class ReposController < ApplicationController
layout 'default'

def index
@dirs = Dir.new(SVN_PARENT_PATH).entries.sort.reject {|p| p == "." or p == ".."}
end

def new
end

def create
svn = SvnAdmin.new(SVN_PARENT_PATH)
repos = params[:repos]

respond_to do |format|

begin
svn.create(repos[:client_name], repos[:project_name])
flash[:notice] = "Successfully created."
format.html { redirect_to :action => "index" }
format.xml { head :ok }
rescue
flash[:error] = "Failed to create structure."
format.html { redirect_to :action => "index" }
format.xml { render :xml => object.errors, :status => :unprocessable_entity }
end
end
end
end

You can also easily create a route and a ProjectsController that allows you to see all of the projects within a repository.

The route in config/routes.rb is simply:

map.connect ':repos/projects/',
:controller => 'projects',
:action => 'index'

And the ProjectsController looks up the :repos param to open the proper repository and list the top-level directories with it:

require "svnadmin"

class ProjectsController < ApplicationController
layout 'default'

def index
repos_path = params[:repos]
svn = SvnAdmin.new(SVN_PARENT_PATH)
@projects = svn.ls(repos_path)
end
end

Hopefully that will help you handle your Subversion administration. It should let you code up your conventions so that they are followed whenever a new repository is created or a new project is started.

DRY your CruiseControl.NET Configuration

Don’t Repeat Yourself (DRY) is one of the principles of good software development. The idea is that there should ideally be one and only one “source of knowledge” for a particular fact or calculation in a system. Basically it comes down to not copying-and-pasting code around or duplicating code if at all possible. The advantages of this are many.

Advantages of DRY

  • There will be less code to maintain
  • If a bug is found, it should only have to be fixed in one place
  • If an algorithm or process is changed, it only needs to be changed in one place
  • More of the code should become reusable because as you do this you will parameterize methods to make them flexible for more cases

If it’s good for code isn’t it good for other things like configuration? Why yes it is.

Using CruiseControl.NET Configuration Builder

The Configuration Preprocessor allows you to define string properties and full blocks of XML to use for substitution and replacement. To start using the Configuration Preprocessor, you add xmlns:cb=”urn:ccnet.config.builder”, an xml namespace, to your document to tell the config parser that you plan to do this.

From there you can define a simple property like:


Or you can make it a full block of XML:


http://svn.example.com/svn/$(client)/$(project)/trunk D:\Builds-Net\projects\$(client)\$(project)\trunk
svn.exe
true


Defining Reusable Blocks

Using these ideas I wanted to come up with a templated approach that would allow me to share configuration among multiple projects. That way, if I added new statistics or change the layout of my build server, I would only have to change it in a single place. Thus keeping things DRY. It also encourages more consistency across multiple projects making things easier to understand.

So, I started defining some reusable blocks in the main ccnet.config file which you can see below. The exact details will depend on your configuration of course.

Full Example of config.xml




http://svn.example.com/svn/$(client)/$(project)/trunk D:\Builds-Net\projects\$(client)\$(project)\trunk
svn.exe
true



C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe
D:\Builds-Net\projects\$(client)\$(project)\trunk

build.proj $(build-args)
$(build-targets)
600
D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll



C:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.exe
D:\Builds-Net\projects\$(client)\$(project)\trunk

build.proj $(build-args)
$(build-targets)
600
D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll





D:\Builds-Net\projects\$(client)\$(project)\trunk\*.Test.xml
D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageMerge.xml
D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageSummary.xml
D:\Builds-Net\projects\$(client)\$(project)\trunk\*.FxCop.xml




D:\Builds-Net\projects\$(client)\$(project)\logs
















At the end of the file you can see the cb:include references. Those are one-line includes to include the configuration of each project. This makes things easier to manage, I think, because you only have to look at the individual project configuration.

Using Reusable Blocks in Individual Configuration Files

From there I need to make use of those defined blocks in in individual file. The first thing I needed to do was to set the parameters that I had defined as simple string replacements in the reusable blocks. Normally you would do that with cb:define as I showed above. But the trick is that you can only have one property with a given name defined. If you include multiple project configurations that doesn’t work. What does work is using cb:scope definitions. This allows for a value to be defined only within a specific scope.


client="ExampleClient"
project="SpecialProject"
build-args="/p:Configuration=Debug"
build-targets="Clean;Test">
...

From there you just need to start including the blocks that you defined in the main ccnet.confg within the scope block.

Full Example of Project Configuration

client="ExampleClient"
project="SpecialProject"
build-args="/p:Configuration=Debug"
build-targets="Clean;Test">













As you can see, the only one I didn’t template out was the email block because that depends on the developers working on each project.

Have fun bringing simplicity and consistency to your Cruise Control.NET configuration!

For the full details see the CruiseControl.NET Configuration Preprocessor documentation.

StringBuilder and my Biggest Pet Peeve

What You Should Know About Strings

In both Java and .NET (and other languages) String objects are immutable. They don’t change. Period. If you “change” a String, it creates a new String. That includes String concatenation using a +


// One string created "foo"
String foo = "foo";
// foo exists, "bar" is created and the combination of foo and "bar" is a third string
String bar = foo + " bar";

Ok, if you don’t know this, fine. But if you don’t know this, why would you EVER use a StringBuilder?

Why Does StringBuilder Exist?

We know that Strings are immutable. If you need to do a bunch of string modification, concatenation, replacement – you will create a bunch of strings. Ok, great…why do I care? We care because we are told that creating a lot of Objects (and then later having to Garbage Collect them) is inefficient. To start with, I will guarantee right now that concatenating strings WILL NOT be the thing that prevents your application from performing. Guaranteed. Period.

Ok fine, it’s not going to be a problem. But you want to be a responsible coder and not do things that are intentionally inefficient if you can help it.

So you use a StringBuilder. StringBuilder is implemented internally as an array of characters. The code manages the allocation and copying of data to new arrays if the buffer gets filled. It sometimes over allocates the new buffer so that it has to perform allocations less often. You sacrifice a bit of memory overhead to avoid some Object creation and Garbage Collection later.

My Biggest Pet Peeve

Your use of StringBuilder is a premature optimization but probably a forgivable one.

So, WHY, OH WHY do you do this:

// One string created "foo"
StringBuilder sb = new StringBuilder();
sb.append("Foo: " + fooValue + " \n");
sb.append("Bar: " + barValue + "\n");

It makes me have violent thoughts. Please stop.

Update:
For some of the reasons why Strings are immutable, see this post on immutability and its positive qualities.

MSBuild Task for PartCover

<![Rant[

I continue to lament the dearth of option for Test Coverage in the .NET world.

In the Java world you have open source tools like Emma and Cobertura that are widely used and supported (and many more) as well as proprietary tools like Clover available.

.NET we have an open source NCover SF that requires you to do odd code instrumentation and is essentially dead it seams, another NCover which is proprietary and costs money and PartCover which is open source, but doesn't seem real active.

Don't get me wrong, NCover.org is a good option if you are willing to spend the money for it. But with a team of 30+ and a CI server, I'm not sure if I want to drop $10k on it. (NCover up until version 1.5.8 was Free Software (GPL) before it was closed. Who has the source and why haven't you forked it yet?)

]]>

If you’re not willing to pay that basically leaves PartCover. But of course you want to integrate your code coverage with your automated build. There was no support for MSBuild out of the box, so I decided to build it.

Creating an MSBuild Task

I need to do 2 things to run PartCover:

  1. Register the native PartCover.CorDriver.dll
  2. Execute the PartCover.exe with the proper options

Register Native DLL

To see the details on how to register a native DLL using .NET code so my earlier post Register and Unregister COM DLL from .NET Code.

Execute PartCover

The MSBuild framework provides a ToolTask base class whose whole purpose is for executing external command line tools. I used this as the base of the task.

1. ToolName

First you override the ToolName property to return the name of the EXE to run. Nothing special here, it’s just the executable name.


protected override string ToolName
{
get { return "PartCover.exe"; }
}

2. Properties

Next to start build the task you go about defining all of the settings that a user will need to set to execute the task. You then create those as Properties on the class and they will be set by MSBuild. Start with the simple things that someone will need to pass to get the tool to execute properly. You can build from there for other properties. If possible give the properties sane defaults so that people don’t have to override them in their build file.


// ...

///

/// The application to execute to get the coverage results.
/// Generally this will be your unit testing exe.
///

public string Target
{
get { return _target; }
set { _target = value; }
}

///

/// The arguments to pass to the executable
///

public string TargetArgs
{
get { return _targetArgs; }
set { _targetArgs = value; }
}

public string WorkingDirectory
{
get { return _workingDirectory; }
set { _workingDirectory = value; }
}

// ...

3. Command Arguments

Then you need to override string GenerateCommandLineCommands() method. The whole purpose of this method is to construct any command line parameters that need to be passed to the ToolName command using the Properties defined in the task.


protected override string GenerateCommandLineCommands()
{
StringBuilder builder = new StringBuilder();
AppendIfPresent(builder, "--target", Target);
AppendIfPresent(builder, "--target-work-dir", WorkingDirectory);
AppendIfPresent(builder, "--target-args", QuoteIfNeeded(TargetArgs));
AppendIfPresent(builder, "--output", Output);

AppendMultipleItemsTo(builder, "--include", Include);
AppendMultipleItemsTo(builder, "--exclude", Exclude);

Log.LogCommandLine(builder.ToString());

return builder.ToString();
}

5. Execute

Finally, if you have anything special to do, you can override the Execute(). In this case, I wanted to handle the registering and de-registering of the Core.dll. Make sure that you call the base.Execute() method so that the TaskTarget can do the work that it needs to do.


public override bool Execute()
{
string corDriverPath = Path.Combine(ToolPath, CorDriverName);
Log.LogMessage("CoreDriver: {0}", corDriverPath);
using (Registrar registrar = new Registrar(corDriverPath))
{
registrar.RegisterComDLL();
return base.Execute();
}
}

To see the whole thing, download the files at the bottom of this post.

How to Use PartCover with MSBuild

Now that you have a Custom task you need to create a Target in your MSBuild file to execute the task.







Target="$(LibDirectory)NUnitnunit-console.exe"
TargetArgs="%(TestAssemblies.FullPath) /xml=%(TestAssemblies.Filename).xml /labels /nologo /noshadow"
WorkingDirectory="$(MSBuildProjectDirectory)"
Output="partcover.xml"
Include="[Zorched.*]*"
Exclude="[*.Test]*"
/>

Download the code:
PartCover MSbuild.zip

Good luck and I hope someone else finds this useful.

Tracking Project Metrics

How do you track the health of your software projects? Ideally you could come up with few, easy to collect metrics and have your Continuous Integration system generate the information and maybe even graph it over time. What we track is going to be based on a set of beliefs and assumptions, so I think we should make that clear.

My Software Metrics Beliefs and Assumptions

  • The simplest code that solves the problem is the best. Simple does not mean rote or repetitive. It means well designed, well abstracted, well factored.
  • Unit Testing improves the quality of code.
  • Overly complex code, code that is not well factored, “big” code is hard to unit test.
  • The metrics need to be easy to interpret and easy to gather or I won’t do it.

Based on those beliefs and assumptions we have defined the kinds of things we care about. We want simple, small classes and methods. We want classes that fit the Single Responsibility Principle. We want unit test coverage. And we want to know when we deviate from those things.

Inevitably this won’t tell you the whole picture of a project. Some deviation is inevitable as well (we’re not perfect). But this is giving us a picture into a project that would let us look at “hot spots” and determine if they are things we want to deal with. It will never tell you if the system does what a user really wants. It will never fully tell you if a project will be successful.

The Metrics I Came Up With

  • Unit Test Coverage – How many code paths are being exercised in our tests.
  • Cyclomatic Complexity – Number of methods over 10, 20, 40
  • Lines of Code – General size information
  • Methods over a certain size – Number of methods over 15, 30, 45 lines
  • Classes over a certain size – Number of classes over 150, 300, 600 lines
  • Afferent/Efferent Coupling – Dead code and code that does too much

Tools

I’m currently thinking mostly about .NET projects because at work we do a lot of small to mid-size projects using .NET. Many of the tools already exist and are Open Source in the Java world and many Java Continuous Integration servers support calculating some of those metrics for you already. So you’re probably in pretty good shape if you’re doing Java development.

FxCop

Out-of-the-box FxCop doesn’t really give you many of these things. But it does provide you a way that you can fairly easily your own rules using their API.

For Example to calculate Cyclomatic Complexity you can implement a class that extends BaseIntrospectionRule and overrides the VistBranch and VisitSwitchInstruction.


public override void VisitBranch(Branch branch)
{
if (branch.Condition != null)
{
Level ++;
}
base.VisitBranch(branch);
}

public override void VisitSwitchInstruction(SwitchInstruction switchInstruction)
{
Level += switchInstruction.Targets.Count;
base.VisitSwitchInstruction(switchInstruction);
}

Coverage

PartCover is an open source code coverage tool for .NET. It does the job but does not have what I’d call any “wow” factor.

NCover is a proprietary coverage tool. High on wow factor, but expensive. NCover was once free and open source and gained a good following, but they switched and closed it and made the decision to charge $150 – $300 a license for it depending on the version.

(I lament the state of the .NET ecosystem with regard to tooling. Either an OK open source version or an expensive commercial version is not a real choice. There are so many good options in the Unit Testing space, but not in the coverage space.)

NDepend Mini-Review

Disclosure: I received a free copy of NDepend from the creator. I would like to think that didn’t influence this at all, but wanted to make that known.

One of the tools that I’ve been looking at for .NET projects is NDepend. It seems to cover all of the cases that I mentioned above except for the Code Coverage case (although it integrates with NCover, but I haven’t looked at that). It has a really cool query languages, that looks a lot like SQL, that lets you customize any of the existing metrics that it tracks or write your own (it comes with so many that it seems like I just customize). It comes with so many metrics that in practice it can be seem quite overwhelming. I think the right thing to do for most projects is to pick the handful that you care about and limit it to that.

NDepend comes with NAnt and MSBuild tasks that will let you integrate it into your build automation. It also comes with an XSL stylesheet to integrate the NDepend output into CruiseControl.NET for reporting purposes.

Some things you might run into:

  • C# support is the primary language (e.g. No support for code level Cyclomatic Complexity for non-C# language, but still supports IL level CC). Only a problem if you are forced to do VB.NET of course, or use another CLR language.
  • A single license is $400 and 20 licenses or more are over $200 each. So price could be a concern to many people. I’m thinking it might make the most sense to run this on a CI server, so in that case you would probably only need a small number of licenses.
  • It integrates with code coverage tools, but currently only NCover. See the previous comments about the cost of NCover combined with the cost of NDepend if price is an issue.

What Else?

What does everyone else think? What do you care about and what are the right metrics for keeping track of the health of a software project?

Register and Unregister COM DLL from .NET Code

Background

The command regsvr32 is used to register a native, unmanaged code DLL so that it is available via COM. With a registered DLL you can use COM Interop to call that code from .NET Managed code.

Regsvr32 is unmanaged code and as such makes use of some existing functions that are defined in the kernel32.dll. Fortunately .NET makes available a pretty easy to use foreign function interface (FFI) in the form of P/Invoke.

In general to call an unmanaged function you just need to use the DllImport annotation on an extern function to tell the CLR how to access the function.

e.g.:

[DllImport("shell32.dll")]
static extern int DllGetVersion(ref DLLVERSIONINFO pdvi);

Registering an Unmanaged DLL with C# Code

regsvr32 actually calls functions defined within the DLL itself in what is known as a self-registering DLL. So assuming your DLL is self-registering then you should be able to use this approach as well. The only thing we need to do is figure out what functions to call.

It ends up there are 2 basic functions: LoadLibrary and GetProcAddress.

LoadLibrary

LoadLibrary returns a handle the module (a pointer to a structure).

After you are done with you library you can clean up by calling FreeLibrary passing it the handle that was returned from LoadLibrary.

GetProcAddress

GetProcAddress finds a function defined in a module and returns a pointer to that function. A function pointer allows you to call a method in a dynamic way. It is functionally equivalent to a delegate in managed code.

In C e.g.:

(*some_func)();

Put it All Together

Now we have a basic algorithm to register a DLL:

  1. LoadLibrary to get a handle to the library
  2. GetProcAddress to get a function pointer to the proper function to register the DLL
  3. Call the function returned from GetProcAddress
  4. Cleanup

Mix that in with some error checking code and I got the following:

public class Registrar : IDisposable
{
private IntPtr hLib;

[DllImport("kernel32.dll", CharSet = CharSet.Ansi, ExactSpelling = true, SetLastError = true)]
internal static extern IntPtr GetProcAddress(IntPtr hModule, string procName);

[DllImport("kernel32.dll", SetLastError = true)]
internal static extern IntPtr LoadLibrary(string lpFileName);

[DllImport("kernel32.dll", SetLastError = true)]
internal static extern bool FreeLibrary(IntPtr hModule);

internal delegate int PointerToMethodInvoker();

public Registrar(string filePath)
{
hLib = LoadLibrary(filePath);
if (IntPtr.Zero == hLib)
{
int errno = Marshal.GetLastWin32Error();
throw new Win32Exception(errno, "Failed to load library.");
}
}

public void RegisterComDLL()
{
CallPointerMethod("DllRegisterServer");
}

public void UnRegisterComDLL()
{
CallPointerMethod("DllUnregisterServer");
}

private void CallPointerMethod(string methodName)
{
IntPtr dllEntryPoint = GetProcAddress(hLib, methodName);
if (IntPtr.Zero == dllEntryPoint)
{
throw new Win32Exception(Marshal.GetLastWin32Error());
}
PointerToMethodInvoker drs =
(PointerToMethodInvoker) Marshal.GetDelegateForFunctionPointer(dllEntryPoint,
typeof(PointerToMethodInvoker));
drs();
}

public void Dispose()
{
if (IntPtr.Zero != hLib)
{
UnRegisterComDLL();
FreeLibrary(hLib);
hLib = IntPtr.Zero;
}
}
}

Note:
The requirement I was dealing with was a build script so I wanted to register the unmanaged DLL, use it, and then unregister it so the computer would be in its previous state. If you want to leave the DLL registered, such as for an install program, you would need to modify the above example.

To call this code you just need to pass it a path to the dll that needs to be registered.

using (Registrar registrar = new Registrar("path\\to\\com.dll"))
{
registrar.RegisterComDLL();
return base.Execute();
}

Resources:
Check out pinvoke.net for a lot of good documentation and example of how to call native methods from managed code.