I’ve released my first public Grails Plugin today.
The Grails Constraint plugin gives you the ability to create custom Constraints that you can apply to your Domain classes to validate them. These are applied and act just like the built in Domain class constraints.
Why would you want this?
Grails provides two generic, catch-all Constraints in the core application:
- validator – a generic closure mechanism for validation
- matches – a regular expression mechanism
While those work, I find myself often wanting to use the same Constraints on multiple Domain classes (think Social Security Number, Phone Number, Zipcode, etc.) and I don’t like to repeat those regular expressions or validations all over the place.
What does this plugin do?
With the Grails Constraints plugin, you can write a custom Constraint class, drop it in /grails-app/utils/ and it will automatically be available to all of your domain classes.
Example
I create a new constrain by hand in /grails-app/utils/ComparisonConstraint.groovy. (You can also use the provided create-constraint script like grails create-constraint com.foo.MyConstraint)
class ComparisonConstraint { static name = "compareTo" static expectsParams = true def validate = { val, target -> def compareVal = target."$params" if (null == val || null == compareVal) return false return val.compareTo(compareVal) == 0 } }
Then you can apply your constraint to your Domain class:
class Login { String password String confirm static constraints = { password(compareTo: 'confirm') } }
See Grails Custom Constraints Plugin for the full documentation on what all of the above means and the source code.
The Struts2 documentation contains examples that are often basic at best which can make it challenging to figure out how to do things sometimes. I was working on creating a form that would allow me to select values from a list to connect 2 objects in a One-to-Many relationship. This is a common relationship for many things. In this example, I’ll use a User and Role class to demonstrate the concept.
For background, here’s a JPA mapped User and Role class.
import java.util.List; import javax.persistence.*; @Entity public class User { private Long id; // ... other member variables private List<Role> roles; @Id @GeneratedValue(strategy = GenerationType.AUTO) public Long getId() { return id; } public void setId(Long id) { this.id = id; } @OneToMany @JoinTable(name = "UserRoles", joinColumns = @JoinColumn(name = "user_Id"), inverseJoinColumns = @JoinColumn(name = "role_Id"), uniqueConstraints = @UniqueConstraint(columnNames = {"user_Id", "role_Id"}) ) public List<Role> getRoles() { return roles; } public void setRoles(List<Role> roles) { this.roles = roles; } // ... other properties } @Entity public class Role { private Long id; @Id @GeneratedValue(strategy = GenerationType.AUTO) public Long getId() { return id; } public void setId(Long id) { this.id = id; } // ... other properties }
A list of Roles exists in the database. When a User is created, they are assigned one or more Roles. The User and Roles are connected in the Database through a join table as defined in the mapping. At this point I created some DAO Repository classes to manage the persistence and an Action to handle the form values. (The details of JPA, setting up Persistence and Actions are beyond the scope of this post.)
The part that caused me the most grief ended up being the form. I wanted to add a checkbox list that contained all of the Roles. The example on the Struts site for checkboxlist, a control with 43 documented properties is:
<s:checkboxlist name="foo" list="bar"/>
Needless to say, there was some ‘figuring out’ to be done.
The form itself is pretty vanilla for the most part. The checkboxlist is the interesting part because it’s what allows us to map the User to the Roles. I knew that I was looking for something to put into the value property of the control that would tell it to pre-select the Role values that were already associated with the User.
I started out with something like:
<s:checkboxlist name="user.roles.id"
list="roles"
listKey="id"
listValue="name"
label="%{getText('label.roles')}"
value="user.roles"/>That didn’t work. When you think about it, that makes sense because the keys in the list are ids and the values supplied are Role objects. So I needed to figure out how to get the Ids from the Roles. I could have done that in the Action class, but it seemed like there should be a better way. A way that would allow me to continue in more of a Domain fashion.
Doing some research into OGNL, I came upon the list projections section which was the key…
The OGNL projection syntax gives us user.roles.{id}. Basically that is a list comprehension that takes of list of Role objects and turns it into a list of Role Ids. That list of Ids becomes the list of values that will be preselected.
Knowing that I can now create a select box that will include the pre-selected values on an edit form:
<s:checkboxlist name="user.roles.id"
list="roles"
listKey="id"
listValue="name"
label="%{getText('label.roles')}"
value="user.roles.{id}"/>Enjoy.
Regular Expressions are both complex and elegant at the same time. They can be made to look like someone was just randomly hammering on their keyboard. They are also an incredibly efficient and elegant solution to describing the structure of text and matching those structures. They are very handy for defining what a string should look like and as such are very good for use in data validation.
To validate a US phone number you might create a simple regular expression \d{3}-\d{3}-\d{4} which will match a phone number like 123-555-1212. Where regular expressions can become difficult though is when the format is not quite as clear cut. What if we need to support (123) 555-1212 and just 555-1212? Well that’s where things can get more complex. But this is not about validating phone numbers. In this post I will look at how to make assertions about the complexity of a string, which is very useful if you want to enforce complexity of a user created password.
The key to making password strength validation easy using Regular Expressions is to understand Zero-width positive lookahead assertions (also know as zero-width positive lookaheads). Now that’s a mouthful isn’t it? Luckily the concept itself is a lot simpler than the name.
Zero-width positive lookahead assertions
Basically a Zero-width positive lookahead assertion is simply an assertion about a match existing or not. Rather than returning a match though, it merely returns true or false to say if that match exists. It is used as a qualification for another match.
The general form of this is:
(?= some_expression)
For example:
- The regular expression z matches the z in the string zorched.
- The regular expression z(?=o) also matches the z in the string zorched. It does not match the zo, but only the z.
- The regular expression z(?=o) does NOT match the z in the string pizza because the assertion of z followed by an o is not true.
Making Assertions About Password Complexity
Now that you know how to make assertions about the contents of a string without actually matching on that string, you can start deciding what you want to actually assert. Remember that on their own these lookaheads do not match anything, but they modify what is matched.
Assert a string is 8 or more characters:
(?=.{8,})
Assert a string contains at least 1 lowercase letter (zero or more characters followed by a lowercase character):
(?=.*[a-z])
Assert a string contains at least 1 uppercase letter (zero or more characters followed by an uppercase character):
(?=.*[A-Z])
Assert a string contains at least 1 digit:
(?=.*[\d])
Assert a string contains at least 1 special character:
(?=.*[\W])
Assert a string contains at least 1 special character or a digit:
(?=.*[\d\W])
These are of course just a few common examples but there are many more that you could create as well.
Applying Assertions to Create a Complexity Validation
Knowing that these make assertions about elements in a string, but not about a match itself, you need to combine this with a matching regular expression to create you match validation.
.* matches zero or more characters.
^ matches the beginning of a string.
$ matches the end of a string.
Put together ^.*$ matches any single line (including an empty line). With what you know about Zero-width positive lookahead assertions now you can combine a “match everything” with assertions about that line to limit what is matched.
If I want to match a line with at least 1 lowercase character then I can use:
^.*(?=.*[a-z]).*$
(Which reads something like: start of string, zero or more characters, assert that somewhere in the string is a lowercase character, zero or more trailing characters, the end of the string)
The part that makes this all interesting is that you can combine any number of assertions about the string into one larger expression that will create your rules for complexity. So if you want to match a string at least 6 characters long, with at least one lower case and at least one uppercase letter you could use something like:
^.*(?=.{6,})(?=.*[a-z])(?=.*[A-Z]).*$
And if you want to throw in some extra complexity and require at least one digit or one symbol you could make a match like:
^.*(?=.{6,})(?=.*[a-z])(?=.*[A-Z])(?=.*[\d\W]).*$
There you go. Now you can create regular expressions to check the complexity of passwords.
For more help with Regular Expressions, you might want to check out:
Scala and Adding New Syntax
One interesting thing about some languages is their support for adding new syntax. While all languages have the ability to add new functions or types some have specific properties that make it easy to add what looks like new built-in syntax.
Scala is an Object Oriented language. You can declare classes and objects, do inheritance and composition, and all the other things you might expect from an OO language. Scala is also a Functional language because functions are first-class citizens (also called a functor). And when I say Scala is an OO language I really mean it: everything is an Object. Even functions are Objects. (Chew on that one for a bit.)
Scala also supports the idea of optional parenthesis for method calls that only take a single argument (Note: This applies to method calls on object only. Not to functions.). This ends up being for a very practical reason. Take the following example:
1 + 2
This is a very nice way to write an addition operation. In reality what’s happening is:
1.+(2)
1 is an object and + is a method on that object that takes a single parameter. Applying the previous rule we get to remove the dot and the the parenthesis. Which allows us to write our previous example 1 + 2.
The good news is they bring this consistency to the language as a whole, so any method call can optionally use the dot. Any call to a method that only takes a single parameter can exclude the parenthesis around its arguments. These features make it pretty easy to emulate the built-in syntax of a language.
Your Own While Loop
Let’s say I want to write my own while loop:
def mywhile(condition: => Boolean)(command: => Unit) { if (condition) { command mywhile(condition)(command) } } var x = 1 mywhile(x < 100000) { println(x) x += 1 }
As you can see, I end up calling mywhile the same as I would call a built-in while. This is implemented as a tail-recursive function. If the condition is met, the command is executed. The function then recurses, calling itself to continue. x < 100000 is an anonymous function that returns a boolean expression.
Your Own Do…While Loop
A while loop can be built using just a single function. What if you want to create a do…while loop instead? In this case you can make use of the OO/functional hybrid.
class Repeater(command: => Unit){ final def aslongas(condition: => Boolean) { command if (condition) aslongas(condition) } } def mydo(command: => Unit): Repeater = { new Repeater(command) } var x = 0 mydo { x += 1 println(x) } aslongas (x < 100000)
In this case I use recursion again to do the looping. But I use an Object to bind the command to and an aslongas method to run that command and check the looping condition. I use a function mydo to bootstrap an instance of the Repeater class. Scala gives us the ability to use functions and objects when they make sense.
Why Should You Care?
Ok, so you’re not going to write your own while loops. The language has them built-in already. But what this allows you to see is how you can add new “syntax”. That ability makes it quite convenient and easy to write higher-order syntax to solve application specific problems or to create DSLs.
Update: Changed the until name to ‘aslongas’ since it really wasn’t until the condition was met.
Using ORM tools allow you to map the data to a database independently of how your object model looks. Grails supports one-to-many and one-to-one relationships if you want to have the data in different table. But what about when you want to map a single table to multiple objects? In Grails a has a relationship where all the data is stored in a single table is defined by using the embedded syntax. (This creates a component in the Hibernate mapping world.)
An example of using the embedded syntax to create a compositional relationship:
class Person { static embedded = ['address'] String name int age Address address = new Address() } class Address { String street String street2 String city String state String zip static constraints = { street2(nullable:true) } }
This is all great, but…
When you attempt to databind against the Person model including Address properties, you end up with an exception:
2009-04-17 20:36:10,058 [13260127@qtp2-0] ERROR errors.GrailsExceptionResolver – java.lang.ClassCastException: Address$__clinit__closure1
org.codehaus.groovy.runtime.InvokerInvocationException: java.lang.ClassCastException: Address$__clinit__closure1
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:92)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:234)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1061)
at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:910)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:892)
at groovy.lang.Closure.call(Closure.java:279)
at groovy.lang.Closure.call(Closure.java:274)
It ends up there is a bug in Grails versions 1.0.4, 1.0.5 and 1.1 that is causing this. That bug is related to the constraints on the embedded class, in this case the Address class. Removing the constraints allows the object to be properly databound, with the obvious downside of lacking constraints.
I have filed a bug about this: GRAILS-4446. So hopefully it will be fixed soon. Hopefully if other people are having this problem they will help them out.
Update:
There is another workaround which is to declare the embedded class in its own file which makes it a first-class Domain object. When that’s done the constraints work as expected. The downside is that this means that a table will be created for the embedded domain class (and never used).
Using Ajax for simple forms can offer users a very clean, simple and fast way to input data. I came across a situation recently where I was looking into replacing a document based workflow with an application. The documents themselves contained a series of different kinds of transactions that could have occurred. There were not a set number of any of the types and the user needed a simple way to enter many rows of data.
Grails offers some very easy to use simple Ajax controls. The formRemote tag has an update parameter that can be used to specify a DOM element to update after the form is submitted. This is Ajax in it’s simplest form. Submit some data and replace the contents of a DOM element with a partial-page update. This works well if you want to update a div for example. But what if you want to add a row to a table? In the standard update method, you would have to re-render the entire table contents. In many cases that will be fine, but I figured I could do better than that and just render the new row and add it to the table.
Defining Domain Classes
For this example we’ll start with some simple classes. An Account can have many Checks posted against it. There are no set number, so we define a hasMany relationship between the Account and the Checks.
class Account { static transients = ['checksTotal'] static hasMany = [checks:Check] Date date = new Date() Double getChecksTotal() { if (! checks) return 0 return checks.inject(0) { current, check-> current + check.amount } } static mapping = { checks joinTable: false } } class Check { String name Double amount String checkNumber String reason }
Creating a Controller
Next we need to create a Controller to mediate between our Domain and our Views. The Controller is setup to generate the default scaffold. We’ll end up overriding the show view to add a form on the page to easily post Checks to that Account. The action is the postCheck action. This is our Ajax action that will post the Check to the Account.
class AccountController { def scaffold = true def postCheck = { def account = Account.get(params.id) if(account) { def check = new Check(params) account.addToChecks(check) if(! account.hasErrors() && account.save()) { render template:'check', bean:check, var:'check' } } } }
The postCheck method calls the render method to render a template and return it as the Ajax response. This template contains the markup for a single Check row.
You can see the check template here:
<tr> <td>${check?.name}</td> <td>${check?.checkNumber}</td> <td><g:formatNumber number="${check?.amount}" format="\$###,##0" /></td> <td>${check?.reason}</td> </tr>
Creating the Ajax Form
Now we need to add the form to our account/show.gsp so that we can call the AccountController.postCheck action. The example form is below. The thing to notice is that we’re not using the update parameter of the formRemote, but rather the onSuccess parameter. The update method tell the Ajax callback what element to replace but we can’t do that with a table. onSuccess on the other hand allows us to handle the returned values in our own way. Here’ we’re going to call the appendTableRow Javascript function to handle the response.
In this case I definitely don’t want to replace the entire node because the first row of my table is actually the form itself. This gives it a very nice look where the data is entered under the appropriate header and then immediately shows up as the following row when the user hits enter.
<h3>Checks</h3> <g:formRemote name="postCheck" url="[action: 'postCheck', params: [id:account?.id]]" onSuccess="appendTableRow('checks', e)"> <table id="checks"> <thead> <tr> <th>Name</th> <th>Check #</th> <th>Amount</th> <th>Reason</th> </tr> </thead> <tbody> <tr> <td><g:textField id="name" name="name"/></td> <td><g:textField id="checkNumber" name="checkNumber"/></td> <td><g:textField id="amount" name="amount"/></td> <td><g:textField id="reason" name="reason"/></td> </tr> <g:render template="check" collection="${account?.checks}" var="check"/> </tbody> </table> </g:formRemote>
Luckily the appendTableRow function is made pretty easy by using the prototype APIs. The event returned to us is an HTML snippet, so in this case we just want to insert that snippet directly into the DOM which is accomplished with the insert prototype function.
function appendTableRow(tableId, e) { $(tableId).down('tbody').down('tr').insert({after: e.responseText}); }
In the end you have a data table that allows for easy, fast updates with almost no hassle.
Running scheduled tasks in web applications is not normally a straightforward thing to do. Web applications are built to respond to requests from users and respond to that request. This request/response lifecycle doesn’t always match well to a long running thread that wakes up to run a task every 10 minutes or at 2 AM every day.
ASP.NET Scheduled Task Options
Using ASP.NET running on Windows, there are a number of different options that you could choose to implement this. Windows built in Scheduled Tasks can be run to periodically perform execute a program. A Windows Service could be constructed that used a Timer or a Thread to periodically do the work. Scheduled Tasks and Windows Service require you to write a standalone program. You can share DLLs from your Web application but in the end it is a separate app that needs to be maintained. Another option if you go this route is to turn the Scheduled Task or Service being run into a simple Web Service or REST client that can call your Web application but doesn’t need any knowledge of the jobs themselves.
Another option is an Open Source tool called Quartz.NET. Quartz.NET is based on the popular Java scheduled task runner called (not surprisingly) Quartz. Quartz.NET is a full-featured system that manages Jobs that do the work and Triggers that allow you to specify when you want those jobs run. It can run in your web application itself or as an external service.
The simplest approach to get started is to run directly in your Web application as a process in IIS. The downside to this is that IIS will periodically recycle it’s processes and won’t necessarily start a new one until a new web request is made. Assuming you can deal with this indeterministic behavior then in an IIS process will be fine. It also creates a relatively easy path that will allow you to migrate to the external service process at a later point if need be.
I’m an ALT.NET kind of .NET developer, so I like to use tools like NHibernate for ORM and Spring.NET for Dependency Injection, AOP and generally wiring everything together. The good news is that Spring.NET supports Quartz.NET through its Scheduling API. Start with that for some basic information on using Quartz.NET with Spring. The bad news is that the documentation is a bit thin and the examples basic. I attempt to remedy that in part here.
Using Quartz.NET, NHibernate and Spring.NET to run Scheduled Tasks
The goal is to integrate an existing Spring managed object like a Service or a DAL that uses NHibernate with a Quartz Job that will run on a periodic basis.
To start with you need to create an interface for your service and then implement that interface. The implementation I’ll leave to you and your problem, but the example below you can image uses one or more NHibernate DALs to lookup Users, find their email preferences, etc.
Implementing Services and Jobs
public interface IEmailService { void SendEveryoneEmails(); }
When implementing your Job you need to know a few details about how Quartz works:
- The first thing to understand is that if you are going to use the AdoJobScheduler to store your Jobs and triggers in the database the Job needs to be Serializable. Generally speaking your DAL classes and NHibernate sessions and the like are not going to be serializable. To get around that, we make the properties set-only so that they will not be serialized when they are stored in the database.
- The second thing to understand is that your Job will not be running in the context of the Web application or a request so anything you have to set up connections (such as an OpenSessionInView filter) will not apply to Jobs run by Quartz. This means that you will need to setup your own NHibernate session for all of the dependent objects to use. Luckily Spring provides some help with this in the SessionScope class. This is the same base class as is used by the OpenSessionInView filter.
Using the Service interface you created, you then create a Job that Quartz.NET can run. Quartz.NET provides the IJob interface that you can implement. Spring.NET provides a base class that implements that interface called QuartzJobObject helps deal with injecting dependencies.
using NHibernate; using Quartz; using Spring.Data.NHibernate.Support; using Spring.Scheduling.Quartz; public class CustomJob : QuartzJobObject { private ISessionFactory sessionFactory; private IEmailService emailService; // Set only so they don't get serialized public ISessionFactory SessionFactory { set { sessionFactory = value; } } public IEmailService EmailService { set { emailService = value; } } protected override void ExecuteInternal(JobExecutionContext ctx) { // Session scope is the same thing as used by OpenSessionInView using (var ss = new SessionScope(sessionFactory, true)) { emailService.SendEveryoneEmails(); ss.Close(); } } }
Wiring Services and Jobs Together with Spring
Now that you have your classes created you need to wire everything together using Spring.
First we have our DALs and Services wired in to Spring with something like the following:
<object id="UserDAL" type="MyApp.DAL.UserDAL, MyApp.Data"> <property name="SessionFactory" ref="NHibernateSessionFactory" /> </object> <object id="EmailService" type="MyApp.Service.EmailService, MyApp.Service"> <property name="UserDAL" ref="UserDAL" /> </object>
Next you create a Job that references the Type of the Job that you just created. The type is referenced instead of the instance because the lifecycle of the Job is managed by Quartz itself. It deals with instantiation, serialization and deserialization of the object itself. This is a bit different than what you might expect from a Spring service normally.
<object id="CustomJob" type="Spring.Scheduling.Quartz.JobDetailObject, Spring.Scheduling.Quartz"> <property name="JobType" value="MyApp.Jobs.CustomJob, MyApp.Jobs" /> </object>
Once your Job is created, you create a Trigger that will run the Job based on your rules. Quartz (and Spring) offer two types of Jobs SimpleTriggers and CronTriggers. SimpleTriggers allow you to specify things like “Run this task every 30 minutes”. CronTriggers follow a crontab format for specifying when Jobs should run. The CronTrigger is very flexible but could be a little confusing if you aren’t familiar with cron. It’s worth getting to know for that flexibility though.
<object id="CustomJobTrigger" type="Spring.Scheduling.Quartz.CronTriggerObject, Spring.Scheduling.Quartz"> <property name="JobDetail" ref="CustomJob"/> <property name="CronExpressionString" value="0 0 2 * * ?" /> <!-- run every morning at 2 AM --> <property name="MisfireInstructionName" value="FireOnceNow" /> </object>
The last piece that needs to be done is the integration of the SchedulerFactory. The SchedulerFactory brings together Jobs and Triggers with all of the other configuration needed to run Quartz.NET jobs.
A couple of things to understand about configuring the SchedulerFactory:
- Specifying <property name=”DbProvider” ref=”DbProvider”/> (where DbProvider is the db:provider setup used by your Nhibernate configuration) tells the SchedulerFactory to use the AdoJobProvider and store the Jobs and Trigger information in the database. The tables will need to exist already and Quartz provides a script for this task.
- Running on SQL Server requires a slight change to Quartz. It uses a locking mechanism to prevent Jobs from running concurrently. For some reason the default configuration uses a FOR UPDATE query that is not supported by SQL Server. (I don’t understand exactly why a .NET utility wouldn’t work with SQL Server out of the box?)
To fix the locking a QuartzProperty needs to be set:
<entry key=”quartz.jobStore.selectWithLockSQL” value=”SELECT * FROM {0}LOCKS WHERE LOCK_NAME=@lockName”/> - The JobFactory is set to the SpringObjectJobFactory because it handles the injection of dependencies into QuartzJobObject like the one we created above.
- SchedulerContextAsMap is a property on the SchedulerFactory that allows you to set properties that will be passed to your Jobs when they are created by the SpringObjectJobFactory. This is where you set all of the Property names and the corresponding instance references to Spring configured objects. Those objects will be set into your Job instances whenever they are deserialized and run by Quartz.
Here’s the whole ScheduleFactory configuration put together:
<object id="SchedulerFactory" type="Spring.Scheduling.Quartz.SchedulerFactoryObject, Spring.Scheduling.Quartz"> <property name="JobFactory"> <object type="Spring.Scheduling.Quartz.SpringObjectJobFactory, Spring.Scheduling.Quartz"/> </property> <property name="SchedulerContextAsMap"> <dictionary> <entry key="EmailService" value-ref="EmailService" /> <entry key="SessionFactory" value-ref="NHibernateSessionFactory" /> </dictionary> </property> <property name="DbProvider" ref="DbProvider"/> <property name="QuartzProperties"> <dictionary> <entry key="quartz.jobStore.selectWithLockSQL" value="SELECT * FROM {0}LOCKS WHERE LOCK_NAME=@lockName"/> </dictionary> </property> <property name="triggers"> <list> <ref object="CustomJobTrigger" /> </list> </property> </object>
Conclusion
Scheduled tasks in ASP.NET applications shouldn’t be too much trouble anymore. Reusing existing Service and DAL classes allows you to easily create scheduled tasks using existing, tested code. Quartz.NET looks to be a good solution for these situations.
Subversion has a default Web UI that is served up by Apache if you run Subversion that way. It is pretty boring and read-only. Then there are things like WebSVN that make it less boring, but still read-only. I got curious about what it would take to make something even less boring and NOT read-only. Why not allow me to create a new repository on the server? Why not allow me to create a new default directory structure for a new project in a repository all through a web interface?
Parent Path Setup
All of my repositories use the idea of the SVNParentPath. This makes Apache assume that every directory under a given path is an SVN Repository. That structure makes it easy to deal with multiple repositories and secure them with a single security scheme. Using that assumption then it is easier to write some code that will list existing repositories and create new ones in a known location.
SvnAdmin With Ruby Subversion Bindings
Subversion provides language bindings for a number of different languages (Java, Python, Perl, PHP and Ruby) in addition to the native C libraries. Using these bindings it becomes fairly easy to deal with Subversion. The only hiccup will be dealing with the apparent lack of documentation for the code. So be prepared to do some exploration, digging and reading of the code.
I chose to try this using Ruby because it was quick and easy and it was a language I was already familiar with.
First you need to know how to create a new repository and open an existing repository. Fortunately those are simple, one-line operations:
Svn::Repos.create(repos_path, {}) repos = Svn::Repos.open(repos_path)
There was nothing special (from what I could tell) that would allow you to determine if a repository already existed, so I just created a simple function using the Ruby File operations to determine if a directory already existed. This code would allow me to determine if I needed to create new repository or open an existing one:
def repository_exists?(repos_path) File.directory?(repos_path) end
Now I have a repository open so I wanted to build a project structure using the default conventions I use for Subversions projects. My convention is to have a repository named after a client, the top-level directories are named for the client’s project and then each project has the standard trunk, branches and tags within that. Depending on the kind of work you do that convention may or may not make sense for you.
With that decided, I created the code to write that structure in a repository. The one thing I found is that interacting with the Subversion repository allowed you to do things within a transaction that would force all of the changes to be recorded as a single commit. I thought this was a good thing, so performed these operations as a transaction:
txn = repos.fs.transaction # create the top-level, project based directory txn.root.make_dir(project_name) # create the trunk, tags and branches for the new project %w(trunk tags branches).each do |dir| txn.root.make_dir("#{project_name}/#{dir}") end repos.commit(txn)
Finally I put all of those things together into a class. The class had the concept of being initialized to a base Parent Path so all of the operations would know to start from that location:
require "svn/repos" class SvnAdmin def initialize(parent_path) @base_path = parent_path end # Provides a list of directory entries. path must be a directory. def ls(client_name, path="/", revision=nil) repos_path = File.join(@base_path, client_name) repos = ensure_repository(repos_path) entries = repos.fs.root(revision).dir_entries(path) entries.keys end def create(client_name, project_name) repos_path = File.join(@base_path, client_name) repos = ensure_repository(repos_path) txn = repos.fs.transaction txn.root.make_dir(project_name) %w(trunk tags branches).each do |dir| txn.root.make_dir("#{project_name}/#{dir}") end repos.commit(txn) end private def ensure_repository(repos_path) if ! repository_exists?(repos_path) Svn::Repos.create(repos_path, {}) end repos = Svn::Repos.open(repos_path) end def repository_exists?(repos_path) File.directory?(repos_path) end end
SvnAdmin from Rails
Now that I had some simple code to create new repositories or add a new project to an existing repository I decided to wrap it in a simple Rails application that would allow me to create repositories using a web-based interface.
To start with, I’m not going to use a database or any ActiveRecord classes in this project (which you might do if you wanted authentication or something else) so I disabled ActiveRecord in the config/environment.rb
Then I created an ReposController to manage the Subversion repositories. This controller contains a couple of simple actions: You can also easily create a route and a ProjectsController that allows you to see all of the projects within a repository. The route in config/routes.rb is simply: And the ProjectsController looks up the :repos param to open the proper repository and list the top-level directories with it: Hopefully that will help you handle your Subversion administration. It should let you code up your conventions so that they are followed whenever a new repository is created or a new project is started.config.frameworks -= [ :active_record ]
require "svnadmin"
class ReposController < ApplicationController
layout 'default'
def index
@dirs = Dir.new(SVN_PARENT_PATH).entries.sort.reject {|p| p == "." or p == ".."}
end
def new
end
def create
svn = SvnAdmin.new(SVN_PARENT_PATH)
repos = params[:repos]
respond_to do |format|
begin
svn.create(repos[:client_name], repos[:project_name])
flash[:notice] = "Successfully created."
format.html { redirect_to :action => "index" }
format.xml { head :ok }
rescue
flash[:error] = "Failed to create structure."
format.html { redirect_to :action => "index" }
format.xml { render :xml => object.errors, :status => :unprocessable_entity }
end
end
end
end
map.connect ':repos/projects/',
:controller => 'projects',
:action => 'index'
require "svnadmin"
class ProjectsController < ApplicationController
layout 'default'
def index
repos_path = params[:repos]
svn = SvnAdmin.new(SVN_PARENT_PATH)
@projects = svn.ls(repos_path)
end
end
DRY your CruiseControl.NET Configuration
Don’t Repeat Yourself (DRY) is one of the principles of good software development. The idea is that there should ideally be one and only one “source of knowledge” for a particular fact or calculation in a system. Basically it comes down to not copying-and-pasting code around or duplicating code if at all possible. The advantages of this are many.
Advantages of DRY
- There will be less code to maintain
- If a bug is found, it should only have to be fixed in one place
- If an algorithm or process is changed, it only needs to be changed in one place
- More of the code should become reusable because as you do this you will parameterize methods to make them flexible for more cases
If it’s good for code isn’t it good for other things like configuration? Why yes it is.
Using CruiseControl.NET Configuration Builder
The Configuration Preprocessor allows you to define string properties and full blocks of XML to use for substitution and replacement. To start using the Configuration Preprocessor, you add xmlns:cb=”urn:ccnet.config.builder”, an xml namespace, to your document to tell the config parser that you plan to do this.
From there you can define a simple property like:
<cb:define client="xxx"/>Or you can make it a full block of XML:
<cb:define name="svn-block"> <sourcecontrol type="svn"> <trunkUrl>http://svn.example.com/svn/$(client)/$(project)/trunk</trunkUrl> <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory> <executable>svn.exe</executable> <autoGetSource>true</autoGetSource> </sourcecontrol> </cb:define>
Defining Reusable Blocks
Using these ideas I wanted to come up with a templated approach that would allow me to share configuration among multiple projects. That way, if I added new statistics or change the layout of my build server, I would only have to change it in a single place. Thus keeping things DRY. It also encourages more consistency across multiple projects making things easier to understand.
So, I started defining some reusable blocks in the main ccnet.config file which you can see below. The exact details will depend on your configuration of course.
Full Example of config.xml
<!-- How to add a new project: Step 1. Create a config file named "<config>-project.xml" Step 2. Add the project reference below --> <cruisecontrol xmlns:cb="urn:ccnet.config.builder"> <!-- cb defines to compose reusable blocks of configuration --> <!-- use <cb:define client="xxx"/> and <cb:define project="yyy"/> to use these --> <cb:define name="svn-block"> <sourcecontrol type="svn"> <trunkUrl>http://svn.example.com/svn/$(client)/$(project)/trunk</trunkUrl> <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory> <executable>svn.exe</executable> <autoGetSource>true</autoGetSource> </sourcecontrol> </cb:define> <cb:define name="msbuild-20-block"> <msbuild> <executable>C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\MSBuild.exe</executable> <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory> <projectFile>build.proj</projectFile> <buildArgs>$(build-args) </buildArgs> <targets>$(build-targets)</targets> <timeout>600</timeout> <logger>D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll</logger> </msbuild> </cb:define> <cb:define name="msbuild-35-block"> <msbuild> <executable>C:\WINDOWS\Microsoft.NET\Framework\v3.5\MSBuild.exe</executable> <workingDirectory>D:\Builds-Net\projects\$(client)\$(project)\trunk</workingDirectory> <projectFile>build.proj</projectFile> <buildArgs>$(build-args) </buildArgs> <targets>$(build-targets)</targets> <timeout>600</timeout> <logger>D:\Program Files\CruiseControl.NET\server\Rodemeyer.MsBuildToCCNet.dll</logger> </msbuild> </cb:define> <cb:define name="merge-block"> <!-- Merge the output of tests, code coverage and fxcop --> <merge> <files> <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.Test.xml</file> <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageMerge.xml</file> <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.CoverageSummary.xml</file> <file>D:\Builds-Net\projects\$(client)\$(project)\trunk\*.FxCop.xml</file> </files> </merge> </cb:define> <cb:define name="loggers-block"> <xmllogger> <logDir>D:\Builds-Net\projects\$(client)\$(project)\logs</logDir> </xmllogger> <rss/> <modificationHistory onlyLogWhenChangesFound="true" /> </cb:define> <cb:define name="stats-block"> <statistics> <statisticList> <firstMatch name="Svn Revision" xpath="//modifications/modification/changeNumber" /> <firstMatch name="Coverage" xpath="//coverageReport/project/@coverage" generateGraph="true"/> <firstMatch name="Warnings" xpath="//msbuild/@warning_count" generateGraph="true"/> <firstMatch name="Errors" xpath="//msbuild/@error_count" generateGraph="true"/> <!-- NDepend --> <!-- <firstMatch name="ILInstructions" xpath="//ApplicationMetrics/@NILInstruction" /> <firstMatch name="Avgerage Complexity" xpath="//ApplicationMetrics/MethodCC/@Avg" /> <firstMatch name="Max Complexity" xpath="//ApplicationMetrics/MethodCC/@MaxVal" /> <firstMatch name="LinesOfCode" xpath="//ApplicationMetrics/@NbLinesOfCode" generateGraph="true"/> <firstMatch name="LinesOfComment" xpath="//ApplicationMetrics/@NbLinesOfComment" generateGraph="true"/> --> </statisticList> </statistics> </cb:define> <cb:include href="config-client-project.xml"/> <cb:include href="config-client2-project-trunk.xml"/> </cruisecontrol>
At the end of the file you can see the cb:include references. Those are one-line includes to include the configuration of each project. This makes things easier to manage, I think, because you only have to look at the individual project configuration.
Using Reusable Blocks in Individual Configuration Files
From there I need to make use of those defined blocks in in individual file. The first thing I needed to do was to set the parameters that I had defined as simple string replacements in the reusable blocks. Normally you would do that with cb:define as I showed above. But the trick is that you can only have one property with a given name defined. If you include multiple project configurations that doesn’t work. What does work is using cb:scope definitions. This allows for a value to be defined only within a specific scope.
<cb:scope client="ExampleClient" project="SpecialProject" build-args="/p:Configuration=Debug" build-targets="Clean;Test"> ... </cb:scope>
From there you just need to start including the blocks that you defined in the main ccnet.confg within the scope block.
Full Example of Project Configuration
<!-- CruiseControl.NET configuration --> <project name="ExampleClient SpecialProject" xmlns:cb="urn:ccnet.config.builder"> <cb:scope client="ExampleClient" project="SpecialProject" build-args="/p:Configuration=Debug" build-targets="Clean;Test"> <cb:svn-block/> <tasks> <cb:msbuild-35-block/> </tasks> <publishers> <cb:merge-block/> <!-- Enable collection of project statistics --> <cb:stats-block/> <cb:loggers-block/> <email mailhost="smtp.example.com" from="ccnet@example.com" includeDetails="true"> <users> <user name="Developer One" group="buildmaster" address="dev1@example.com" /> <user name="Developer Two" group="developers" address="dev2@example.com" /> </users> <groups> <group name="developers" notification="change" /> <group name="buildmaster" notification="change" /> </groups> </email> </publishers> </cb:scope> </project>
As you can see, the only one I didn’t template out was the email block because that depends on the developers working on each project.
Have fun bringing simplicity and consistency to your Cruise Control.NET configuration!
For the full details see the CruiseControl.NET Configuration Preprocessor documentation.