Using Ruby Subversion Bindings to Create Repositories

Subversion has a default Web UI that is served up by Apache if you run Subversion that way. It is pretty boring and read-only. Then there are things like WebSVN that make it less boring, but still read-only. I got curious about what it would take to make something even less boring and NOT read-only. Why not allow me to create a new repository on the server? Why not allow me to create a new default directory structure for a new project in a repository all through a web interface?

Parent Path Setup

All of my repositories use the idea of the SVNParentPath. This makes Apache assume that every directory under a given path is an SVN Repository. That structure makes it easy to deal with multiple repositories and secure them with a single security scheme. Using that assumption then it is easier to write some code that will list existing repositories and create new ones in a known location.

SvnAdmin With Ruby Subversion Bindings

Subversion provides language bindings for a number of different languages (Java, Python, Perl, PHP and Ruby) in addition to the native C libraries. Using these bindings it becomes fairly easy to deal with Subversion. The only hiccup will be dealing with the apparent lack of documentation for the code. So be prepared to do some exploration, digging and reading of the code.

I chose to try this using Ruby because it was quick and easy and it was a language I was already familiar with.

First you need to know how to create a new repository and open an existing repository. Fortunately those are simple, one-line operations:

Svn::Repos.create(repos_path, {})
repos = Svn::Repos.open(repos_path)

There was nothing special (from what I could tell) that would allow you to determine if a repository already existed, so I just created a simple function using the Ruby File operations to determine if a directory already existed. This code would allow me to determine if I needed to create new repository or open an existing one:

def repository_exists?(repos_path)
File.directory?(repos_path)
end

Now I have a repository open so I wanted to build a project structure using the default conventions I use for Subversions projects. My convention is to have a repository named after a client, the top-level directories are named for the client’s project and then each project has the standard trunk, branches and tags within that. Depending on the kind of work you do that convention may or may not make sense for you.

With that decided, I created the code to write that structure in a repository. The one thing I found is that interacting with the Subversion repository allowed you to do things within a transaction that would force all of the changes to be recorded as a single commit. I thought this was a good thing, so performed these operations as a transaction:


txn = repos.fs.transaction

# create the top-level, project based directory
txn.root.make_dir(project_name)

# create the trunk, tags and branches for the new project
%w(trunk tags branches).each do |dir|
txn.root.make_dir("#{project_name}/#{dir}")
end

repos.commit(txn)

Finally I put all of those things together into a class. The class had the concept of being initialized to a base Parent Path so all of the operations would know to start from that location:

require "svn/repos"

class SvnAdmin
def initialize(parent_path)
@base_path = parent_path
end

# Provides a list of directory entries. path must be a directory.
def ls(client_name, path="/", revision=nil)
repos_path = File.join(@base_path, client_name)
repos = ensure_repository(repos_path)

entries = repos.fs.root(revision).dir_entries(path)
entries.keys
end

def create(client_name, project_name)
repos_path = File.join(@base_path, client_name)
repos = ensure_repository(repos_path)

txn = repos.fs.transaction
txn.root.make_dir(project_name)
%w(trunk tags branches).each do |dir|
txn.root.make_dir("#{project_name}/#{dir}")
end

repos.commit(txn)
end

private
def ensure_repository(repos_path)
if ! repository_exists?(repos_path)
Svn::Repos.create(repos_path, {})
end
repos = Svn::Repos.open(repos_path)
end

def repository_exists?(repos_path)
File.directory?(repos_path)
end

end

SvnAdmin from Rails

Now that I had some simple code to create new repositories or add a new project to an existing repository I decided to wrap it in a simple Rails application that would allow me to create repositories using a web-based interface.

To start with, I’m not going to use a database or any ActiveRecord classes in this project (which you might do if you wanted authentication or something else) so I disabled ActiveRecord in the config/environment.rb
config.frameworks -= [ :active_record ]

Then I created an ReposController to manage the Subversion repositories. This controller contains a couple of simple actions:

  1. An index action to list the existing repositories (directories)
  2. A new action to display a form to enter the client and project names
  3. A create action to use the SvnAdmin class to create a new repository and/or project


require "svnadmin"

class ReposController < ApplicationController layout 'default' def index @dirs = Dir.new(SVN_PARENT_PATH).entries.sort.reject {|p| p == "." or p == ".."} end def new end def create svn = SvnAdmin.new(SVN_PARENT_PATH) repos = params[:repos] respond_to do |format| begin svn.create(repos[:client_name], repos[:project_name]) flash[:notice] = "Successfully created." format.html { redirect_to :action => "index" }
format.xml { head :ok }
rescue
flash[:error] = "Failed to create structure."
format.html { redirect_to :action => "index" }
format.xml { render :xml => object.errors, :status => :unprocessable_entity }
end
end
end
end

You can also easily create a route and a ProjectsController that allows you to see all of the projects within a repository.

The route in config/routes.rb is simply:

map.connect ':repos/projects/',
:controller => 'projects',
:action => 'index'

And the ProjectsController looks up the :repos param to open the proper repository and list the top-level directories with it:

require "svnadmin"

class ProjectsController < ApplicationController layout 'default' def index repos_path = params[:repos] svn = SvnAdmin.new(SVN_PARENT_PATH) @projects = svn.ls(repos_path) end end

Hopefully that will help you handle your Subversion administration. It should let you code up your conventions so that they are followed whenever a new repository is created or a new project is started.

Windows Subversion Maintenance Scripts

Previously I wrote about the Subversion Maintenance Scripts that I use for doing things like backups and upgrades. They were bash shell scripts that automated dealing with multiple Subversion repositories. The secret was that those guys were being run using Cygwin on Windows. Recently we got a new, more powerful server to run our Virtuals on. I decided to rewrite the scripts using native Windows Batch files so that I wouldn’t have to install Cygwin again. Hopefully someone else will find this useful too.

Windows Batch Script for Creating a Default Repository Structure

I work for a custom software development consulting firm so we have a lot of different clients. We don’t want to mix different clients’ code so we create a repository per client. We often get multiple projects per client though (yeah, they like us) so we go ahead and create a project under the client. We also like to use the trunk, tags and branches directories by default.

With all of these rules: Automate!

create_repo.bat

@echo off

REM Check command line args to make sure a customer and project name were passed
IF (%1)==() GOTO CMDLINE
IF (%2)==() GOTO CMDLINE

REM If svnadmin is not in your path, uncomment the following
REM PATH=%PATH%;"D:\Program Files\Subversion\bin"

REM Set up variables with the directories
SET CUSTOMER=%1
SET PROJECT=%2
SET CUR_DIR=%~dp0
SET CUR_DIR=%CUR_DIR:\=/%
SET SVN_DIR=%CUR_DIR%/repos/%CUSTOMER%
SET PROJ_DIR=file:///%SVN_DIR%/%PROJECT%

echo Creating directory in %SVN_DIR%

svnadmin create "%SVN_DIR%"
svn -m "Create default structure." mkdir %PROJ_DIR% %PROJ_DIR%/trunk %PROJ_DIR%/tags %PROJ_DIR%/branches
GOTO END

:CMDLINE
echo Usage: %0 ^ ^

:END
echo Done

Some of the interesting pieces:
%~dp0 gives you the directory that the current script lives in.
%CUR_DIR:\=/% replaces the backslash (\) with forward slashes (/) so that the SVN file:// URL will work correctly.

Windows Batch Script to Dump Your Subversion Repositories

The next thing you might want to do is to dump all of your repositories so that you can back them up or upgrade SVN or move to a new server. With just one repository it’s easy to do by hand. With 20+ repositories it’s much nicer to run a script and then go do something else.

dump_all.bat

@echo off

SET DUMP_DIR=dumps
SET REPOS_DIR=repos

IF NOT EXIST %DUMP_DIR% (
echo Creating directory %DUMP_DIR%
md %DUMP_DIR%
)

echo Dumping...

FOR /f %%i IN ('dir /B /A:D %REPOS_DIR%\*') DO (
echo Dumping %REPOS_DIR%\%%i to %DUMP_DIR%\%%i.dump
svnadmin dump %REPOS_DIR%\%%i > %DUMP_DIR%\%%i.dump
)

Some of the interesting pieces:
FOR /f %%i IN (‘dir /B /A:D %REPOS_DIR%\*’) DO is the part that finds each directory (in this case an SVN repository). Then for each repository you run the svnadmin dump command and send the dump file to a different directory.

Windows Batch Script to Load Your Subversion Repositories

After you’ve dumped all of your repositories and you need to restore them, because you’ve upgraded or moved them to a new server we need to do the reverse of the dump script.

restore_all.bat

@echo off

SET DUMP_DIR=dumps
SET REPOS_DIR=repos

echo Restoring....

REM %%~ni becomes the dump name without .dump

FOR /f %%i IN ('dir /B %DUMP_DIR%\*.dump') DO (
echo Restoring %DUMP_DIR%\%%i to %REPOS_DIR%\%%~ni
svnadmin create %REPOS_DIR%\%%~ni
svnadmin load %REPOS_DIR%\%%~ni < %DUMP_DIR%\%%i )

Some of the interesting pieces:
FOR in this case is the same as the dump, except it's looking for all the dump files.
%%i holds the name of the dump file (e.g. repo.dump). %%~ni strips off the extension to give you just the name of the file (e.g. repo).

Now you can manage your repositories with ease if you're stuck on a Windows machine.

Automated Subversion Tagging With MSBuild

I’ve written previously about using MSBuild With Nunit as well as a bit of a Manifesto on Relentless Build Automation. I believe that automating the build and deployment process is a necessary step to ensure the reliable delivery of quality software.

Release Management

One of the things that we as software developers have to do is regularly make releases, either to QA or to a customer. When we make releases we need to be able to continue to develop the software to add new features and fix bugs. Tagging is a safety process that we use in a version control system to allow us to easily get to the code that we used to build a specific release. When you Tag, you can always rebuild that release. So, if weeks or months down the road you need to fix a critical bug, but don’t want to release new features, you can get back to the Tag, create a Branch, fix the bug and release the new build to your users.

How Do We Ensure We Can Recreate Releases

How can we ensure that we will be able to recreate a release that we make to either QA or a customer? Use Automation to Tag your builds when you create them of course.

I’ve contributed a new SvnCopy Task to the MSBuild Community Tasks project which was just accepted and committed. It is currently in the Subversion repository for the project but should be available shortly in an official build. This will allow you to easily automate the process of Tagging or Branching your builds when your release. Subversion uses the Copy metaphor for both Branching and Tagging operations which is different from some other version control systems.

Example:












You can then integrate the process of creating a Tag every time you generate a build by tying together Tasks with dependencies. In the example below, the GenerateTestBuild calls GenerateCabFiles and Tag to automatically build the installer and Tag Subversion with the current revisions number.














Hopefully this will help you get started on some more automation.

Update:
MSBuild Community Tasks version 1.2 has been released containing this code. You can get it here.

Resources

MSBuild Community Tasks
Subversion

Pragmatic Version Control Using Subversion

When I was in grade school I had to do a biographical book review every year. Probably 5 years, from 2nd grade to 7th grade we would be required to do book reviews and inevitably we’d have to do a biography. We’ll, I hated biographies, so every year I would do the same person: Jacque Cousteau. So, I’m not going to do any biographies, but I will review some technical books that I read.

Pragmatic Version Control Using Subversion

Pragmatic Version Control Using Subversion by Mike Mason

Index:

  1. Introduction
  2. What is Version Control?
  3. Getting Started with Subversion
  4. How To…
  5. Common Subversion Commands
  6. File Locking and Binary Files
  7. Organizing Your Repository
  8. Using Tags and Branches
  9. Creating a Project
  10. Third-Party Code

This book is not written as an exhaustive reference of of all of the Subversion commands. For that you can use the Subversion Red Bean book. Instead, this book is written as a series of scripts and best practices that you can use on your project. It’s more of a manual on how to use Subversion on an average project. I found this very helpful because it tackled the problems from a solution-oriented “why” approach, as opposed to merely a how approach.

Chapters 1 – 3

As would be expected, the early chapters offer a general introduction to the topic of the book. In the style of many of the Extreme Programming and Pragmatic Programmers books, these introduction include stories to illustrate the reasons and techniques that are outlined in more detail throughout the book. I find these stories to be very helpful because I think many people can identify with them.

Chapters 4 – 6

These chapters do a good job of covering the basics of source control. If you’ve been using source control such as Subversion or CVS, these chapters will likely not offer you a lot of new information. If you are new to using version control systems, then these chapters will give you a great overview of the basic day-to-day operations that you need to be an effective version control user.

Chapter 7

Chapter 7 is entirely devoted to file locking and binary files. While file locking is interesting in certain situations, I’ve never personally found the need for it. I guess this is a good chapter to skim just to know the possibilities in case you ever run across a situation where you need to lock binary files. If you’re in a hurry, skip this one I guess.

Chapters 8 – 10

These chapters constituted some of the more interesting information in the book for me. They deal with the aspects of of source control that make it truly powerful. Branching, tagging and general project organization. Often people avoid using branching and tagging because it can be confusing. But this book lays out some really good strategies for leveraging these features.

I really liked the covering of using branching and tagging. In Subversion both of these operations are are performed using the same “svn copy” command. If both of these logical operations are performed using the same actual Subversion command, then why would you need to use both? Well these chapters answer this question well, for example, by laying out the use of tags to demarcate longer-running bug fix session for easy merging into the trunk or a release branch. I found these chapters to be the most interesting by far to someone who has used Subversion for a long time, but maybe hasn’t utilized it to it’s fullest extent (*ahem* that would be me probably).

Conclusion

Pragmatic Version Control does a great job of walking through the day-to-day tasks of using Subversion on a project. It covers the basics of updating, merging, committing, etc. It also offers some really good advice on branching and tagging (and the reasons to use both) for releases as well as for development reasons such as bug fixing. It also covers things such as dealing with vendor branches and dealing with shared projects by using svn:externals references to import other projects into your main subversion project.

There is also a version of this book written for CVS users. I’m a big fan of Subversion and so I would recommend any CVS users to seriously consider moving to Subversion. There is a great script cvs2svn that handles converting a CVS repository into a Subversion repository. It does a great job of collecting individual CVS file revisions into Subversion atomic commits. It also handles creating branches and tags for you so you won’t lose any of your version history.

Pragmatic Version Control using Subversion – Mike Mason – ISBN: 0-9776166-5-7

Subversion Maintenance Scripts

With the recent release of Subversion 1.4 I’m going through the process of updating repositories. I often like to use multiple repositories because Subversion versions the entire repository with each checkin (Atomic commits rule!). When I do that, I group all of the repositories in the same directory. It makes it really easy to configure Apache with the SVNParentPath directive. And if you’re using svn+ssh or something like that, then it’s just easier to remember where they are.

SVNParentPath Example

With this configuration it’s very easy to bring new repositories up because it doesn’t require any configuration change in Apache.



DAV svn
SVNParentPath s:/svn
SVNListParentPath on
SVNAutoversioning on

# authentication
AuthType Basic
AuthName "Subversion Access"
AuthUserFile s:/logins.txt
# Access is configured in the access file.
AuthzSVNAccessFile s:/access.txt
Require valid-user

The one obvious downside to having multiple repositories is managing things like backups and upgrades. You have a lot more to do than just dump and load a single repository. So to help handle that task, I created some simple shell scripts.

Dump All Repositories

This script will dump all of the SVN repositories in a given directory.


#!/bin/sh

# Assumes each directory is an SVN repository
# and creates a dump file for each of them.

for i in *; do
svnadmin dump $i > ${i}.dump;
done;

Recreate/Load All Dump Files

This script will take all of the dump files and create new repositories and then load the dump file into them. This is good for major revision changes such as to 1.4 where they have changed some structures and improved the efficiency of storing binary files for example.


#!/bin/sh

# Assumes a directory full of dump files
# Creates a new SVN repository and loads the dump file into it

for i in *.dump; do
# The %% syntax is a substring command in bash to strip off the
# last occurrence of the string that follows so Foo.dump -> Foo
repos=${i%%.dump};
svnadmin create $repos;
svnadmin load $repos < $i; done

Create New Repository Script

This is a handy one I keep around to make it easy to help enforce best practices. This creates a repository and then automatically creates the trunk/ branches/ and tags/ directories.


#!/bin/sh

if [ -z "$1" ]
then
echo "Usage: $0 ";
exit 1;
fi

# set up directories
cur_dir=`pwd`
# use these 2 if you're running on Cygwin under Windows
# cur_cyg_dir=`pwd`
# cur_dir=`cygpath -m $cur_cyg_dir`
svn_dir=${cur_dir}/${1}

svnadmin create "$1"
svn -m "Create default structure." mkdir file:///${svn_dir}/trunk file:///${svn_dir}/tags file:///${svn_dir}/branches

echo "done"

Not that these are incredibly complex shell scripts, but they can do a lot of work while you do other things (like post to your blog). I hope they'll help someone out there.

And just to be warned, if you're doing this on a bunch of big repositories it can take some time!

Getting the Revision Number of your Subversion Working Copy

I was asked today “How do I find out the revision number of my Subversion directory?” and I didn’t know the answer. The good news is there is an easy answer:
svnversion .

From the svnversion help command:

The version number will be a single number if the working
copy is single revision, unmodified, not switched and with
an URL that matches the TRAIL_URL argument. If the working
copy is unusual the version number will be more complex:

This command takes a little while to run, because it runs against the remote repository as well. It gives more detailed results also such as whether or not you have local modifications or if the repository has been changed since your last update. These details can be interesting in certain cases. But sometimes all you want to know is what revision you currently have checked out…

If all you want to do is check on the current revision of your local working copy you can use some Unix magic to do that.
grep revision .svn/entries | awk -F\" '{print $2}'

Redirect this output to a file or use it as an argument to a build script and you can have the revision number put into your application.
ant -Drevision=grep revision .svn/entries | awk -F\" '{print $2}'
(notice the backticks ` and this needs to be on one line)

A little Unix scripting and voila!

(Thanks to Ross Greinke for this little snippet)