Check Multiple Mercurial Repositories for Incoming Changes

Currently I have a whole bunch of Mercurial repositories in a directory. All of these are cloned from a central repository that the team pushes their changes to. I like to generally keep my local repositories up-to-date so that I can review changes. Manually running hg incoming -R some_directory on 20 different projects is a lot of work. So I automated it with a simple shell script.

This script will run incoming (or outgoing) on all of the local repositories and print the results to the console. Then I can manually sync the ones that have changed if I want.

I called this file hgcheckall.sh and run it like: ./hgcheckall.sh incoming

#!/bin/bash
 
# Find all the directories that are mercurial repos
dirs=(`find . -name ".hg"`)
# Remove the /.hg from the path and that's the base repo dir
merc_dirs=( "${dirs[@]//\/.hg/}" )
 
case $1 in
    incoming)
    for indir in ${merc_dirs[@]}; do
        echo "Checking: ${indir}"
        hg -R "$indir" incoming
    done
    ;;
    outgoing)
    for outdir in ${merc_dirs[@]}; do
        echo "Checking: ${outdir}"
        hg -R "$outdir" outgoing
    done
    ;;
    *)
    echo "Usage: hgcheckall.sh [incoming|outgoing]"
    ;;
esac

I guess the next major improvement would be to capture the output and then automatically sync the ones that have changed, but I haven’t gotten around to that yet.

Start a New Branch on your Remote Git Repository

Git is a distributed version control system so it allows you to create branches locally and commit against them. It also supports a more centralized repository model. When using a centralized repository you can push changes to it so that others can pull them more easily. I have a tendency to work on multiple computers. Because of this, I like to use a centralized repository to track the branches as I work on them. That way no matter what machine I’m on, I can still get at my branches.

The Workflow

My workflow is generally something like this:

  1. Create a remote branch
  2. Create a local branch that tracks it
  3. Work, Test, Commit (repeat) – this is all local
  4. Push (pushes commits to the remote repository)

Git commands can be a bit esoteric at times and I can’t always seem to remember how to create a remote git branch and then start working on new code. There also seems to be multiple ways of doing it. I’m documenting the way that seem to work for me so that I can remember it. Maybe it will help someone else too.

Creating a Remote Branch

  1. Create the remote branch

    git push origin origin:refs/heads/new_feature_name

  2. Make sure everything is up-to-date

    git fetch origin

  3. Then you can see that the branch is created.

    git branch -r

This should show ‘origin/new_feature_name’

  1. Start tracking the new branch
    git checkout --track -b new_feature_name origin/new_feature_name

This means that when you do pulls that it will get the latest from that branch as well.

  1. Make sure everything is up-to-date
    git pull

Cleaning up Mistakes

If you make a mistake you can always delete the remote branch

git push origin :heads/new_feature_name

(Ok Git’ers – that has to be the least intuitive command ever.)

Use the Branch from Another Location

When you get to another computer or clone the git repository to a new computer, then you just need to start tracking the new branch again.

git branch -r

to show all the remote branches

git checkout --track -b new_branch origin/new_feature_name

to start tracking the new branch

Automate it A Bit

That’s a pretty easy thing to automate with a small shell script luckily

</p>
 
<h1>!/bin/sh</h1>
 
<h1>git-create-branch <branch_name></h1>
 
<p>if [ $# -ne 1 ]; then
         echo 1>&amp;2 Usage: $0 branch_name
         exit 127
fi</p>
 
<p>set branch_name = $1
git push origin origin:refs/heads/${branch_name}
git fetch origin
git checkout --track -b ${branch_name} origin/${branch_name}
git pull

For further help, you might want to check out:

Pragmatic Project Automation

In a previous post, I wrote a review of Pragmatic Version Control Using Subversion which covers effectively using version control systems on software development projects. This entry covers another book by the Pragmatic Programmers in their Pragmatic Starter Series.

Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps

Pragmatic Project Automation by Mike Clark

Index:

  1. Introduction
  2. One-Step Builds
  3. Scheduled Builds
  4. Push-Button Releases
  5. Installation and Deployment
  6. Monitoring

Pragmatic Project Automation shares a lot with the rest of the Pragmatic Starter Series. As the name implies these books are intended to get you started down the path without necessarily showing you the entire way. The book is relatively short coming in at a slim 152 pages. This makes it a quick read and allows it to cover the topic from a high level. It sells the idea of automation and shows how to do automation in all aspects of software development. It shows examples everything from the simplest scripts using Unix shell and Windows tools all the way up to purpose-build programs that will make automation very easy for larger and more complex projects.

Chapter 1 – Introduction

The introduction lays the groundwork for automation. It categorizes automation into 3 types:

  • Scheduled Automation
  • Triggered Automation
  • Commanded Automation

Commanded Automation is probably the most familiar to most Java developers. This is the idea that you run a small script or an Ant build. This kind of automation is user driven and run when they want to do a certain task. Scheduled Automation, as it’s name implies, is something that is run based on time while Triggered Automation is automation that happens in response to a change in environment such as checking in code to a version control system. This groudwork is built upon during the rest of the book.

Chapter 2 – One-Step Builds

This is the most common kind of automation from what I’ve seen. This chapter covers building source code using a script. Being a Pragmatic book, it starts with a simple example using a shell script. It then does a good job of building a good template for an Ant build. One of the interesting things in this chapter is that it discusses using Groovy, a Java based scripting language, that has built-in support for Ant scripting. In many ways this Groovy example is very interesting because it’s a real scripting language so you can easily create your own custom code without the overhead of creating an Ant task. Maybe JRuby will catch on here (hint, hint JRuby folks) as I like the Ruby syntax a lot more than Groovy.

Chapter 3 – Scheduled Builds

Chapter 3 starts with the easiest way of doing a scheduled build, using cron on Unix or at on Windows to periodically run a build script. I really like the fact that the authors mentioned this because I really think it lowers the barrier to people getting into automation. There’s no need to over engineer a solution if you have simple needs. Of course this simple technique is fairly limited so they don’t stop there.

This chapter also covers using CruiseControl for doing both Scheduled and Triggered builds. It walks through the setup of your first project in CruiseControl. I’m a huge fan of Continuous Integration because it’s an always on backup for the processes that your team wants to enforce.

Chapter 4 – Push-Button Releases

Doing a build which you do everyday, hopefully multiple times a day, is one thing, but how do you move code from source control to QA, a staging server or a production server? Well you automate it of course! This chapter covers creating a script that you can use to turn your code into a form you can send to a customer. The simple examples are just creating the proper artifacts and creating archives like Zip or Tar files. It also talks about a number of processes including using automation to manage branching and tagging in your version control system. The branching and tagging automation not only makes the “release manager’s” life easier, it really helps to create a consistent process that means you’ll always be able to find and recreate any version in the future.

Chapter 5 – Installation and Deployment

This chapter covers 2 slightly unrelated (to me) but interesting ideas. The most obvious based on the name is creating an installer. It discusses everything from a simple Zip file “install” to using a full-blown installation system. The book uses the Open Source Nullsoft Scripted Installer (NSIS) as an example of building a Windows install. I’ve never used NSIS before so I can’t speak for its strengths or weaknesses, but it is used in a lot of Open Source projects. The introduction is very brief, so plan on doing some other reading if you want to use Nullsoft.

Chapter 6 – Monitoring

To wrap the book up, we get the Make chapter of the book. Now that you have triggered and scheduled builds running and applications in production, how do you know when there is a problem? This chapter talks about a number of good ideas starting with the easy ones of email and cell phones but taking some unanticipated turns into things like RSS feeds as well.

Some of the more interesting ideas where using custom applications to drive visual feedback monitors. Two of the more interesting examples where using Lava Lamps and Ambient Orbs to signal when a build is broken. The Lava Lamps would sit in the office and turn on/off red or green Lava Lamps based on the build status. Even more interesting is the Ambient Orb. It can glow in multiple colors. It communicates over a national wireless network like the pager network. So now you can take your magic orb with you and it will notify you when there’s a problem with the build.

The only problem with this chapter is that it really doesn’t have any implementations for these cool ideas. It leaves a lot up to the imagination.

Conclusion

The book is 152 pages, as I said above, which means that it is by no means a comprehensive reference to any of the topics covered. If you are already sold on the idea of automation, this would be good ammunition to convince other people on a team or in management to support the idea. If you are completely unfamiliar with the topic, then this book will offer you a good basis for getting started. If you are already familiar with the topic and are looking to expand on what you know or get ideas for more ways to automate then you probably won’t get a lot out of this book and should probably look elsewhere.

Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps

Other Related

Ant
Cruise Control
Java Development with Ant

Pragmatic Version Control Using Subversion

When I was in grade school I had to do a biographical book review every year. Probably 5 years, from 2nd grade to 7th grade we would be required to do book reviews and inevitably we’d have to do a biography. We’ll, I hated biographies, so every year I would do the same person: Jacque Cousteau. So, I’m not going to do any biographies, but I will review some technical books that I read.

Pragmatic Version Control Using Subversion

Pragmatic Version Control Using Subversion by Mike Mason

Index:

  1. Introduction
  2. What is Version Control?
  3. Getting Started with Subversion
  4. How To…
  5. Common Subversion Commands
  6. File Locking and Binary Files
  7. Organizing Your Repository
  8. Using Tags and Branches
  9. Creating a Project
  10. Third-Party Code

This book is not written as an exhaustive reference of of all of the Subversion commands. For that you can use the Subversion Red Bean book. Instead, this book is written as a series of scripts and best practices that you can use on your project. It’s more of a manual on how to use Subversion on an average project. I found this very helpful because it tackled the problems from a solution-oriented “why” approach, as opposed to merely a how approach.

Chapters 1 – 3

As would be expected, the early chapters offer a general introduction to the topic of the book. In the style of many of the Extreme Programming and Pragmatic Programmers books, these introduction include stories to illustrate the reasons and techniques that are outlined in more detail throughout the book. I find these stories to be very helpful because I think many people can identify with them.

Chapters 4 – 6

These chapters do a good job of covering the basics of source control. If you’ve been using source control such as Subversion or CVS, these chapters will likely not offer you a lot of new information. If you are new to using version control systems, then these chapters will give you a great overview of the basic day-to-day operations that you need to be an effective version control user.

Chapter 7

Chapter 7 is entirely devoted to file locking and binary files. While file locking is interesting in certain situations, I’ve never personally found the need for it. I guess this is a good chapter to skim just to know the possibilities in case you ever run across a situation where you need to lock binary files. If you’re in a hurry, skip this one I guess.

Chapters 8 – 10

These chapters constituted some of the more interesting information in the book for me. They deal with the aspects of of source control that make it truly powerful. Branching, tagging and general project organization. Often people avoid using branching and tagging because it can be confusing. But this book lays out some really good strategies for leveraging these features.

I really liked the covering of using branching and tagging. In Subversion both of these operations are are performed using the same “svn copy” command. If both of these logical operations are performed using the same actual Subversion command, then why would you need to use both? Well these chapters answer this question well, for example, by laying out the use of tags to demarcate longer-running bug fix session for easy merging into the trunk or a release branch. I found these chapters to be the most interesting by far to someone who has used Subversion for a long time, but maybe hasn’t utilized it to it’s fullest extent (*ahem* that would be me probably).

Conclusion

Pragmatic Version Control does a great job of walking through the day-to-day tasks of using Subversion on a project. It covers the basics of updating, merging, committing, etc. It also offers some really good advice on branching and tagging (and the reasons to use both) for releases as well as for development reasons such as bug fixing. It also covers things such as dealing with vendor branches and dealing with shared projects by using svn:externals references to import other projects into your main subversion project.

There is also a version of this book written for CVS users. I’m a big fan of Subversion and so I would recommend any CVS users to seriously consider moving to Subversion. There is a great script cvs2svn that handles converting a CVS repository into a Subversion repository. It does a great job of collecting individual CVS file revisions into Subversion atomic commits. It also handles creating branches and tags for you so you won’t lose any of your version history.

Pragmatic Version Control using Subversion – Mike Mason – ISBN: 0-9776166-5-7

Subversion Maintenance Scripts

With the recent release of Subversion 1.4 I’m going through the process of updating repositories. I often like to use multiple repositories because Subversion versions the entire repository with each checkin (Atomic commits rule!). When I do that, I group all of the repositories in the same directory. It makes it really easy to configure Apache with the SVNParentPath directive. And if you’re using svn+ssh or something like that, then it’s just easier to remember where they are.

SVNParentPath Example

With this configuration it’s very easy to bring new repositories up because it doesn’t require any configuration change in Apache.

<Location /svn>
    DAV svn
    SVNParentPath s:/svn
    SVNListParentPath on
    SVNAutoversioning on
 
    # authentication
    AuthType Basic
    AuthName "Subversion Access"
    AuthUserFile s:/logins.txt
    # Access is configured in the access file.
    AuthzSVNAccessFile s:/access.txt
    Require valid-user
</Location>

The one obvious downside to having multiple repositories is managing things like backups and upgrades. You have a lot more to do than just dump and load a single repository. So to help handle that task, I created some simple shell scripts.

Dump All Repositories

This script will dump all of the SVN repositories in a given directory.

#!/bin/sh
 
# Assumes each directory is an SVN repository
# and creates a dump file for each of them.
 
for i in *; do
    svnadmin dump $i > ${i}.dump;
done;

Recreate/Load All Dump Files

This script will take all of the dump files and create new repositories and then load the dump file into them. This is good for major revision changes such as to 1.4 where they have changed some structures and improved the efficiency of storing binary files for example.

#!/bin/sh
 
# Assumes a directory full of dump files
# Creates a new SVN repository and loads the dump file into it
 
for i in *.dump; do
    # The %% syntax is a substring command in bash to strip off the 
    # last occurrence of the string that follows so Foo.dump -> Foo
    repos=${i%%.dump};
    svnadmin create $repos;
    svnadmin load $repos < $i;
done

Create New Repository Script

This is a handy one I keep around to make it easy to help enforce best practices. This creates a repository and then automatically creates the trunk/ branches/ and tags/ directories.

#!/bin/sh
 
if [ -z "$1" ]
then
    echo "Usage: $0 <repository>";
    exit 1;
fi
 
# set up directories
cur_dir=`pwd`
# use these 2 if you're running on Cygwin under Windows
# cur_cyg_dir=`pwd`
# cur_dir=`cygpath -m $cur_cyg_dir`
svn_dir=${cur_dir}/${1}
 
svnadmin create "$1"
svn -m "Create default structure." mkdir file:///${svn_dir}/trunk file:///${svn_dir}/tags file:///${svn_dir}/branches
 
echo "done"

Not that these are incredibly complex shell scripts, but they can do a lot of work while you do other things (like post to your blog). I hope they’ll help someone out there.

And just to be warned, if you’re doing this on a bunch of big repositories it can take some time!