Java 7 Code Coverage with Gradle and Jacoco

Thanks

Steven Dicks’ post on Jacoco and Gradle is a great start to integrating Jacoco and Gradle, this is a small iteration on top of that work.

Java 7 Code Coverage

The state of Code Coverage took a serious turn for the worst when Java 7 came out. The byte-code changes in Java 7 effectively made Emma and Cobertura defunct. They will not work with Java 7 constructs. Fortunately there is a new player in town called JaCoCo (for Java Code Coverage). JaCoCo is the successor to Emma which is being built on the knowledge gained over the years by the Eclipse and Emma teams on how best to do code coverage. And works with Java 7 out-of-the-box.

The advantage of using established tools is that they generally are well supported across your toolchain. JaCoCo is fairly new and so support in Gradle isn’t so smooth. Fortunately Steven’s post got me started down the right path. The one thing that I wanted to improve right away was to use transitive dependency declarations as opposed to having local jar files in my source repository. JaCoCo is now available in the Maven repos so we can do that. One thing to note is that the default files build in the Maven repo are Eclipse plugins, so we need to reference the “runtime” classifier in our dependency

The Gradle Script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
configurations {
    codeCoverage
    codeCoverageAnt
}
dependencies {
    codeCoverage 'org.jacoco:org.jacoco.agent:0.5.10.201208310627:runtime@jar'
    codeCoverageAnt 'org.jacoco:org.jacoco.ant:0.5.10.201208310627'
}
test {
    systemProperties = System.properties
    jvmArgs "-javaagent:${configurations.codeCoverage.asPath}=destfile=${project.buildDir.path}/coverage-results/jacoco.exec,sessionid=HSServ,append=false",
            '-Djacoco=true',
            '-Xms128m',
            '-Xmx512m',
            '-XX:MaxPermSize=128m'
}
task generateCoverageReport << {
    ant {
        taskdef(name:'jacocoreport', classname: 'org.jacoco.ant.ReportTask', classpath: configurations.codeCoverageAnt.asPath)

        mkdir dir: "build/reports/coverage"

        jacocoreport {
            executiondata {
                fileset(dir: "build/coverage-results") {
                    include name: 'jacoco.exec'
                }
            }
            structure(name: project.name) {
                classfiles {
                    fileset(dir: "build/classes/main") {
                        exclude name: 'org/ifxforum/**/*'
                        exclude name: 'org/gca/euronet/generated**/*'
                    }
                }
                sourcefiles(encoding: 'CP1252') {
                    fileset dir: "src/main/java"
                }
            }

            xml  destfile: "build/reports/coverage/jacoco.xml"
            html destdir:  "build/reports/coverage"
        }
    }
}

A Few Details

The magic is in the jvmArgs of the test block. JaCoCo is run as a Java Agent which uses the runtime instrumentation feature added in Java 6 to be able to inspect the running code. Extra arguments can be added to JaCoCo there including things like excludes to exclude specific classes from coverage. The available parameters are the same as the maven JaCoCo parameters.

The generateCoverageReport task converts the jacoco.exec binary into html files for human consumption. If you’re just integrating with a CI tool, like Jenkins, then you probably don’t need this, but it’s handy for local use and to dig into the details of what’s covered.

Loose Ends

One problem that I ran into was referencing project paths like the project.buildDir from within an Ant task. Hopefully someone will come along and let me know how that’s done.

Posted in Automation, Code, Groovy, Java | Tagged , , | 2 Comments

Groovy Mocking a Method That Exists on Object

In Groovy, the find method exists on Object. The find method is also an instance method on EntityManager which is commonly used in JPA to get an instance based on a database id. I was trying to create a Mock of EntityManager like:

1
2
3
def emMock = new MockFor(EntityManager)
emMock.demand.find { Class clazz, Object rquid -> return new Request(rqUID: rquid) }
def em = emMock.proxyDelegateInstance()

This gave me an error though:

1
2
3
4
5
6
groovy.lang.MissingMethodException: No signature of method: com.onelinefix.services.RequestRepositoryImplTest$_2_get_closure1.doCall() is applicable for argument types: (groovy.mock.interceptor.Demand) values: [groovy.mock.interceptor.Demand@a27ebd9]
Possible solutions: doCall(java.lang.Class, java.lang.Object), findAll(), findAll(), isCase(java.lang.Object), isCase(java.lang.Object)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) ~[groovy-all-1.8.6.jar:1.8.6]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877) ~[groovy-all-1.8.6.jar:1.8.6]
    at groovy.lang.Closure.call(Closure.java:412) ~[groovy-all-1.8.6.jar:1.8.6]
    at groovy.lang.Closure.call(Closure.java:425) ~[groovy-all-1.8.6.jar:1.8.6]

It ends up that find is a DefaultGroovyMethod that’s added to all Objects and this was causing the mock.demand to get confused because it thought I was trying to call the Object.find.

Luckily there is a workaround and that’s to use the ordinal argument for find.

1
emMock.demand.find(1) { Class clazz, Object rquid -> return new Request(rqUID: rquid) }

That’s enough of a change to invoke mock version instead of the DefaultGroovyMethod version.
Now if I only knew why….

Posted in Code, Groovy, Unit Testing | Tagged , , , | Leave a comment

Testing and Internal Implementation in .NET

Switching back and forth between Java and .NET lets you see some of the differences between the two platforms more easily. This happened to me the other day when I switched from Java to .NET and was writing Unit Tests. In Java, the access modifiers include public, private, protected and default. In C# they are public, private, protected and internal. In general, the public, private access modifiers are very similar. Protected is slightly different in that Java allows both derived classes as well as classes in the same package to access those elements where C# only allows derived classes to access them. Where things diverge more is in the default/internal differences. Default in java restricts access to the same package while internal in C# restricts access to the same Assembly (generally a single DLL).

What does this have to do with testing you might ask?

It’s a good OO design principle to expose only those things that are part of the contract to a class or package and to leave the implementation hidden as much as possible. This is called encapsulation. You can make methods private or default/internal. You can make entire classes default/internal and only publicly expose an interface that clients need to use.

A common practice in the Java world is to mimic the package layout of your main source code in your test code. When you mimic that layout then your test classes and implementation classes end up being in the same packages. Because of this your test classes can access all those default members to test them. In C# because it’s not based on a namespace, but rather an Assembly this doesn’t work.

Luckily there’s an easy workaround.

In the AssemblyInfo.cs of your main project add:

1
[assembly: InternalsVisibleTo("someOther.AssemblyName.Test")]

Where SomeOther.AssemblyName.Test is the name of the Assembly that contains your tests for the target assembly. Then the test code can access internal details of the assembly. And you can easily test the things that other calling code might not have access to.

Posted in .NET | Tagged , , | 1 Comment

Using Groovy AST to Add Common Properties to Grails Domain Classes

Groovy offers a lot of runtime meta-programming capabilities that allow you to add reusable functionality in a shared fashion. Grails plugins make use of this ability to enhance your project. One of the things that you can’t do with runtime meta-programming in Grails is to add persistent Hibernate properties to your domain classes. If you want to add a persistent property in a plugin (or otherwise using meta-programming) for your Grails project you have to make use of “compile-time” meta-programming. In Groovy this is done with AST Transformations.

(If you are unfamiliar with the concept of the Abstract Syntax Tree, see the Wikipedia article on Abstract Syntax Tree.)

AST Transformations are made up of two parts: (1) An annotation and (2) an ASTTransformation implementation. During compilation the Groovy compiler finds all of the Annotations and calls the ASTTransformation implementation for the annotation passing in information about.

To create your own Transformation you start by creating an Annotation. The key to the annotation working is that your annotation has to itself be annotated with @GroovyASTTransformationClass. The values passed to the GroovyASTTransformationClass define the Transformation that will be called on classes, methods or other code prior to it being compiled.

Example Annotation

1
2
3
4
5
6
7
8
9
10
package net.zorched.grails.effectivity;

import org.codehaus.groovy.transform.GroovyASTTransformationClass;
import java.lang.annotation.*;

@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@GroovyASTTransformationClass({"net.zorched.grails.effectivity.EffectivizeASTTransformation"})
public @interface Effectivize {
}

Notice the reference to net.zorched.grails.effectivity.EffectivizeASTTransformation. That’s the important part because it defines the class that will be used to perform the transformation.

Example Transformation

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
package net.zorched.grails.effectivity;

import org.codehaus.groovy.ast.*;
import org.codehaus.groovy.ast.builder.AstBuilder;
import org.codehaus.groovy.ast.expr.*;
import org.codehaus.groovy.ast.stmt.*;
import org.codehaus.groovy.control.*;
import org.codehaus.groovy.transform.*;
import java.util.*;
import static org.springframework.asm.Opcodes.*;

@GroovyASTTransformation(phase = CompilePhase.CANONICALIZATION)
public class EffectivizeASTTransformation implements ASTTransformation {

    // This is the main method to implement from ASTTransformation that is called by the compiler
    public void visit(ASTNode[] nodes, SourceUnit sourceUnit) {
        if (null == nodes) return;
        if (null == nodes[0]) return;
        if (null == nodes[1]) return;
        if (!(nodes[0] instanceof AnnotationNode)) return;

        ClassNode cNode = (ClassNode) nodes[1];
        addProperty(cNode, "effectiveStart", Date.class, createGenerateStartMethodCall())
        addProperty(cNode, "effectiveEnd", Date.class, createGenerateEndMethodCall())

    }

    // This method returns an expression that is used to initialize the newly created property
    private Expression createGenerateStartMethodCall() {
        return new ConstructorCallExpression(new ClassNode(Date.class), ArgumentListExpression.EMPTY_ARGUMENTS);
    }

    private Expression createGenerateEndMethodCall() {
        return new MethodCallExpression(
                new ConstructorCallExpression(new ClassNode(Date.class), ArgumentListExpression.EMPTY_ARGUMENTS),
                "parse",
                new ArgumentListExpression(new ConstantExpression("yyyy/MM/dd"), new ConstantExpression("2099/12/31")));
    }

    // This method adds a new property to the class. Groovy automatically handles adding the getters and setters so you
    // don't have to create special methods for those
    private void addProperty(ClassNode cNode, String propertyName, Class propertyType, Expression initialValue) {
        FieldNode field = new FieldNode(
                propertyName,
                ACC_PRIVATE,
                new ClassNode(propertyType),
                new ClassNode(cNode.getClass()),
                initialValue
        );

        cNode.addProperty(new PropertyNode(field, ACC_PUBLIC, null, null));
    }
}

This example code gets called for each annotated class and adds two new Date properties called effectiveStart and effectiveEnd to it. Those properties are seen by Grails and Hibernate and will become persistent and behave the same as if you typed them directly in your Domain.

It’s a lot of work to add a simple property to a class, but if you’re looking to consistently add properties and constraints across many Grails Domain classes, this is the way to do it.

Posted in Code, Groovy | Tagged , , | 5 Comments

Check Multiple Mercurial Repositories for Incoming Changes

Currently I have a whole bunch of Mercurial repositories in a directory. All of these are cloned from a central repository that the team pushes their changes to. I like to generally keep my local repositories up-to-date so that I can review changes. Manually running hg incoming -R some_directory on 20 different projects is a lot of work. So I automated it with a simple shell script.

This script will run incoming (or outgoing) on all of the local repositories and print the results to the console. Then I can manually sync the ones that have changed if I want.

I called this file hgcheckall.sh and run it like: ./hgcheckall.sh incoming

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash

# Find all the directories that are mercurial repos
dirs=(`find . -name ".hg"`)
# Remove the /.hg from the path and that's the base repo dir
merc_dirs=( "${dirs[@]//\/.hg/}" )

case $1 in
    incoming)
    for indir in ${merc_dirs[@]}; do
        echo "Checking: ${indir}"
        hg -R "$indir" incoming
    done
    ;;
    outgoing)
    for outdir in ${merc_dirs[@]}; do
        echo "Checking: ${outdir}"
        hg -R "$outdir" outgoing
    done
    ;;
    *)
    echo "Usage: hgcheckall.sh [incoming|outgoing]"
    ;;
esac

I guess the next major improvement would be to capture the output and then automatically sync the ones that have changed, but I haven’t gotten around to that yet.

Posted in Automation | Tagged , , , | 2 Comments

Java Return from Finally

try…catch…finally is the common idiom in Java for exception handling and cleanup. The thing that people may not know is that returning from within a finally block has the unintended consequence of stoping an exception from propagating up the call stack. It “overrides” the throwing of an exception so that the caller will never get to handle it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class Main {
    public static void main(String[] args) throws Throwable {
        System.out.println("Starting");
        method();
        System.out.println("No way to know that an exception was thrown");
    }

    public static void method() throws Throwable {
        try {
            System.out.println("In method about to throw an exception.");
            throw new RuntimeException();
        } catch (Throwable ex) {
            System.out.println("Caught exception, maybe log it, and then rethrow it.");
            throw ex;
        } finally {
            System.out.println("return in finally prevents an exception from being passed up the call stack.");
            return; // remove the return to see the real behavior
        }
    }
}

I recently came across code like this. This is Real Bad. Returning from within finally prevents the propagation of regular exceptions, which is bad enough, but worse, prevents the propagation of runtime exceptions which are generally programmer errors. This one small mistake can hide programmer errors so that you’ll never see them and never know why things aren’t working as expected. One of the interesting things is that the Java compiler understands this as well. If you return from within a finally block where an exception would otherwise be thrown, the compiler does not force you to declare that exception in the method’s throws declaration.

Long story short. Don’t return from with finally. Ever.

Posted in Java | Tagged , , | 2 Comments

USB to Serial Adapters in VMWare

I needed to do some work using a pin pad (a device that allows you to enter a numeric code at a point-of-sale or other system) and needed to test it in a 32bit Windows environment. The pin pad uses a serial port to communicate to a computer. Of course no portable computers (maybe no desktops either, it’s been years since I’ve had one) have serial ports anymore, so you have to use a USB to Serial adapter. It seems like many brands of these converters use a chip by a company called Prolific. Prolific makes the chip that lives in the converter that does the translation from serial to USB and back. I found the drivers from prolific and the device loaded just fine on Windows 7. I then created a Windows XP image in VMWare.

This is where the fun started.

I loaded the driver in Windows XP but no matter what I tried I couldn’t get VMWare to take control of the adapter. Every time I chose Virtual Machine -> Removable Devices -> Prolific USB device -> Connect it would give me an error: “Driver error”. Nice and specific and helpful right? I rebooted the host and the guest, tried without the driver installed on the host and rebooted again. Try as I might nothing worked.

Long story short, I was plugging the adapter into a USB3 port and either VMWare or the driver didn’t like USB3. So If you are seeing a similar problem, try and find out what kind of USB port it is. I was using Windows 7 on a Lenovo W510. This machine has USB2 and USB3 ports. The 2 obvious USB ports on the left hand side are USB3. There is a USB2 port on the back, but I keep another device plugged into that one so I didn’t even think to try it. It turns out though that the W510 has dual ESATA/USB2 port next to the USB3 ports. That dual port looks physically different than a normal USB port so I assumed it was just ESATA. I plugged the usb to serial adapter into this dual port and everything worked flawlessly.

Hopefully that will helps someone save a bunch of time that I wasted trying to get it to work.

Posted in Code | Tagged , , | 6 Comments

What is the Cost of Not Doing Things?

We’re really good at measuring the cost of some things. We’re good at measuring the cost of a new computers for everyone on the team, we’re good at measuring the cost per hour of a resource on a project and we’re good at measuring the time it will take to complete a new feature.

It seems like people are not good at is measuring the cost of not doing things. What is the cost of maintaining an application on 10 year old technology instead of upgrading it to newer versions as they come out? What is the cost of not having unit tests and automated test suites? What is the cost of running many different versions of a framework or a virtual machine?

Unfortunately this leaves us with a problem. When we cannot quantify the cost of inaction it often looks like a reasonable choice because we assume that it’s free. That assumption is the root of a lot of problems.

This post is just me ranting, I wish I had the answer.

Posted in Code | Tagged , | 3 Comments

NoSQL with MongoDB and Ruby Presentation

I presented at the Milwaukee Ruby User’s Group tonight on NoSQL using MongoDB and Ruby.

Code Snippets for the Presentation

Basic Operations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// insert data
db.factories.insert( { name: "Miller", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Lakefront", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Point", metro: { city: "Steven's Point", state: "WI" } } );
db.factories.insert( { name: "Pabst", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Blatz", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Coors", metro: { city: "Golden Springs", state: "CO" } } );

// simple queries
db.factories.find()
db.factories.findOne()
db.factories.find( { "metro.city" : "Milwaukee" } )
db.factories.find( { "metro.state": {$in : ["WI", "CO"] } } )

// update data
db.factories.update( { name: "Lakefront"}, { $set : { thebest : true } } );
db.factories.find()

// delete data
db.factories.remove({name:"Coors"})
db.factories.remove()

Ruby Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
require 'rubygems'
require 'mongo'
include Mongo

db   = Connection.new.db('sample-db')
coll = db.collection('factories')

coll.remove

coll.insert( { :name => "Miller",    :metro => { :city => "Milwaukee", :state => "WI" } } )
coll.insert( { :name => "Lakefront", :metro => { :city: "Milwaukee", :state => "WI" } } )
coll.insert( { :name => "Point",     :metro => { :city => "Steven's Point", :state => "WI" } } )
coll.insert( { :name => "Pabst",     :metro => { :city => "Milwaukee", :state => "WI" } } )
coll.insert( { :name => "Blatz",     :metro => { :city => "Milwaukee", :state => "WI" } } )
coll.insert( { :name => "Coors",     :metro => { :city => "Golden Springs", :state => "CO" } } )

puts "There are #{coll.count()} factories. Here they are:"
coll.find().each { |doc| puts doc.inspect }
coll.map_reduce("function () { emit(this.metro.city, this.name); }", "function (k, vals) { return vals.join(","); }").each { |r| puts r.inspect }

Map Reduce Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
db.factories.insert( { name: "Miller", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Lakefront", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Point", metro: { city: "Steven's Point", state: "WI" } } );
db.factories.insert( { name: "Pabst", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Blatz", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Coors", metro: { city: "Golden Springs", state: "CO" } } );

var fmap = function () {
    emit(this.metro.city, this.name);
}
var fred = function (k, vals) {
    return vals.join(",");
}
res = db.factories.mapReduce(fmap, fred)
db[res.result].find()
db[res.result].drop()

The Presentation

Download NoSQL with MongoDB and Ruby Slides

Thanks to Meghan at 10Gen for sending stickers and a copy of MongoDB: The Definitive Guide that I gave out as a door prize. I read the book quickly this weekend before the talk and found it quite good, so I recommend it if you want to get started with MongoDB.

Posted in Code, Ruby | Tagged , , | 2 Comments

MongoDB: MapReduce Functions for Grouping

SQL GROUP BY allows you to perform aggregate functions on data sets; To count all of the stores in each state, to average a series of related numbers, etc. MongoDB has some aggregate functions but they are fairly limited in scope. The MongoDB group function also suffers from the fact that it does not work on sharded configurations. So how do you perform grouped queries using MongoDB? By using MapReduce functions of course (you read the title right?)

Understanding MapReduce

Understanding MapReduce requires, or at least is made much easier by, understanding functional programming concepts. map and reduce (fold, inject) are functions that come from Lisp and have been inherited by a lot of languages (Scheme, Smalltalk, Ruby, Python).

map
A higher-order function which transforms a list by applying a function to each of its elements. Its return value is the transformed list. In MongoDB terms, the map is a function that is run for each Document in a collection and can return a value for that row to be included in the transformed list.
reduce
A higher-order function that iterates an arbitrary function over a data structure and builds up a return value. The reduce function takes the values returned by map and allows you to run a function to manipulate those values in some way.

Some Examples

Let’s start with some sample data:

1
2
3
4
5
6
7
db.factories.insert( { name: "Miller", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Lakefront", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Point", metro: { city: "Steven's Point", state: "WI" } } );
db.factories.insert( { name: "Pabst", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Blatz", metro: { city: "Milwaukee", state: "WI" } } );
db.factories.insert( { name: "Coors", metro: { city: "Golden Springs", state: "CO" } } );
db.factories.find()

Lets say I want to count the number of factories in each of the cities (ignore the fact that I could have the same city in more than one state, I don’t in my data). For a count, I write a function that “emits” the group by key and a value that you can count. It can be any value, but for simplicity I’ll make it 1. emit() is a MongoDB server-side function that you use to identify a value in a row that should be added to the transformed list. If emit() is not called then the values for that row will be excluded from the results.

1
2
3
mapCity = function () {
    emit(this.metro.city, 1);
}

The next piece is the reduce() function. The reduce function will be passed a key and an array of values that were collected by the map() function. I know my map function returns a 1 for each row keyed by city. So the reduce function will be called with a key “Golden Springs” and a single-element array containing a 1. For “Milwaukee” it will be passed an 4-element array of 1s.

1
2
3
4
5
6
7
reduceCount = function (k, vals) {
    var sum = 0;
    for (var i in vals) {
        sum += vals[i];
    }
    return sum;
}

With those 2 functions I can call the mapReduce function to perform my Query.

1
2
res = db.factories.mapReduce(mapCity, reduceCount)
db[res.result].find()

This results in:

1
2
3
{ "_id" : "Golden Springs", "value" : 1 }
{ "_id" : "Milwaukee", "value" : 4 }
{ "_id" : "Steven's Point", "value" : 1 }

Counting is not the only thing I can do of course. Anything can be returned by the map function including complex JSON objects. In this example I combine the names of all of the Factories in a given City into a simple comma-separated list.

1
2
3
4
5
6
7
8
mapCity = function () {
    emit(this.metro.city, this.name);
}
reduceNames = function (k, vals) {
    return vals.join(",");
}
res = db.factories.mapReduce(mapCity, reduceNames)
db[res.result].find()

Give you:

1
2
3
{ "_id" : "Golden Springs", "value" : "Coors" }
{ "_id" : "Milwaukee", "value" : "Miller,Lakefront,Pabst,Blatz" }
{ "_id" : "Steven's Point", "value" : "Point" }

Conclusion

These are fairly simple examples, but I think it helps to work through this kind of simple thing to fully understand a new technique before you have to work with harder examples.

For more on MongoDB check out these books:

Posted in Code | Tagged , , , | 6 Comments