Java micro benchmark with jmh and Netbeans

Note that jmh has evolved significantly since I wrote this post and some of the information below might be obsolete.

jmh (Java Microbenchmark Harness) is an open source micro-benchmarking tool for java, part of the OpenJDK. I have been using it for a few weeks and found it easy to use and very useful. One advantage it has over Caliper is that it runs on Windows.

Installation

The installation process is fairly straight-forward using Maven. For example, with Netbeans, it can be done following those steps:

  • Download source (you need to have Mercurial installed):

  • Open, compile and install the library:

    • Netbeans then proposes to open the project: click Open Projects
    • Select the top project and click open
    • Right click on the project > Custom > Goals
    • In Goals, type: clean install -DskipTests=true

Create a Microbenchmark Project

  • Menu File > New Project
  • Select Maven / Java Application > Next
  • Let’s call it performance
  • Enter a GroupID (I use com.assylias for this example)
  • Click Finish

Let’s now configure the dependencies and allow the project to run jmh:

  • In the project’s Project Files, select and edit pom.xml
  • Use the following dependencies and build settings:

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <artifactId>jmh-core</artifactId>
        <version>1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <finalName>microbenchmarks</finalName>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>org.openjdk.jmh.Main</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Sample benchmark

Let’s try to benchmark something to see if it works. We could for example try to find the best method to copy an array. The four candidates we are going to test are:

  • Object[] newArray = originalArray.clone();
  • Object[] newArray = Arrays.copyOf(originalArray, originalArray.length);
  • System.arrayCopy(originalArray, 0, newArray, 0, originalArray.length);
  • and a plain old loop

In your project source package, right click and add a new class, let’s call it ArrayCopy, and copy the following code:

import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.BenchmarkType;
import org.openjdk.jmh.annotations.GenerateMicroBenchmark;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

/**
 *
 */
@State(Scope.Thread)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class ArrayCopy {

    private int[] array = {1, 2, 3, 4, 5, 6, 7, 8, 9};

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] clone_() {
        int[] copy = array.clone();
        return copy;
    }

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] arrayCopy() {
        int[] copy = new int[array.length];
        System.arraycopy(array, 0, copy, 0, array.length);
        return copy;
    }

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] copyOf() {
        int[] copy = Arrays.copyOf(array, array.length);
        return copy;
    }

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] loop() {
        int[] copy = new int[array.length];
        for (int i = 0; i < array.length; i++) {
            copy[i] = array[i];
        }
        return copy;
    }    
}

Finally, let’s create the launcher that will run the micro-benchmark – I use this runner class (alternatively you can run it from the command line but I’m lazy and prefer running it from the IDE):

import java.io.IOException;
import org.openjdk.jmh.Main;

public class RunTest {
    private static final String TEST = ".*ArrayCopy.*"; //uses regexp

    public static void main(String[] args) throws IOException {
        Main.main(getArguments(TEST, 5, 5000, 1));
    }

    private static String[] getArguments(String className, int nRuns, int runForMilliseconds, int nThreads) {
        return new String[]{className,
            "-i", "" + nRuns,
            "-r", runForMilliseconds + "ms",
            "-t", "" + nThreads,
            "-w", "5000ms",
            "-wi", "3",
            "-v"
        };
    }
}

Clean and Build the project (CTRL+F11) and run it (SHIFT+F6 with the RunTest class selected).

You should get a detailed output of the performance of the various methods and a summary table that looks like this:

Benchmark                       Thr    Cnt  Sec         Mean   Mean error          Var    Units
c.a.p.g.a.ArrayCopy.arrayCopy     1     10    1       11.947        0.049        0.002  nsec/op
c.a.p.g.a.ArrayCopy.clone_        1     10    1       11.801        0.368        0.128  nsec/op
c.a.p.g.a.ArrayCopy.copyOf        1     10    1       11.783        0.115        0.013  nsec/op
c.a.p.g.a.ArrayCopy.loop          1     10    1       17.985        0.109        0.011  nsec/op

Next steps

The jmh project comes with a few samples which are very interesting and useful to read. It is also useful to check the usage, for example with the printUsage method in the example above (or by running it from the command line with no argument: java -jar microbenchmarks.jar).

About these ads
Tagged , , , ,

2 thoughts on “Java micro benchmark with jmh and Netbeans

  1. We had committed the simple Maven archetype for the benchmark project generation yesterday. See JMH page at OpenJDK for the examples, it should be more convenient than messing with POMs on your own.

    P.S. Also, Caliper is not working on Windows?! O_o

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: