BenchmarkRunner.java - Benchmark Runner Program

JVM Tutorials - Herong's Tutorial Examples

∟BenchmarkRunner.java - Benchmark Runner Program

This section provides a tutorial example on how to write a benchmark runner program that takes care of JVM warming up, JIT compilation, measuring time, and other non-test related tasks.

The best way to write a micro benchmark program is probably to separate it in 2 parts:

Test Code - The specific piece of code to be tested.
Test Runner - Taking care of JVM warming up, JIT compilation, measuring time, and other tasks.

Here my test runner program, BenchmarkRunner.java:

/* BenchmarkRunner.java
 * Copyright (c) HerongYang.com. All Rights Reserved.
 */
class BenchmarkRunner {
   static java.io.PrintStream out = System.out;
   static java.io.InputStream in = System.in;
   long startTime = 0;
   long endTime = 0;
   long[] timeRecords = null;

   public static void main(String[] a) {
      if (a.length<5) {
        out.println("Missing arguments. Usage: ");
        out.println(
           "BenchmarkRunner class method warmups runs steps");
        return;
      }
      try {
         String className = a[0];
         String methodName = a[1];
         int numberOfWarmups = Integer.parseInt(a[2]);
         int numberOfRuns = Integer.parseInt(a[3]);
         int numberOfSteps = Integer.parseInt(a[4]);

         // Warming up the JVM
         out.println("Are you ready?");
         in.read(new byte[1]);

         // Loading the benchmark class and method
        Class<?> testClass = Class.forName(className);
        java.lang.reflect.Method testMethod
           = testClass.getMethod(methodName, int.class,
           BenchmarkRunner.class);
         BenchmarkRunner testRunner
            = new BenchmarkRunner(numberOfRuns);
         Object testObject
            = testClass.getDeclaredConstructor().newInstance();

         // JIT warmup
         out.println();
         out.println("Waking up the JIT compiler...");
         for (int i=0; i<numberOfWarmups; i++) {
            Object testResult =
            testMethod.invoke(testObject, numberOfSteps, testRunner);
            out.println("Run: "+i+", Time: "+testRunner.returnTime()
               +", Test returns: "+testResult);
         }

         // Benchmark runs
         out.println();
         out.println("Starting benchmark test runs...");
         for (int i=0; i<numberOfRuns; i++) {
            Object testResult =
            testMethod.invoke(testObject, numberOfSteps, testRunner);
            testRunner.recordTime(i);
            out.println("Run: "+i+", Time: "+testRunner.returnTime()
               +", Test returns: "+testResult);
         }

         // Benchmark report
         out.println();
         out.println("Benchmark test time report...");
         testRunner.report(numberOfRuns, numberOfSteps);

      } catch (Exception e) {
         e.printStackTrace();
      }

   }

   // Constructor
   public BenchmarkRunner(int runs) {
      timeRecords = new long[runs];
   }

   // Starting the timer - to be called by test method
   public void startTimer() {
      startTime = System.nanoTime();
   }

   // Stopping the timer - to be called by test method
   public void stopTimer() {
      endTime = System.nanoTime();
   }

   // Returning time from the timer
   public long returnTime() {
      return endTime - startTime;
   }

   // Recording time from the timer
   public void recordTime(int i) {
      timeRecords[i] = endTime - startTime;
   }

   // Reporting benchmark result
   public void report(int runs, int steps) {
      long total = 0;
      long minimum = Long.MAX_VALUE;
      long maximum = 0;
      for (int i=0; i<runs; i++) {
         long t = timeRecords[i];
         total += t;
         if (t>maximum) maximum = t;
         if (t<minimum) minimum = t;
      }
      long average = total/runs;
      out.println("Runs: "+runs+", Ave: "+average/steps
         +", Min: "+minimum/steps
         +", Max: "+maximum/steps
         +" - Per step in nanoseconds");
      out.println("Runs: "+runs+", Ave: "+average
         +", Min: "+minimum+", Max: "+maximum
         +" - All steps in nanoseconds");
      out.println("Runs: "+runs+", Ave: "+average/1000000
         +", Min: "+minimum/1000000
         +", Max: "+maximum/1000000
         +" - All steps in milliseconds");
      out.println("Runs: "+runs+", Ave: "+average/1000000000
         +", Min: "+minimum/1000000000
         +", Max: "+maximum/1000000000
         +" - All steps in seconds");
   }

   // Constructor needed as a sample benchmark test class
   public BenchmarkRunner() {
   }

   // A sample benchmark test method
   public static long sampleTest(int steps, BenchmarkRunner runner) {
      long total = 0;
      runner.startTimer();
      for (int i=0; i<steps; i++) {
         total += i;
      }
      runner.stopTimer();
      return total;
   }
}

Note that:

A simple output and input dialog step is used to warm up the JVM. You can wait as long as you needed to make sure that the JVM is fully warmed up.
The test class and test method are explicitly loaded and prepared.
In order to warm up the JIT compiler, the test method is called in a loop. The number of iterations of this loop is controlled by the "warmups" parameter, which should be large enough to fully wake up the JIT compiler - triggering it to compile, and sometimes recompile, the test method into native codes.
After JIT compilation, the test method is called in the benchmark test loop. The number of iterations of this loop is controlled by the "runs" parameter, which allows you to perform the benchmark test multiple times and take the average execution time.
The "steps" parameter is passed to the test method, just in case the test code needs to be repeated in multiple times.
Two callback methods, startTimer() and stopTimer(), are provided for the test method to call back to provide execution time measurements.
Other methods are provided to record execution time of each test run and report test result.