Tag Archives: java

Lambdas and serialization

With the introduction of intersection cast in Java 8, it is now possible to write: Object o = (SomeType & SomeOtherType) otherObject;

One very interesting use case is Serialization as this allows us to automagically serialize a lambda effortlessly:

    File file = Files.createTempFile("lambda", "ser").toFile();
    try (ObjectOutput oo = new ObjectOutputStream(new FileOutputStream(file))) {
        Runnable r = (Runnable & Serializable) () -> System.out.println("I can be serialized!");
        oo.writeObject(r);
    }

    try (ObjectInput oi = new ObjectInputStream(new FileInputStream(file))) {
        Runnable  r = (Runnable) oi.readObject();
        r.run();
    }
Tagged , , ,

Memory conscious toList collector with Java 8

The following code:

    List source = Arrays.asList(1, 2, 3, 4, 5);
    List result = new ArrayList<> (source.size());
    for (int i : source) result.add(2 * i);

can easily be transformed to use lambdas in Java 8, for example:

    List result = source.stream()
                                 .map(i -> 2 * i)
                                 .collect(toList());

There is however an important difference: in the first code, the result list has a capacity of 5, whereas in the second code it has a capacity of 10, hence using more memory*.

The Collectors utility class also offers a toCollection method which accepts a Supplier<Collection>. That method allows to use a properly sized collection, for example:

    Supplier s = () -> new ArrayList<>(source.size());
    List result = source.stream()
                                 .map(i -> 2 * i)
                                 .collect(toCollection(s));

The code is now equivalent to the original code without lambdas and the result list has an optimal capacity.


*The capacity of the backing array can easily be checked with:

    Field f = ArrayList.class.getDeclaredField("elementData");
    f.setAccessible(true);
    System.out.println(((Object[]) f.get(result)).length);


Tagged , , ,

Transparent background with JavaFX

Something that can be useful is to have a transparent background. For example if one wants to show several components as if they were free-floating windows while keeping them all within the same Stage.

It can be achieved by making each layer transparent (Stage, Scene, Nodes). Here is a simple example that shows how it can be done:

public class TransparentWindow extends Application {

    @Override
    public void start(Stage stage) throws Exception {
        try {
            Label lbl = new Label("LABEL");
            VBox p = new VBox(lbl);

            //make the background of the label white and opaque
            lbl.setStyle("-fx-background-color: rgba(255, 255, 255, 1);");

            //add some borders to visualise the element' locations
            lbl.setBorder(new Border(new BorderStroke(Color.BLUE, BorderStrokeStyle.SOLID, null, null)));
            p.setBorder(new Border(new BorderStroke(Color.RED, BorderStrokeStyle.SOLID, null, null)));

            Scene scene = new Scene(p);
            stage.setScene(scene);

            //this is where the transparency is achieved:
            //the three layers must be made transparent
            //(i)  make the VBox transparent (the 4th parameter is the alpha)
            p.setStyle("-fx-background-color: rgba(0, 0, 0, 0);");
            //(ii) set the scene fill to transparent
            scene.setFill(null);
            //(iii) set the stage background to transparent
            stage.initStyle(TRANSPARENT);

            stage.setWidth(200);
            stage.setHeight(100);
            stage.show();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) {
        launch(args);
    }
}

And the result looks like this (with my IDE in the background):

Capture

The next step is to add some custom buttons to allow the user to close the window (if you run the example as is you will need to kill the application).

Tagged , , ,

Installing a web server for a Java Webstart Deployment

The goal is to install a HTTP server on a linux box (debian) and place the files from a Java project in there to allow a deployment via Java Webstart. This is a reminder for myself, nothing that can’t be found by googling a little.

Install a HTTP server

I used apache as a server (it just works, there might be better options):

apt-get update
apt-get install apache2 apache2-doc apache2-utils

Check that the procedure worked by trying to access the server at http://hostnameOrIp, which should show a default web page.

That default page is located in /var/www, so you can start placing files in there as you wish.

Optional: make the folder available to Windows computers on the network via Samba:

Add the following profile to /etc/samba/smb.conf

[www]
   comment = Web Server
   path = /var/www
   guest ok = yes
   browseable = yes
   writeable = yes
   create mask = 0777
   directory mask = 0777

Change the permissions:

sudo chmod 777 -R /var/www

And restart Samba:

/etc/init.d/samba restart

Prepare the Java project files

I assume that the project is packaged as a JAR file, with a certain number of dependencies, available as JAR files too.

You can then sign and deploy the files to the server, with the associated JNLP file.

Prepare the index.html

A very basic html page containing a "launch" button:

<html><body>
    <script src= "http://www.java.com/js/deployJava.js"></script>
    <script>
        // using JavaScript to get location of JNLP
        // file relative to HTML page
        var dir = location.href.substring(0, location.href.lastIndexOf('/')+1);
        var url = dir + "YourJnlp.jnlp";
        deployJava.createWebStartLaunchButton(url);
    </script>
</body></html>
Tagged , ,

JavaFX 2 & Maven

One of the annoying thing with JavaFX 2 is that it is co-bundled with the JDK since Java 7u7 but is not on the classpath by default.

One way, in Netbeans, is to create a Maven JavaFX project (although I have been somewhat unlucky with that approach).

There are various alternatives proposed on the web – one that worked for me consists in playing with the pom file to include JavaFX in the classpath with:

<dependency>
    <groupId>javafx</groupId>
    <artifactId>jfxrt</artifactId>
    <scope>system</scope>
    <systemPath>${java.home}/lib/jfxrt.jar</systemPath>
    <version>2.2.4</version>
</dependency>

Although this might not be the best approach, it seems to work fine.

Tagged , ,

Java micro benchmark with jmh and Netbeans

Note that jmh has evolved significantly since I wrote this post and some of the information below might be obsolete.

jmh (Java Microbenchmark Harness) is an open source micro-benchmarking tool for java, part of the OpenJDK. I have been using it for a few weeks and found it easy to use and very useful. One advantage it has over Caliper is that it runs on Windows.

Installation

The installation process is fairly straight-forward using Maven. For example, with Netbeans, it can be done following those steps:

  • Download source (you need to have Mercurial installed):

  • Open, compile and install the library:

    • Netbeans then proposes to open the project: click Open Projects
    • Select the top project and click open
    • Right click on the project > Custom > Goals
    • In Goals, type: clean install -DskipTests=true

Create a Microbenchmark Project

  • Menu File > New Project
  • Select Maven / Java Application > Next
  • Let’s call it performance
  • Enter a GroupID (I use com.assylias for this example)
  • Click Finish

Let’s now configure the dependencies and allow the project to run jmh:

  • In the project’s Project Files, select and edit pom.xml
  • Use the following dependencies and build settings:

<dependencies>
    <dependency>
        <groupId>org.openjdk.jmh</groupId>
        <artifactId>jmh-core</artifactId>
        <version>1.0-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.0</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <finalName>microbenchmarks</finalName>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>org.openjdk.jmh.Main</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Sample benchmark

Let’s try to benchmark something to see if it works. We could for example try to find the best method to copy an array. The four candidates we are going to test are:

  • Object[] newArray = originalArray.clone();
  • Object[] newArray = Arrays.copyOf(originalArray, originalArray.length);
  • System.arrayCopy(originalArray, 0, newArray, 0, originalArray.length);
  • and a plain old loop

In your project source package, right click and add a new class, let’s call it ArrayCopy, and copy the following code:

import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.BenchmarkType;
import org.openjdk.jmh.annotations.GenerateMicroBenchmark;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

/**
 *
 */
@State(Scope.Thread)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class ArrayCopy {

    private int[] array = {1, 2, 3, 4, 5, 6, 7, 8, 9};

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] clone_() {
        int[] copy = array.clone();
        return copy;
    }

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] arrayCopy() {
        int[] copy = new int[array.length];
        System.arraycopy(array, 0, copy, 0, array.length);
        return copy;
    }

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] copyOf() {
        int[] copy = Arrays.copyOf(array, array.length);
        return copy;
    }

    @GenerateMicroBenchmark(BenchmarkType.AverageTimePerOp)
    public int[] loop() {
        int[] copy = new int[array.length];
        for (int i = 0; i < array.length; i++) {
            copy[i] = array[i];
        }
        return copy;
    }    
}

Finally, let’s create the launcher that will run the micro-benchmark – I use this runner class (alternatively you can run it from the command line but I’m lazy and prefer running it from the IDE):

import java.io.IOException;
import org.openjdk.jmh.Main;

public class RunTest {
    private static final String TEST = ".*ArrayCopy.*"; //uses regexp

    public static void main(String[] args) throws IOException {
        Main.main(getArguments(TEST, 5, 5000, 1));
    }

    private static String[] getArguments(String className, int nRuns, int runForMilliseconds, int nThreads) {
        return new String[]{className,
            "-i", "" + nRuns,
            "-r", runForMilliseconds + "ms",
            "-t", "" + nThreads,
            "-w", "5000ms",
            "-wi", "3",
            "-v"
        };
    }
}

Clean and Build the project (CTRL+F11) and run it (SHIFT+F6 with the RunTest class selected).

You should get a detailed output of the performance of the various methods and a summary table that looks like this:

Benchmark                       Thr    Cnt  Sec         Mean   Mean error          Var    Units
c.a.p.g.a.ArrayCopy.arrayCopy     1     10    1       11.947        0.049        0.002  nsec/op
c.a.p.g.a.ArrayCopy.clone_        1     10    1       11.801        0.368        0.128  nsec/op
c.a.p.g.a.ArrayCopy.copyOf        1     10    1       11.783        0.115        0.013  nsec/op
c.a.p.g.a.ArrayCopy.loop          1     10    1       17.985        0.109        0.011  nsec/op

Next steps

The jmh project comes with a few samples which are very interesting and useful to read. It is also useful to check the usage, for example with the printUsage method in the example above (or by running it from the command line with no argument: java -jar microbenchmarks.jar).

Tagged , , , ,

A simple logback appender for mongoDB

logback is a logging framework that, among other things, allows to log data to a database. However mongoDB is not one of the supported ones.

But logback is fully configurable so I’ve written a simple appender to use logback with mongo.

/*
 * Copyright 2013 Yann Le Tallec.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.assylias.logging;

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.classic.spi.IThrowableProxy;
import ch.qos.logback.classic.spi.ThrowableProxyUtil;
import ch.qos.logback.core.CoreConstants;
import ch.qos.logback.core.UnsynchronizedAppenderBase;
import com.mongodb.BasicDBObject;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;
import com.mongodb.MongoException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.net.UnknownHostException;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import org.slf4j.Marker;

/**
 * A logback appender that uses Mongo to log messages.
 * <p>
 * Typical configuration:
 * <pre> {@code
 * <appender name=\"TEST\" class=\"com.assylias.logging.MongoAppender\">
 *     <host>192.168.1.1</host>
 *     <port>27017</port>
 *     <db>log</db>
 *     <collection>test</collection>
 * </appender>
 * } </pre>
 * The log messages have the following JSON format (the {@code marker}, {@code exception} and {@code stacktrace} fields are optional):
 * <pre> {@code
 * { "_id" : ObjectId("514b2d529234d98131221578"),
 *   "logger" : "com.assylias.logging.MongoAppenderTest",
 *   "timestamp" : ISODate("2013-03-21T15:54:58.357Z"),
 *   "level" : "ERROR",
 *   "marker" : "Marker",
 *   "thread" : "TestNG",
 *   "message" : "An error occurend in the test",
 *   "exception" : "java.lang.RuntimeException: java.lang.Exception",
 *   "stacktrace" : [ "at com.assylias.logging.MongoAppenderTest.testCausedBy(MongoAppenderTest.java:129) ~[test-classes/:na]",
 *                    "at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_17]",
 *                    "at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_17]",
 *                    "Caused by: java.lang.Exception: null",
 *                    "at com.assylias.logging.MongoAppenderTest.testCausedBy(MongoAppenderTest.java:126) ~[test-classes/:na]",
 *                    "... 20 common frames omitted" ],
 * } } </pre>
 * If an error occurs while logging, the message might also contain a {@code logging_error} field:
 * <pre> {@code
 *    "logging_error" : "Could not log all the event information: com.mongodb.MongoInterruptedException: A driver operation has been interrupted
 *                       at com.mongodb.DBPortPool.get(DBPortPool.java:216)
 *                       at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:440)
 *                       ..."
 * } </pre>
 */
public class MongoAppender extends UnsynchronizedAppenderBase<ILoggingEvent> {

    private String host;
    private int port;
    private String db;
    private String collection;
    private DBCollection logCollection;

    public MongoAppender() {
    }

    @Override
    public void start() {
        try {
            connect();
            super.start();
        } catch (UnknownHostException | MongoException e) {
            addError("Can't connect to mongo: host=" + host + ", port=" + port, e);
        }
    }

    private void connect() throws UnknownHostException {
        MongoClient client = new MongoClient(host, port);
        DB mongoDb = client.getDB(db == null ? "log" : db);
        logCollection = mongoDb.getCollection(collection == null ? "log" : collection);
    }

    @Override
    protected void append(ILoggingEvent evt) {
        if (evt == null) return; //just in case

        DBObject log = getBasicLog(evt);
        try {
            logException(evt.getThrowableProxy(), log);
            logCollection.insert(log);
        } catch (Exception e) {
            try {
                StringWriter sw = new StringWriter();
                e.printStackTrace(new PrintWriter(sw));
                log.put("logging_error", "Could not log all the event information: " + sw.toString());
                log.put("level", "ERROR");
                logCollection.insert(log);
            } catch (Exception e2) { //really not working
                addError("Could not insert log to mongo: " + evt, e2);
            }
        }
    }

    private DBObject getBasicLog(ILoggingEvent evt) {
        DBObject log = new BasicDBObject();
        log.put("logger", evt.getLoggerName());
        log.put("timestamp", new Date(evt.getTimeStamp()));
        log.put("level", String.valueOf(evt.getLevel())); //in case getLevel returns null
        Marker m = evt.getMarker();
        if (m != null) {
            log.put("marker", m.getName());
        }
        log.put("thread", evt.getThreadName());
        log.put("message", evt.getFormattedMessage());
        return log;
    }

    private void logException(IThrowableProxy tp, DBObject log) {
        if (tp == null) return;
        String tpAsString = ThrowableProxyUtil.asString(tp); //the stack trace basically
        List<String> stackTrace = Arrays.asList(tpAsString.replace("\t","").split(CoreConstants.LINE_SEPARATOR));
        if (stackTrace.size() > 0) {
            log.put("exception", stackTrace.get(0));
        }
        if (stackTrace.size() > 1) {
            log.put("stacktrace", stackTrace.subList(1, stackTrace.size()));
        }
    }

    public String getHost() {
        return host;
    }

    public void setHost(String host) {
        this.host = host;
    }

    public int getPort() {
        return port;
    }

    public void setPort(int port) {
        this.port = port;
    }

    public String getDb() {
        return db;
    }

    public void setDb(String db) {
        this.db = db;
    }

    public String getCollection() {
        return collection;
    }

    public void setCollection(String collection) {
        this.collection = collection;
    }
}

Here is an excerpt from the class javadoc.

Typical configuration:

<appender name=\"TEST\" class=\"com.assylias.logging.MongoAppender\">
    <host>192.168.1.1</host>
    <port>27017</port>
    <db>log</db>
    <collection>test</collection>
</appender>

The log messages have the following JSON format (the marker, exception and stacktrace fields are optional):

{ "_id" : ObjectId("514b2d529234d98131221578"),
  "logger" : "com.assylias.logging.MongoAppenderTest",
  "timestamp" : ISODate("2013-03-21T15:54:58.357Z"),
  "level" : "ERROR",
  "marker": "A_MARKER",
  "thread" : "TestNG",
  "message" : "An error occurend in the test",
  "exception" : "java.lang.RuntimeException: java.lang.Exception",
  "stacktrace" : [ "at com.assylias.logging.MongoAppenderTest.testCausedBy(MongoAppenderTest.java:129) ~[test-classes/:na]",
                   "at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_17]",
                   "at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_17]",
                   "Caused by: java.lang.Exception: null",
                   "at com.assylias.logging.MongoAppenderTest.testCausedBy(MongoAppenderTest.java:126) ~[test-classes/:na]",
                   "... 20 common frames omitted" ],
} 

If an error occurs while logging, the message might also contain a logging_error field:

  "logging_error" : "Could not log all the event information: com.mongodb.MongoInterruptedException: A driver operation has been interrupted
                  at com.mongodb.DBPortPool.get(DBPortPool.java:216)
                  at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:440)
                  ..."
Tagged , , ,

Java Memory Model and reordering

This is not another post about how to solve the double checked locking idiom. The aim here is to understand what could go wrong without synchronization.

One of the most important promises of the Java Memory Model (JMM) is:

If a program is correctly synchronized, then all executions of the program will appear to be sequentially consistent.
This is an extremely strong guarantee for programmers. Programmers do not need to reason about reorderings to determine that their code contains data races. Therefore they do not need to reason about reorderings when determining whether their code is correctly synchronized. Once the determination that the code is correctly synchronized is made, the programmer does not need to worry that reorderings will affect his or her code.

The following code, taken from Java Concurrency in Practice, is not thread safe because accesses to the shared resource variable are not properly synchronized:

public class UnsafeLazyInitialization {
    private static Resource resource;

    public static Resource getInstance() {
        if (resource == null) //1
            resource = new Resource();  //2
        return resource; //3
    }
}

I will detail below some of the problems that could happen when running that innocent piece of code in a multi-threaded environment. In particular:

  • resource could be instantiated more than once
  • getInstance could return an object in an inconsistent state
  • more interestingly, getInstance could return null

Multiple instantiation

This is the most obvious issue – the code uses a check-then-act pattern so it is very possible that two threads could arrive at the same time in the method, both see resource as null and both initialise the variable. We how have 2 instances of what was supposed to be a singleton.

Improperly constructed object

This one is less obvious – line 2 looks like it is atomic but it isn’t as the JVM needs to (among other things):

  1. allocate some memory
  2. create the new object
  3. initialise its fields with their default value (false for boolean, 0 for other primitives, null for objects)
  4. run the constructor, which includes running parent constructors too
  5. assign the reference to the newly constructed object to resource

Because there is no synchronization, the JMM allows a JVM to perform these steps in virtually any order. See for example this famous discussion about double checked locking that shows that some JIT compilers do run step 5 before step 4.
So getInstance could return a reference to a non-null but inconsistent object (with un-initialised fields).

getInstance can return null

This is even less obvious. It is difficult to imagine an execution path that could return null with such a simple code. However the JMM allows it. To understand why this is possible, we need to analyse the reads and writes in details and assess whether there is a happens-before relationship between them. The code can be rewritten as follows to clearly show the reads and writes:

                               Thread 0
---------------------------------------------------------------------
 10: resource = null; //default value                                  //write
=====================================================================
           Thread 1               |          Thread 2                
----------------------------------+----------------------------------
 11: a = resource;                | 21: x = resource;                  //read
 12: if (a == null)               | 22: if (x == null)               
 13:   resource = new Resource(); | 23:   resource = new Resource();   //write
 14: b = resource;                | 24: y = resource;                  //read
 15: return b;                    | 25: return y;                    

The JLS #17.4.5 gives the rules for a read to be allowed to observe a write:

We say that a read r of a variable v is allowed to observe a write w to v if, in the happens-before partial order of the execution trace:

  • r is not ordered before w (i.e., it is not the case that hb(r, w)), and
  • there is no intervening write w’ to v (i.e. no write w’ to v such that hb(w, w’) and hb(w’, r)).

In this example, both 21 and 24 are therefore allowed to observe 10 or 13 and a legal execution of the program is (assuming thread 1 sees resource null and initialises it):

  1. 21: x = not null (reads the write line 13)
  2. 22: false
  3. 24: y = null (reads the write line 10)
  4. 25: return null

Instructions reordering

In practice, T2 is not going to see a null value after having seen a non-null value, but either the compiler or the JVM or the JIT can reorder instructions in a way that will produce a similar execution. For eaxmple, a possible reordering (with a theoretical execution) would be:

public class UnsafeLazyInitialization {
    private static Resource resource;

    public static Resource getInstance() {
        Resource temp = resource; //null in T1 and T2
        if (resource == null) //null in T1 but not in T2 because it has been initialised by T1 in the meantime
            resource = temp = new Resource(); //only executed by T1
        return temp; //T1 returns the new value, T2 returns null
    }
}

This reordering, although it makes little sense, is perfectly valid because it does not affect the intra-thread semantics (if run in a single-threaded environment, it will produce the same result as the original code).

Conclusion

This example shows that even on a fairly contrived example the outcome of an improperly synchronized program can be quite surprising. Although it is unlikely that any compilers would actually perform that reordering, a more complex situation could quickly become impossible to analyse.
Bottom line: saving on synchronization is not an option.

Tagged , ,

Size of objects in Java

I need to optimise the size taken by some of my objects as there will be millions of them and a factor of 3 starts to make a difference on a standard desktop PC, especially if it is a 32 bit machine.

JVM parameters

To get (fairly) accurate results using Runtime.getRuntime().freeMemory() I have used the following JVM parameters (on hotspot 7u11 x64 – some parameters might not be available on other JVMs): -server -Xms2000m -Xmx2000m -verbose:gc -XX:-UseTLAB -XX:+UseCompressedOops. The various flags do the following:

  • -Xms and -Xmx are well known and respectiely define the initial and maximum heap size. I set them high (2GB) to prevent the garbage collector from running during the test.
  • -verbose:gc makes the JVM print to the console when a GC is run. That enables to visually control that no GC ran during the tests.
  • -XX:-UseTLAB asks the JVM not to allocate memory in chunks (which it otherwise does for efficiency). Turning that option off gives more accurate and stable results.
  • -XX:+UseCompressedOops asks the JVM to compress references from 8 to 4 bytes. Note that it is on by default on Hotspot 7 x64.

Tests

The objective of my test was to determine the size of various classes that I could use to store tick data. The issue being that some ticks come with extra information and some don’t. I was wondering whether I should declare all the possible fields and leave them null when not available, or have an array or EnumMap instead. The only constraint here being memory footprint.

There are a few interesting observations – they are obviously empirical and might vary depending on several factors, including architecture (32/64 bits), JVM, other?:

  • an int takes 4 bytes
  • BUT an empty class (no members) and a class that has an int take the same space in memory (16 bytes), due to memory alignement
  • a null reference takes 4 bytes (same as int, due to compressed oop), with the same alignement observation
  • whether members are instantiated when declared or via a constructor does not make a difference (4 bytes)
  • an Object takes 16 bytes in memory
  • Adding 6 null Strings to my class (from TickBasic to TickComplete) adds 24 bytes, i.e. a size increase of 50%
  • The natural design is better than trying to use a “lazy” data structure with EnumMap

The test code at the bottom is fairly extensive but not very complicated. The output is (64 bits machine):

Object: 16 bytes
Object array (empty): 4 bytes
Object array (full) - incremental size: 16 bytes
Object with 1 int: 16 bytes
Object with 2 ints: 24 bytes
Object with 3 ints: 24 bytes
Object with 1 long: 24 bytes
Object with 2 longs: 32 bytes
Object with 3 longs: 40 bytes
Object with 1 null Object: 16 bytes
Object with 2 null Objects: 24 bytes
Object with 3 null Objects: 24 bytes
Object with 1 allocated Object: 16 bytes
Object with 2 allocated Objects: 24 bytes
Object with 3 allocated Objects: 24 bytes
TickBasic: 56 bytes
TickComplete: 80 bytes
TickCompleteWithConstructor: 80 bytes
TickArray: 128 bytes
TickEnumMap: 112 bytes

Test Code

public class TestMemory {

    private static final int SIZE = 100_000;
    private static Runnable r;
    private static Object[] array;
    private static Object o;
    private static Object o1 = new Object();
    private static Object o2 = new Object();
    private static Object o3 = new Object();

    private static void test(Runnable r, String name, int numberOfObjects) {
        long mem = Runtime.getRuntime().freeMemory();
        r.run();
        System.out.println(name + ": " + (mem - Runtime.getRuntime().freeMemory()) / numberOfObjects + " bytes");
    }

    public static void main(String[] args) throws Exception {
        System.out.println(System.getProperty("java.vm.name"));
        DateTime date = new DateTime(); //for some reason the result gets biased if we don't initialise DateTime first

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new Object(); } };
        test(r, "Object", SIZE);

        r = new Runnable() { public void run() { array = new Object[SIZE]; } };
        test(r, "Object array (empty)", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) array[i] = new Object(); } };
        test(r, "Object array (full) - incremental size", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith1Int(); } };
        test(r, "Object with 1 int", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith2Ints(); } };
        test(r, "Object with 2 ints", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith3Ints(); } };
        test(r, "Object with 3 ints", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith1Long(); } };
        test(r, "Object with 1 long", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith2Longs(); } };
        test(r, "Object with 2 longs", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith3Longs(); } };
        test(r, "Object with 3 longs", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith1NullObject(); } };
        test(r, "Object with 1 null Object", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith2NullObjects(); } };
        test(r, "Object with 2 null Objects", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith3NullObjects(); } };
        test(r, "Object with 3 null Objects", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith1Object(); } };
        test(r, "Object with 1 allocated Object", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith2Objects(); } };
        test(r, "Object with 2 allocated Objects", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new ObjectWith3Objects(); } };
        test(r, "Object with 3 allocated Objects", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new TickBasic(); } };
        test(r, "TickBasic", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new TickComplete(); } };
        test(r, "TickComplete", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new TickCompleteWithConstructor(new DateTime(), TickType.TRADE); } };
        test(r, "TickCompleteWithConstructor", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new TickArray(); } };
        test(r, "TickArray", SIZE);

        r = new Runnable() { public void run() { for (int i = 0; i < SIZE; i++) o = new TickEnumMap(); } };
        test(r, "TickEnumMap", SIZE);
    }

    public static class ObjectWith1Int  { int i; }
    public static class ObjectWith2Ints { int i, j; }
    public static class ObjectWith3Ints { int i, j, k; }
    public static class ObjectWith1Long  { long i; }
    public static class ObjectWith2Longs { long i, j; }
    public static class ObjectWith3Longs { long i, j, k; }
    public static class ObjectWith1NullObject  { Object o; }
    public static class ObjectWith2NullObjects { Object o, p; }
    public static class ObjectWith3NullObjects { Object o, p, q; }
    public static class ObjectWith1Object  { Object o = o1; }
    public static class ObjectWith2Objects { Object o = o1; Object p = o2; }
    public static class ObjectWith3Objects { Object o = o1; Object p = o2; Object q = o3; }

    public static class TickBasic {
        DateTime time = new DateTime();
        TickType type = TickType.TRADE;
        double value;
        int size;
    }

    public static class TickComplete {
        DateTime time = new DateTime();
        TickType type = TickType.TRADE;
        double value;
        int size;
        String cc;
        String ec;
        String mc;
        String bbc;
        String bsc;
        String rc;
    }

    public static class TickCompleteWithConstructor {
        DateTime time;
        TickType type;
        double value;
        int size;
        String cc;
        String ec;
        String mc;
        String bbc;
        String bsc;
        String rc;

        public TickCompleteWithConstructor(DateTime time, TickType type) {
            this.time = time;
            this.type = type;
        }
    }

    public static class TickArray {
        Object[] o = new Object[TickFields.values().length];
    }

    public static class TickEnumMap {
        Map values = new EnumMap(TickFields.class);
    }

    public static enum TickFields {

        TIME("time"),
        TYPE("type"),
        VALUE("value"),
        SIZE("size"),
        CONDITION_CODE("conditionCode"),
        EXCHANGE_CODE("exchangeCode"),
        MIC_CODE("micCode"),
        BROKER_BUY_CODE("brokerBuyCode"),
        BROKER_SELL_CODE("brokerSellCode"),
        RPS_CODE("rpsCode");
        private String code;

        TickFields(String code) {
            this.code = code;
        }
    }

    public static enum TickType {

        TRADE,
        BID,
        MID,
        ASK;
    }
}
Tagged , , ,

Bloomberg Java API Wrapper: jBloomberg

jBloomberg is an open source (Apache 2) Java library that wraps the low level Bloomberg Java API. A couple of usage examples are given at the bottom of the package javadoc page .

The main advantages of this library vs. the Bloomberg API are:

  • Less string based configuration: whenever possible I have used enums to remove the typos issues
  • Straight forward: retrieving historical data literally takes 5 lines of code, whereas when using the Bloomberg API, the code gets quickly cluttered with parsing, error handling and so on
  • Builder based design: most queries to Bloomberg are prepared with builders using the fluent interface pattern
  • The library takes thread safety seriously (so does the Bloomberg API) – all actions / objects are thread safe and can be used in a multi threaded application
  • Uses the standard java.util.concurrent package objects, so the syntax / way of doing things should look familiar to Java developers. For example, a historical data request returns a Future<RequestResult>
Tagged ,