Off-heap Hash Map in Java, part 2

I spent some time trying to re-produce issue we had in production which justified usage of Off-heap hash map. And it was a total fail! In theory I knew that it is related to cases when app needs huge maps consuming almost all heap memory and those maps are not static, they are constantly changed triggering Full GC cycles. Anyway I got some interesting results just comparing time and memory usage of map population.

So I had this simple code to create a map. OHM uses same interface as HashMap so it is pretty simply to test them both.

public class HashMapTest {
    public static void main(String[] args) {
        //final Map<String, String> map = new HashMap<>(15_000_000, 0.9f);
        final Map<String, String> map = new OHMap<>(15_000_000);
 
        for (int i = 0; i < 10; i++) {
            map.clear();
 
            System.out.print("Loading map...");
            long start = System.currentTimeMillis();
            populateMap(map);
            long end = System.currentTimeMillis();
            System.out.println("Done in " + (end - start) + "ms.");
        }
    }
 
    private static void populateMap(Map<String, String> map) {
        for (int i = 0; i < 10 * 1000_000; i++) {
            map.put(String.valueOf(i), UUID.randomUUID().toString());
        }
    }
}

I started with OpenJDK 8 and as expected OHM was slower than HashMap: 33ms vs 23ms. But memory consumption is quit opposite! I had to pump -Xmx to 3Gb to make HashMap test work and total memory used by Java process was 3181Mb. OHM worked even with -Xmx1G though total memory consumption was also close to 3Gb.

Now the most interesting results (prompted me to post this) I got from using OpenJDK 11! The performance difference between HashMap and OHM was shocking: 17ms vs. 34ms!!! And memory consumption for HasMap test with -Xmx3G was lower than 3Gb!

Undoubtedly Java engineers did a good job with core JDK11. With such results I may no need of OHM in production when we switch to Java 11. But I still wasn’t able to re-produce the state of continuous GC cycles with huge hash maps. My next try will be adding multi-threading to get closer to production use case.

Off-heap Hash Map in Java, part 1

A month ago I had to deal with an interesting case: huge hash maps in Java! When input started to grow we switched to larger AWS instance. But that didn’t help because input kept growing. And I observed huge heap consumption and very long GC pauses. Essentially application was partly doing its job and partly doing GC. But when it started to hit max heap I had to do something. And I started investigating off-heap solutions.

My “googling” quickly led me to two Java solutions: Java Large Off Heap Cache and Binary Off Heap Hash Map. Both solutions are treating keys and values as blob of bytes. I chose BinaryOffheapHashMap because it is a small code which I can understand. Even that code was experimental it solved my task: creating a hash map outside of GC world. You can read more about that project here. OHC looks more “professional” and is something I will try next time.

So my experiments allowed me to look on Java from a different angle: “greedy” memory consumption and really nasty GC cycles. I will publish my test results in my next post.

Java Tutorial: JNDI Trail Tips for OpenLDAP

With every major release of JDK I quickly review Oracle’s Java Tutorial for any updates. I did that for JDK 8 and will do that for JDK 9 soon. Usually I skip trails like JNDI or JavaFX because I don’t use them at my job. But few months ago I decided to read JNDI trail and want to share some tips I had to use.

Server Setup

So you will need an LDAP server and tutorial refers reader to few vendors. I try to use implementations for Linux and sure there is one – OpenLDAP. Given that my desktop is Windows I have to run it in virtual machine. And for that I use VritualBox + Vagrant. Here is my Vagrantfile (configuration for Vagrant):

Vagrant.configure("2") do |config| 
  config.vm.box = "ubuntu/trusty64" 
  config.vm.network "forwarded_port", guest: 389, host: 1389
  config.vm.provision "shell", inline: &lt;&lt;-SHELL 
    export DEBIAN_FRONTEND=noninteractive 
    apt-get update 
    apt-get install -y slapd ldap-utils gnutls-bin ssl-cert
  SHELL 
end

It deploys Ubuntu an installs all necessary tools. After VM is up and running, you need to login (vagrant ssh) and re-configure slapd for tutorial needs. This official guide helped me a lot.

So first thing is to re-configure OpenLDAP:

sudo dpkg-reconfigure slapd

It will ask for domain name. Use something easy, like example.com. Then it will ask for Organization Name. Enter “JNDITutorial”. Then it will ask for administrator password. Don’t forget it 🙂 For any further questions you can safely use default values.

Next step is to update LDAP with schemas used by tutorial:

sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/java.ldif
sudo ldapadd -Q -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/corba.ldif

Next thing is populating DB with test data. JNDI trail has a link to tutorial.ldif. You need to download and update its DN names to our installed server: if we used example.com as domain name, then our full DN will be o=JNDITutorial,dc=example,dc=com and we have to ensure that in the file:

sed -i 's/o=JNDITutorial/o=JNDITutorial,dc=example,dc=com/' tutorial.ldif

Now you can upload test data (this is where you need to use admin password):

ldapadd -x -D cn=admin,dc=example,dc=com -W -f tutorial.ldif

There is a big chance you will get something like this:

ldap_add: Object class violation (65)
 additional info: invalid structural object class chain (alias/organizationalUnit)

Ignore that – it doesn’t affect tutorial.

Connection and Authentication

The connection string in JNDI examples must be slightly modified – you have to specify full DN and correct port. Given our configuration and domain example.com env initialization should look like this:

Hashtable&lt;String, Object&gt; env = new Hashtable&lt;&gt;();
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
env.put(Context.PROVIDER_URL, "ldap://localhost:1389/o=JNDITutorial,dc=example,dc=com");

Examples where something is updated or created require authentication. By default OpenLDAP excepts simple authentication. In this case you have to add additional settings to env:

env.put(Context.SECURITY_PRINCIPAL, "cn=admin,dc=example,dc=com");
env.put(Context.SECURITY_CREDENTIALS, "password");

Digest-MD5

Example with Digest-MD5 will not work w/o additional modifications. This is what I did to make it functional (thanks StackOverflow). First of all sasldb must be accessible by slapd:

sudo adduser openldap sasl

Then OpenLDAP hast to be configured to use sasldb directly. Create sasldb.ldif file:

dn: cn=config
changetype: modify
replace: olcSaslAuxprops
olcSaslAuxprops: sasldb

And update OpenLDAP configuration with it:

sudo ldapmodify -Q -Y EXTERNAL -H ldapi:/// -f sasldb.ldif

Last thing is to create user in SASLDB. For example user “test”:

sudo saslpasswd2 -c test

That’s it! Now you will be able to connect to OpenLDAP using these environment configuration:

Hashtable<String, Object> env = new Hashtable<>();
env.put(Context.INITIAL_CONTEXT_FACTORY, 
  "com.sun.jndi.ldap.LdapCtxFactory");
env.put(Context.PROVIDER_URL, 
  "ldap://localhost:1389/o=JNDITutorial,dc=example,dc=com");
env.put(Context.SECURITY_AUTHENTICATION, "DIGEST-MD5");
env.put(Context.SECURITY_PRINCIPAL, "test");
env.put(Context.SECURITY_CREDENTIALS, "*****");

SSL and Custom Sockets

OpenLDAP does not support SSL/LDAPS out of the box. Instead server guide instructs you how to configure TLS which allows to negotiate encrypted connection using the same server port. Though TLS case is slightly different then just SSL protocol – it requires to use JSSE extension. There is a detail trail here. In short: environment settings are the same as for unencrypted connection, but all your work has to be done inside encrypted TLS session:

StartTlsResponse tls = (StartTlsResponse) ctx.extendedOperation(
  new StartTlsRequest());
tls.negotiate();
// Do your work with LDAP context
tls.close();

The important step to make that work is adding server certificate to JRE keystore. Otherwise your connection will fail. So if you followed OpenLDAP guide then copy /etc/ssl/certs/ldap01_slapd_cert.pem to your local machine (or /vagrant for Vagrant). And then use keytool to import it:

keytool -importcert -alias jnditutorial ^
-file ldap01_slapd_cert.pem ^
-keystore "C:\Program Files\Java\jre1.8.0_151\lib\security\cacerts"

Although this is a Windows example, Linux or Unix would be very similar. Note that keystore is called cacerts (not jseecacerts). Also note a little caveat: if you have both JDK and JRE installed there is a big chance calling “java” runs JRE’s JVM, not JDK’s one.