Java OutOfMemoryError. Out of swap space?

Recently we faced an issue wherein the java process on Solaris (SUN JDK 1.6.0_24) crashed with the following message (Note that the JDK 64 bit install on Solaris is a 2 step process – you would need to first install the 32 bit JVM and then the 64 bit is layered on top of it):

This happened on a Solaris box. Investigated:

  • Checked the GC logs and did not see the heap as been stressed or an abnormal frequency of GC occurring. The heap was well withing the boundary of 2G that it had been assigned as the maximum.
  • Checked the OS swap memory and there seemed to be a lot of that available. [there seem to be a suggestion as to disable swap in server nodes. We do not follow this approach as of now.]. Details of the commands and the parameters that I checked are here.
  • Checked the overall memory available in the server node and it was much more than what was required.
  • Checked the total virtual memory utilized by the JVM before it spewed out the OutOfMemoryError and it was approaching 2.9G. [Again please refer to the link pasted earlier on commands and parameters that I used in this investigation]. This was a useful pointer that will come handy in theorizing a cause and replicating the issue.
  • Checked the number of threads that were spawned by the processes on the server node and the JVM before the error.
  • Checked the JVM: whether it was client or server.
  • A colleague, in the meantime, had pointed out that the VM was 32 bit.

As a result of the facets collected as part of the investigation and outlined above, it seemed that the issue was that the 32 bit JVM was unable to address more memory even though the node had plenty of it and neither the heap or the permgen or anything else was being stressed.

To replicate it, wrote a program that spawned a configurable amount of threads and in each thread a native call was made (you could simply call the Deflater and that leads to a native call – a snippet is provided below) and these threads were programmed to sleep for a certain time, invoke the native code ‘x’ number of times and start all over again. The virtual memory was monitored at all times and after a few iterations had the magic number of threads that would cause the “OutOfMemory swap space” issue.

Now the next step was to upgrade the JVM to 64 bit, run the process as 64 bit and make sure that the “OutOfMemoryError” does not happen. That was also demonstrated.

Conclusion

If your JVM process reaches 3G and it is 32 bit then you can expect these issues. Note that you would need to do the investigation and rule out any other cause such as a memory leak, JVM bug etc. The solution is to up the model to 64 bits.

Java tips

  • java -version => check to see if the default is a “server” VM or a “client” VM. On my mac (the default is a server VM):
    $ java -version
    java version “1.6.0_26″
    Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)
    Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)
  • Make sure that you are running 32 bit or 64 bit VM by running the following command:
    $ file /usr/bin/java
    /usr/bin/java: Mach-O universal binary with 3 architectures
    /usr/bin/java (for architecture x86_64):    Mach-O 64-bit executable x86_64
    /usr/bin/java (for architecture i386):    Mach-O executable i386
    /usr/bin/java (for architecture ppc7400):    Mach-O executable ppc
  • On Solaris, you would need to install 64 bit support explicitly (as of Solaris-Sparc 10). All you need to do is to download the 64 bit self installing sh file and run it in the same folder as the existing 32 bit installation.
  • To test the Virtual Memory that your process utilized run the following command:
    ps -eafl -o vsz,rss => here vsz: is the total memory including in RAM and rss: is the total memory in RAM.
  • To confirm the bitness (32 vs 64) of a running JVM process, use jinfo:
    jinfo -d64 -sysprops [pid]
    And look for “sun.arch.data.model = 64″ property.
  • To view the swap available, you need to either use top, vmstat (on solaris, check the swap column => that specifies the swap available)
  • On solaris, use prstat to figure out the no of threads that a process is spawning.