Ad-hoc diagnostic tools for JVM

There are bunch of graphical tools helping you to look inside of running JVM (VisualVm and JConsole are extremely helpful). Unfortunately, some times (almost always in my case), you may find yourself in SSH console on headless server side by side with your JVM process, trying to investigate problem.

Why CLI is important?

You can connect to remote JVM process via JMX in VisualVM and there are other interesting tools like CRaSH offering a lot of goodies for troubleshooting. But …

• You may be behind firewall, using broker SSH relay as only way to access environment.
• All remote tools require ahead of time setup on JVM side.
• Remote connections should be secured properly – that is huge burden.

CLI tools are leveraging OS security and will work in your SSH console, reliving you from all pains above.

Stock tools

Of cause there are CLI tools in your JDK package. Let me highlight few tools from Oracle’s stock JDK.

jps

This one could list you JVM process (instead of doing ps … | grep java). Similar to ps it could display command line arguments of process. It is useful to find PID of JVM you are interested in, which will be required for other tool.

jmap

This little tool will allow you to take adhoc head dumps and calculate memory footprint histograms by classes. It also can be used to enfore full GC on target JVM. Be careful,  some of jmap operations could cause STW pauses in target JVM.

jstack

Dump your thread stacks, look though your locks.

jstat

JVM exposed a lot of internal details (e.g. memory size, compilation statistics, etc). jstat could report some of this data. Output is fairly cryptic (and machine oriented), but never less helpful.

jinfo

Introspect –XX options of running JVM. You could also change some of them for live JVM process (e.g. enable -XX:+PrintGCDetails to add GC details in application log).

Behind the scene

Behind the scene JVM have several internal protocols which could be used by diagnostic tools (not to count JMX):

• Attach API – lightweight protocol for connection to JVM processes.
• Perf data – shared memory based protocol for JVM to expose its performance counters. Using shared memory makes it very lightweight.
• JDI, JDWP, JVMTI – components of Java Platform Debugger Architecture

Both Attach API and perf data are lightweight, fairly unintrusive for monitored application and could be easily used from Java code.

Swiss Java Knife – CLI tools for ad hoc JVM diagnostic

Some time ago I was blogging about few tools using Attach API and JMX – jtop and gcrep. Since then, few things have changed:

• wrapper around attach API has been factored out in separate module org.gridkit.lab:jvm-attach-api:1.1
• tools have been consolidated into single JAR and modularized (so it is easy to add new commands or build jar with specific set of commands)

But most important new cool features have been added:

java –jar sjk.jar jps

Stock jps can print command line, but it is as far as it can help you. SJK’s version of jps allows you to choose which information about process you would like to be shown. E.g. you can add value of specific system property or –XX flag to the output. Another improvement is build-in filter option. You can filter java processes by command line (which includes main class name) and system properties.

java –jar sjk.jar ttop

Thread top command was also improved. Sort option has been added and you also can limit number of threads in output to top N. Additinally, filter option will help you to monitor only certain threads.

java –jar sjk.jar mxdump

This command will dump all MBeans from process to json format.

java –jar sjk.jar hh

“Heap histo” command is extended version of jmap –histo. Killer feature is --dead flag which will display histogram of garbage in heap (actually tool will take 2 histograms: all heap and live object – and show difference).

You can download sjk jars here.
Or build from sources https://github.com/aragozin/jvm-tools