SJK is learning new tricks

SJK or (Swiss Java Knife) was my secret weapon for firefighting various types of performance problems for long time.

A new version of SJK was released not too long ago and it contains а bunch of new and powerful features I would like to highlight.

ttop contention monitoring

SJK is living it's name by bundling a number of tool into single executable jar. Though, ttop is a likely single most commonly used tool under SJK roof.

ttop is a kind top for threads of JVM process. Besides CPU usage counter (provided by OS) and allocation rate (tracked by JVM), a new thread contention metrics was introduced in recent SJK release.

Thread contention metrics are calculated by JVM, which counts and times when Java threads enters into BLOCKED or WAITING state.

If enabled, SJK is using these metrics to display rates and percentage of time spent in either state.

2018-05-29T14:20:03.382+0300 Process summary 
  process cpu=231.09%
  application cpu=212.78% (user=195.86% sys=16.92%)
  other: cpu=18.31% 
  thread count: 157
  GC time=4.72% (young=4.72%, old=0.00%)
  heap allocation rate 976mb/s
  safe point rate: 6.3 (events/s) avg. safe point pause: 8.24ms
  safe point sync time: 0.07% processing time: 5.09% (wallclock time)
[000180] user=19.40% sys= 0.31% wait=183.6/s(75.77%) block=    0/s( 0.00%) alloc=  110mb/s - hz._hzInstance_2_dev.cached.thread-8
[000094] user=16.92% sys= 0.16% wait=58.50/s(81.54%) block=    0/s( 0.00%) alloc=   94mb/s - hz._hzInstance_3_dev.generic-operation.thread-0
[000057] user=15.05% sys= 0.62% wait=56.91/s(82.35%) block= 0.20/s( 0.01%) alloc=   91mb/s - hz._hzInstance_2_dev.generic-operation.thread-0
[000095] user=15.21% sys= 0.00% wait=55.61/s(82.32%) block= 0.30/s( 0.04%) alloc=   87mb/s - hz._hzInstance_3_dev.generic-operation.thread-1
[000022] user=14.59% sys= 0.00% wait=56.01/s(83.42%) block= 0.30/s( 0.08%) alloc=   86mb/s - hz._hzInstance_1_dev.generic-operation.thread-1
[000058] user=13.97% sys= 0.16% wait=56.91/s(84.13%) block= 0.10/s( 0.02%) alloc=   81mb/s - hz._hzInstance_2_dev.generic-operation.thread-1

An important fact about these metrics is - CPU time + WAITING + BLOCKED should be 100% in ideal world.

In reality, you a likely to see a gap. A few reason why equation above is not holding:

• GC pauses are freezing thread execution, but not accounted by thread contention monitoring,
• thread may be waiting for IO operation, but it is not accounted as BLOCKED or WAITING state by JVM,
• system may starve on CPU resource and thread is waiting for CPU core on OS level (which is also not accounted by JVM).

Contention monitoring is not enabled by default, use -c flag with ttop command to enabled it.

HTML5 based flame graph

SJK was able to produce flame graphs for sometime already. Though, old flame graphs were generated as svg with limited interactivity.

New version offers a new type of flame graphs based on HTML5 and interactive. Right in browser it allows:

• filtering data by threads,
• zoom into specific paths or by presence of specific frame,
• filtering data by thread state (if state information is available).

HTML5 report is 100% self contained file with no dependencies, it can sent it by email and open on any machine. Here is an example of new flame graph you can play right now.

New flame command is used to generate HTML5 flame graphs.

`jstack` dump support

SJK is accepting a number of input data formats for thread sampling data, which is used for flame graphs and other types of performance analysis.

A new format added in 0.10 version is text thread dump formats produced by jstack. Full list of input formats now:

• SJK native thread sampling format
• JVisualVM sampling snapshots (.nps)
• Java Flight Recorder recording (.jfr)
• jstack produced text thread dumps

HeapUnit - Test your Java heap content

There are usually a number of tests which you would like to run for each build to make sure what your code does make sense. Typically, such tests would be focusing on business function of your code.

Though, on a rare occasion, you would really like to test certain non-functional aspects. A memory/resource would be a good example.

How would you test memory leak?

This is quite a challenge, right?

You can use debugger or profiler to inspect internal state of your system. Though, that approach assumes manual testing.

You can write test which would stress your system provoking OutOfMemoryError which would fail your test if code has defect. That generally works, though adding a stress test to mostly functional automatic test pack may not be a best idea. That approach may not work for other kind of resource leaks.

You can exploit weak or phantom reference to trace garbage collector work. This approach makes test more lightweight compared to fully fledged stress testing, but it is not applicable in many cases. E.g. you may not have a reference to leak suspected objects.

For some time I was actively practising automated inspection of JVM heap dumps for diagnostic purposes. JVM could easily produce its own heap dump (using JVM attach interface) and that dump can be inspected via API to assert certain invariants (e.g. number of live instances of particular type). Why not use it for resource leak testing and similar cases?

Resurrecting object from dump

Heap dump API allows you to inspect fields of dumped objects; there is also heap path notation for writing sophisticated selectors. Though, you cannot invoke methods, not even toString() or equals(), on objects from dump. For quantitative analysis of, this is ok. But for asserting complex conditions typical to test scenario, dealing with Java objects may be much more convenient, though.

Heap dump doesn?t contain full class information. But if dump is produced from JVM we are running in we can relay on class metedata available through reflection.

Objenesis library and Java reflection is used to convert instance data from heap dump back to normal Java objects.

At the end, usage of HeapUnit is fairly simple. Using API you can

• take heap dump
• select certain types of instance from dump by class or heap path notation
• inspect instance?s fields using symbolic names
• or rehydrate instance into Java object

Example

Below is a simple example listing Socket objects in JVM

@Test
public void printSockets() throws IOException {

    ServerSocket ss = new ServerSocket();
    ss.bind(sock(5000));

    Socket s1 = new Socket();
    Socket s2 = new Socket();

    s1.connect(sock(5000));
    s2.connect(sock(5000));

    ss.close();
    s1.close();
    // s2 remains unclosed

    HeapImage hi = HeapUnit.captureHeap();

    for(HeapInstance i: hi.instances(SocketImpl.class)) {
        // fd field in SocketImpl class is nullified when socket gets closed
        boolean open = i.value("fd") != null;
        System.out.println(i.rehydrate() + (open ? " - open" : " - closed"));
    }
}

HeapUnit library is available in Maven Central repo. You can bring it to your project using Maven coordinates below.

<dependency>
    <groupId>org.gridkit.heapunit</groupId>
    <artifactId>heapunit</artifactId>
    <version>0.2</version>
</dependency>

HotSpot JVM garbage collection options cheat sheet (v4)

After three years, I have decided to update my GC cheat sheet.

New version finally includes G1 options, thankfully there are not very many of them. There are also few useful options introduced to CMS including parallel inital mark and initiating concurrent cycles by timer.

Finally, I made separate cheat sheet versions for Java 7 and Java 8.

Below are links to PDF versions

• Hotspot JVM GC options cheat sheet for Java 8
• Hotspot JVM GC options cheat sheet for Java 7

Java 8 GC cheat sheet

Java 7 GC cheat sheet