Safepoints in HotSpot JVM

Term Stop-the-World pause is usually associated with garbage collection. Indeed GC is a major contributor to STW pauses, but not the only one.

Safepoints

In HotSpot JVM Stop-the-World pause mechanism is called safepoint. During safepoint all threads running java code are suspended. Threads running native code may continue to run as long as they do not interact with JVM (attempt to access Java objects via JNI, call Java method or return from native to java, will suspend thread until end of safepoint).

Stopping all threads are required to ensure what safepoint initiator have exclusive access to JVM data structures and can do crazy things like moving objects in heap or replacing code of method which is currently running (On-Stack-Replacement).

How safepoints work?

Safepoint protocol in HotSpot JVM is collaborative. Each application thread checks safepoint status and park itself in safe state in safepoint is required.

For compiled code, JIT inserts safepoint checks in code at certain points (usually, after return from calls or at back jump of loop). For interpreted code, JVM have two byte code dispatch tables and if safepoint is required, JVM switches tables to enable safepoint check.

Safepoint status check itself is implemented in very cunning way. Normal memory variable check would require expensive memory barriers. Though, safepoint check is implemented as memory reads a barrier. Then safepoint is required, JVM unmaps page with that address provoking page fault on application thread (which is handled by JVM’s handler). This way, HotSpot maintains its JITed code CPU pipeline friendly, yet ensures correct memory semantic (page unmap is forcing memory barrier to processing cores).

When safepoints are used?

Below are few reasons for HotSpot JVM to initiate a safepoint:

• Garbage collection pauses
• Code deoptimization
• Flushing code cache
• Class redefinition (e.g. hot swap or instrumentation)
• Biased lock revocation
• Various debug operation (e.g. deadlock check or stacktrace dump)

Trouble shooting safepoints

Normally safepoints just work. Thus, you can care less about them (most of them, except GC ones, are extremely quick). But if something can break it will break eventually, so here is useful diagnostic:

• -XX:+PrintGCApplicationStoppedTime – this will actually report pause time for all safepoints (GC related or not). Unfortunately output from this option lacks timestamps, but it is still useful to narrow down problem to safepoints.
• -XX:+PrintSafepointStatistics –XX:PrintSafepointStatisticsCount=1 – this two options will force JVM to report reason and timings after each safepoint (it will be reported to stdout, not GC log).

References

·         How does JVM handle locks – quick info about biased locking

·         HotSpot JVM thread management

Back to school, tech meet-ups in Moscow this September

I’m regularly hosting meeting of technology enthusiasts in Moscow, and it is time to announce couple of events for this month. But wait, we are in Moscow so let me speak Russian.

Сегодня, я хочу сделать анонс очередных двух круглых столов IT энтузиастов в Москве.

13ое сентября

Вячеслав Воробьёв расскажет нам о проблеме автоматического планирования.

Проблема всплыла за кружкой пива после последней нашей встречи посвящённой автоматизации деплоймента и Вячеслав любезно согласился сделать ликбез посвящённый академическому подходу к проблеме.<

27ое сентября

Data persistence, lies, darn lies and delusions - флейм на тему СУБД от вашего покорного слуги.

Я хочу вспомнить о фундаментальных основах систем хранения данных: механизмы изоляции транзакций, WAL и undo логи, двух фазный коммит и paxos, BTrees и LSM-Trees – а потом хорошенько потролить NoSQL, SQL и вообще всё что как-то хранит данные.

Зарегистрироваться на круглые столы и подписаться на анонс мероприятий  можно по ссылке.

http://aragozin.timepad.ru/

Story of relentless CMS

Recently, a comment on other article of this blog has led me to noteworthy issue with CMS (Concurrent Mark Sweep) garbage collector.

Problem has appeared after minor release of application. JVM was configured to use CMS and it was working fine, but after a change its behavior has changed. Normally CMS is doing collection cycle when old space usage meets certain threshold, so you can see famous saw of heap usage.

Most time CMS is staying idle, just occasionally doing a collection. But after release, heap usage diagram have changed to something like this.

For some reason CMS is not waiting for heap usage threshold anymore and doing one old space GC cycle right after another. Heap usage diagram may not look bad by itself, but continuous background GC means that at least one core is constantly occupied by marking and sweeping old space. In addition, sweeping over memory also impacts cache utilization in CPUs causing additional impact to performance.

My first guess about –XX:+UseCMSInitiatingOccupancyOnly flag being missed was wrong (you know without that flag JVM could adjust CMS initiation threshold at runtime according to internal heuristics), CMS setup was fine.

After scanning through options, –XX:+CMSClassUnloadingEnabled flag has drawn my attention. By default CMS will not collect permanent space; you should use that flag to enable it. Permanent space is a special memory space used by Java class objects and some JVM data structures; it is not part of application heap and being sized separately. It means, in particular, that permanent space has its own memory usage which is not correlated with old space memory usage.

So, if CMS for permanent space is enabled, GC cycle will be triggered if either old space or permanent space has reached usage threshold. This turned out to be a problem. Permanent space was a little too small for application, so CMS were trying to collect it relentlessly.

Increasing permanent space size (-XX:PermSize=size) has solved an issue.

Alternative approach could be using different threshold for old and permanent space (i.e. –XX: CMSInitiatingPermOccupancyFraction=percent). Also it may make a sense to turn off permanent space collection at all, many applications just do not need it (it was called “permanent” for reason after all).