Analyzing Thread Dumps in Middleware - Part 3

This posting is the third section in the series Analyzing Thread Dumps in Middleware

This section details with tools for analyzing thread dumps along with limitations of such tools.

Thread Dump Analysis Tools

Earlier in the series, we saw how to capture thread dumps and analyze manually the threads and dependencies and bottlenecks. Sometimes the number of threads can be quite large with real big stack depths making manual  analysis quite tedious and easier to miss problems and optimizations. Using a tool can help make the analysis more efficient and more consistent.

We will briefly cover a few tools and then  drill down deeper into a specific tool enhanced by the author of this blog in collaboration with his colleague.

Samurai Thread Dump Analyzer

Samurai is a open source Thread Dump Analyzer (TDA) tool that can handle large thread dumps and report on deadlocks and contentions. It uses a source and graphical view to toggle between threads and also supports comparison of threads across multiple thread dumps. It can be downloaded from

http://www.softpedia.com/progDownload/Samurai-Download-38475.html. Its also capable of analyzing GC Logs.

Sample snapshot of Samurai TDA

 

 It highlights the threads in different color coding and clicking on the thread names take the user to the thread stack traces. Its useful to quickly see blocked threads and compare threads across thread dumps.

Thread Dump Analyzer (TDA)

TDA is another open source analyzer developed by Ingo Rockel (with contributions from Robert Whitehurst) that can be run as a standalone analyzer as well as a Java VisualVM Plugin.

The latest version can be downloaded from http://java.net/projects/tda/downloads. Version 2.2 supports Sun Hotspot and JRockit Thread Dumps. It reports the analysis in a graphical view with threads listed under different categories (All threads, blocked, waiting / sleeping for events) and Locks (termed as monitors). It is also capable of merging threads across different threads and reporting the thread contents.

Clicking on the Monitors shows the lock chains (holder of the lock) and blocked threads:

 

Thread navigation is made easy by just clicking on the threads. Similarly click on the lock info to navigate the lock chains. 

Comparing the thread dumps results in a merged view of threads and their stack traces.

 

Clicking on a thread shows its stack traces from across the compared thread dumps. The tool can also analyze thread dumps directly from clipboard rather than reading only from a log file. Another key strength is its very low memory consumption while handling real large thread dumps.

Limitations of TDA Tools

The above mentioned TDA tools eases analysis of Thread Dumps by highlighting threads in various states as well as navigating lock chains. 

But they still do not report on the specific activity within the threads - what is each thread doing? is there some optimization possible? should something be avoided or removed? Can a thread be treated as idle or benign or does it require a closer look? What are the hotspots even if there are no locks involved? Are there any application server specific recommendations that can be mined and utilized?

These can be simple but valuable things that can be highlighted resulting in more detailed and useful thread dump analysis that tends to get overlooked with currently available analyzers tools. The real analysis is left to the user who might just glance over the thread stacks when there are too many threads or very deep stacks and miss the details.

Summary

In this section, we discussed some of the tools that can help with thread dump analysis as well as their limitations. In the next and final section in this series, we will look a new tool developed by the author of this blog in collaboration with his colleague in Oracle to fill the gaps mentioned with some of existing TDA tools in general as well as look at some real world thread dump analysis.