ThreadLogic version 0.95 available

 

ThreadLogic version 0.95 is available for public download now.
 
Additional Features

Biggest change is support for externalizing the Advisories and group definitions. Users can use the pre-defined AdvisoryMap.xml to come up with custom advisories with their own definition of what constitutes an advisory - name, pattern/keyword, health, description, advice and let ThreadLogic tag the matching threads with your custom advisory when it finds a match. Similarly the group definitions can be modified or enhanced to include new groupings. This is in addition to other smaller bug fixes and code cleanup.

Before we go into how to add customized advisories and groups, it would help to understand how the Advisory and Groups are designed.

Advisory

The advisories are loaded from a file named AdvisoryMap.xml. A copy of the advisory map is located under com/oracle/ateam/threadlogic/resources folder of the threadlogic jar file.

Example of Advisory:

            <Advisory>

                        <Name>Custom NIO Select Advisory</Name>

                        <Health>IGNORE</Health>

                        <Keyword>sun.nio.ch.SelectorImpl.select</Keyword>

                        <Descrp>Using Native IO via Select or Poll</Descrp>

                        <Advice>Ignore</Advice>

            </Advisory>

Each Advisory entry has a Name, Health, Keyword (pattern to search against), Description and Advice.

• The Name is used as short id/reference to an Advice
• The Health can be one of the following - IGNORE, NORMAL, WATCH, WARNING, FATAL (increasing level of severity).
• The Keyword is the pattern or marker to look for in a thread. It can be in thread stack content or in the thread name/labels etc. The keyword can be a package, class name, method name combination or just a specific method name that can be a unique identifier for that specific advisory.
• The Descrp describes what was found via the advisory
• The Advice provides any suggestions or tips to resolve any problem reported via the advisory.

<Advisory>     <Name>WLS Idle Thread</Name>     <Health>IGNORE</Health>     <Keyword>ExecuteThread.waitForRequest</Keyword>     <Descrp>WebLogic Idle Thread waiting for new request</Descrp>     <Advice>Ignore - its an idle thread waiting for a new request to execute</Advice> </Advisory>

The Thread with a certain set of advisories is marked with the health level that matches the most severe of its tagged advisories. Similarly, the critical advisories across multiple threads in the thread group are promoted to the group and so on to the Thread dump.

Advisory Pattern Matching

Use '.' (period) instead of / package paths in the keyword entry.

Use wildcard as needed; Use . (period character) for escaping $, ?, _ etc.

The advisory is a match for a thread when the keyword search against the thread is successful. There can be multiple keywords separated by a ", ".

For example:

<Advisory>

  <Name>Database Query Execution</Name>

  <Health>WATCH</Health>   

  <Keyword>PreparedStatement.execute, executeQuery</Keyword>

  <Descrp>Executing operation or query on DB</Descrp>

  <Advice>Check/Monitor Database SQL Executions if it takes longer and also check for socket connection disruption to database if thread continues to show same pattern</Advice>

</Advisory>

In above example, there are 2 keywords (PreparedStatement.execute and executeQuery) both covering some form of DB operation. Use ',' as a separator for specifying multiple patterns within an advisory.

For multi-line pattern, use PatternA.*PatternB to match against all lines that start off  with PatternA with some content in the middle and ending with PatternB.

Use with caution as it can do greedy grab of everything within the specified patterns.

Example:

<Advisory>

  <Name>Test Wild card pattern</Name>

  <Health>WARNING</Health>   

  <Keyword>myPackage.myTest.*nextFewLines.doSomething</Keyword>

  <Descrp>Test description</Descrp>

  <Advice>Test advice</Advice>

</Advisory>

 

The Name of the advice can be used within the GroupDefns (explained in next section) to downgrade certain advisories for specific thread groups. Ensure the Advice Name in AdvisoryMap matches with the AdvisoryId inside GroupDefns.

For Example:

<ComplexGroup>

  <Name>Oracle AQ Adapter</Name>

  <Visible>true</Visible>

  <Inclusions>

    <SimpleGroupId>Oracle AQ AdapterTemp</SimpleGroupId>

  </Inclusions>

  <Exclusions>

    <SimpleGroupId>Oracle SOA DFW</SimpleGroupId>

  </Exclusions>

  <ExcludedAdvisories>

    <AdvisoryId>Database Query Execution</AdvisoryId>

    <AdvisoryId>Socket Read</AdvisoryId>

  </ExcludedAdvisories>       

</ComplexGroup>

In the above example, the default health levels associated with Database Query Execution (WATCH) and Socket Read (WATCH) Advisories are excluded against the thread group: Oracle AQ Adapter as the AQ Adapter threads functionality is to poll for data from DB over sockets repeatedly and its expected normal behavior.

Note: Avoid using '&' as it would confuse the XML Parser and result in exceptions.

If '&' is really required, use '&amp;' as xml parsing would fail against &.

Thread Group Definitions

The Thread Groups are created based on Group definitions declared in two xml files: WLSGroups.xml & NonWLSGroups.xml (both packaged under com/oracle/ateam/threadlogic/resources folder of the threadlogic jar file). The WLS Groups would be created first before Non WLS Groups.

The thread groups association is based on the execution path in the threads or thread names.

The Thread Group definition can be a simple group (match a set of patterns) or complex (include some groups and exclude others). A set of advisories can also be referred as ignorable or excluded while determining the health of a thread or a group.

            <SimpleGroup>

                        <Name>Oracle Service Bus (OSB)</Name>

                        <Visible>true</Visible>

                        <Inclusion>true</Inclusion>

                        <MatchLocation>stack</MatchLocation>

                        <PatternList>

                                    <Pattern>com.bea.wli.sb.transports</Pattern>

                                    <Pattern>com.bea.wli.sb.pipeline</Pattern>

                        </PatternList>

            </SimpleGroup>

            <ComplexGroup>

                        <Name>Oracle AQ Adapter</Name>

                        <Visible>true</Visible>

                        <Inclusions>

                                    <SimpleGroupId>Oracle AQ AdapterTemp</SimpleGroupId>

                        </Inclusions>

                        <Exclusions>

                                    <SimpleGroupId>Oracle SOA DFW</SimpleGroupId>

                        </Exclusions>

                        <ExcludedAdvisories>

                                    <AdvisoryId>Database Query Execution</AdvisoryId>

                                    <AdvisoryId>Socket Read</AdvisoryId>

                        </ExcludedAdvisories>       

            </ComplexGroup>

Users can define with their own custom thread groups or enhance/modify by editing the two group definition files.

Group Definitions

The GroupDefn can contain multiple Thread Group Definitions. Each of the thread group definition can be Simple or Complex Group Type.

SimpleGroup

A SimpleGroup can contain direct patterns for threads to match against (either against thread name or stack). A ComplexGroup is comprised of individual SimpleGroup entities.

SimpleGroup has following elements:

• Name of the Simple Group. SimpleGroup names can be used as reference for building ComplexGroups.
• Visible implies the Group should be visible in the Analysis Tree View
• Inclusion means to include threads matching the specified pattern. Specify true or false. If false, means exclude those threads that demonstrate a given pattern
• Patterns can be code pattern (combination of package, class and or method names). Wild cards are also supported.  Use '.' instead of '/' for the package names. There can be multiple pattern entries or all merged with a '|' delimiter to form a choice. Using individual pattern entries makes for easier reading.
• MatchLocation is used to locate a matching pattern against thread stack or thread name. Allowed values are either stack or name.
• Visible determines whether this group should appear in the thread group tree or not. Allowed values are true or false. If not false, then the simple group might be used as a building block for complex groups but not exposed as a node within the groups

Note: Simple Groups can have multiple patterns but all patterns within that group can only be applied against the Thread Name or Stack, similar restriction applies for inclusions or exclusion.

The SimpleGroup definition listed below tries to categorize multiple threads whose thread name matches "GC task" or "VM Periodic task" or "Attach Listener" etc. and categorize them under the JVM Group and marking it as a visible group.

<SimpleGroup>

            <Name>JVM</Name>

            <Visible>true</Visible>

            <Inclusion>true</Inclusion>

            <MatchLocation>name</MatchLocation>

            <PatternList>

                        <Pattern>GC task</Pattern>

                        <Pattern>Low Memory Detector</Pattern>

                        <Pattern>CompilerThread</Pattern>

                        <Pattern>Finalizer</Pattern>

                        <Pattern>VM Periodic Task</Pattern>

                        <Pattern>Attach Listener</Pattern>

                        <Pattern>Attach .andler</Pattern>

                        <Pattern>OperatingSystemMXBean</Pattern>

                        <Pattern>MemoryPoolMXBean</Pattern>

                        <Pattern>Code Generation Thread</Pattern>

                        <Pattern>Code Optimization Thread</Pattern>

                        <Pattern>VM Thread</Pattern>

            </PatternList>

</SimpleGroup>     

Or use '|' as delimiter to provide a choice of patterns:

            <SimpleGroup>

                        <Name>WebLogic Muxer</Name>

                        <Visible>true</Visible>

                        <Inclusion>true</Inclusion>

                        <MatchLocation>stack</MatchLocation>

                        <PatternList>                                                                          <Pattern>EPollSocketMuxer|DevPollSocketMuxer|PosixSocketMuxer|NTSocketMuxer|JavaSocketMuxer|NIOSocketMuxer</Pattern>

                        </PatternList>

<ExcludedAdvisories>

    <AdvisoryId>Socket Read</AdvisoryId>

</ExcludedAdvisories>   

            </SimpleGroup>

Above example shows how to match multiple threads based on some pattern in the stack - match threads containing "EPollSocketMuxer" or "DevPollSocketMuxer" as belonging to WebLogic Muxer group by searching for those keyword patterns against the thread stack and make it a visible group.

Also, one can see that it carries exclusion - exclude health levels associated with Socket Read advisory even if the threads in this Muxer group match that advisory. The AdvisoryId should match name of an advisory defined in the AdvisoryMap.xml described earlier.

Complex Groups

Complex groups are comprised of Simple Groups. They cannot directly refer to patterns unlike SimpleGroup definition. Each of the underlying simple groups within a Complex group can be inclusion or exclusion. The Complex group itself can be visible or not, just like the SimpleGroup, to be displayed as a Thread Group.

<SimpleGroup>

            <Name>Oracle AQ AdapterTemp</Name>

            <Visible>false</Visible>

            <Inclusion>true</Inclusion>

            <MatchLocation>stack</MatchLocation>

            <PatternList>                                                                 

                        <Pattern>oracle.integration.platform.blocks.event.saq</Pattern>

            </PatternList>

</SimpleGroup>

<ComplexGroup>

            <Name>Oracle AQ Adapter</Name>

            <Visible>true</Visible>

            <Inclusions>

                        <SimpleGroupId>Oracle AQ AdapterTemp</SimpleGroupId>

            </Inclusions>

            <Exclusions>

                        <SimpleGroupId>Oracle SOA DFW</SimpleGroupId>

            </Exclusions>

            <ExcludedAdvisories>

                        <AdvisoryId>Database Query Execution</AdvisoryId>

                        <AdvisoryId>Socket Read</AdvisoryId>

            </ExcludedAdvisories>       

</ComplexGroup>

In above sample, the Complex Group "Oracle AQ Adapter" uses 2 underlying simple groups:  Oracle AQ AdapterTemp (as inclusion) and Oracle SOA DFW (as exclusion). All threads belong to the simple group referred by "Oracle AQ AdapterTemp" should be included while all threads matching the "Oracle SOA DFW" should be excluded. Use non-visible Simple Groups to build more ComplexGroups that would be visible.

Also both Simple and Complex Groups can exclude certain advisories - as in disgregard the health of those advisories in determining overall thread health consideration. Note: The AdvisoryId should match a predefined Advisory name in the AdvisoryMap.xml file described earlier.

Thread Groups

Based on the parsing of the WLS & NonWLS Groups.xml, the thread groups get divided into two buckets - WLS and non-WLS related threads. The JVM threads, LDAP and other unknown custom threads go under the non-WLS bucket while all the WLS, Muxer, ADF, Coherence, Oracle, SOA, JMS, Oracle Adapter threads are all under the WLS bucket. The classification can be changed by modifying the underlying GroupsDefn xml files. 

The Threads are first filtered against WLS Group based on definitions in the WLSGroups.xml before categorizing remaining threads within the non-WLS groups that are defined in the NonWLSGroups.xml file.

Customizing Advisories & Grouping

Users can enhance the advisories and grouping by adding their own definitions into the mix, overriding the ones packaged within Threadlogic distribution. This means users dont have be limited or not be able to categorize their custom groups while adding advisories for patterns they frequently hit or look out against.

Create two sub-directories named advisories and groups (can be named differently also) where they expect to run the theadlogic. Copy over the AdvisoryMap.xml into the advisories folder and copy the WLSGroups.xml and NonWLSGroups.xml into the groups folder. All the xml files are packaged under the com/oracle/ateam/threadlogic/resources folder of the jar.

Edit the copied files to modify or include new definitions. For advisories, you can create additional files by using the base AdvisoryMap.xml as template to create or modify new advisory definitions. The groups definitions are limited to just two files: WLSGroups.xml and NonWLSGroups.xml.

Run threadlogic tool while following the instructions mentioned below to load customized advisories and or groups.

Customized Advisories

Use -Dthreadlogic.advisories=<advisories-directory> command line argument to pick your customized advisories from a mentioned directory to be used in ADDITION TO the ones packaged within Threadlogic jar file.

There can be multiple advisories definition files for each subsystem or product and these would all get loaded ahead of the packaged AdvisoryMap.xml definition.

Note: Use unique keywords and names to avoid conflicts.

Example:

java -Dthreadlogic.advisories=/user/test/tlogic/advisories \

  -jar /user/test/tlogic/threadlogic-jar

Customized Grouping

Use -Dthreadlogic.groups=<groups-directory> command line argument to pick your customized set of group definitions from the specified directory INSTEAD OF ones packaged within Threadlogic jar file.

ThreadLogic expects the groups definition files to be just two - WLSGroups.xml and NonWLSGroups.xml. All WebLogic related thread groups are expected in the WLSGroups and would be created ahead of the NonWLS Groups. The defnitions picked this way from user defined directory would override the packaged (Non)WLSGroups.xml definitions. This customization behavior for groups (substitution) is different from customized advisories (addition)

Ensure the names for the files are retained as its necessary for creating the WLS & NonWLS parent groups.

Example: :

java -Dthreadlogic.groups=/user/test/tlogic/groups  \

  -jar /user/test/tlogic/threadlogic-jar

To add custom advisories and override groups, add both command line arguments pointing to your customized advisories/group folders:

java \

   -Dthreadlogic.groups=/user/test/tlogic/groups \

   -Dthreadlogic.advisories=/user/test/tlogic/advisories  \

   -jar /user/test/tlogic/threadlogic-jar

Reporting Time stamps

JVM and Timestamp reporting against individual Thread Dump

Thread Dump reporting has been enhanced to report time stamp (whenever available) from the dumps. Merge has been enhanced to report on the progress of the thread across the thread dumps while reporting time stamp.

Merged view reporting of individual thread stacks with time stamp

Merged view showing progress information of individual threads with time stamps

Summary

Thanks to customized advisories and groups, its should be easy for users to quickly add and customize patterns and groups, as well as highlight the patterns or anti-patterns already implemented in the advisory list.

Support for WLST (weblogic scripting tool) generated thread dumps would be available in the next release (available in Revision 72  in the online repository) and parsing of partial dumps that don't carry any JVM specific markers.

Feedback and suggestions welcome.