Analyzing Thread Dumps in Middleware - Part 1
This posting deals with analysis of Thread dumps, for improving Middleware Performance (at App Server or Application level) as well as for general troubleshooting. It will also go into details of WebLogic Application Server specific Thread Dump Analysis and fine tuning.
This section details with basics of thread states and locking in general. Subsequent sections will deal with capturing thread dumps and analysis with particular emphasis on WebLogic Application Server Thread dumps.
Thread Dumps
A Thread Dump is a brief snapshot in textual format of threads within a Java Virtual Machine (JVM). This is equivalent to process dump in the native world. Data about each thread including the name of the thread, priority, thread group, state (running/blocked/waiting/parking) as well as the execution stack in form of thread stack trace is included in the thread dump. All threads - the Java VM threads (GC threads/scavengers/monitors/others) as well as application and server threads are all included in the dump. Newer versions of JVMs also report blocked thread chains (like ThreadA is locked for a resource held by ThreadB) as well as deadlocks (circular dependency among threads for locks).
Different JVM Vendors display the data in different formats (markers for start/end of thread dumps, reporting of locks and thread states, method signatures) but the underlying data exposed by the thread dumps remains the same across vendors.
Sample of a JRockit Thread Dump:
===== FULL THREAD DUMP ===============
Mon Feb 06 11:38:58 2012
Oracle JRockit(R) R28.0.0-679-130297-1.6.0_17-20100312-2123-windows-ia32
"Main Thread" id=1 idx=0x4 tid=4184 prio=5 alive, in native
at java/net/PlainSocketImpl.socketConnect(Ljava/net/InetAddress;II)V(Native Method)
at java/net/PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
^-- Holding lock: java/net/SocksSocketImpl@0x10204E50[biased lock]
at java/net/PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java/net/PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java/net/SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java/net/Socket.connect(Socket.java:525)
at java/net/Socket.connect(Socket.java:475)
at sun/net/NetworkClient.doConnect(NetworkClient.java:163)
at sun/net/www/http/HttpClient.openServer(HttpClient.java:394)
at sun/net/www/http/HttpClient.openServer(HttpClient.java:529)
^-- Holding lock: sun/net/www/http/HttpClient@0x10203FB8[biased lock]
at sun/net/www/http/HttpClient.<init>(HttpClient.java:233)
at sun/net/www/http/HttpClient.New(HttpClient.java:306)
at sun/net/www/http/HttpClient.New(HttpClient.java:323)
at sun/net/www/protocol/http/HttpURLConnection.getNewHttpClient(HttpURLConnection.java:860)
at sun/net/www/protocol/http/HttpURLConnection.plainConnect(HttpURLConnection.java:801)
at sun/net/www/protocol/http/HttpURLConnection.connect(HttpURLConnection.java:726)
at sun/net/www/protocol/http/HttpURLConnection.getOutputStream(HttpURLConnection.java:904)
^-- Holding lock: sun/net/www/protocol/http/HttpURLConnection@0x101FAD88[biased lock]
at post.main(post.java:29)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Signal Handler)" id=2 idx=0x8 tid=4668 prio=5 alive, daemon
"(OC Main Thread)" id=3 idx=0xc tid=6332 prio=5 alive, native_waiting, daemon
"(GC Worker Thread 1)" id=? idx=0x10 tid=1484 prio=5 alive, daemon
"(GC Worker Thread 2)" id=? idx=0x14 tid=5548 prio=5 alive, daemon
"(Code Generation Thread 1)" id=4 idx=0x30 tid=8016 prio=5 alive, native_waiting, daemon
"(Code Optimization Thread 1)" id=5 idx=0x34 tid=3596 prio=5 alive, native_waiting, daemon
"(VM Periodic Task)" id=6 idx=0x38 tid=1352 prio=10 alive, native_blocked, daemon
"(Attach Listener)" id=7 idx=0x3c tid=6592 prio=5 alive, native_blocked, daemon
"Finalizer" id=8 idx=0x40 tid=1576 prio=8 alive, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:183)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"Reference Handler" id=9 idx=0x44 tid=3012 prio=10 alive, native_waiting, daemon
at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$100(Reference.java:11)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace
"(Sensor Event Thread)" id=10 idx=0x48 tid=980 prio=5 alive, native_blocked, daemon
"VM JFR Buffer Thread" id=11 idx=0x4c tid=6072 prio=5 alive, in native, daemon
===== END OF THREAD DUMP ===============
Sample of a Sun Hotspot Thread Dump (executing same code as above)
2012-02-06 11:37:30
Full thread dump Java HotSpot(TM) Client VM (16.0-b13 mixed mode):
"Low Memory Detector" daemon prio=6 tid=0x0264bc00 nid=0x520 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x02647400 nid=0x1ae8 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x02645800 nid=0x1480 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x02642800 nid=0x644 waiting on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x02614800 nid=0x1e70 in Object.wait() [0x1882f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x04660b18> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x04660b18> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x02610000 nid=0x1b84 in Object.wait() [0x1879f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x04660a20> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x04660a20> (a java.lang.ref.Reference$Lock)
"main" prio=6 tid=0x00ec9400 nid=0x19e4 runnable [0x0024f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
- locked <0x04642958> (a java.net.SocksSocketImpl)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:525)
at java.net.Socket.connect(Socket.java:475)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
- locked <0x04642058> (a sun.net.www.http.HttpClient)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:860)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:801)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:726)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:904)
- locked <0x04639dd0> (a sun.net.www.protocol.http.HttpURLConnection)
"VM Thread" prio=10 tid=0x0260d000 nid=0x4dc runnable
"VM Periodic Task Thread" prio=10 tid=0x02656000 nid=0x16b8 waiting on condition
JNI global references: 667
Heap
def new generation total 4928K, used 281K [0x04660000, 0x04bb0000, 0x09bb0000)
eden space 4416K, 6% used [0x04660000, 0x046a6460, 0x04ab0000)
from space 512K, 0% used [0x04ab0000, 0x04ab0000, 0x04b30000)
to space 512K, 0% used [0x04b30000, 0x04b30000, 0x04bb0000)
tenured generation total 10944K, used 0K [0x09bb0000, 0x0a660000, 0x14660000)
the space 10944K, 0% used [0x09bb0000, 0x09bb0000, 0x09bb0200, 0x0a660000)
compacting perm gen total 12288K, used 1704K [0x14660000, 0x15260000, 0x18660000)
the space 12288K, 13% used [0x14660000, 0x1480a290, 0x1480a400, 0x15260000)
No shared spaces configured.
Sample of an IBM Thread dump
NULL ------------------------------------------------------------------------
0SECTION THREADS subcomponent dump routine
NULL =================================
NULL
1XMCURTHDINFO Current Thread Details
NULL ----------------------
NULL
1XMTHDINFO All Thread Details
NULL ------------------
NULL
2XMFULLTHDDUMP Full thread dump J9 VM (J2RE 6.0 IBM J9 2.4 Windows Vista x86-32 build jvmwi3260sr4ifx-20090506_3499120090506_034991_lHdSMr,
native threads):
3XMTHREADINFO "main" TID:0x00554B00, j9thread_t:0x00783AE4, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1E48, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/oti/vm/BootstrapClassLoader.loadClass(BootstrapClassLoader.java:65)
4XESTACKTRACE at sun/net/NetworkClient.isASCIISuperset(NetworkClient.java:122)
4XESTACKTRACE at sun/net/NetworkClient.<clinit>(NetworkClient.java:83)
4XESTACKTRACE at java/lang/J9VMInternals.initializeImpl(Native Method)
4XESTACKTRACE at java/lang/J9VMInternals.initialize(J9VMInternals.java:200(Compiled Code))
4XESTACKTRACE at java/lang/J9VMInternals.initialize(J9VMInternals.java:167(Compiled Code))
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.getNewHttpClient(HttpURLConnection.java:783)
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.plainConnect(HttpURLConnection.java:724)
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.connect(HttpURLConnection.java:649)
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.getOutputStream(HttpURLConnection.java:827)
4XESTACKTRACE at post.main(post.java:29)
3XMTHREADINFO "JIT Compilation Thread" TID:0x00555000, j9thread_t:0x00783D48, state:CW, prio=10
3XMTHREADINFO1 (native thread ID:0x111C, native priority:0xB, native policy:UNKNOWN)
3XMTHREADINFO "Signal Dispatcher" TID:0x6B693300, j9thread_t:0x00784210, state:R, prio=5
3XMTHREADINFO1 (native thread ID:0x1E34, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)
4XESTACKTRACE at com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:54)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B693800, j9thread_t:0x0078EABC, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1AA4, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B695500, j9thread_t:0x0078ED20, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x14F8, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B695A00, j9thread_t:0x0078EF84, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x9E0, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B698800, j9thread_t:0x0078F1E8, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1FB8, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B698D00, j9thread_t:0x0078F44C, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1A58, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B69BB00, j9thread_t:0x0078F6B0, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1430, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B69C000, j9thread_t:0x029D8FE4, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0xBC4, native priority:0x5, native policy:UNKNOWN)
NULL ------------------------------------------------------------------------
Basics of Thread States
A Thread can be in one of the following states:
- New
- Runnable
- Non-Runnable
- Sleep for a time duration
- Wait for a condition/event
- Blocked for a lock
- Dead
In a thread dump, we are looking at threads that have been created already and are either in running or non-running states. So, the new (unless a new thread just got created at the exact moment the thread dump was generated) & dead states are really not of value or present in a thread dump.
Running state implies the thread is actively working on something. Coming to the non-runnable states, its possible a thread has nothing to do and sleeps for some duration and periodically checks for condition to start work. Wait for a condition implies the thread is waiting for some form of notification or an event and can start work once there is a green light. Its much more efficient to use waiting for a condition pattern instead of regular sleep-wake up pattern for optimal usage of resources. If there are multiple threads regularly doing a sleep and wake up periodically, they can be optimized to wake up on notify of an event and only one thread would be successful in getting the notify call instead of all doing the regular check for event in the sleep pattern.
Blocked implies it cannot proceed with its work till it can obtain a lock which is currently held by someone else. This is similar to obtaining a lock to a Critical Region or Semaphore (in OS semantics) before proceeding with the work.
States
Each of the thread entry in a thread dump specifies the state along with name and priority.
In Sun Hotspot, the state is also part of the individual thread entry. The main thread appears as RUNNABLE while the Finalizer GC thread appears in a WAITING state.
"main" prio=6 tid=0x00ec9400 nid=0x19e4 runnable [0x0024f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)"Finalizer" daemon prio=8 tid=0x02614800 nid=0x1e70 in Object.wait() [0x1882f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x04660b18> (a java.lang.ref.ReferenceQueue$Lock)
In a JRockit thread dump, there is no state mentioned for running thread. The Finalizer appears in native_waiting state.
"Main Thread" id=1 idx=0x4 tid=4184 prio=5 alive, in native
at java/net/PlainSocketImpl.socketConnect(Ljava/net/InetAddress;II)V(Native Method)
at java/net/PlainSocketImpl.doConnect(PlainSocketImpl.java:333)"Finalizer" id=8 idx=0x40 tid=1576 prio=8 alive, native_waiting, daemon at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method) at jrockit/memory/Finalizer.access$700(Finalizer.java:12)
In IBM thread dump, the state is specified by the field state. CW stands for Condition Wait.
"main" TID:0x00554B00, j9thread_t:0x00783AE4, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1E48, native priority:0x5, native policy:UNKNOWN)
4XESTACKTRACE at com/ibm/oti/vm/BootstrapClassLoader.loadClass(BootstrapClassLoader.java:65)
4XESTACKTRACE at sun/net/NetworkClient.isASCIISuperset(NetworkClient.java:122)
4XESTACKTRACE at sun/net/NetworkClient.<clinit>(NetworkClient.java:83)
....................
4XESTACKTRACE at sun/net/www/protocol/http/HttpURLConnection.getOutputStream(HttpURLConnection.java:827)
4XESTACKTRACE at post.main(post.java:29)
3XMTHREADINFO "JIT Compilation Thread" TID:0x00555000, j9thread_t:0x00783D48, state:CW, prio=10
3XMTHREADINFO1 (native thread ID:0x111C, native priority:0xB, native policy:UNKNOWN)
3XMTHREADINFO "Gc Slave Thread" TID:0x6B693800, j9thread_t:0x0078EABC, state:CW, prio=5
3XMTHREADINFO1 (native thread ID:0x1AA4, native priority:0x5, native policy:UNKNOWN) Locks
What are locks? Locks are regions that act as speed beakers or gatekeepers to ensure only one thread can obtain a temporary ownership of a resource and start work on something. This is mainly to ensure multiple threads don't work in the same region and mess up the final outcome or to ensure ordering of execution. This can be equated to only one person can operate on an ATM machine at a time as you don't want multiple different users to withdraw or deposit at the same time from the same machine without the ATM machine being able to confirm each of the operation being carried out (like credit multiple times a single deposit or credit to wrong accounts). Similar to Writer/Readers problem in OS scheduling, we don't want the multiple writers to intersperse their writes to the same page or readers to read incomplete content. JVM provides implicit locks whenever a code demarcates a method call as synchronized or a region of code within a method. The lock can be on an instance or class level or method level. JDK 1.6 provides higher level abstractions in form of concurrent.locks package (Rentrant Locks) similar to the jvm locks.
What happens when a thread requests for a Lock? If the lock is not owned by anyone, the thread becomes the new owner till it relinquishes it. What if another thread attempts to obtain ownership of the same lock when its already owned by a different thread? The new bidder gets added to a blocked list of contenders for the lock. As more threads join the waiting contenders list, the chances of getting ownership decreases among them. Normally the owner of the lock might be done finishing its job in a short duration and would relinquish the lock in a short while and one of the threads from the blocked list is chosen to become the new owner. But if the owner has to do heavy weight lifting and continues to own the lock for lot longer, and the lock is required by multiple threads, this can create a bottleneck in application or server execution as other threads cannot proceed without getting the lock which is held by the long or slow running owner. This can lead to blocked thread chains.
JRockit Thread blocking:
"ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" id=131 idx=0x248 tid=8047 prio=5 alive, blocked, native_blocked, daemon
-- Blocked trying to get lock: weblogic/utils/classloaders/GenericClassLoader@0xd1bacb10[fat lock]
at jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1411)
at jrockit/vm/Locks.lockFat(Locks.java:1512)
at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1054)[optimized]
at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1005)[optimized]
at jrockit/vm/Locks.monitorEnter(Locks.java:2179)[optimized]
at java/lang/ClassLoader.loadClass(ClassLoader.java:292)
at java/lang/ClassLoader.loadClass(ClassLoader.java:248)
at weblogic/utils/classloaders/GenericClassLoader.loadClass(GenericClassLoader.java:179)
Sun Hotspot Thread blocking:
"ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'"
waiting for lock java.util.Collections$SynchronizedSet@1dc8b68c BLOCKED
weblogic.management.provider.internal.RegistrationManagerImpl.invokeRegistrationHandlers(RegistrationManagerImpl.java:211)
weblogic.management.provider.internal.RegistrationManagerImpl.unregister(RegistrationManagerImpl.java:105)
weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:289)
weblogic.messaging.common.PrivilegedActionUtilities$2.run(PrivilegedActionUtilities.java:56)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
What if ThreadA owning LockA now needs LockB which is held by ThreadB. ThreadA will block till it gets ownership of LockB. Everyone else waiting for LockA will continue to block for LockA till ThreadA releases it. Deadlock is the condition that occurs if ThreadA is blocked for LockB while holding LockA and ThreadB is also blocked for LockA while holding LockB. Its a mutual deadlock where neither thread can proceed as its blocked for the other to release a resource it needs for its completion. The level of circular dependency can be simply between two thread or more threads. Restart of the JVM is the only way to clear the Deadlock.
Synchronization and Wait
A thread might acquire a lock but find itself unable to proceed as it has to wait for a condition to happen (like plane boarded and ready to fly but has to wait for take off signal from Control tower). In those cases, there might be others waiting to obtain lock owned by it. The owner thread can do a wait on the lock object which will make it relinquish the lock voluntarily and place itself on the waiting list till it gets a notification that has re-obtained a lock and rechecks the condition before proceeding ahead.
The usage pattern is for the thread that has changed the condition to obtain a lock and call notify on it before releasing the lock so the waiters can obtain the lock and check for the condition.
The following table shows three threads working on the same lock object but doing different activities.
| Thread1 acquires lock on synchronized code segment and then waits for event notification. Will show up as having locked, released and now waiting for notification on the same Lock object ( LockObj@0x849bb) |
|
Thread2 acquires lock on synchronized code segment and then sends notification
Will show as holding (or locked) Lock object (LockObj@0x849bb)
|
|
| Thread3 waits to acquire lock on synchronized code segment. Will show up as Blocked or Waiting for Lock (on LockObj@0x849bb)
|
|
Reducing Locks
Now we know how locks can lead to blocked threads and affect application performance, how to reduce locking in application?
There are multiple options:
- Try to use the concurrent.locks package which provide extended capabilities compared to synchronized regions or method calls for timed lock acquisition, fairness among threads etc. But use with care as not following the guidelines can lead to locks never getting released after acquiring it and lot more suffering.
- Avoid synchronized methods. Go with smaller synchronized regions whenever possible. Try to use synchronizing on a specific object (locking a room or a cabinet) that needs to be protected rather than a bigger parent object (compared to locking the entire building).
- Increase the number of resources, where access to them leads to locking. For example, when there are finite resources, access requires owner lock the resource and release once done. If the number of resources are too little (like JDBC Connections or some pool for user objects) compared to user threads requesting for the resource, there would be higher contention. Try to increase the number of resources.
- Try to cache resources if each call to create the resource requires synchronized calls.
- Try to avoid the synchronized call entirely if possible by changing the logic of execution.
- Try to control the order of locking in cases of deadlocks. For example, every thread has to obtain LockA before obtaining LockB and not mix up the order of obtaining locks.
- If the owner of the lock has to wait for an event, then do a synchronization wait on the lock which would release the lock and put the owner itself on the blocked list for the lock automatically so other threads can obtain the lock and proceed.
In this section, we went over basics of thread states and thread locking. In the next section, we will drill deeper into capturing and analyzing Thread Dumps with special look into WebLogic Application Server specific thread dumps.
Comments
Post a Comment