Hung thread detection in WebSphere Application Server

A Java dump, also known as a Java core, Java thread dump, or a thread dump is a file that contains the information about thread like active thread, hung thread, dead lock, runnable and inactive thread info.
The thread dump is a snapshot of exactly what's executing at a moment in time.

We are finding thread hung in logs like this.

WSVR0605W: Thread threadname has been active for hangtime and may be hung.  There are totalthreads  threads in total in the server that may be hung.

Example:

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758692 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758693 milliseconds and may be hung.  There is/are 2 thread(s) in total in the server that may behung .

.

.

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 75869N milliseconds and may be hung.  There is/are n thread(s) in total in the server that may be hung.

Automated Thread dump generation.

Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom propertie>New-

com.ibm.websphere.threadmonitor.dump.java    when value is set to 'true', creates a java core when a hung thread is detected false

ManualThread dump generation

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

$AdminControl invoke $jvm dumpThreads.

Genarate through python Script

Download the file windows_hang.py and copy the file to your <PROFILE_ROOT>\bin directory. If instead copied to <WAS_HOME>\bin, the default server, which may be the deployment manger (dmgr), will be accessed when wsadmin.bat is launched.


NOTE: This script only works for WebSphere Application Server 6.1 and higher.

To launch the script to produce 3 javacores spaced 2 minutes apart, run this command:

wsadmin -lang jython -f windows_hang.py -j -s SERVER_NAME

Replacing SERVER_NAME with your server's name in AppServer profile root.

EX:

javacore.date.time.id.txt

For example: javacore.20120909.204717.27050.t

IN Linux.

use kill -3 PID on unix/linux machines.

Thread Monitoring settings

• Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom properties
• Add the following 4 custom properties

#	Property	Description	Default
A	com.ibm.websphere.threadmonitor.interval	How frequently thread monitor should check all the managed threads for hung threads	180
B	com.ibm.websphere.threadmonitor.threshold	After how many seconds a thread can be considered as hung	600
C	com.ibm.websphere.threadmonitor.false.alarm.threshold	The number of times that false alarms can occur before automatically increasing the threshold (T)	100
D	com.ibm.websphere.threadmonitor.dump.java	when set to 'true', creates a java core when a hung thread is detected	false

• Click OK and save the changes.
• Sync the changes and restart the servers for the changes to take effect.