Enabling trace

Trace logs show trace events such as function entries and exits, component events, and debugging activities. Use the administration console to enable trace for a process.
You can configure the IBM^® WebSphere^® XML Document Management Server Component to start in a trace-enabled state by setting the appropriate configuration properties.
You can control how much detail each logger records by adjusting the log level details. Because the loggers are grouped hierarchically, setting the trace level on one logger also sets all subsequent loggers to the same level. Altering the tracing levels impact the performance of the system.
Enable and configure trace by completing the following steps:

1.     Log in to the Integrated Solutions Console.

a.     Open a browser and navigate to the following URL: https://host_name:port/ibm/console.

Where:

§  host_name is the fully qualified host name of the server where the application or the network deployment manager is deployed.

§  port is the secured port used to access the console. The default is 9043.

Note: The default unsecured port is 9060. If you use 9060, you must have "http" instead of "https" in the URL.

b.    Enter an administrator user ID and password. If security is not enabled leave the password field blank.

c.     Click Log in.

2.     In the navigation panel, click Troubleshooting > Logs and Trace.

3.     Click server_name in the Server column of the table.

4.     Click Diagnostic Trace.

5.     Configure your trace options:

a.     Display the Configuration tab.

b.    Select Enable log.

c.     Click Change Log Level Details(if you required new logs traces default =INFO.*).

d.    Click Components to view all loggers for the individual components.

e.     Click + to show the children of the logger.

f.     Click logger_name to change the log details. To enable tracing on specific components of IBM WebSphere XML Document Management Server Component, use com.ibm.xdms.*=all as a logger group name.

§  com.ibm.xcap.*

§  com.ibm.xdms.agp.*

§  com.ibm.xdms.common.*

§  com.ibm.xdms.function.*

§  com.ibm.xdms.install.*

§  com.ibm.xdms.osgi.*

§  com.ibm.xdms.sip.*

§  com.ibm.xdms.utils.*

§  com.ibm.xdms.xcap.*

g.    Choose the appropriate level of tracing.

Remember: When you change the level for a logger, the change is propagated to the children of the logger.

For additional information regarding trace levels, click ? in the title bar of the panel to open the help page.

6.     Click OK.

7.     Click Save.

8.     Restart the server_name server. 

The Flow of J2EE Application

The typical application flow is as follows:

1. A Web client requests a URL in the browser (input page).

2. The request is routed to the Web server over the Internet.

3. The Web server immediately passes the request to the Web server plug-in.

All requests go to the Web server plug-in first.

4. The Web server plug-in examines the URL, verifies the list of host name

aliases from which it will accept traffic based on the virtual host information,

and chooses a server to handle the request.

5. A stream is created. A stream is a connection to the Web container. It is

possible to maintain a connection (stream) over a number of requests. The

Web container receives the request and, based on the URL, dispatches it to

the proper servlet.

6. If the servlet class is not loaded, the dynamic class loader loads the servlet

(servlet init(), then doGet() or doPost()).

7. JNDI is used for lookup of either datasources or EJBs required by the servlet.

8. Depending upon whether a datasource is specified or an EJB is requested,

the JNDI directs the servlet:

– To the corresponding database and gets a connection from its connection

pool in the case of a data source.

– To the corresponding EJB container, which then instantiates the EJB when

an EJB is requested.

9. If the EJB request involves an SQL transaction, it goes back to the JNDI to

look up the datasource.

10.The SQL statement is executed and the retrieved data is sent back either to

the servlet or to the EJB.

11.Data beans are created and handed off to JSPs in the case of EJBs.

12.The servlet sends data to JSPs.

13.The JSP generates the HTML that is sent back through the plug-in to the Web

server.

14.The Web server sends the output page (output HTML) to the browser.

Heap Dumps generation

Heap Dumps

Heap dump is a text file which keeps record of all objects in the java heap. It contains the information like size and address of all objects, as well as the addresses of all objects that it references.

Heap dump is nothing but a snap shot of our JVM’s Heap memory,contains all active java objects and their activities which object doing what and how much memory it is occupying.

whenever OutOfMemoryException occurs Websphere application Server creates heap dump in profile's home directory.

It is useful in situations, most notable for memory leak problems or memory allocation problems for exaple OutOfMemoryExceptions.

Automated heap dump generation

Click Servers > Application servers in the administrative console navigation tree.

Click server_name >Performance and Diagnostic Advisor Configuration.

Click the Runtime tab.

Select the Enable automatic heap dump collection check box.

Click OK.

Generating Heap dump Manually

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

For generating heap dump:

$AdminControl invoke $jvm generateHeapDump.

In order to manually generate a heapdump, you must first enable signal-based heapdumps by setting the IBM_HEAPDUMP=TRUE and IBM_NOSIGHANDLER=TRUE environmental variables.

What is the effect of changing max heap and min heap size of a jvm

                             Max Heap Size: When you have established the maximum heap size that you need,  you might want to set the minimum heap size to the same value;  Using the same values is not usually a good idea, because it delays the start of garbage collection until the heap is full. The first time that the Garbage Collector runs, therefore, becomes a very expensive operation. Also, the heap is more likely to be fragmented and require a heap compaction. Again this is a very expensive operation.

                             Min Heap Size: The recommendation is to start your application with the minimum heap size that it needs. When it starts up, the Garbage Collector will run often and, because the heap is small, it runs efficiently.

Whats the max heap allowed by AIX

                             Minimum heap size allowed

                             AIX:             16 MB

                             LINUX:          16MB

                             WINDOWS:    16 MB

                             Maximum heap size allowed

          AIX: Half the available memory with a minimum of 16 MB and a maximum of 512 MB

          LINUX:          Half the available memory with a minimum of 16 MB and a maximum of 512 MB

          WINDOWS: Half the real (physical or RAM) memory with a minimum of 16 MB and a maximum of 2 GB

What are the effects of large initial heap/max heap

                   The time between garbage collections will increase, but the garbage collection will take longer, which indirectly effects the turn-around-time of the applicaiton performance.

                                                (or)

                             Because it delays the start of garbage collection until the heap is full. The first time that the Garbage Collector runs, therefore, becomes a very expensive operation. Also, the heap is more likely to be fragmented and require a heap compaction. Again this is a very expensive operation.

Hung thread detection in WebSphere Application Server

A Java dump, also known as a Java core, Java thread dump, or a thread dump is a file that contains the information about thread like active thread, hung thread, dead lock, runnable and inactive thread info.
The thread dump is a snapshot of exactly what's executing at a moment in time.

We are finding thread hung in logs like this.

WSVR0605W: Thread threadname has been active for hangtime and may be hung.  There are totalthreads  threads in total in the server that may be hung.

Example:

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758692 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758693 milliseconds and may be hung.  There is/are 2 thread(s) in total in the server that may behung .

.

.

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 75869N milliseconds and may be hung.  There is/are n thread(s) in total in the server that may be hung.

Automated Thread dump generation.

Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom propertie>New-

com.ibm.websphere.threadmonitor.dump.java    when value is set to 'true', creates a java core when a hung thread is detected false

ManualThread dump generation

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

$AdminControl invoke $jvm dumpThreads.

Genarate through python Script

Download the file windows_hang.py and copy the file to your <PROFILE_ROOT>\bin directory. If instead copied to <WAS_HOME>\bin, the default server, which may be the deployment manger (dmgr), will be accessed when wsadmin.bat is launched.


NOTE: This script only works for WebSphere Application Server 6.1 and higher.

To launch the script to produce 3 javacores spaced 2 minutes apart, run this command:

wsadmin -lang jython -f windows_hang.py -j -s SERVER_NAME

Replacing SERVER_NAME with your server's name in AppServer profile root.

EX:

javacore.date.time.id.txt

For example: javacore.20120909.204717.27050.t

IN Linux.

use kill -3 PID on unix/linux machines.

Thread Monitoring settings

• Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom properties
• Add the following 4 custom properties

#	Property	Description	Default
A	com.ibm.websphere.threadmonitor.interval	How frequently thread monitor should check all the managed threads for hung threads	180
B	com.ibm.websphere.threadmonitor.threshold	After how many seconds a thread can be considered as hung	600
C	com.ibm.websphere.threadmonitor.false.alarm.threshold	The number of times that false alarms can occur before automatically increasing the threshold (T)	100
D	com.ibm.websphere.threadmonitor.dump.java	when set to 'true', creates a java core when a hung thread is detected	false

• Click OK and save the changes.
• Sync the changes and restart the servers for the changes to take effect. 

IBM IHS Webserver Installation

Follow the steps below to install and add IBM HTTP Server 6.1
to WebSphere Application Server
1) Run install.exe from installation directory. Welcome window will
appear click Next

2-a) Software License Agreement will appear select “I accept both the
IBM and the non-IBM terms”

2-b) Enter the location where you want to install IBM HTTP Server
and click next

3) Enter HTTP Port 80 and HTTP Administration Port 8008 and click
next

4-a) Select the check boxes “Run IBM HTTP Server as a Windows
Service” and “Run IBM HTTP Administration as a Windows
Service” and also select the radio button “Log on as a local
system account”

4-b) In the HTTP Administration Server Authentication window select
the checkbox and fill up User ID and password fields. Click Next

5) Select the check box to install HTTP Server plug-in and specify
the webserver name and host name of application server

6) Following window will show you installation summary before
installation begins Click next to install product.

7) After the Installation completes following window 
will appear and
shows successful installation message

IBM HTTP Server has been successfully installed and we shall now
follow the steps below to add the IBM HTTP Server to existing cell.


1) Launch the administrative console.
a) Enter User Id and Password and Login
2) Create a new unmanaged node for the Web server.
a) In the navigation tree, expand System administration and
click Nodes.
b) Click the Add Node button.
c) In the Add Node window select Unmanaged node and click
Next

3) In the Nodes window specify configuration information for the
node. Specify the name ihsnode and Host Name where you
creating Node. The Platform Type is Windows. Click OK.

4) Save the changes. The node ihsnode should be added to the list
of nodes
Node has been created now add the Web server to the configuration
In this step the Web server definition will be added to the ihsnode.

5) Add the Web server to the ihsnode configuration.
a. In the administrative console select Servers —> Web
servers.
b. Click New to add a Web server.
c. On Step 1 of creating a new Web server entry select the
node ihsnode from the drop-down menu. Specify the server
name of webserver00.
Click next

6) Specify the Web server template. Ensure that IHS is selected
and click Next

7) Specify the properties for the new Web server. Enter the
following information in the fields as provided
Field Name                                         Value
Port                                                          80
Installation location                                  <ihs_root>
Service name                                       IBMHTTPServer6.1
Plug-in installation location                          <plugin_root>
Application mapping All                              Port 8008
Username                                                         ihsadmin
Password                                                      was1edu
8) Click Next

9) On the confirmation window click finish and save the changes.

.

How to enable JMX in websphere

By default JMX remote port is not enabled in WebSphere, We have to manually enable the JMX remote port. Here I am giving some steps to enable the JMX remote port in Websphere.

This has been done with Websphere 7.0.
After installation of web sphere application server 7.0, fallow the fallowing steps to configure Remote JMX port.

STEP 1:
Login to Admin console of the web sphere any profile(server), short cut will be available in start menu programs.
deploy the PerfServletApp.ear application if not deployed already.
GO TO Applications IN LEFT PANE CLICK WebSphere Enterpise Applications TO CHECK PerfServletApp.ear IS DEPLOYED OR NOT.
IF NOT THEN CLICK New Application UNDER Applications. BROWSE FROM WebSphere directory -> AppServer -> InstallableApps.
FOLLOW THE STEPS.
STEP 2:
Enable the PMI Data and set all the statistics enabled.
GO TO Monitoring and Tuning IN LEFT PANE CLICK ON Performance Monitoring Infrastructure(PMI) IN CONIFGURATION TAB ENABLE THE PMI AND
SET THE ALL STATISTICS. ALSO SET THE ALL STATISTICS IN Runtime Tab. SAVE THE CHANGES.

STEP 3:

Set the generic jvm argument = -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote

in Severs -> Server Types -> WebSphere Application Servers
shows the servers list. click on the server you want.
In the right pane -> Server Infrastructure -> Java and Process Management  click on Process definition, again in Additional Properties of Configuration tab
click on Java Virtual Machine. put the -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote in Generic Jvm Argument field.
and save changes.

STEP 4:

To enable the JMX remote port open the below properties file and add the code below.

FILE : WebSphere directory \AppServer\java\jre\lib\management\management.properties

CODE :
com.sun.management.jmxremote.port=9001
com.sun.management.jmxremote.ssl=false
com.sun.management.jmxremote.authenticate=false

STEP 5:

SAVE THE MASTER DATA AND STOP THE SERVER AND START THE SERVER TO LOAD THE CHANGES……

Verify the JMX is enable or not…

If you have a default Websphere installation, the JNDI port number will likely be 2809, 2810, ... depending on how many servers there are installed on one system and the specific one you want to connect to. To be sure, when starting Websphere, check the logs, as it will dump a line like

0000000a RMIConnectorC A   ADMC0026I: The RMI Connector is available at port 2810

If you don't get this line, open the Websphere admin console and go to

Application servers > server1 > Administration Services > JMX connectors

to see if you need to add or change the config.

Second important bit of information is that the following JAR is always needed when doing JMX with the serverin Java dir:

com.ibm.ws.admin.client_6.1.0.jar

Configuring ports

When you configure WebSphere Application Server resources or assign port numbers to other applications, you must avoid conflicts with other assigned ports. In addition, you must explicitly enable access to particular port numbers when you configure a firewall.

Procedure

1.     Review the port number settings, especially when you are planning to coexist.

2.     Optional: Change the port number settings.

During installation, you can use the Installation wizard.

You can set port numbers when configuring the product after installation.

o    During profile creation using the manageprofiles command, you can accept the default port values or you can specify your port settings. If you want to specify ports, you can do so in any of the following ways:

§  Specify the use of a port file that contains the port values.

§  Specify the use of a starting port value.

§  Specify the use of the default port values.

o    During profile creation using the Profile Management tool, you can accept the port settings recommended by the tool or you can specify your port settings.

You can perform one of the following actions to change port settings after installation:

o    Use the updatePorts tool to change port settings.

o    Edit the profile_root/config/cells/cell_name/nodes/node_name/serverindex.xml file to change the port settings, or use scripting to change the values.