The Flow of J2EE Application

The typical application flow is as follows:

1. A Web client requests a URL in the browser (input page).

2. The request is routed to the Web server over the Internet.

3. The Web server immediately passes the request to the Web server plug-in.

All requests go to the Web server plug-in first.

4. The Web server plug-in examines the URL, verifies the list of host name

aliases from which it will accept traffic based on the virtual host information,

and chooses a server to handle the request.

5. A stream is created. A stream is a connection to the Web container. It is

possible to maintain a connection (stream) over a number of requests. The

Web container receives the request and, based on the URL, dispatches it to

the proper servlet.

6. If the servlet class is not loaded, the dynamic class loader loads the servlet

(servlet init(), then doGet() or doPost()).

7. JNDI is used for lookup of either datasources or EJBs required by the servlet.

8. Depending upon whether a datasource is specified or an EJB is requested,

the JNDI directs the servlet:

– To the corresponding database and gets a connection from its connection

pool in the case of a data source.

– To the corresponding EJB container, which then instantiates the EJB when

an EJB is requested.

9. If the EJB request involves an SQL transaction, it goes back to the JNDI to

look up the datasource.

10.The SQL statement is executed and the retrieved data is sent back either to

the servlet or to the EJB.

11.Data beans are created and handed off to JSPs in the case of EJBs.

12.The servlet sends data to JSPs.

13.The JSP generates the HTML that is sent back through the plug-in to the Web

server.

14.The Web server sends the output page (output HTML) to the browser.

Heap Dumps generation

Heap Dumps

Heap dump is a text file which keeps record of all objects in the java heap. It contains the information like size and address of all objects, as well as the addresses of all objects that it references.

Heap dump is nothing but a snap shot of our JVM’s Heap memory,contains all active java objects and their activities which object doing what and how much memory it is occupying.

whenever OutOfMemoryException occurs Websphere application Server creates heap dump in profile's home directory.

It is useful in situations, most notable for memory leak problems or memory allocation problems for exaple OutOfMemoryExceptions.

Automated heap dump generation

Click Servers > Application servers in the administrative console navigation tree.

Click server_name >Performance and Diagnostic Advisor Configuration.

Click the Runtime tab.

Select the Enable automatic heap dump collection check box.

Click OK.

Generating Heap dump Manually

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

For generating heap dump:

$AdminControl invoke $jvm generateHeapDump.

In order to manually generate a heapdump, you must first enable signal-based heapdumps by setting the IBM_HEAPDUMP=TRUE and IBM_NOSIGHANDLER=TRUE environmental variables.

What is the effect of changing max heap and min heap size of a jvm

                             Max Heap Size: When you have established the maximum heap size that you need,  you might want to set the minimum heap size to the same value;  Using the same values is not usually a good idea, because it delays the start of garbage collection until the heap is full. The first time that the Garbage Collector runs, therefore, becomes a very expensive operation. Also, the heap is more likely to be fragmented and require a heap compaction. Again this is a very expensive operation.

                             Min Heap Size: The recommendation is to start your application with the minimum heap size that it needs. When it starts up, the Garbage Collector will run often and, because the heap is small, it runs efficiently.

Whats the max heap allowed by AIX

                             Minimum heap size allowed

                             AIX:             16 MB

                             LINUX:          16MB

                             WINDOWS:    16 MB

                             Maximum heap size allowed

          AIX: Half the available memory with a minimum of 16 MB and a maximum of 512 MB

          LINUX:          Half the available memory with a minimum of 16 MB and a maximum of 512 MB

          WINDOWS: Half the real (physical or RAM) memory with a minimum of 16 MB and a maximum of 2 GB

What are the effects of large initial heap/max heap

                   The time between garbage collections will increase, but the garbage collection will take longer, which indirectly effects the turn-around-time of the applicaiton performance.

                                                (or)

                             Because it delays the start of garbage collection until the heap is full. The first time that the Garbage Collector runs, therefore, becomes a very expensive operation. Also, the heap is more likely to be fragmented and require a heap compaction. Again this is a very expensive operation.

Hung thread detection in WebSphere Application Server

A Java dump, also known as a Java core, Java thread dump, or a thread dump is a file that contains the information about thread like active thread, hung thread, dead lock, runnable and inactive thread info.
The thread dump is a snapshot of exactly what's executing at a moment in time.

We are finding thread hung in logs like this.

WSVR0605W: Thread threadname has been active for hangtime and may be hung.  There are totalthreads  threads in total in the server that may be hung.

Example:

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758692 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 758693 milliseconds and may be hung.  There is/are 2 thread(s) in total in the server that may behung .

.

.

WSVR0605W: Thread "WebContainer : 4" (0000003b) has been active for 75869N milliseconds and may be hung.  There is/are n thread(s) in total in the server that may be hung.

Automated Thread dump generation.

Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom propertie>New-

com.ibm.websphere.threadmonitor.dump.java    when value is set to 'true', creates a java core when a hung thread is detected false

ManualThread dump generation

set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]

Where server1 is the name of application server for which we want heapdump.

$AdminControl invoke $jvm dumpThreads.

Genarate through python Script

Download the file windows_hang.py and copy the file to your <PROFILE_ROOT>\bin directory. If instead copied to <WAS_HOME>\bin, the default server, which may be the deployment manger (dmgr), will be accessed when wsadmin.bat is launched.


NOTE: This script only works for WebSphere Application Server 6.1 and higher.

To launch the script to produce 3 javacores spaced 2 minutes apart, run this command:

wsadmin -lang jython -f windows_hang.py -j -s SERVER_NAME

Replacing SERVER_NAME with your server's name in AppServer profile root.

EX:

javacore.date.time.id.txt

For example: javacore.20120909.204717.27050.t

IN Linux.

use kill -3 PID on unix/linux machines.

Thread Monitoring settings

• Navigate to Servers --> Applicaiton Servers --> server_name --> administration --> custom properties
• Add the following 4 custom properties

#	Property	Description	Default
A	com.ibm.websphere.threadmonitor.interval	How frequently thread monitor should check all the managed threads for hung threads	180
B	com.ibm.websphere.threadmonitor.threshold	After how many seconds a thread can be considered as hung	600
C	com.ibm.websphere.threadmonitor.false.alarm.threshold	The number of times that false alarms can occur before automatically increasing the threshold (T)	100
D	com.ibm.websphere.threadmonitor.dump.java	when set to 'true', creates a java core when a hung thread is detected	false

• Click OK and save the changes.
• Sync the changes and restart the servers for the changes to take effect. 

IBM IHS Webserver Installation

Follow the steps below to install and add IBM HTTP Server 6.1
to WebSphere Application Server
1) Run install.exe from installation directory. Welcome window will
appear click Next

2-a) Software License Agreement will appear select “I accept both the
IBM and the non-IBM terms”

2-b) Enter the location where you want to install IBM HTTP Server
and click next

3) Enter HTTP Port 80 and HTTP Administration Port 8008 and click
next

4-a) Select the check boxes “Run IBM HTTP Server as a Windows
Service” and “Run IBM HTTP Administration as a Windows
Service” and also select the radio button “Log on as a local
system account”

4-b) In the HTTP Administration Server Authentication window select
the checkbox and fill up User ID and password fields. Click Next

5) Select the check box to install HTTP Server plug-in and specify
the webserver name and host name of application server

6) Following window will show you installation summary before
installation begins Click next to install product.

7) After the Installation completes following window 
will appear and
shows successful installation message

IBM HTTP Server has been successfully installed and we shall now
follow the steps below to add the IBM HTTP Server to existing cell.


1) Launch the administrative console.
a) Enter User Id and Password and Login
2) Create a new unmanaged node for the Web server.
a) In the navigation tree, expand System administration and
click Nodes.
b) Click the Add Node button.
c) In the Add Node window select Unmanaged node and click
Next

3) In the Nodes window specify configuration information for the
node. Specify the name ihsnode and Host Name where you
creating Node. The Platform Type is Windows. Click OK.

4) Save the changes. The node ihsnode should be added to the list
of nodes
Node has been created now add the Web server to the configuration
In this step the Web server definition will be added to the ihsnode.

5) Add the Web server to the ihsnode configuration.
a. In the administrative console select Servers —> Web
servers.
b. Click New to add a Web server.
c. On Step 1 of creating a new Web server entry select the
node ihsnode from the drop-down menu. Specify the server
name of webserver00.
Click next

6) Specify the Web server template. Ensure that IHS is selected
and click Next

7) Specify the properties for the new Web server. Enter the
following information in the fields as provided
Field Name                                         Value
Port                                                          80
Installation location                                  <ihs_root>
Service name                                       IBMHTTPServer6.1
Plug-in installation location                          <plugin_root>
Application mapping All                              Port 8008
Username                                                         ihsadmin
Password                                                      was1edu
8) Click Next

9) On the confirmation window click finish and save the changes.

.

How to enable JMX in websphere

By default JMX remote port is not enabled in WebSphere, We have to manually enable the JMX remote port. Here I am giving some steps to enable the JMX remote port in Websphere.

This has been done with Websphere 7.0.
After installation of web sphere application server 7.0, fallow the fallowing steps to configure Remote JMX port.

STEP 1:
Login to Admin console of the web sphere any profile(server), short cut will be available in start menu programs.
deploy the PerfServletApp.ear application if not deployed already.
GO TO Applications IN LEFT PANE CLICK WebSphere Enterpise Applications TO CHECK PerfServletApp.ear IS DEPLOYED OR NOT.
IF NOT THEN CLICK New Application UNDER Applications. BROWSE FROM WebSphere directory -> AppServer -> InstallableApps.
FOLLOW THE STEPS.
STEP 2:
Enable the PMI Data and set all the statistics enabled.
GO TO Monitoring and Tuning IN LEFT PANE CLICK ON Performance Monitoring Infrastructure(PMI) IN CONIFGURATION TAB ENABLE THE PMI AND
SET THE ALL STATISTICS. ALSO SET THE ALL STATISTICS IN Runtime Tab. SAVE THE CHANGES.

STEP 3:

Set the generic jvm argument = -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote

in Severs -> Server Types -> WebSphere Application Servers
shows the servers list. click on the server you want.
In the right pane -> Server Infrastructure -> Java and Process Management  click on Process definition, again in Additional Properties of Configuration tab
click on Java Virtual Machine. put the -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote in Generic Jvm Argument field.
and save changes.

STEP 4:

To enable the JMX remote port open the below properties file and add the code below.

FILE : WebSphere directory \AppServer\java\jre\lib\management\management.properties

CODE :
com.sun.management.jmxremote.port=9001
com.sun.management.jmxremote.ssl=false
com.sun.management.jmxremote.authenticate=false

STEP 5:

SAVE THE MASTER DATA AND STOP THE SERVER AND START THE SERVER TO LOAD THE CHANGES……

Verify the JMX is enable or not…

If you have a default Websphere installation, the JNDI port number will likely be 2809, 2810, ... depending on how many servers there are installed on one system and the specific one you want to connect to. To be sure, when starting Websphere, check the logs, as it will dump a line like

0000000a RMIConnectorC A   ADMC0026I: The RMI Connector is available at port 2810

If you don't get this line, open the Websphere admin console and go to

Application servers > server1 > Administration Services > JMX connectors

to see if you need to add or change the config.

Second important bit of information is that the following JAR is always needed when doing JMX with the serverin Java dir:

com.ibm.ws.admin.client_6.1.0.jar

Configuring ports

When you configure WebSphere Application Server resources or assign port numbers to other applications, you must avoid conflicts with other assigned ports. In addition, you must explicitly enable access to particular port numbers when you configure a firewall.

Procedure

1.     Review the port number settings, especially when you are planning to coexist.

2.     Optional: Change the port number settings.

During installation, you can use the Installation wizard.

You can set port numbers when configuring the product after installation.

o    During profile creation using the manageprofiles command, you can accept the default port values or you can specify your port settings. If you want to specify ports, you can do so in any of the following ways:

§  Specify the use of a port file that contains the port values.

§  Specify the use of a starting port value.

§  Specify the use of the default port values.

o    During profile creation using the Profile Management tool, you can accept the port settings recommended by the tool or you can specify your port settings.

You can perform one of the following actions to change port settings after installation:

o    Use the updatePorts tool to change port settings.

o    Edit the profile_root/config/cells/cell_name/nodes/node_name/serverindex.xml file to change the port settings, or use scripting to change the values.

HTTP 404 Troubleshooting.

Troubleshooting:

Verify that the Web server is responding

Verify that the application server is started

Verify that the application is started

Collect data

If the type of error is not apparent to you or you did not obtain the URL

information, collect the following diagnostic data:

Errors displayed by the user browser, including the URL that failed.

SystemOut log data for the application server

If your application is deployed to a cluster, you might need to collect the logs

from each active server in the cluster

Web server access and error logs

If you are running an IBM HTTP Server, see “IBM HTTP Server, HTTP Server

(powered by Apache) log files

_

Analyze data:

The two key determinations that you can make from the diagnostic data are:

The URL that failed.

Where the failure occurred (Web server, plug-in, application server)

Analyze the diagnostics in the following order:

The browser error display

1.     The page cannot be displayed

2.     JSP error or JSF error

3.     Failed to find resource

4.     File not found

5.     WebGroup/virtual host not defined

The errors displayed by the browser can indicate the failure type and URL that

failed. This might be enough information to resolve the problem.

.SystemOut  Errors

To analyze SystemOut, search for:

1.Web container messages: SRVExxxxE or SRVExxxxW messages

2.JSP messages: JSPGxxxxE or JSPGxxxxW messages

3.JSF messages: JSFGxxxxE or JSFGxxxxW messages

4. Error messages that are related to the application startup

If the failure occurs in the Web container, SystemOut has the most informative

error messages. These might be displayed by the browser also, depending on

the error handling of the application. If no error messages are seen in the

JVM logs, the request most likely did not reach the application server

Web server log

Search for 404 status codes in the access log

A log entry in the NCSA format looks like this:

Remote_host- - date_time "request" status_code bytes

For 404 errors:

1. Search for “404” in status_code.

2 Note the remote_host address, the date_time stamp, and the request URL.

Search for errors in the error log

An error entry in the error log resembles this example:

[date_time] [error] [client Remote_host] error_message

For errors in this entry:

. Search for “error.”

. Correlate the access log entry that you noted with the error log entry by

comparing the date_time and remote_host entries.

3. Note the error_message text

If no other error indications are found, look at the Web server log, which can

tell you the type of error (404 or 500) and the URL that caused the


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The Flow of Request happens from

Load Balancer >> Webserver >> Appserver >> DB

so the investigation for such issues should also follow the same routes .. ( well thats how i follow )

1) Try to check if the url is responding frm your end. ( this will eliminate if its specific to a user or its error for every one)

2) Check the Webserver if its running or not , If its not running start it

3) Check if the application server and application is running .. if not start it

4) Try to access the Aplication directly from the Appserver .. ie using the http transport port. ie wc_defalut .. ( this will identify if the error is due to App server or is with the Webserver/plugin)

5) check for the errors in the Appserver and web server logs

6) Check for the config in the plugin file to ensure that the url which is being hit is avaliable in the plugin-xml.cfg ( If it is not then it could be possible that the plugin was not regeneraed ad propagated after the deployment)

7) Enable the trace in the plugin-xml.cfg to understand in the plugin logs whether the plugin is forwarding the req to the appservers or not .

8) Lastly also check if the page which you are requesting is avaliable within the web modules

If you see one of the following messages, you
might have a problem with the virtual host
configuration: SRVE0255E: A WebGroup/Virtual Host to handle
url has not been defined. (V6.1) SRVE0017W: A WebGroup/Virtual Host to handle
url has not been defined. (V6.0) Error messages with prefixes JSPG, JSFG, or
SRVE have information about the error, including
the URL that caused the problem