聚沙成塔-小哈的记事薄 » hang

应用程序死锁导致服务器挂起的介绍

hashei — Mon, 17 Aug 2009 08:09:00 +0000

原来好东西都躲到Metalink上去了

Problem Description

An inadvertent deadlock in the application code can cause a server to hang. For example, a situation in which thread1 is waiting for resource1 and is holding a lock on resource2, while thread2 needs resource2 and is holding the lock on resource1. Neither thread can progress.

Problem Troubleshooting

This Application Deadlock pattern should be used only after doing all the steps in the Generic Server Hang pattern. One indicator that this is an application deadlock problem is that thread dumps will show the threads are in the application methods. Several thread dumps taken a few seconds apart will show that the threads are not progressing. Troubleshooting this problem will involve reviewing the application code. There exists a thread analyzer tool at BEA dev2dev which has proven useful in analysis of the thread dumps.

Quick Links

Why does the problem occur?

Fundamentaly, this problem happens because the design and implementation of the application has introduced the possibility of deadlocks. These types of problems may only show up under heavy load. Therefore, these applications often pass through QA testing and become problems in production.

Coding problems to look for:

Unnecessary use of synchronized java classes, e.g., using Hashtable (synchronized) versus the use of HashMap(unsynchronized)
Application has a synchronized method that contains synchronized object method calls. See example below.
import java.util.Vector;   <– Vector is a synchronized java class
Public class Employee {
           Vector names = new Vector();
            Employees () {
                   Object object = new Object();
                    synchronized (object) {
                            names.add("al");
                            names.add("Saganich");
                     }
            }
            synchronized String getName (int index) {
                      String name = (String) names.elementAt(index);
                       return name;
            }
Using synchronization around long running complex code.
Threads are waiting on resources that will never become available.

Application Design

The application uses up all of the configured number of threads. This can happen when an executing thread reaches a point where it must wait for work done by another thread to complete. The timing may be that the waited for method which this thread is trying to enter is long running. Eventually, all the threads must reach this long running method. After running awhile, the application will find that the threads will be lined up waiting for this long running method. No new work can be introduced because the allocated number of threads is all used up. See example below.
import java…..;
import java…..;
public class myAppMethods {
          public String getName(String name) {
                     String lastname = getLastName(name);
                     return lastname;
          }
          public synchronized String getLastName (String name) {
                     do a DB Lookup    <———— takes mucho time to get a last name
                      return lastname;
         }

If the database if very slow, the server can appear to be hung because the threads will line up trying to get access to the database and all the available threads could eventually be used up.
The application running inside a WebLogic Server invokes a service on another WebLogic instance on a remote machine. The remote service invoked on the remote machine makes a call back to the first server. This sets up the opportunity for a deadlock on the first server especially under heavy load. The first server has an execution thread that is tied up waiting for an inbound response. This inbound response will require a thread from the same execute pool as the thread that is waiting to receive the response.
If the first server is faster than the remote server, eventually all the threads in the execute pool will be exhausted by the server making outbound requests with fewer threads available for processing inbound responses. As the load grows, the number of outgoing requests that cannot complete their work grows while they wait for an inbound response to complete. Below is an example of a thread in the waitForDataResponseImpl.java method of.

"ExecuteThread: ‘52′ for queue: ‘default’" daemon prio=5 tid=0×4b3e40b0 nid=0×1170 waiting on monitor [0x4c74f000..0x4c74fdbc]
at java.lang.Object.wait(Native Method)
at
weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:72)

Topic	Pattern Name	Link
RMI, RJVM responses – all threads tied up waiting for RJVM, RMI responses.	EJB_RMI Server Hang	EJB_RMI Server Hang
Application Deadlock – thread locks resource1 then waits for lock for resource2. Another thread locks resource2 and then waits for lock for resource1.	Application Deadlock Causes Server Hang	Application Dead Lock
Threads are all used up, none available for new work.	Thread Usage Server Hang	TBD
Garbage Collection taking too much time.	Garbage Collection Server Hang	TBD
JSP improper settings for servlet times, e.g. PageCheckSeconds.	JSP cause Server Hang	TBD
Long Running JDBC calls or JDBC deadlocks lead to a hang.	JDBC Causes Server Hang	JDBC Causes Server Hang
JVM hang during (code optimization), looks like server hang.	Server Hang in Code Optimization	TBD
JSP compilation causes server hang under heavy load.	JSP Compilation Server Hang	TBD
SUN JVM bugs, e.g. Light weight thread library.	Sun JVM Bugs that Cause Server Hangs	TBD

聚沙成塔-小哈的记事薄 » hang

应用程序死锁导致服务器挂起的介绍

原来好东西都躲到Metalink上去了

Problem Description

Problem Troubleshooting

Quick Links

Why does the problem occur?

Application Design

Known WebLogic Server Issues

External Resources

JDBC引发的服务器hang解决思路

JDBC Causes Server Hang

应用服务器发生hang的诊断方法

写在前面

Generic Hang