聚沙成塔-小哈的记事薄

Even a transaction timeout will not kill or timeout any action that is done by the resources that are enlisted in this transaction. The actions will run as long as they take, without interruption. A transaction timeout will set a flag on the transaction that will mark it as rollback only, so that any subsequent request to commit this transaction will fail with a TimedOutException or RollbackException. However, as mentioned above, the long running JDBC calls can lead to blocked WebLogic Server execute threads, which can finally lead to a hanging instance, if all threads are blocked and no execute thread remains available for handling incoming requests.

More recent WebLogic Server versions have a health check functionality that regularly checks if a thread does not react for a certain period of time (the default is 600 seconds). If this happens, an error message is printed to your log file similar to following:

Top of Page

Problem Troubleshooting
Different programming techniques or JDBC connection pool configurations can lead to deadlocks or long running JDBC calls that lead to hanging WebLogic Server instances. General information about how to troubleshoot and analyze a hanging WebLogic Server instance is provided in Generic Server Hang Pattern.

This pattern addresses JDBC calls causing a server hang and other well known JDBC-related causes for common problems leading to hanging WebLogic Server instance.  Other Support Patterns referenced in this pattern are at the WebLogic Server Support Patterns Site.

Quick Links

• Why does the problem occur?
• Analysis of a hanging WebLogic Server instance
• Tips and Tricks to optimize your JDBC code and JDBC connection pool configuration

Why does the problem occur?
The following are some different possible reasons that can cause JDBC calls to lead to a hanging WebLogic Server instance:

• Use of DriverManager.getConnection() in your JDBC code.
• SQL Queries issued to the database take unexpectedly long time to return.
• Database for which the JDBC connection pool is configured hangs and does not return from calls in a timely manner.
• A slow or overloaded network causes database calls to slow down or hang.
  
• A deadlock causes all execute threads to hang and wait forever.
• RefreshMinutes or TestFrequencySeconds property in the JDBC connection pool causes hang periods in WebLogic Server.
• JDBC connection pool shrinking and re-creation of database connections causes long response times.

Top of Page

Synchronized DriverManager.getConnection()
Older JDBC application code sometimes uses DriverManager.getConnection() calls to retrieve a database connection using a certain driver. This technique is not recommended as it can cause deadlocks or at least relatively low performance for your connection requests. The reason behind this is, that all DriverManager calls are class-synchronized, meaning that one DriverManager call in one thread will block all other DriverManager calls in any other thread inside one WebLogic Server instance.

In addition to that, the constructor for a SQLException makes a DriverManager call, and most drivers have DriverManager.println() calls for logging, so any of these can block all other threads that issue a DriverManager call.

DriverManager.getConnection() can take a relatively long time until it returns with the physical connection created to the database. Even if no deadlock occurs, all other calls need to wait until that one thread gets its connection. This is not a best practice in a multi-threaded system like WebLogic Server.

If you prefer to use JDBC connections in your JDBC code, you should use a WebLogic Server JDBC connection pool, define a DataSource for it, and get the connection from the DataSource. This will give you all advantages from a pool (resource sharing, connection reuse, connection refresh if a database was down, etc). It also will help you avoid the deadlocks that may happen with DriverManager calls. See detailed information on how to use JDBC connection pools, DataSources, and other JDBC objects in WebLogic Server at: http://e-docs.bea.com/wls/docs81/jdbc/intro.html#1036718 and http://e-docs.bea.com/wls/docs81/jdbc/programming.html#1054307.

A typical thread blocked in a DriverManager.getConnection() call looks like:

A thread hanging in a long running SQL call will show a very similar stack in a thread dump as the one for a hanging database. Please compare the next section for details.

Hanging Database
Good database performance is key for the performance of an application that relies on this database. Consequently, a hanging database can block many or all available execute threads in a WebLogic Server instance and finally lead to a hanging server. To diagnose this, you should take 5 to 10 thread dumps from your hanging WebLogic Server instance and check your execute threads (in the default queue or your application thread queue) to see if they are currently in SQL calls and waiting for a result from the database. A typical stack trace for a thread that currently issues a sql query could look similar to following example:

Top of Page

Slow Network
Communication between WebLogic Server and the database relies on a well-performing and reliable network in order to serve the requests in a timely manner. Slow network performance can therefore lead to hanging or blocking execute threads waiting for results of SQL queries. The related stack traces will look similar to example above in Hanging Database section. It is not possible to find the root cause of the hanging or slow SQL queries by solely analyzing the WebLogic Server thread dumps. These give the first hint that something is wrong with the performance of the SQL calls. The next step is to check if there is a database or network problem that causes poorly performing SQL calls.

Deadlock
Both an application level deadlock as well as a deadlock on the database level can lead to hanging threads. You should check your thread dumps to see if there is an application level deadlock. Information on how to do this is provided in Server Hang – Application Deadlock Pattern. A database deadlock can be detected either in the database log or by the SQL Exception that can be found in the WebLogic Server log file. An example for a related SQL Exception is:

RefreshMinutes or TestFrequencySeconds
If you see recurring periods of low database performance, slow SQL calls, or connection peaks, the setting of the RefreshMinutes or TestFrequencySeconds configuration property in your JDBC connection pools could be the reason. This is described in detail in Investigating JDBC Problems Pattern. Unless you do not have a firewall between your WebLogic Server instance and your database, you should disable this functionality.

Pool Shrinking
Physical connections to a database are resources that should be opened once and kept open as long as possible, as a new connection request is a considerable resource overhead for the database, the operating system kernel, and the WebLogic Server. Consequently, pool shrinking should be disabled on production systems in order to keep this overhead at a minimum. If pool shrinking is enabled, idle pool connections will be closed and reopened once connection requests to the pool cannot be satisfied.

As these activities can take some time, the related application requests may take an unexpectedly long time which can lead users to assume that the system hangs. Information on how to optimize JDBC connection pool configurations is provided in Investigating JDBC Problems Pattern.

Top of Page

Analysis of a hanging WebLogic Server instance
General information on how to analyze a hanging WebLogic Server instance is provided in Generic Server Hang Pattern.

Most times it will be helpful to start with taking thread dumps from the hanging system in order to find out what is going on, e.g., what the different threads are doing and why they hang. Generally, thread dumps can be taken on production systems, however caution is necessary for very old versions of the JVM (<1.3.1_09), as they may crash during thread dumps. Also if the WebLogic Server instance has a huge number of threads, it will mean that the thread dump will take awhile to complete, while the rest of the threads are blocked.

Please take more than one thread dump (5 to 10) with a delay of some seconds in between. This gives you the possibility to check the progress of the different threads. Also it will show if the system actually hangs (no progress at all) or if the throughput is extremely slow, which can seem to be a hanging system.

Information on how to take thread dumps is provided in “Generic Server Hang” support pattern or in our documentation: http://e-docs.bea.com/wls/docs81/cluster/trouble.html.

Also please check if the complete WebLogic Server instance hangs or if it is the application that hangs. “Generic Server Hang” support pattern also includes this information.

Analyzing the thread dumps can show if one of the reasons mentioned in the previous section Why does the problem occur? actually is responsible for your hanging instance. If for example all your threads are in a DriverManager method like getConnection() then you have identified the root cause and need to change your application to use a DataSource or Driver.connect() instead of DriverManager.getConnection().

A very useful tool, Samurai, can be used to analyze thread dumps and to monitor the progress of threads between different thread dumps. This can be downloaded from dev2dev at:  http://dev2dev.bea.com/resourcelibrary/utilitiestools/adminmgmt.jsp.

A whitepaper on analyzing thread dumps on dev2dev: http://dev2dev.bea.com/products/wlplatform81/articles/thread_dumps.jsp will also be helpful in going deeper into the thread dumps to find out more about the server hang.

Top of Page

Tips and Tricks to optimize your JDBC code and JDBC connection pool configuration
There are some best practices both in the development of JDBC code and also in the configuration practice of JDBC connection pools that can help to avoid common problems and optimize resource usage so that hanging server instances should not happen.

JDBC Programming
In order to optimize resource usage in WebLogic Server and conserve database resources, you should use JDBC connection pools for your application’s JDBC calls. Connections created and destroyed in your application code generate an unnecessary overhead which should be avoided. For generic documentation on JDBC programming, see: http://e-docs.bea.com/wls/docs81/jdbc/rmidriver.html#1028977. Also details on JDBC performance tuning are at: http://e-docs.bea.com/wls/docs81/jdbc/performance.html#1027791.

You can view comprehensive information on JDBC that will help to optimize your JDBC code and the utilization of your JDBC resources on dev2dev Java Database Connectivity page at: http://dev2dev.bea.com/technologies/jdbc/index.jsp.

JDBC Connection Pool Configuration
The Investigating JDBC Problems Pattern has recommendations on how to configure a connection pool for production environments. In order to avoid hangs or bad performance, these configuration tips should be considered.

• WLS 8.1 Release Notes
• WLS 7.0 Release Notes
• WLS 6.1 Release Notes

Please note that changes have been made in WLS 8.1 SP3 to resolve CR134921, where for certain JDBC connections, the call to roll back a transaction was not being handled immediately because the driver had to wait for any currently-executing statement to return. 

Searching will also return Release Notes, as well as other Support Solutions and CR-related information as noted at Need Further Help?.  Contract customers who are logged in at http://support.bea.com/ will also see a Browse portlet for both Solutions and Bug Central where latest available CRs can be browsed by Product version.

1. Query AskBEA at http://support.bea.com/ using “jdbc server hang”, as an example, to discover other published solutions.  Contract Support Customers: Ensure you are logged to access available CR-related information.
2. Ask a more detailed question on one of BEA’s newsgroups at http://forums.bea.com
   

If this does not resolve your issue and you have a valid Support Contract, you can open a Support Case by logging in at: http://support.bea.com/ .

FEEDBACK

Please provide us input on whether or not this Support Diagnostic Pattern “JDBC Causes Server Hang” helped, any clarifications you needed, and any requests for new topics to Support Diagnostic Patterns.

DISCLAIMER NOTICE:

BEA Systems, Inc. provides the technical tips and patches on this Website for your use under the terms of BEA’s maintenance and support agreement with you. While you may use this information and code in connection with software you have licensed from BEA, BEA makes no warranty of any kind, express or implied, regarding the technical tips and patches.

Any trademarks referenced in this document are the property of their respective owners. Consult your product manuals for complete trademark information.

• The server does not respond to new requests.
• Requests time out.
• Requests take longer and longer to process (may be on the way to a hang).
• A server crash is not usually a symptom of a hung server but may follow.

• Why does the problem occur?
• Potential Causes of Server Hang
• Basic Steps
• Known WebLogic Server Issues
• Collecting Thread Dumps
• Analysis of a Thread Dump

Why does the problem occur?
A server can hang for a variety of reasons (refer to Potential Causes of Server Hang). Generally, a server hangs because of a lack of some resource. Lack of a resource prevents the server from servicing requests. For example, because of a problem (deadlock) or volume of requests there may be no execute threads available to do any work; all are busy or busy with previous requests.

Top of Page

When a server is hanging, first ping the server using java weblogic.Admin t3://server:port PING. If the server can respond to the ping, it may be that the application is hanging and not the server itself.

Ensure that the server is actually hanging and not doing garbage collection. To verify, restart the server with -verbosegc turned on, and redirect stdout and stderr to one file. When the server stops responding, it can be determined if it’s doing garbage collection or it is really hanging.  If the garbage collection is taking too long (>10 seconds), the server may miss the heartbeats that servers use to keep each other informed of the topoplogy of the cluster.

WebLogic Server uses the ‘default’ thread queue or a configured application specific thread queue to service client requests. Client requests will only be handled in the default queue if no application specific thread queue is defined.  Please see Tuning WebLogic Server Applications, Tuning the Default Execute Queue Threads, and Tuning WebLogic Server Performance Parameters for more information on defining application specific thread queues.

In release 8.1, a change was made to the thread architecture in WebLogic Server.  A specific kernel thread group for internal WebLogic tasks was created.  This was found to be necessary to avoid deadlocks that occurred in earlier releases when all threads in the ‘default’ thread queue were used and none were thus available for WebLogic internal tasks.

The threads in the ‘default’ queue or the application specific thread queue (if one has been configured) are the threads that should be examined in the event of a server hang. Here’s an example of what one of these threads looks like in a thread dump. Execute Thread ‘14′ from the ‘default’ queue looks like in a thread dump when the thread is waiting for work. The latest method called by this thread is Object.wait(). This thread is in a state “waiting on monitor”.

Below is an example of  threads that  may  be hanging.  ExecuteThread ‘9′ is waiting to lock some object <dde51520>.   Notice the “waiting to lock <dde51520>” line in the stack trace for this thread.  ExecuteThread ‘6′ is also “waiting to lock the same object <dde51520>”.  The third thread, ExecuteThread ‘5′ has locked this object <dde51520>and is doing work.  This  example demonstrates why one thread dump is not enough.  If the server is hanging, and it is suspected that the cause is the locked object <dde51520>, then subsequent thread dumps will show whether or not that object was released and a new thread has locked object <dde51520>.  If after several thread dumps,  you do not see that the threads have progressed, that object <dde51520> has not been released, you may suspect that there is a problem with the routine(s) in the ExecuteThread ‘5′ call stack because the lock is not being released.

If the Execute Queue has idle threads, it is possible that not enough socket reader threads are allocated. By default, a WebLogic Server instance creates three socket reader threads upon booting. If a cluster system utilizes more than three sockets during peak periods, increase the number of socket reader threads.

The number of socket reader threads should usually be small. However, configure one thread for each Weblogic Server that acts as a client of the server instance that is hanging.

If using a JDBC connection pool, ensure that the JDBC connections have been configured to be equivalent to the number of simultaneous requests, i.e., execute threads, for the pool.

Top of Page

The possibility exists that a problem with JDBC could produce deadlock. Check the version and service pack level of the server found in the beginning of the weblogic.log. Then check above the version and service pack lines for any temporary patches that have already been applied to the server classpath. The patches will tell what problems have already been addressed.

Top of Page

The way to take a thread dump is dependent on the operating system where the hung server instance is installed. Information about taking a thread dump on various operating systems can be found at http://e-docs.bea.com/wls/docs81/cluster/trouble.html#gc. Redirection of both standard error and standard out places the thread dump information in the proper context with server information and other messages and provides more useful logs.

Unix Systems (Solaris, HP, AIX)
Use kill –3 <weblogic process id> to create the necessary thread dumps to diagnose a problem. Ensure this is done several times on each server, spaced about 5 to 10 seconds apart, to help diagnose deadlocks. For this to work, nohup the process when starting the server (refer to Solutions S-12292 and S-15924).

Windows, XP, NT
Each server requires <Ctrl>-<Break> to create the necessary thread dumps to diagnose a problem. Ensure this is done several times on each server, spaced about 5 to 10 seconds apart, to help diagnose deadlocks. On NT, in the command shell type CTRL-Break.

If you have installed WebLogic as a Windows service, you will not be able to see the messages from the JVM or WebLogic Server that are printed to standard out or standard error.  To view these messages, you must direct standard out and standard error to a file.  To do this, take the following steps:

1. Create a backup copy of the WL_HOME\server\bin\installSvc.cmd master script.
2. In a text editor, open the WL_HOME\server\bin\installSvc.cmd master script.
3. In installSvc.cmd, the last command in the script invokes the beasvc utility.
4. At the end of the beasvc command, append the command -log:”pathname”
   where pathname is a fully qualified path and filename of the file that you want to store the server’s standard out and standard error messages.
5. The modified beasvc command will resemble the following command:
   “%WL_HOME%\server\bin\beasvc” -install
   -svcname:”%DOMAIN_NAME%_%SERVER_NAME%”
   -javahome:”%JAVA_HOME%” -execdir:”%USERDOMAIN_HOME%”
   -extrapath:”%WL_HOME%\server\bin” -password:”%WLS_PW%”
   -cmdline:%CMDLINE%
   -log:”d:\bea\user_projects\domains\myWLSdomain\myWLSserver-stdout.txt”
6. If you started WebLogic with nohup, the log messages will show up in nohup.out.

Linux
The Linux operating system views threads differently than other operating systems. Each thread is seen by the operating system as a process. To take a thread dump on Linux, find the process id from which all the other processes were started. Use the commands:

• To obtain the root PID, use:
  

  ps -efHl | grep ‘java’ **. **

Use a grep argument that is a string that will be found in the process stack that matches the server startup command. The first PID reported will be the root process, assuming that the ps command has not been piped to another routine.

• Use the weblogic.Admin command THREAD_DUMP

Another method of getting a thread dump is to use the THREAD_DUMP admin command. This method is independent of the OS on which the server instance is running.

java weblogic.Admin -url ManagedHost:8001 -username weblogic -password weblogic THREAD_DUMP

NOTE: This command cannot be used if unable to ping the server instance.

If the JVM in use is Sun’s, the thread dump goes to stdout. Sun has enhanced the thread dump format between JVM 1.3.1 and 1.4. To obtain Sun’s 1.4 style of thread dump add the following option to the java command line for starting the 1.3.1 JVM:

-XX:+JavaMonitorsInStackTrace

Top of Page

The most useful tool in analyzing a server hang is a set of thread dumps. A thread dump provides information on what each of the threads is doing at a particular moment in time. A set of thread dumps (usually 3 or more taken 5 to 10 seconds apart) can help analyze the change or lack of change in each thread’s state from one thread dump to another. A hung server thread dump would typically show little change in thread states from the first to the last dump.

Threads can be in one of the following states:

More information on thread states can be found at http://java.sun.com/developer/onlineTraining/Programming/JDCBook/stack.html#states.

There is also a thread analysis tool at http://dev2dev.bea.com/resourcelibrary/utilitiestools/adminmgmt.jsp.
Download the tool and read the instructions at the link.

What to Look at in the Thread Dump
All requests enter the WebLogic Server through the ListenThread. If the ListenThread is gone, no work can be received and therefore no work can be done. Verify that a ListenThread exists in the thread dump. The ListenThread should be in the socketAccept method. The following example shows what the Listen Thread looks like:

“ExecuteThread: ‘1′ for queue: ‘weblogic.socket.Muxer’” daemon prio=10 tid=0×000 35fc8 nid=74 lwp_id=6888067 runnable [0x1b1b0000..0x1b1b0530] at weblogic.socket.PosixSocketMuxer.poll(Native Method)
at
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:99)
– locked <0×25c01198> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)

“ExecuteThread: ‘0′ for queue: ‘weblogic.socket.Muxer’” daemon prio=10 tid=0×000 35e68 nid=73 lwp_id=6888066 waiting for monitor entry [0x1b231000..0x1b231530]
at
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:92)
- waiting to lock <0×25c01198> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)

Allocating execute threads to act as socket reader threads increases the speed and the ability of the server to accept client requests. It is essential to balance the number of execute threads that are devoted to reading messages from a socket and those threads that perform the actual execution of tasks in the server.

In release 8.1, the socket reader threads no longer use “ExecuteThreads” in the default queue.  Instead they have their own thread group named.

Next Steps
The next steps require a further analysis of the thread dump. Look in the thread dump to see what each the threads are doing at the time of the hang. This will help to analyze the next stage of the investigation. For example, if there are many threads involved in JSP compilation, refer to Potential Causes of Server Hang for further diagnosis and actions to test.

Top of Page