聚沙成塔-小哈的记事薄

使用32位JDK时，JVM一般设置最大设置为1.7G，而现在服务器普遍内存都很大，当然可以通过多个server建立垂直集群来更好的利用资源，但不妨使用64位JDK。虽然WebLogic可以直接在setDomainEnv里指定JAVA_HOME来更改JDK，但肯定会遇到BEA-000438的错，原因在于缺少对应64位JDK的native io libaray（位于weblogic/server/native）。一种方式是从别处拷贝一份过来，还有一种是下载wls_generic.jar形式的安装文件，而不是已经带有JDK的。然后下载64位JDK安装（Jrockit下载），用java –jar wls_generic.jar来安装就可以了。

————————————————————————————

附一个错误分析，和native libaray相关，但并不是由于64位的关系，而是没有执行权限。

启动过程中发现

<Apr 28, 2010 6:27:15 PM GMT+08:00> <Error> <Socket> <BEA-000438> <Unable to loa
d performance pack. Using Java I/O instead. Please ensure that a native performa
nce library is in: ‘/opt/java1.5/jre/lib/IA64N:/opt/java1.5/jre/lib/IA64N/server
:/opt/java1.5/jre/../lib/IA64N::/opt/weblogic/bea/weblogic90/server/native/hpux1
1/IPF64:/opt/weblogic/bea/weblogic90/server/native/hpux11/PA_RISC:/opt/weblogic/
bea/weblogic90/server/native/hpux11/PA_RISC/oci920_8:/usr/lib’

没有启动native io，导致系统性能低下（这里要注意HP-UX里IA64N下的是32位JDK，IA64W下的才是64位JDK），而且java io配置的值较小，产生如下报错

<Apr 28, 2010 6:15:03 PM GMT+08:00> <Warning> <Socket> <BEA-000402> <There are:
5 active sockets, but the maximum number of socket reader threads allowed by the
configuration is: 4. You may want to alter your configuration.>

在应用使用过程中从而出现

<Apr 28, 2010 6:14:10 PM GMT+08:00> <Error> <Console> <BEA-240003> <Console enco
untered the following error javax.servlet.jsp.JspException: Broken pipe (errno:3
2)
        at com.bea.console.taglib.html.tree.TreeTag.print(TreeTag.java:231)
        at com.bea.console.taglib.html.tree.TreeTag.doEndTag(TreeTag.java:192)

观察控制台的thread信息

Self-Tuning Thread Pool    
Active Execute Threads    Execute Thread Total Count    Execute Thread Idle Count    Queue Length    Pending User Request Count    Completed Request Count    Hogging Thread Count    Standby Thread Count    Throughput    Health
16    58    15    6048    0    144840    4    38    4.577865205875421    OK

排队的请求数多达6000个，导致了OutOfMemory，在JAVA堆还很空的情况下

观察发现/opt/weblogic/bea/weblogic90/server/native/hpux11/IPF32下面和native io相关的libmuxer.so没有执行权限，chmod +x 后再次启动错误信息不再出现

本文原文地址为http://support.bea.com/support_news/product_troubleshooting/Investigating_Out_of_Memory_Memory_Leak_Pattern.html

但现在这个网址已经无法访问，我在Metalink找到这篇文章并与大家分享。文章发表的较早，但是OOM发生的原理和解决的方法不变。文中提到的/3GB参数和垃圾回收日志分析的方法我在《JAVA性能优化—IBM JDK JVM参数设置》和《JAVA性能优化-GC日志分析》都提到过。如果英文的看起来比较麻烦，我google到台湾人做的一份翻译JavaAPsvr_A_200411_MemoryLeak，不过我看着“记忆体”觉得更别扭。

Problem Description

Out Of Memory (OOM) – An application displays Out of Memory errors due to memory exhaustion, either in java heap or native memory.

Memory Leak – Constant memory growth in either java heap or native memory, which will eventually end up in out of memory situation. The techniques to debug the memory leak situations are the same as the out of memory situations.

Problem Troubleshooting

Please note that not all of the following items would need to be done. Some issues can be solved by only following a few of the items.

Quick Links:

• Java heap, Native memory, Process size
• Difference between process address space and physical memory
• Why does the OOM problem occur and What does the JVM do in this situation?
• Steps to debug the problem
  • Determine if Java OOM or Native OOM
  • Java OOM
  • Native OOM
• HP JVM specific tools/tips>
• Jrockit specific features
• Bibliography

Java heap This is the memory that the JVM uses to allocate java objects. The maximum value of java heap memory is specified using �Xmx flag in the java command line. If the maximum heap size is not specified, then the limit is decided by the JVM considering factors like the amount of physical memory in the machine and the amount of free memory available at that moment. It is always recommended to specify the max java heap value.

Native memory This is the memory that the JVM uses for its own internal operations. The amount of native memory heap that will be used by the JVM depends on the amount of code generated, threads created, memory used during GC for keeping java object information and temporary space used during code generation, optimization etc.

If there is a third party native module, it could also use the native memory. For example, native JDBC drivers allocate native memory.

The max amount of native memory is limited by the virtual process size limitation on any given OS and the amount of memory already committed for the java heap with -Xmx flag. For example, if the application can allocate a total of 3 GB and if the max java heap is 1 GB, then the max possible native memory is approximately 2 GB.

Process size Process size will be the sum of the java heap, native memory and the memory occupied by the loaded executables and libraries. On 32 bit operating systems, the virtual address space of a process can go up to 4 GB. Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 ~ 2 GB). The remaining is available for the application.

Windows ：By default, 2 GB is available for the application and 2 GB is reserved for Kernel’s use. However, on some variants of Windows, there is a /3GB switch which can be used to change this ratio such that the application gets 3 GB. More details on the /3GB switch can be found at:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/4gt_ram_tuning.asp

RH Linux AS 2.1 ： 3 GB is available for the application.

For other operating systems, please refer to the OS documentation for your configuration.

Top of Page

阅读全文…

weblogic的集群很早以前做过，当时写了一份文档，最近真好又有机会实验了Weblogic 10.3的集群配置，两相比较，同时对照着文档过了一遍，确定下文对weblogic9和10都适用。截了图也方便新手学习。

环境

平台：两台Windows 2003 SP2（SP版本要一样），必须位于同一网段，并且必须是IP广播(UDP)可到达的

软件：Weblogic9.1

拓扑

在A机上建立一个Administrator Server，作为管理节点；在AB机上分别建立Managed Server，加入集群appCluster，作为应用程序的运行环境；在B机上分别建立Proxy Server，作为提供外部访问的服务地址。

安装过程

在机器A上

创建新的Domain

选择“Tool- Configuration Wizard”，单击“下一步”按钮

选择安装域源

选择安装“WebLogic Server”，单击“下一步”按钮

配置管理员用户名和密码

输入Weblogic管理员管理服务器（AdminServer）的用户名和密码，在此以“weblogic”作用登录管理服务器的用户名和密码，单击“下一步”按钮

阅读全文…

上周在AIX6.1下安装weblogic10.3.0，并配置了hacmp集群环境，但是接下来的几天遇到了挂起问题，为此还加班了一天。

现象描述：

Weblogic启动后，10到30分钟就会hang住，应用和管理控制台都无法访问。强制kill -9 pid后端口无法释放，使用rmsock 命令查看端口显示Wait for exiting processes to be cleaned up before removing the socket。

分析及处理过程

1. 用ps –ef | grep java找到weblogic进程，每隔三分种执行kill -3 pid，在domain目录下生成javacore文件

2. 分析weblogic日志，发现如下内容

<Aug 21, 2009 4:33:37 AM CDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘1′ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “620″ seconds working on the request

“weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@20de20de”, which is more than the configured time (StuckThreadMaxTime) of “600″ seconds. Stack trace:

java.net.SocketOutputStream.socketWrite0(Native Method)

java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:103)

……

<Aug 21, 2009 4:34:37 AM CDT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘1′ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “680″ seconds working on the request

“weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@20de20de”, which is more than the configured time (StuckThreadMaxTime) of “600″ seconds. Stack trace:

java.net.SocketOutputStream.socketWrite0(Native Method)

java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:103)

……

3. 用IBM Thread and Monitor Dump Analyzer for java分析刚才生成的thread dump，找到如下两个线程信息：

3XMTHREADINFO “[ACTIVE] ExecuteThread: ‘5′ for queue: ‘weblogic.kernel.Default (self-tuning)’” TID:0×39CBED00, j9thread_t:0×3751C83C, state:R, prio=5

3XMTHREADINFO1 (native thread ID:0xCE1DB, native priority:0×5, native policy:UNKNOWN)

4XESTACKTRACE at java/net/PlainSocketImpl.socketClose0(Native Method)

4XESTACKTRACE at java/net/PlainSocketImpl.socketPreClose(PlainSocketImpl.java:706)

4XESTACKTRACE at java/net/PlainSocketImpl.close(PlainSocketImpl.java:540)

4XESTACKTRACE at java/net/SocksSocketImpl.close(SocksSocketImpl.java:1041)

4XESTACKTRACE at java/net/Socket.close(Socket.java:1343)

4XESTACKTRACE at weblogic/socket/SocketMuxer.closeSocket(SocketMuxer.java:475)

4XESTACKTRACE at weblogic/socket/SocketMuxer.cancelIo(SocketMuxer.java:813)

4XESTACKTRACE at weblogic/socket/SocketMuxer$TimerListenerImpl.timerExpired(SocketMuxer.java:1021(Compiled Code))

4XESTACKTRACE at weblogic/timers/internal/TimerImpl.run(TimerImpl.java:273(Compiled Code))

4XESTACKTRACE at weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516(Compiled Code))

4XESTACKTRACE at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201(Compiled Code))

4XESTACKTRACE at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)

3XMTHREADINFO “ExecuteThread: ‘7′ for queue: ‘weblogic.socket.Muxer’” TID:0×35381D00, j9thread_t:0×35385864, state:R, prio=5

3XMTHREADINFO1 (native thread ID:0xB916F, native priority:0×5, native policy:UNKNOWN)

4XESTACKTRACE at weblogic/socket/PosixSocketMuxer.poll(Native Method)

4XESTACKTRACE at weblogic/socket/PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102(Compiled Code))

4XESTACKTRACE at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)

4XESTACKTRACE at weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)

4XESTACKTRACE at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)

4XESTACKTRACE at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)

4. 执行线程只有这两个是running状态，一个做CLOSE()，一个做POLL()。别的都是blocked或者wait状态。

5. 经过metalink查询以及和800支持人员确认，这是Weblogic在AIX的JVM上由来已久的bug，从8.1.4就开始在不同版本间出现。原因是IBM的JVM底层socket实现和weblogic配合问题，需要打patch CR370915_1030GA.jar解决。

操作过程

1．在weblogic的启动脚本中，找到CLASSPATH一行

2．在CLASSPATH变量的第一位添加补丁jar包
Eg: CLASSPATH=”${CLASSPATH}${CLASSPATHSEP}${MEDREC_WEBLOGIC_CLASSPATH}”
—>
CLASSPATH=/路径/CR370915_1030GA.jar:”${CLASSPATH}${CLASSPATHSEP}${MEDREC_WEBLOGIC_CLASSPATH}”

3．以上操作仅对这个domain起作用，为了对所有domain起作用，可以添加到common/bin/的目录中的commEnv.sh文件中WEBLOGIC_CLASSPATH=最前面

总结

这个bug在weblgoic和IBM的JVM相组合的平台上出现较为普遍，如果出现相关日志信息，基本可以断定需要打CR370915补丁。

更新：我这里的补丁仅仅 for weblogic 10.3.0.0，其它版本的可以自行用Smart Update下载

Patches for WLS 8.x can be found in My Oracle Support. Open the Patches & Updates tab. Search for patch ID 8173442 for the patches for WLS 8.1mp3, 8.1mp4, and 8.1mp5. Search for patch ID 8179792 for the patch for WLS 8.1mp6.

Patches for WLS 9.x and higher can be downloaded from Smart Update using these patch IDs and passcodes:

——————————————
PATCH REPOSITORY INFORMATION
——————————————
WLS Version | Patch ID |  Passcode
————+———-+—————-
9.2      |  T4DV    |  7C7PYV9B
9.2mp1   |  HZHQ    |  PTUYCCSI
9.2mp2   |  WJD2    |  GU1CW2AB
9.2mp3   |  GNLT    |  8J9L6Q4Y
10.0     |  PMAJ    |  9UQ69LLT
10.0mp1  |  ITVL    |  K8RBHQQ2
10.3     |  9YT5    |  I1DB5QSV

如果生产机无法联网，可以

1. You can apply the patch to the offline system manually by extracting the actual patch and adding it to the classpath on the offline system:Extract the actual patch jar file. If you downloaded the patch using SmartUpdate, it will be in the form <patch_id>.jar (for example: E5W8.jar). Inside this jar file is the actual patch jar file, which will be of the form CR326566_92mp3.jar. Extract the latter file for the following steps.
2. Add the extracted jar file as the first element of the classpath of the Admin server as well as the managed servers in the domain.
3. If you are starting servers using the WebLogic startup script, update the classpath in the startup script like this:set CLASSPATH=<PATCH_DIR>\jars\CR326566_92mp3.jar;%CLASSPATH% (Windows)CLASSPATH=<PATCH_DIR>/jars/CR326566_92mp3.jar:$CLASSPATH (UNIX)where PATCH_DIR is the directory on your local machine where you extracted/saved the patch file.
4. Similarly, if you are starting servers using Node Manager, add the patch jar to the beginning of the Class Path argument in the Server Start tab for the server(s).

我一般用第二种，对于单个补丁快捷方便，SmartUpdate可以单独安装，但是会让你选择应用到哪个BEA的主目录，不同的版本和平台能下的补丁不一样。在Windows平台上当然没有AIX的BEA版本，不过只要自己建个目录，然后拷贝一份register.xml进去就可以了。

原来好东西都躲到Metalink上去了

Problem Description

An inadvertent deadlock in the application code can cause a server to hang. For example, a situation in which thread1 is waiting for resource1 and is holding a lock on resource2, while thread2 needs resource2 and is holding the lock on resource1. Neither thread can progress.

Problem Troubleshooting

This Application Deadlock pattern should be used only after doing all the steps in the Generic Server Hang pattern. One indicator that this is an application deadlock problem is that thread dumps will show the threads are in the application methods. Several thread dumps taken a few seconds apart will show that the threads are not progressing. Troubleshooting this problem will involve reviewing the application code. There exists a thread analyzer tool at BEA dev2dev which has proven useful in analysis of the thread dumps.

Quick Links

• Why does the problem occur?
• Known WebLogic Server Issues
• External Resources

阅读全文…