聚沙成塔-小哈的记事薄 » weblogic http://www.hashei.me 一个系统工程师的絮叨 Tue, 10 Jan 2012 18:03:42 +0000 http://wordpress.org/?v=2.9 en hourly 1 如何在一台服务器上部署多个相同url的应用？ http://www.hashei.me/2009/12/use_same_context_root_for_different_application_on_one_server.html?utm_source=rss&utm_medium=rss&utm_campaign=use_same_context_root_for_different_application_on_one_server http://www.hashei.me/2009/12/use_same_context_root_for_different_application_on_one_server.html#comments Tue, 15 Dec 2009 14:16:00 +0000 hashei http://www.hashei.me/2009/12/use_same_context_root_for_different_application_on_one_server.html 现在的服务器性能越来越强，内存动辄4G、8G，而32位JDK下，单个java堆一般不超过1.7G。那多出来的内存怎么办？建垂直集群（Vertical clusters：垂直集群是指同一机器上部署多个服务器，充分利用硬件资源）可以，简单的部署多个appserver也行。但是应用如何处理？如果多个项目都需要用相同的context root，比如说/，那怎么办？

其实做过网站的朋友都知道，一台服务器上一般不可能只放一个网站，那如何访问到这些网站，有三种方法：

  1. IP不同，一个网卡上绑定多个IP，不同的IP对应不同的网站。
  2. 端口不同，不同的网站运行在不同的端口上。
  3. 虚拟主机不同，通过不同的虚拟主机名（一般都是域名）来访问。

那对于webshpere或者weblogic，很明显可以实行的是上述第三种方案。部署应用时会让我们选择部署到哪台虚拟主机，默认的都为default host，所以我们只要新建一个虚拟主机，名称一般为域名。那么就能实现通过一级域名直接访问应用。即使一台app server上部署了多个应用。

这种设置对于提供互联网服务的应用尤其有效，因为大家本来就是通过域名来访问的。而且即使只有一个应用，设置非默认的虚拟主机名可以起到只能通过域名来访问，而IP地址访问则会得到应用服务找到的错。虽说提高的安全性不多，但能满足年底全市各部门关于“只能通过域名访问，而IP无法访问”的要求。

至于内网用户，可以在内部dns上配置域名解析，或者编写hosts文件，达到同样效果。

更新：对于webshpere，我测试下来即使不同的虚拟主机，相同的上下文根也无法同时启动。在日志里会看到已经有重复的Context root了，这样的话只能配置apache使用它的虚拟主机解决

Copyright © 2008 This feed is for personal, non-commercial use only
聚沙成塔-小哈的记事薄 by hashei 如果喜欢，欢迎订阅feed.hashei.com
Digital Fingerprint: 10f920a9f2bae51c3c73c4f5fb50a949]]> http://www.hashei.me/2009/12/use_same_context_root_for_different_application_on_one_server.html/feed 2 官方JRockit JVM调优文档 http://www.hashei.me/2009/09/weblogic_profiling_and_performance_tuning.html?utm_source=rss&utm_medium=rss&utm_campaign=weblogic_profiling_and_performance_tuning http://www.hashei.me/2009/09/weblogic_profiling_and_performance_tuning.html#comments Mon, 07 Sep 2009 00:38:51 +0000 hashei http://www.hashei.me/2009/09/weblogic_profiling_and_performance_tuning.html 转自BEA，原文链接已经无法访问，文中的许多链接也更改过地址，我把能找到的都重新做了连接。以前一直以为Jrockit和Sun的JVM配置差不多，看了这篇文章和最后参考资料中的信息，发现区别不是一点点，很多常用参数的使用都不一样。Jrockit的自动化设置应该说做的不错，在WebLogic上我很少更改它的默认配置（除了堆最大最小值），不过不影响这篇文章存在的价值。

摘要

本文的目的是以清单的方式提供BEA JRockit JVM的调优信息。从深奥的命令行选项到迭代性能测试，本文涵盖了许多方面。大部分数据都是我与用户合作过程中收集的。您要是也有什么技巧的话，请告诉我，在本文的下一版中，我会尝试将它们添加进去。

具体的产品版本信息都已在适当的地方列出；但是，本文所提供的通用指南适用于JRockit的大多数版本。每个版本的JRockit都增加了新的设置和优化，所以请查看 发行说明 和 JRockit产品中心。

验证当前的JRockit环境

首先需要确定您的运行时应用程序服务器所使用的JRockit的版本。为此，可以查看相应应用程序服务器的日志文件。也可以使用适当的脚本设置系统环境，然后执行java –version命令来确定JRockit的版本。

接着，收集当前JVM标志，开发和/或生产阶段需要用到它们：

-server -Xms1024m -Xmx1536m -Xverboselog:gc.log -Xverbose:memory-Xgcprio:throughput

这将告诉您当前JRockit实例的配置情况。

确定应用程序的目标

确定应用程序的目标是什么。是“响应快”还是“性能高”？根据目标的不同，需要设置不同的垃圾收集算法。

例如，如果应用程序的目标是实现高性能，则确保设置了Dynamic Garbage Collector "-Xgcprio:throughput" 选项。如果目标是响应时间短，那么需要将-Xgcprio:pausetime -Xpausetarget=XXX’中的pausetarget设置为最佳值。有关更多细节，请查看JRockit 调优文档。

收集故障诊断数据

如果JVM性能有问题，那么最好是先收集一些分析数据。该工作可以由团队中有相关经验的人员来完成，您也可以将这些信息发送给BEA Support做进一步分析。

首先，出现问题时需要收集大约10分钟的运行时JRockit Recording（JRA)数据。可以使用jrcmd.sh实用工具或JRockit Mission Control（JRMC）完成此操作。请阅读“性能测试期间的JRCMD/JRA”和“JRockit Mission Control”两节的内容。有关详细信息，请参阅 JRockit Mission Control文档。Latency Analysis一节提供许多有价值的内容，我们可以从中了解任何潜在的延迟问题（在JRockit中需要一个许可证就可以使用它）。

然后，需要收集问题发生时的一些详细日志。方法是在启动服务器实例的时候在JVM命令行输入以下参数：

-Xverboselog:perTestGC.log-Xverbose:opt,memory,gcpause,memdbg,compaction,gc,license-Xverbosetimestamp -Xgcreport

这样会将有价值的分析数据收集到刚才配置的perTestGC.log文件中。团队成员和/或BEA Support可以对这些数据进行分析。

最后一点：通常，应用程序不会请求执行垃圾收集（也就是在应用程序代码中调用System.gc()）。但如果您怀疑它有问题，那么可以在启动服务器实例的时候，在Java命令行使用-XXnoSystemGC参数来禁用它。

现在，我将介绍如何通过迭代性能测试方法解决这些问题。

迭代性能测试方案及其方法

完成初始数据的收集和分析后，我们可以通过迭代方法来调优JVM。此处介绍的测试方案是在JRockit JVM层执行迭代调优的通用方法，可以找到哪些设置可能有益于特定应用程序。假定您有测量性能结果的方法；然后，可以将它们与“基准”（您应该已经有了） 进行比较。

测试1：线程本地区域大小和大对象大小

在本测试中，我们将查看线程本地区域大小。这很重要，因为如果这些标志的默认设置对于应用程序不是最佳的（多数情况下是这样），那么就会造成堆锁定，这将对性能产生影响。将大部分对象限制在一定范围内对整体性能有益。

  • 分析收集的JRA Recording数据
  • 分析结果，查看-XXtlasize和-XXlargeobjectlimit是否需要调优（请记住，对于多数应用程序，根据eDocs，线程本地区域大小应该至少是大对象大小的两倍）。这些内容位于JRA Recording首页的右上方。请查看下面关于 tlaSize 和largeObjectLimit 的信息。在JRockit R27.3之前的多数情况下，这些都不必调优。

注意：为了确保在稳定状态下获取配置文件和度量，应用程序需要有足够的热启动时间。要在性能分析过程中检查是否符要求，可以查看JRA Recording中的优化标签，其中性能分析前后的优化数量和优化时间应该大致（理论上应该是准确地）相等。

测试2：锁性能分析

现在让我们看一下锁性能分析，它会显示在应用程序中是否有过多的锁定。如果确实如此的话，那么将对整体性能造成影响。

  • 运行测试（启用-Djrockit.lockprofiling），分析这些结果。确定没有通过JVM启用的日志。该标志大概会占用5%到10%的系统开销，一个单独的测试将收集该数据，此时性能被忽略，唯一做完的分析是锁定分析。
-server -Xms1536m -Xmx1536m -Djrockit.lockprofiling
  • 在相同测试中，使用jrcmd.sh实用工具或JRockit Mission Control（JRMC）热启动应用程序后，运行10分钟的JRA Recording。有关如何使用该工具的信息，请参见电子文档。
  • 使用top和iostat选项监控操作系统，如果需要的话还可使用ctrhandler.act文件指定的信息执行线程转储。
  • 分析结果。
测试3：调优tlaSize和largeObjectLimit

在这个测试中，我们将根据前面测试的结果调优线程本地区域大小和大对象限制。

  • 确定JVM未启用日志。将 –XxtlaSize和-XXlargeObjectLimit的值设置较高一点可能会有所帮助。但是，要验证和比较这点则需要需要长时间运行测试。对于 R27.2，将preferredSize设置为16k可能有所帮助。您可以查看关于这一问题的 详细信息。为此要，更改TLA设置并使用与测试1相同的Java命令行选项重新运行测试；增加-XXtlaSize和-XXlargeobjectlimit的TLA值设置。相关信息，请参见 tlaSize。注意：在R27.3之前，提高性能通常不需要调优这些标志。事实上，过度调优这些标志很可能带来负面影响。
  • 在相同测试中，使用jrcmd.sh实用工具或JRockit Mission Control（JRMC）热启动应用程序后，运行10分钟的JRA Recording。关于如何使用该工具的信息，请参见电子文档。
  • 使用top和iostat选项监控操作系统，如果需要的话还可使用ctrhandler.act文件指定的信息执行线程转储。
  • 分析结果。
测试4：调优垃圾收集算法

本节测试的目的是运行各种不同的垃圾收集算法设置，并查看哪种设置对于应用程序最佳。关于 -XXsetGC标志 的详细令牌，请阅读以下内容。JRockit将以调优的nursery大小运行并移除-Xgcprio:throughput标志。该 throughput选项将在这些双版本的垃圾收集器之间自动切换，但进行直接的选择可能带来一些额外的性能上的好处。Nursery调优的目的是使被提 升的对象保持较少的数量，因为这是nursery收集中代价高的部分。通过增加和减小nursery的大小对其调优。nursery的大小主要依赖于对象 生存的时间，因为如果它们生存着，则在YC期间会得到提升。运行jrcmd <PID>版本查看哪个当前垃圾收集器策略是活动的。

  • a. 测试4-1：
    • 使用–XxsetGC垃圾收集算法设置运行测试。该测试将设置垃圾收集选项为单倍行距加上并行标记算法和并行扫描算法；还要手动调优nursery大小：
-server -Xms1536m -Xmx1536m -Xns:384m  -Xverboselog:perTestGC.log-Xverbose:opt,memory,gcpause,memdbg,compaction,gc,license-Xverbosetimestamp -Xgcreport -XXnoSystemGC -XXsetGC:singleparpar
    • 在相同测试中，使用jrcmd.sh实用工具或JRockit Mission Control（JRMC）热启动应用程序后，运行10分钟的JRA Recording。关于如何使用该工具的信息，请参见电子文档。
    • 使用top和iostat选项监控操作系统，如果需要的话还可使用ctrhandler.act文件指定的信息执行线程转储。
    • 分析结果。
  • b. 测试4-2：
    • 用 –XxsetGC运行一个使用垃圾收集算法集的测试。该测试将设置垃圾收集选项为分代的（双倍行距）加上并行标记算法和并行扫描算法；还要手动调优nursery大小：
-server -Xms1536m -Xmx1536m -Xns:384m  -Xverboselog:perTestGC.log
-Xverbose:opt,memory,gcpause,memdbg,compaction,gc,license
-Xverbosetimestamp -Xgcreport -XXnoSystemGC -XXsetGC:genparpar
    • 在做这个测试的时候，使用jrcmd.sh实用工具或JRockit Mission Control (JRMC)收集应用程序热启动并运行后10分钟的JRA Recording。关于如何使用该工具，请参见eDocs。
    • 使用top、iostat，需要的话，还可用带有ctrhandler.act文件给定的信息的线程转储来监控正在运行的系统。
    • 分析结果。
  • c. 测试4-3：根据前面的-XXsetGC:genparpar测试向上或向下调优nursery大小。
  • d. 测试4-4：用-Xgc:gencon -Xns50m（和设为收集规格的日志）试试。
  • e. 测试4-5：用-Xgc:parallel -XXcompactratio:1（和设为收集规格的日志）试试。
测试5：调优垃圾收集线程

本测试的目的是查看gcthreads标志设置对整体性能的影响。

  • 根据前面的结果，调优-XXgcthreads标志为实际物理CPU的数量并重新运行测试(由于这些值默认是基于机器上核心和硬件线程的数量，所以这应该是自动调优的)。您可以查看在“收集故障诊断数据”一节收集的详细输出日志来对其验证。更多细节请参见 gcThreads flag。
测试6：调优锁争用

如果在胖锁（fat lock）上存在锁争用，则可以用-XXdisableFatSpin禁止它们，或者用-Djrockit.useAdaptiveFatSpin= true让JRockit自适应地禁止它们。当测试2启用-Djrockit.lockprofiling时，可以通过在JRA中查看那个标签来确定这一 点。更多细节，请参见locking in JRockit。

测试7：调优Xeon硬件

如果运行在Xeon硬件之上，那么可以添加-XXallocPrefetch和-XXallocRedoPrefetch，它们与TLA和LargeObjectLimit一起将有助于减少内存分配的开销。有关详细信息，请参见 allocPrefetch标志。为了得到最佳的结果，您可能会在BIOS中禁用硬件预取指令。虽然操作方式取决于BIOS的牌子，但参数的名称通常都为“Hardware Prefetcher”、“Adjacent Sector Prefetcher”、“Adjacent Cache Line Prefetcher”等。更多信息请参见 Intel on this subject。

测试8：将堆放入largePage

这会将堆锁入内存，使操作系统不能将其交换出来。更多信息请参见 largePages标志，有关Linux操作系统端配置的更多信息还可以参考 在Linux上配置-XXlargePages 一节。在JRockit R27版中，该选项的名称为-XlargePages。根据前面的结果，调优-XlargePages标志可能有所帮助，但可能也没有。使用该标志运行测试，查看结果否对整体性能有帮助。

测试9：用-XXaggressive标志测试

本节中的这些配置将使JVM高速运行并尽快达到稳定状态。为了实现此目标，JVM在启动时需要更多内部资源；但当目标一旦达成，它所需要的自适应优化将更少。我们推荐您为了那些单独工作的、运行时间长的、内存敏感的应用程序使用这个选项。更多细节请参见 aggressive标志。使用-XXaggressive标志运行测试，查看结果否对整体性能有帮助。

测试10：用-XX:+UseNewHashFunction标志测试

这个选项支持一个新的、更快的HashMap散列函数，它在Java 5.0 Update 8中引入，从R27.1.0开始也是BEA JRockit的一部分。这个散列函数能够通过改进的散列扩展提高性能而不改变HashMap中元素存放的顺序。更多细节请参见UseNewHashFunction标志。使用这个新的-XX:+UseNewHashFunction运行测试，，查看结果否对整体性能有帮助。

测试11：将暗物质减到最少

“暗物质”指被浪费的堆内存，它使堆成为许多碎片。了解如何最大程序地减少暗物质，以便当堆需要压缩时，整体吞吐量不受影响。请看以下选项：

  • 使用分代的垃圾收集器 (-Xgc:gencon or -Xgc:genpar)。在初始收集（nursery垃圾收集）期间，在nursery中被发现生存的对象被迁移到旧的一代。这样有好的副作用，当迁移对象时可以压缩它们。
  • 提高压缩比（-XXcompactionRatio=nn）。通过将对象迁移到压缩块中、消除它们之间的暗物质，压缩减少了暗物质。
  • 通 过-XXminBlockSize:<memSize>选项修改规范（将什么视为暗物质？）。堆上小于最小块大小的块视为暗物质。因此，通过 减小最小块大小，您最终减少了暗物质。但请注意，因为JRockit为了释放堆空间必须做更细密的搜索，所以垃圾收集的时间会更长。最小块大小默认为2 KB。
进一步测试：调优应用程序服务器层

最后，查看上述调优建议，调优WebLogic Server实例层。

结束语

本文提供的信息绝非一个完整的清单。但它会让您开始更好地理解和调优JRockit JVM层！

参考资料
附录

以下内容为正文引用过的额外信息。

性能测试期间的JRCMD/JRA

使用命令行或者基于JRockit Mission Control (JRMC) Eclipse的工具都能够执行JRA Recoding。我们可以使用JRMC连接到多个 JRockit JVM，收集JRA recording，查看JVM的实时数据，检测和排除内存泄漏，以及查看应用程序内的潜伏物（执行缓慢“点”）。有关如何运行JRockit Mission Control，请参见下一节。

  • 下载 文档许可证。
  • 将下载的license.bea文件添加到<JROCKIT_HOME>/jre目录中。这样，完整路径将如下所示：<JROCKIT_HOME>/jre/license.bea
  • 运 行JRCMD如下：jrcmd.sh <PID> jrarecording filename=myrecording.xml time=600（该文件被写入本地目录或者指定的完整路径／文件名）或jrcmd.sh <PID> print_threads。
  • 使用脱机JRA工具分析最后得到的myrecording.xml.zip文件。通过在<JROCKIT_HOME>/bin中执行JRA binary，将该JRA工具放入此目录下。
  • 请查看 JRCMD文档。
JRockit Mission Control
  • 使用JRockit将下列内容添加到WebLogic Instance的启动行中：java -Xmanagement:autodiscovery=true,ssl=false,authenticate=false,port=7091
  • 使用<jrockit-install-directory>/bin/jrmc.exe(sh)启动JRockit Mission Control
  • 如有必要，将JR Mission Control的license.bea file文件添加到：<JROCKIT_HOME>/jre/license.bea
  • 在Mission Control的帮助下，JRA Recordings、内存泄漏、潜伏物探查和监控能够全部在一处完成。
堆/线程的测试期快照
  • 创建一个简单的、名称ctrhandler.act的文件并将其存放在<JROCKIT_HOME>/jre/bin/jrockit目录。
  • 执行线程转储命令时，例如，kill -3)，JRockit会查看此文件并执行命令列表。
  • ctrlhandler.act文件应该包含以下内容：
#ctrlhander.act file located in the <JROCKIT_HOME>/jre/bin/jrockit directoryset_filename filename=./jrocket_control_breakoutput.txt append=truetimestampprint_threadstimestampversionprint_class_summaryprint_object_summary increaseonly=trueprint_threadsprint_threads nativestack=trueprint_utf8pooltimestampprint_memusagetimestampheap_diagnosticstimestamp# The following is optional and is another way to generate a JRArecordingjrarecording filename=./myjra.xml time=600
在Linux上配置-XXlargePages

问题：为何使用largePages？

回答：使用largePages的优点是可以锁定堆内存，并且不适用页面交换（还可以减少IOWait和GC）。在实际内存中，访问堆中的对象明显变快了。因此，为了实现性能目标，largePages选项是一个理想的选择。

  • 如果机器支持大页面，则cat /proc/meminfo的输出将如下所示：
HugePages_Total: xxxHugePages_Free:  yyyHugepagesize:    zzz KB

如果xxx为0，则没有分配任何大页面。

  • 如果不为0，则需要使用CONFIG_HUGETLBFS(位于“File systems”下面)和CONFIG_HUGETLB_PAGE（当CONFIG_HUGETLBFS被选时，自动被选上）配置选项构建Linux kernel。
  • 接着，在Linux上分配大页。注意：只允许根用户分配大页面。
    • 装载文件系统。JRockit使用hugepages文件系统，它驻留在内存中。这些步骤可以完成文件系统的安装。每次机器重启时都需要完成实际装载和chmod命令，或者也可以将其添加到一个/etc/rc.d/rc.local类型的文件：
                 mkdir -p  /mnt/hugepages        mount -t hugetlbfs nodev /mnt/hugepages        chmod 777 /mnt/hugepages
    • 分配大页面。这是通过指定应该分配的内存数量来自动执行的。在分配过程中，这些面页将被保留起来，不能像普通页面那样使用。它们的分配和解除分配方式如下：
echo 20 > /proc/sys/vm/nr_hugepages

此处，数字20指应该保留的页面数量。要解除分配，可以将分配页面设置为0。（注意：请参见后面Q&A部分以确定正确的数量。）

如果并非所有请求的页面都被保留，那么可用内存就不够了。如果确要如此，内存很可能会有太多的碎片，那么我们的建议是重启机器。请注意，大页面不能被交换，所以一切都必须放在物理内存中。

在RHEL3上，该文件的名称为/proc/sys/vm/hugetlb_pool，所以可以使用以下命令执行：

echo 500 > /proc/sys/vm/hugetlb_pool

这里的数字500指请求的大小为多少MB，而不是页数。如果JRockit不能够直接消除临时的大页文件，请别忘记在执行之后进行消除。否则， 那些页在被释放之前将不可用。这对RedHat kernel build 2.4.18-e.25.smp适用，而2.4.18-e.12.smp却不行。

问题：如何确定发给/proc/sys/vm/nr_hugepages的正确数量呢？

回答：因为要将整个Java堆放入largePages，所以必须根据堆的大小和页面大小来确定。为/proc/sys/vm/nr_hugepages确定正确的数量就像下面的例子那样简单，信息如下：

JVM Max Heap = 1536MB (1572864 KB approx)HPAGE_SIZE = 2 MB then  the value sent to /proc/sys/vm/nr_hugepages would be approximately 7.7.  (1572864 MB / 2 MB)

注意：

尽管动态设置大页面的数量在理论上是可行的，但实践中并非如此。减少大页面的数量决不会有问题，但是增加大页面的数量就会产生问题。原因是，为了创建一个大页，Linux需要在内存中找到足够大的连续的区域。如果找不到，就不能创建大页面。

刚启动系统时，内存的碎片还不是很多，所以要找到足够大的连续的区域并不是问题。但机器运行时间一长，内存会产生更多的碎片。所以，如果要确保能够分配足够大的页面，就必须在启动刚完成时通过启动脚本或手动设置它们。

Copyright © 2008 This feed is for personal, non-commercial use only
聚沙成塔-小哈的记事薄 by hashei 如果喜欢，欢迎订阅feed.hashei.com
Digital Fingerprint: 10f920a9f2bae51c3c73c4f5fb50a949]]> http://www.hashei.me/2009/09/weblogic_profiling_and_performance_tuning.html/feed 1 JDBC引发的服务器hang解决思路 http://www.hashei.me/2009/08/jdbc_causes_server_hang.html?utm_source=rss&utm_medium=rss&utm_campaign=jdbc_causes_server_hang http://www.hashei.me/2009/08/jdbc_causes_server_hang.html#comments Sun, 16 Aug 2009 03:58:36 +0000 hashei http://www.hashei.me/2009/08/jdbc_causes_server_hang.html 这篇也是转自BEA的官方文档，源地址在BEA被Oracle收购后就转到Oracle官网了，所以留为备份。

JDBC Causes Server Hang


Problem Description
A JDBC connection which is used by an application or by WebLogic Server itself will block one WebLogic Server execute thread for the complete duration of the calls that are made via this connection. The JVM will ensure that the CPU is given to runnable threads by its thread scheduling mechanism, while the thread that blocks on a SQL query needs to wait. However, the thread occupied by the JDBC call will be reserved and used for the application until the call returns from the SQL query.

Even a transaction timeout will not kill or timeout any action that is done by the resources that are enlisted in this transaction. The actions will run as long as they take, without interruption. A transaction timeout will set a flag on the transaction that will mark it as rollback only, so that any subsequent request to commit this transaction will fail with a TimedOutException or RollbackException. However, as mentioned above, the long running JDBC calls can lead to blocked WebLogic Server execute threads, which can finally lead to a hanging instance, if all threads are blocked and no execute thread remains available for handling incoming requests.

More recent WebLogic Server versions have a health check functionality that regularly checks if a thread does not react for a certain period of time (the default is 600 seconds). If this happens, an error message is printed to your log file similar to following:


####<Nov 6, 2004 1:42:30 PM EST> <Warning> <WebLogicServer> <mydomain> <myserver> <CoreHealthMonitor>
<kernel identity> <>
<000337> <ExecuteThread: ‘64′ for queue: ‘default’ has been busy for “740″ seconds working on the request “Scheduled Trigger”,
which is more than the configured time (StuckThreadMaxTime) of “600″ seconds.>


This does not interrupt the thread, as this is just a notification for the administrator. The only way a stuck thread becomes unstuck again is when the request it is handling finishes. In this case, you will find a message similar to following in your WebLogic Server’s log file:


####<Nov 7, 2004 4:17:34 PM EST> <Info> <WebLogicServer><mydomain> <myserver> <ExecuteThread: ‘66′
for queue: ‘default’>
<kernel identity> <> <000339> <ExecuteThread: ‘66′ for queue: ‘default’ has become “unstuck”.>


The time interval for the health check functionality is configurable. Please check StuckThreadMaxTime property in the <Server> tag of your config.xml file: http://e-docs.bea.com/wls/docs81/config_xml/Server.html#StuckThreadMaxTime or the “Detecting stuck threads” section in the WebLogic Server administration console help: http://e-docs.bea.com/wls/docs81/perform/WLSTuning.html#stuckthread.

Top of Page

Problem Troubleshooting
Different programming techniques or JDBC connection pool configurations can lead to deadlocks or long running JDBC calls that lead to hanging WebLogic Server instances. General information about how to troubleshoot and analyze a hanging WebLogic Server instance is provided in Generic Server Hang Pattern.

This pattern addresses JDBC calls causing a server hang and other well known JDBC-related causes for common problems leading to hanging WebLogic Server instance.  Other Support Patterns referenced in this pattern are at the WebLogic Server Support Patterns Site.

Quick Links

Why does the problem occur?
The following are some different possible reasons that can cause JDBC calls to lead to a hanging WebLogic Server instance:

Top of Page

Synchronized DriverManager.getConnection()
Older JDBC application code sometimes uses DriverManager.getConnection() calls to retrieve a database connection using a certain driver. This technique is not recommended as it can cause deadlocks or at least relatively low performance for your connection requests. The reason behind this is, that all DriverManager calls are class-synchronized, meaning that one DriverManager call in one thread will block all other DriverManager calls in any other thread inside one WebLogic Server instance.

In addition to that, the constructor for a SQLException makes a DriverManager call, and most drivers have DriverManager.println() calls for logging, so any of these can block all other threads that issue a DriverManager call.

DriverManager.getConnection() can take a relatively long time until it returns with the physical connection created to the database. Even if no deadlock occurs, all other calls need to wait until that one thread gets its connection. This is not a best practice in a multi-threaded system like WebLogic Server.


This information is taken from http://forums.bea.com/bea//thread.jspa?forumID=2022&threadID=200063365&messageID=202311284&start=-1#202311284.
Also our documentation clearly states that DriverManager.getConnection() should not be used: http://e-docs.bea.com/wls/docs81/faq/jdbc.html#501044.

If you prefer to use JDBC connections in your JDBC code, you should use a WebLogic Server JDBC connection pool, define a DataSource for it, and get the connection from the DataSource. This will give you all advantages from a pool (resource sharing, connection reuse, connection refresh if a database was down, etc). It also will help you avoid the deadlocks that may happen with DriverManager calls. See detailed information on how to use JDBC connection pools, DataSources, and other JDBC objects in WebLogic Server at: http://e-docs.bea.com/wls/docs81/jdbc/intro.html#1036718 and http://e-docs.bea.com/wls/docs81/jdbc/programming.html#1054307.

A typical thread blocked in a DriverManager.getConnection() call looks like:
“ExecuteThread-39″ daemon prio=5 tid=0×401660 nid=0×33 waiting for monitor entry [0xd247f000..0xd247fc68]
  at java.sql.DriverManager.getConnection(DriverManager.java:188)
  at com.bla.updateDataInDatabase(MyClass.java:296)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:865)
  at weblogic.servlet.internal.ServletStubImpl.invokeServlet
(ServletStubImpl.java:120)
  at weblogic.servlet.internal.ServletContextImpl.invokeServlet
(ServletContextImpl.java:945)
  at weblogic.servlet.internal.ServletContextImpl.invokeServlet
(ServletContextImpl.java:909)
  at weblogic.servlet.internal.ServletContextManager.invokeServlet
(ServletContextManager.java:269)
  at weblogic.socket.MuxableSocketHTTP.invokeServlet(MuxableSocketHTTP.java:392)
  at weblogic.socket.MuxableSocketHTTP.execute(MuxableSocketHTTP.java:274)
  at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:130)

Top of Page

Long Running SQL Queries
Long running SQL queries block execute threads for their duration and until they return their result to the calling application. This means that a WebLogic Server instance needs to be configured to be able to handle enough calls simultaneously as they are requested by the application load. Limiting factors here are the number of execute threads and the number of connections in the JDBC connection pools. A general rule of thumb is to set the number of connections in the pool equally to the number of execute threads to enable optimal resource utilization. If JTS is used, some more connections in the pools should be available because connections may be reserved for transactions that are actually not active.

A thread hanging in a long running SQL call will show a very similar stack in a thread dump as the one for a hanging database. Please compare the next section for details.

Hanging Database
Good database performance is key for the performance of an application that relies on this database. Consequently, a hanging database can block many or all available execute threads in a WebLogic Server instance and finally lead to a hanging server. To diagnose this, you should take 5 to 10 thread dumps from your hanging WebLogic Server instance and check your execute threads (in the default queue or your application thread queue) to see if they are currently in SQL calls and waiting for a result from the database. A typical stack trace for a thread that currently issues a sql query could look similar to following example:


“ExecuteThread: ‘4′ for queue: ‘weblogic.kernel.Default’” daemon prio=5 tid=0×8e93c8 nid=0×19 runnable [e137f000..e13819bc]
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at oracle.net.ns.Packet.receive(Unknown Source)
  at oracle.net.ns.DataPacket.receive(Unknown Source)
  at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
  at oracle.net.ns.NetInputStream.read(Unknown Source)
  at oracle.net.ns.NetInputStream.read(Unknown Source)
  at oracle.net.ns.NetInputStream.read(Unknown Source)
  at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:931)
  at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:893)
  at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:375)
  at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1983)
  at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:1250)
  – locked <e8c68f00> (a oracle.jdbc.ttc7.TTC7Protocol)
  at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:2529)
  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout
(OracleStatement.java:2857)
  at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:608)
  – locked <e5cc44d0> (a oracle.jdbc.driver.OraclePreparedStatement)
  – locked <e8c544c8> (a oracle.jdbc.driver.OracleConnection)
  at oracle.jdbc.driver.OraclePreparedStatement.executeQuery
(OraclePreparedStatement.java:536)
  – locked <e5cc44d0> (a oracle.jdbc.driver.OraclePreparedStatement)
  – locked <e8c544c8> (a oracle.jdbc.driver.OracleConnection)
  at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:80)
  at myPackage.query.getAnalysis(MyClass.java:94)
  at jsp_servlet._jsp._jspService(__jspService.java:242)
  at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
  at weblogic.servlet.internal.ServletStubImpl$
ServletInvocationAction.run(ServletStubImpl.java:971)
  at weblogic.servlet.internal.ServletStubImpl.invokeServlet
(ServletStubImpl.java:402)
  at weblogic.servlet.internal.ServletStubImpl.invokeServlet
(ServletStubImpl.java:305)
  at weblogic.servlet.internal.RequestDispatcherImpl.include
(RequestDispatcherImpl.java:607)
  at weblogic.servlet.internal.RequestDispatcherImpl.include
(RequestDispatcherImpl.java:400)
  at weblogic.servlet.jsp.PageContextImpl.include(PageContextImpl.java:154)
  at jsp_servlet._jsp.__mf1924jq._jspService(__mf1924jq.java:563)
  at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
  at weblogic.servlet.internal.ServletStubImpl$
ServletInvocationAction.run(ServletStubImpl.java:971)
  at weblogic.servlet.internal.ServletStubImpl.invokeServlet
(ServletStubImpl.java:402)
  at weblogic.servlet.internal.ServletStubImpl.invokeServlet
(ServletStubImpl.java:305)
  at weblogic.servlet.internal.WebAppServletContext$
ServletInvocationAction.run(WebAppServletContext.java:6350)
  at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:317)
  at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:118)
  at weblogic.servlet.internal.WebAppServletContext.invokeServlet
(WebAppServletContext.java:3635)
  at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2585)
  at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
  at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)


The thread will be in running state. You should compare the threads in your different thread dumps in order to see if they receive the return from the SQL call in a timely manner or if they hang in this same call for a longer period of time. If the thread dumps seem to imply long response times from SQL calls, the corresponding database logs should be checked to see if problems in the database cause this slow performance or hang situation.

Top of Page

Slow Network
Communication between WebLogic Server and the database relies on a well-performing and reliable network in order to serve the requests in a timely manner. Slow network performance can therefore lead to hanging or blocking execute threads waiting for results of SQL queries. The related stack traces will look similar to example above in Hanging Database section. It is not possible to find the root cause of the hanging or slow SQL queries by solely analyzing the WebLogic Server thread dumps. These give the first hint that something is wrong with the performance of the SQL calls. The next step is to check if there is a database or network problem that causes poorly performing SQL calls.

Deadlock
Both an application level deadlock as well as a deadlock on the database level can lead to hanging threads. You should check your thread dumps to see if there is an application level deadlock. Information on how to do this is provided in Server Hang – Application Deadlock Pattern. A database deadlock can be detected either in the database log or by the SQL Exception that can be found in the WebLogic Server log file. An example for a related SQL Exception is:


java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
  at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:170)
  at oracle.jdbc.oci8.OCIDBAccess.check_error(OCIDBAccess.java:1614)
  at oracle.jdbc.oci8.OCIDBAccess.executeFetch(OCIDBAccess.java:1225)
  at oracle.jdbc.oci8.OCIDBAccess.parseExecuteFetch(OCIDBAccess.java:1338)
  at oracle.jdbc.driver.OracleStatement.executeNonQuery(OracleStatement.java:1722)
  at oracle.jdbc.driver.OracleStatement.doExecuteOther(OracleStatement.java:1647)
  at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2167)
  at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate
(OraclePreparedStatement.java:404)


As it generally can take some time until a database detects a deadlock and resolves it by rolling back one or more transactions that cause the deadlock, one or more execute threads will be blocked until the rollback has finished.

RefreshMinutes or TestFrequencySeconds
If you see recurring periods of low database performance, slow SQL calls, or connection peaks, the setting of the RefreshMinutes or TestFrequencySeconds configuration property in your JDBC connection pools could be the reason. This is described in detail in Investigating JDBC Problems Pattern. Unless you do not have a firewall between your WebLogic Server instance and your database, you should disable this functionality.

Pool Shrinking
Physical connections to a database are resources that should be opened once and kept open as long as possible, as a new connection request is a considerable resource overhead for the database, the operating system kernel, and the WebLogic Server. Consequently, pool shrinking should be disabled on production systems in order to keep this overhead at a minimum. If pool shrinking is enabled, idle pool connections will be closed and reopened once connection requests to the pool cannot be satisfied.

As these activities can take some time, the related application requests may take an unexpectedly long time which can lead users to assume that the system hangs. Information on how to optimize JDBC connection pool configurations is provided in Investigating JDBC Problems Pattern.

Top of Page

Analysis of a hanging WebLogic Server instance
General information on how to analyze a hanging WebLogic Server instance is provided in Generic Server Hang Pattern.

Most times it will be helpful to start with taking thread dumps from the hanging system in order to find out what is going on, e.g., what the different threads are doing and why they hang. Generally, thread dumps can be taken on production systems, however caution is necessary for very old versions of the JVM (<1.3.1_09), as they may crash during thread dumps. Also if the WebLogic Server instance has a huge number of threads, it will mean that the thread dump will take awhile to complete, while the rest of the threads are blocked.

Please take more than one thread dump (5 to 10) with a delay of some seconds in between. This gives you the possibility to check the progress of the different threads. Also it will show if the system actually hangs (no progress at all) or if the throughput is extremely slow, which can seem to be a hanging system.

Information on how to take thread dumps is provided in “Generic Server Hang” support pattern or in our documentation: http://e-docs.bea.com/wls/docs81/cluster/trouble.html.

Also please check if the complete WebLogic Server instance hangs or if it is the application that hangs. “Generic Server Hang” support pattern also includes this information.

Analyzing the thread dumps can show if one of the reasons mentioned in the previous section Why does the problem occur? actually is responsible for your hanging instance. If for example all your threads are in a DriverManager method like getConnection() then you have identified the root cause and need to change your application to use a DataSource or Driver.connect() instead of DriverManager.getConnection().

A very useful tool, Samurai, can be used to analyze thread dumps and to monitor the progress of threads between different thread dumps. This can be downloaded from dev2dev at:  http://dev2dev.bea.com/resourcelibrary/utilitiestools/adminmgmt.jsp.

A whitepaper on analyzing thread dumps on dev2dev: http://dev2dev.bea.com/products/wlplatform81/articles/thread_dumps.jsp will also be helpful in going deeper into the thread dumps to find out more about the server hang.

Top of Page

Tips and Tricks to optimize your JDBC code and JDBC connection pool configuration
There are some best practices both in the development of JDBC code and also in the configuration practice of JDBC connection pools that can help to avoid common problems and optimize resource usage so that hanging server instances should not happen.

JDBC Programming
In order to optimize resource usage in WebLogic Server and conserve database resources, you should use JDBC connection pools for your application’s JDBC calls. Connections created and destroyed in your application code generate an unnecessary overhead which should be avoided. For generic documentation on JDBC programming, see: http://e-docs.bea.com/wls/docs81/jdbc/rmidriver.html#1028977. Also details on JDBC performance tuning are at: http://e-docs.bea.com/wls/docs81/jdbc/performance.html#1027791.

You can view comprehensive information on JDBC that will help to optimize your JDBC code and the utilization of your JDBC resources on dev2dev Java Database Connectivity page at: http://dev2dev.bea.com/technologies/jdbc/index.jsp.

JDBC Connection Pool Configuration
The Investigating JDBC Problems Pattern has recommendations on how to configure a connection pool for production environments. In order to avoid hangs or bad performance, these configuration tips should be considered.

Top of Page

Known Issues
You can periodically review the Release Notes for your version of WLS for more information on Known Issues or Resolved Issues in Service Packs and browse for JDBC server hang-related issues.  For your convenience, see the following:

Please note that changes have been made in WLS 8.1 SP3 to resolve CR134921, where for certain JDBC connections, the call to roll back a transaction was not being handled immediately because the driver had to wait for any currently-executing statement to return. 

Searching will also return Release Notes, as well as other Support Solutions and CR-related information as noted at Need Further Help?.  Contract customers who are logged in at http://support.bea.com/ will also see a Browse portlet for both Solutions and Bug Central where latest available CRs can be browsed by Product version.


Need Further Help?
If you have followed the pattern, but still require additional help, you can:
  1. Query AskBEA at http://support.bea.com/ using “jdbc server hang”, as an example, to discover other published solutions.  Contract Support Customers: Ensure you are logged to access available CR-related information.
  2. Ask a more detailed question on one of BEA’s newsgroups at http://forums.bea.com

If this does not resolve your issue and you have a valid Support Contract, you can open a Support Case by logging in at: http://support.bea.com/ .


FEEDBACK

Please provide us input on whether or not this Support Diagnostic Pattern “JDBC Causes Server Hang” helped, any clarifications you needed, and any requests for new topics to Support Diagnostic Patterns.



DISCLAIMER NOTICE:

BEA Systems, Inc. provides the technical tips and patches on this Website for your use under the terms of BEA’s maintenance and support agreement with you. While you may use this information and code in connection with software you have licensed from BEA, BEA makes no warranty of any kind, express or implied, regarding the technical tips and patches.

Any trademarks referenced in this document are the property of their respective owners. Consult your product manuals for complete trademark information.

Copyright © 2008 This feed is for personal, non-commercial use only
聚沙成塔-小哈的记事薄 by hashei 如果喜欢，欢迎订阅feed.hashei.com
Digital Fingerprint: 10f920a9f2bae51c3c73c4f5fb50a949]]> http://www.hashei.me/2009/08/jdbc_causes_server_hang.html/feed 2 应用服务器发生hang的诊断方法 http://www.hashei.me/2009/08/java_generic_server_hang.html?utm_source=rss&utm_medium=rss&utm_campaign=java_generic_server_hang http://www.hashei.me/2009/08/java_generic_server_hang.html#comments Sat, 15 Aug 2009 15:04:26 +0000 hashei http://www.hashei.me/2009/08/java_generic_server_hang.html 写在前面

其实这是BEA官网上的一篇文档，是在weblogic8.1的时候推出的。在BEA被Oracle收购后，所有的support文章也就被重定向到Oracle的官网首页= =，而且google的快照也没有了。这篇来自无意间google到的一个外国论坛，虽然是写在8.1时，但是解决问题的方法和思路现在依旧有效。本想理解之后结合案例来写一篇，但是最近一直没有遇到相关的问题，而且觉得那样也许会破坏文章的完整性，所以放出原文，既在网上留个副本，也能让大家各取所需，见仁见智。

从内容看，你会发现除了这篇，还有EJB_RMI Server Hang、Application Dead Lock、JDBC Causes Server Hang，但是那个论坛里还能找到的仅有JDBC Causes Server Hang一篇。所以如果你接触weblogic比较早，保存过另两篇文章，或者在网上看到了，那请留言说明，万分感谢。

Generic Hang

Problem Description
A server hang is suspected when:

  • The server does not respond to new requests.
  • Requests time out.
  • Requests take longer and longer to process (may be on the way to a hang).
  • A server crash is not usually a symptom of a hung server but may follow.
Problem Troubleshooting
Please note that not all of the following items would need to be done. Some issues can be solved by only following a few of the items.Quick Links:

Why does the problem occur?
A server can hang for a variety of reasons (refer to Potential Causes of Server Hang). Generally, a server hangs because of a lack of some resource. Lack of a resource prevents the server from servicing requests. For example, because of a problem (deadlock) or volume of requests there may be no execute threads available to do any work; all are busy or busy with previous requests.

Top of Page
Topic
Pattern Name
Link
RMI, RJVM responses – all threads tied up waiting for RJVM, RMI responses. EJB_RMI Server Hang
Application Deadlock – thread locks resource1 then waits for lock for resource2. Another thread locks resource2 and then waits for lock for resource1. Application Deadlock Causes Server Hang
Threads are all used up, none available for new work. Thread Usage Server Hang TBD
Garbage Collection taking too much time. Garbage Collection Server Hang TBD
JSP improper settings for servlet times, e.g. PageCheckSeconds. JSP cause Server Hang TBD
Long Running JDBC calls or JDBC deadlocks lead to a hang. JDBC Causes Server Hang JDBC Causes Server Hang
JVM hang during (code optimization), looks like server hang. Server Hang in Code Optimization TBD
JSP compilation causes server hang under heavy load. JSP Compilation Server Hang TBD
SUN JVM bugs, e.g. Light weight thread library. Sun JVM Bugs that Cause Server Hangs TBD
Top of Page


When a server is hanging, first ping the server using java weblogic.Admin t3://server:port PING. If the server can respond to the ping, it may be that the application is hanging and not the server itself.

Ensure that the server is actually hanging and not doing garbage collection. To verify, restart the server with -verbosegc turned on, and redirect stdout and stderr to one file. When the server stops responding, it can be determined if it’s doing garbage collection or it is really hanging.  If the garbage collection is taking too long (>10 seconds), the server may miss the heartbeats that servers use to keep each other informed of the topoplogy of the cluster.

WebLogic Server uses the ‘default’ thread queue or a configured application specific thread queue to service client requests. Client requests will only be handled in the default queue if no application specific thread queue is defined.  Please see Tuning WebLogic Server Applications, Tuning the Default Execute Queue Threads, and Tuning WebLogic Server Performance Parameters for more information on defining application specific thread queues.

In release 8.1, a change was made to the thread architecture in WebLogic Server.  A specific kernel thread group for internal WebLogic tasks was created.  This was found to be necessary to avoid deadlocks that occurred in earlier releases when all threads in the ‘default’ thread queue were used and none were thus available for WebLogic internal tasks.

The threads in the ‘default’ queue or the application specific thread queue (if one has been configured) are the threads that should be examined in the event of a server hang. Here’s an example of what one of these threads looks like in a thread dump. Execute Thread ‘14′ from the ‘default’ queue looks like in a thread dump when the thread is waiting for work. The latest method called by this thread is Object.wait(). This thread is in a state “waiting on monitor”.
“ExecuteThread: ‘14′ for queue: ‘default’” daemon prio=5 tid=0×8b0ab30 nid=0×1f4 waiting on monitor [0x96af000..0x96afdc4]
at
java.lang.Object.wait(Native Method)
at
java.lang.Object.wait(Object.java:420)
at
weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:94)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:118)
Threads can be in one of several states.  Please see the table below for a description of the thread states.
The format of the thread dump varies with the vendor.  Check on the vendor’s website for information regarding the format.

Below is an example of  threads that  may  be hanging.  ExecuteThread ‘9′ is waiting to lock some object <dde51520>.   Notice the “waiting to lock <dde51520>” line in the stack trace for this thread.  ExecuteThread ‘6′ is also “waiting to lock the same object <dde51520>”.  The third thread, ExecuteThread ‘5′ has locked this object <dde51520>and is doing work.  This  example demonstrates why one thread dump is not enough.  If the server is hanging, and it is suspected that the cause is the locked object <dde51520>, then subsequent thread dumps will show whether or not that object was released and a new thread has locked object <dde51520>.  If after several thread dumps,  you do not see that the threads have progressed, that object <dde51520> has not been released, you may suspect that there is a problem with the routine(s) in the ExecuteThread ‘5′ call stack because the lock is not being released.
“ExecuteThread: ‘9′ for queue: ‘weblogic.kernel.Default’” daemon prio=5 tid=0xf684c8 nid=0×13 waiting for monitor entry [cc2ff000..cc2ffc24]
at weblogic.cluster.MemberManager.done(MemberManager.java:306)
- waiting to lock <dde51520> (a weblogic.cluster.MemberManager)
at weblogic.cluster.MulticastManager.execute(MulticastManager.java:399)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

“ExecuteThread: ‘6′ for queue: ‘weblogic.kernel.Default’” daemon prio=5 tid=0×9df020 nid=0×10 waiting for monitor entry [cc5ff000..cc5ffc24]
at weblogic.cluster.MemberManager.getRemoteMembers(MemberManager.java:396)
- waiting to lock <dde51520> (a weblogic.cluster.MemberManager)
at weblogic.cluster.ClusterService.getRemoteMembers(ClusterService.java:238)
at weblogic.servlet.internal.HttpServer.setServerList(HttpServer.java:388)
at weblogic.servlet.internal.HttpServer.clusterMembersChanged(HttpServer.java:418)
- locked <ddf32360> (a weblogic.servlet.internal.HttpServer)
at weblogic.cluster.MemberManager$2.execute(MemberManager.java:421)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

“ExecuteThread: ‘5′ for queue: ‘weblogic.kernel.Default’” daemon prio=5 tid=0×9df020 nid=0×12 waiting for monitor entry [cc5ff000..cc5ffc24]
. . .

at weblogic.cluster.MemberManager.checkTimeouts(MemberManager.java:346)
- locked <dde51520> (a weblogic.cluster.MemberManager)
at weblogic.cluster.MulticastManager.trigger(MulticastManager.java:291)
at weblogic.time.common.internal.ScheduledTrigger.run(ScheduledTrigger.java:243

Determine if the”default” ExecuteThread queue is overloaded. Use the console to determine if any of the ExecuteThreads in the ‘default’ queue are idle. If none are idle, then the application probably needs to be configured with a larger number of ExecuteThreads. This value can be changed through the console and is in the config.xml file.

If the Execute Queue has idle threads, it is possible that not enough socket reader threads are allocated. By default, a WebLogic Server instance creates three socket reader threads upon booting. If a cluster system utilizes more than three sockets during peak periods, increase the number of socket reader threads.

The number of socket reader threads should usually be small. However, configure one thread for each Weblogic Server that acts as a client of the server instance that is hanging.

If using a JDBC connection pool, ensure that the JDBC connections have been configured to be equivalent to the number of simultaneous requests, i.e., execute threads, for the pool.

Top of Page


The possibility exists that a problem with JDBC could produce deadlock. Check the version and service pack level of the server found in the beginning of the weblogic.log. Then check above the version and service pack lines for any temporary patches that have already been applied to the server classpath. The patches will tell what problems have already been addressed.

Top of Page


The way to take a thread dump is dependent on the operating system where the hung server instance is installed. Information about taking a thread dump on various operating systems can be found at http://e-docs.bea.com/wls/docs81/cluster/trouble.html#gc. Redirection of both standard error and standard out places the thread dump information in the proper context with server information and other messages and provides more useful logs.

Unix Systems (Solaris, HP, AIX)
Use kill –3 <weblogic process id> to create the necessary thread dumps to diagnose a problem. Ensure this is done several times on each server, spaced about 5 to 10 seconds apart, to help diagnose deadlocks. For this to work, nohup the process when starting the server (refer to Solutions S-12292 and S-15924).

Windows, XP, NT
Each server requires <Ctrl>-<Break> to create the necessary thread dumps to diagnose a problem. Ensure this is done several times on each server, spaced about 5 to 10 seconds apart, to help diagnose deadlocks. On NT, in the command shell type CTRL-Break.

If you have installed WebLogic as a Windows service, you will not be able to see the messages from the JVM or WebLogic Server that are printed to standard out or standard error.  To view these messages, you must direct standard out and standard error to a file.  To do this, take the following steps:

  1. Create a backup copy of the WL_HOME\server\bin\installSvc.cmd master script.
  2. In a text editor, open the WL_HOME\server\bin\installSvc.cmd master script.
  3. In installSvc.cmd, the last command in the script invokes the beasvc utility.
  4. At the end of the beasvc command, append the command -log:”pathname”
    where pathname is a fully qualified path and filename of the file that you want to store the server’s standard out and standard error messages.
  5. The modified beasvc command will resemble the following command:
    “%WL_HOME%\server\bin\beasvc” -install
    -svcname:”%DOMAIN_NAME%_%SERVER_NAME%”
    -javahome:”%JAVA_HOME%” -execdir:”%USERDOMAIN_HOME%”
    -extrapath:”%WL_HOME%\server\bin” -password:”%WLS_PW%”
    -cmdline:%CMDLINE%
    -log:”d:\bea\user_projects\domains\myWLSdomain\myWLSserver-stdout.txt”
  6. If you started WebLogic with nohup, the log messages will show up in nohup.out.

Linux
The Linux operating system views threads differently than other operating systems. Each thread is seen by the operating system as a process. To take a thread dump on Linux, find the process id from which all the other processes were started. Use the commands:

  • To obtain the root PID, use:

    ps -efHl | grep ‘java’ **. **

Use a grep argument that is a string that will be found in the process stack that matches the server startup command. The first PID reported will be the root process, assuming that the ps command has not been piped to another routine.

  • Use the weblogic.Admin command THREAD_DUMP

Another method of getting a thread dump is to use the THREAD_DUMP admin command. This method is independent of the OS on which the server instance is running.

java weblogic.Admin -url ManagedHost:8001 -username weblogic -password weblogic THREAD_DUMP

NOTE: This command cannot be used if unable to ping the server instance.

If the JVM in use is Sun’s, the thread dump goes to stdout. Sun has enhanced the thread dump format between JVM 1.3.1 and 1.4. To obtain Sun’s 1.4 style of thread dump add the following option to the java command line for starting the 1.3.1 JVM:

-XX:+JavaMonitorsInStackTrace

Top of Page


The most useful tool in analyzing a server hang is a set of thread dumps. A thread dump provides information on what each of the threads is doing at a particular moment in time. A set of thread dumps (usually 3 or more taken 5 to 10 seconds apart) can help analyze the change or lack of change in each thread’s state from one thread dump to another. A hung server thread dump would typically show little change in thread states from the first to the last dump.

Threads can be in one of the following states:

Running or runnable thread A runnable state means that the threads could be running or are running at that instance in time.
Suspended thread Thread has been suspended by the JVM.
Thread waiting on a condition variable Threads in a condition wait state can be thought of as waiting for an event to occur.
Thread waiting on a monitor lock Monitors are used to manage access to code that should only be run by a single thread at a time

More information on thread states can be found at http://java.sun.com/developer/onlineTraining/Programming/JDCBook/stack.html#states.

There is also a thread analysis tool at http://dev2dev.bea.com/resourcelibrary/utilitiestools/adminmgmt.jsp.
Download the tool and read the instructions at the link.

What to Look at in the Thread Dump
All requests enter the WebLogic Server through the ListenThread. If the ListenThread is gone, no work can be received and therefore no work can be done. Verify that a ListenThread exists in the thread dump. The ListenThread should be in the socketAccept method. The following example shows what the Listen Thread looks like:
“ListenThread.Default” prio=10 tid=0×00037888 nid=93 lwp_id=6888343 runnable [0x 1a81b000..0x1a81b530] at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
- locked <0×26d9d490> (a java.net.PlainSocketImpl)
at
java.net.ServerSocket.implAccept(ServerSocket.java:439)
at
java.net.ServerSocket.accept(ServerSocket.java:410)
at
weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:24)
at
weblogic.t3.srvr.ListenThread.accept(ListenThread.java:713)
at
weblogic.t3.srvr.ListenThread.run(ListenThread.java:290)
Socket Reader Threads accept the incoming request from the Listen Thread Queue and put it on the Execute Thread Queue. If there are no socket reader threads in the thread dump, then there is a bug somewhere that is causing the socket reader thread to vanish. There should always be at least 3 socket reader threads. One socket reader thread is usually in the poll function, while the other two are available to process requests. Below are Socket Reader threads from a sample thread dump.
“ExecuteThread: ‘2′ for queue: ‘weblogic.socket.Muxer’” daemon prio=10 tid=0×000 36128 nid=75 lwp_id=6888070 waiting for monitor entry [0x1b12f000..0x1b12f530]
at
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:92)
- waiting to lock <0×25c01198> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)

“ExecuteThread: ‘1′ for queue: ‘weblogic.socket.Muxer’” daemon prio=10 tid=0×000 35fc8 nid=74 lwp_id=6888067 runnable [0x1b1b0000..0x1b1b0530] at weblogic.socket.PosixSocketMuxer.poll(Native Method)
at
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:99)
– locked <0×25c01198> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)

“ExecuteThread: ‘0′ for queue: ‘weblogic.socket.Muxer’” daemon prio=10 tid=0×000 35e68 nid=73 lwp_id=6888066 waiting for monitor entry [0x1b231000..0x1b231530]
at
weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:92)
- waiting to lock <0×25c01198> (a java.lang.String)
at
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:178)
at
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:151)
The ThreadPoolPercentSocketReaders attribute sets the maximum percentage of execute threads that are set to read messages from a java socket. The optimal value for this attribute is application-specific. The default value is 33, and the valid range is 1 to 99.

Allocating execute threads to act as socket reader threads increases the speed and the ability of the server to accept client requests. It is essential to balance the number of execute threads that are devoted to reading messages from a socket and those threads that perform the actual execution of tasks in the server.

In release 8.1, the socket reader threads no longer use “ExecuteThreads” in the default queue.  Instead they have their own thread group named.

Next Steps
The next steps require a further analysis of the thread dump. Look in the thread dump to see what each the threads are doing at the time of the hang. This will help to analyze the next stage of the investigation. For example, if there are many threads involved in JSP compilation, refer to Potential Causes of Server Hang for further diagnosis and actions to test.

Top of Page

Copyright © 2008 This feed is for personal, non-commercial use only
聚沙成塔-小哈的记事薄 by hashei 如果喜欢，欢迎订阅feed.hashei.com
Digital Fingerprint: 10f920a9f2bae51c3c73c4f5fb50a949]]> http://www.hashei.me/2009/08/java_generic_server_hang.html/feed 0 注册Weblogic9为Windows服务及外一篇 http://www.hashei.me/2009/08/install-weblogic-as-service-and-recovery-password.html?utm_source=rss&utm_medium=rss&utm_campaign=install-weblogic-as-service-and-recovery-password http://www.hashei.me/2009/08/install-weblogic-as-service-and-recovery-password.html#comments Fri, 07 Aug 2009 04:34:23 +0000 hashei http://www.hashei.me/2009/08/install-weblogic-as-service-and-recovery-password.html Weblogic8时代将weblogic注册成为Windows服务是一件很简单的事，只要执行Domain目录下的installService.cmd即可。不过进入Weblogic9之后，domain目录下就没有了那个脚本，而是出现在%WL_HOME%/server/bin下，直接运行会生成一个“bea _”的服务，运行的话。。。没有任何结果，BEA的edocs上也没有找到有用的帮助。

以下内容，是我盯着installService.cmd和startWeblogic.cmd出神良久后试验成功的。

  1. 复制一份installServer.cmd到domain目录下，
  2. 编辑installServer.cmd，在set WL_HOME=……   下添加set DOMAIN_HOME=<你的domain目录>
  3. 将下一行的call “%WL_HOME%\common\bin\commEnv.cmd”修改成call “%DOMAIN_HOME%\bin\setDomainEnv.cmd”（如果这里的setDomainEnv.cmd写完整路径，那么上一步DOMAIN_HOME可以不用设置，因为setDomainEnv.cmd里还会声明一遍的）
  4. 另起一行，添加两行“set USERDOMAIN_HOME=%DOMAIN_HOME%”“set DOMAIN_NAME=<你的domain名字>”（这是脚本最末一段Install the service里beasvc命令所需要的参数）
  5. 运行installServer.cmd，会在服务里新建一个名为“beasvc %DOMAIN_NAME%_%SERVER_NAME%”的服务，自动运行，帐户属性的是localsystem。

beasvc命令各个参数的详细意义可以参考beasvc /help

以上内容对于Oracle Weblogic 10依旧有效。

weblogic密码遗忘补救方法

  1. 备份   %DOMAIN_HOME%/security下的DefaultAuthenticatorInit.ldift文件，和%DOMAIN_HOME%下的boot.properties文件（如果不在这个目录，那么找找 %DOMAIN_HOME%/servers/AdminServer/security目录），这样如果下列步骤操作失败还能还原回去。
  2.   运行java -cp /bea/weblogic92/server/lib/weblogic.jar weblogic.security.utils.AdminAccount <新用户名> <新密码> .(注意最后有个点的，点之前有空格，不要和原先的一样)
  3. 执行完后在当前目录会生成一个新的DefaultAuthenticatorInit.ldift文件，然后替换原来的%DOMAIN_HOME%/security/DefaultAuthenticatorInit.ldift文件。
  4. 从%DOMAIN_HOME%/servers/AdminServer/data/ldap/子目录中删除初始化状态文件DefaultAuthenticatormyrealmInit.initialized，删除boot.properties。
  5. 运行startWebLogic.sh，启动过程中会让你输入用户名和密码，输入新建立的那个
  6. 用新的用户名密码登录管理控制台，在“安全领域”里点击“myrealm”，在用户和组里，找到之前的那个用户，修改密码，保存，那么以后可以继续使用原来的用户名和密码登录，
  7. 修改boot.properties中的用户密码为上一步修改过的用户密码（明文）格式为
    username=用户名
    password=密码
  8. 重启服务器，现在不用输入用户名密码了，成功启动后，weblogic会加密用户密码的，现在就可以用旧用户登录，新用户可以删除。

Related posts:

Copyright © 2008 This feed is for personal, non-commercial use only
聚沙成塔-小哈的记事薄 by hashei 如果喜欢，欢迎订阅feed.hashei.com
Digital Fingerprint: 10f920a9f2bae51c3c73c4f5fb50a949]]> http://www.hashei.me/2009/08/install-weblogic-as-service-and-recovery-password.html/feed 6