但现在这个网址已经无法访问，我在Metalink找到这篇文章并与大家分享。文章发表的较早，但是OOM发生的原理和解决的方法不变。文中提到的/3GB参数和垃圾回收日志分析的方法我在《JAVA性能优化—IBM JDK JVM参数设置》和《JAVA性能优化-GC日志分析》都提到过。如果英文的看起来比较麻烦，我google到台湾人做的一份翻译JavaAPsvr_A_200411_MemoryLeak，不过我看着“记忆体”觉得更别扭。

Problem Description

Out Of Memory (OOM) – An application displays Out of Memory errors due to memory exhaustion, either in java heap or native memory.

Memory Leak – Constant memory growth in either java heap or native memory, which will eventually end up in out of memory situation. The techniques to debug the memory leak situations are the same as the out of memory situations.

Problem Troubleshooting

Please note that not all of the following items would need to be done. Some issues can be solved by only following a few of the items.

Quick Links:

• Java heap, Native memory, Process size
• Difference between process address space and physical memory
• Why does the OOM problem occur and What does the JVM do in this situation?
• Steps to debug the problem
  • Determine if Java OOM or Native OOM
  • Java OOM
  • Native OOM
• HP JVM specific tools/tips>
• Jrockit specific features
• Bibliography

Java heap This is the memory that the JVM uses to allocate java objects. The maximum value of java heap memory is specified using �Xmx flag in the java command line. If the maximum heap size is not specified, then the limit is decided by the JVM considering factors like the amount of physical memory in the machine and the amount of free memory available at that moment. It is always recommended to specify the max java heap value.

Native memory This is the memory that the JVM uses for its own internal operations. The amount of native memory heap that will be used by the JVM depends on the amount of code generated, threads created, memory used during GC for keeping java object information and temporary space used during code generation, optimization etc.

If there is a third party native module, it could also use the native memory. For example, native JDBC drivers allocate native memory.

The max amount of native memory is limited by the virtual process size limitation on any given OS and the amount of memory already committed for the java heap with -Xmx flag. For example, if the application can allocate a total of 3 GB and if the max java heap is 1 GB, then the max possible native memory is approximately 2 GB.

Process size Process size will be the sum of the java heap, native memory and the memory occupied by the loaded executables and libraries. On 32 bit operating systems, the virtual address space of a process can go up to 4 GB. Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 ~ 2 GB). The remaining is available for the application.

Windows ：By default, 2 GB is available for the application and 2 GB is reserved for Kernel’s use. However, on some variants of Windows, there is a /3GB switch which can be used to change this ratio such that the application gets 3 GB. More details on the /3GB switch can be found at:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/4gt_ram_tuning.asp

RH Linux AS 2.1 ： 3 GB is available for the application.

For other operating systems, please refer to the OS documentation for your configuration.

Top of Page

Each process gets its own address space. In 32 bit operating systems, this address space will range from 0 to 4 GB. This is independent of the available RAM or swap space in the machine. The total physical memory available on the machine is the sum of RAM and the swap space available on that machine. All the running processes share this physical memory.

The memory address within a process is virtual. The kernel maps this virtual address to the physical address. The physical address points to a location somewhere in the physical memory. At any given time, the sum of all the virtual memory used by the running processes in a machine cannot exceed the total physical memory available on that machine.

Top of Page

Out of memory in java heap

The JVM throws java out of memory (java OOM) error if it is not able get more memory in java heap to allocate more java objects. The JVM cannot allocate more java objects if the java heap is full of live objects and it is not able to expand the java heap anymore.

In this situation, the JVM lets the application decide on what to do after throwing the java.lang.OutOfMemoryError. For example, the application may handle this error and decide to shut down itself in a safe way or decide to run ignoring this error. If the application doesn’t handle this error, then the thread that throws this error will exit (you will not see this thread if you take a java thread dump).

In case of weblogic server, this error is handled if it is thrown by an execute thread and the error is logged. If this error is being thrown continuously, then the core health monitor thread shuts down the weblogic server.

Out of memory in native heap

The JVM throws native out of memory (native OOM) if it is not able to get any more native memory. This usually happens when the process reaches the process size limitation on that OS or the machine runs out of RAM and swap space.

When this happens, the JVM handles the native OOM condition, logs a message saying that it ran out of native memory or unable to acquire memory and exits. If the JVM or any other loaded module (like libc or a third party module) doesn’t handle this native OOM situation, then the OS will send a sigabort signal to the JVM which will make the JVM exit. Usually, the JVMs will generate a core file when it gets a sigabort signal.

Top of Page

Determine whether it is a Java OOM or Native OOM

• If the stdout/stderr message says that this is a java.lang.OutOfMemoryError, then this is Java OOM
• If the stdout/stderr message says that it failed to acquire memory, then this is a Native OOM

Please note that the above messages goes to stdout or stderr and not to the application specific log files like weblogic.log

Top of Page

For Java OOM:

1. Collect and analyze the verbose gc output
   1. Add “-verbosegc" flag in the java command line. This will print GC activity info to stdout/stderr. Redirect the stdout/stderr to a file. Run the application until the problem gets reproduced.
   2. Make sure that the JVM does the following before throwing java OOMFull GC run:Does a full GC and all the un-reachable, phantomly, weakly and softly reachable objects are removed and those spaces are reclaimed. More details on different levels of object reachability can be found at:http://java.sun.com/developer/technicalArticles/ALT/RefObj

      You can check whether full GC was done before the OOM message. A message like the following is printed when a full GC is done (format varies depending on the JVM , Check JVM help message to understand the format)

      [memory ] 7.160: GC 131072K->130052K (131072K) in 1057.359 ms

      The format of the above output follows (Note: the same format will be used throughout this Pattern):

      [memory ] <start>: GC <before>K-><after>K (<heap>K), <total> ms
      [memory ] <start> – start time of collection (seconds since jvm start)
      [memory ] <before> – memory used by objects before collection (KB)
      [memory ] <after> – memory used by objects after collection (KB)
      [memory ] <heap> – size of heap after collection (KB)
      [memory ] <total> – total time of collection (milliseconds)

      However, there is no way to conclude whether the soft/weak/phantomly reachable objects are removed using the verbose messages. If you suspect that these objects are still around when OOM was thrown, contact the JVM vendor.

      If the garbage collection algorithm is a generational algorithm (gencopy or gencon in case of Jrockit and the default algorithm in case of other JDKs), you will also see verbose output something like this:

      [memory ] 2.414: Nursery GC 31000K->20760K (75776K), 0.469 ms

      The above is the nursery GC (or young GC) cycle which will promote live objects from nursery (or young space) to old space. This cycle is not important for our analysis. More details on generational algorithms can be found in JVM documentation.

      If the GC cycle doesn’t happen before java OOM, then it is a JVM bug.

      Full compaction:

      Make sure that the JVM does proper compaction work and the memory is not fragmented which could prevent large objects being allocated and trigger a java OOM error.

      Java objects need the memory to be contiguous. If the available free memory is fragmented, then the JVM will not be able to allocate a large object, as it may not fit in any of the available free chunks. In this case, the JVM should do a full compaction so that more contiguous free memory can be formed to accommodate large objects.

      Compaction work involves moving of objects (data) from one place to another in the java heap memory and updating the references to those objects to point to the new location. JVMs may not compact all the objects unless if there is a need. This is to reduce the pause time of GC cycle.

      We can check whether the java OOM is due to fragmentation by analyzing the verbose gc messages. If you see output similar to the following where the OOM is being thrown even whether there is free java heap available, then it is due to fragmentation.

      [memory ] 8.162: GC 73043K->72989K (131072K) in 12.938 ms
      [memory ] 8.172: GC 72989K->72905K (131072K) in 12.000 ms
      [memory ] 8.182: GC 72905K->72580K (131072K) in 13.509 ms
      java.lang.OutOfMemoryError

      In the above case you can see that the max heap specified was 128MB and the JVM threw OOM when the actual memory usage is only 72580K. The heap usage is only 55%. Therefore, the effect of fragmentation in this case is to throw OOM even when there is 45% of free heap. This is a JVM bug or limitation.You should contact the JVM vendor.

2. If the JVM does its work properly (all the things mentioned in the above step), then the java OOM could be an application issue. The application might be leaking some java memory constantly, which may cause this problem. Or, the application uses more live objects and it needs more java heap memory. The following things can be checked in the application:
   • Caching in the application – If the application caches java objects in memory, then we should make sure that this cache is not growing constantly. There should be a limit for the number of objects in the cache. We can try reducing this limit to see if it reduces the java heap usage.Java soft references can also be used for data caching as softly reachable objects are guaranteed to be removed when the JVM runs out of java heap.
   • Long living objects – If there are long living objects in the application, then we can try reducing the life of the objects if possible. For example, tuning HTTP session timeout will help in reclaiming the idle session objects faster.
   • Memory leaks : One example of memory leak is when using database connection pools in application server. When using connection pools, the JDBC statement and resultset objects must be explicitly closed in a finally block. This is due to the fact that calling close() on the connection objects from pool will simply return the connection back to the pool for re-use and it doesn’t actually close the connection and the associated statement/resultset objects.It is recommended to follow the coding practices suggested in the following documents to avoid memory leaks in your application.JDBC – http://e-docs.bea.com/wls/docs81/jdbc/troubleshooting.html#1026696JNDI – http://e-docs.bea.com/wls/docs81/jndi/jndi.html#472853

     JMS – http://e-docs.bea.com/wls/docs81/jms/implement.html#1194127

   • Increase the java heap – We can also try increasing the java heap if possible to see whether that solves the problem.
   • Workaround – As a temporary work around, the application may be gracefully re-started when the java heap usage goes about 90%. When following this work around, the java max heap can be set to as high as possible so that the application will take more time to fill all the java heap. The java heap usage can be monitored by adding ‘-verbosegc’ flag in the java command line which will send the GC/ heap usage info to stdout or stderr.
3. If none of the above suggestion is applicable to the application, then we need to use a JVMPI (JVM Profiler Interface) based profiler like Jprobe or OptimizeIt to find out which objects are occupying the java heap. The profilers also give details on the place in the java code from where these objects are being created. This document doesn’t cover the details on each profiler. The profiler documentation can be referred to understand how to set and start the application with this profilers. In general, JVMPI based profilers have high overhead and drastically reduce the performance of the application. Therefore, it is not advisable to use these profilers in production environments.http://www.borland.com/optimizeithttp://www.quest.com/jprobe

Top of Page

For Native OOM Problem

1. Collect the following information:
   1. �verbosegc output to monitor the java heap usage. This will help to understand the java memory requirement for this application.It should be noted that independent of the actual java heap usage by the application, the amount of max heap specified (using �Xmx flag in the java command line) is reserved at the JVM startup and this reserved memory is not available for any other purpose.In case of Jrockit, use -verbose instead of -verbosegc as this gives codegen information in addition to GC information.
   2. Record the process virtual memory size periodically from the time the application was started until the JVM runs out of native memory. This will help to understand whether the process really hits the size limitation on that OS.In case of Windows, use the following procedure to monitor the virtual process size:
      1. 1. In the Start -> Run\dialog, enter “perfmon” and click OK.
         2. In the “Performance” window that pops up, click on the “+” button (above the graph).
         3. Select the following options in the resulting dialog:
            • Performance object: Process (not the default Processor)
            • Select counter from list: Virtual Bytes
            • Select instances from list: Select the JVM (java) instance
            • Click “Add”, then “Close”

      In case of Unix or Linux, for a given PID, the virtual memory size can be found using this command : ps -p <PID> -o vsz.

      In case of Linux, each java thread within a single JVM instance is shown as a separate process. It is enough if we take the PID of the root java process. The root java process can be found using the �forest option forps command. For example, ps �lU <user> --forest will give a ASCII tree art for all the processes started by the specified user. You can find the root java from the tree art.

2. Memory availability in the machineIf the machine doesn’t have enough RAM and swap space, then the OS will not be able to give more memory to this process that could also result in out of memory. Make sure that the sum of RAM and swap space in the disk is sufficient to cater to all the running processes in that machine.
3. Tuning java heapIf the java heap usage is well within the max heap, then reducing the java max heap will give more native memory to the JVM. This is not a solution but a workaround that can be tried. Since the OS limits the process size, we need to strike a balance between the java heap and the native heap.
4. Native memory usage by the JVMThe amount of native memory usage by the JVM is expected to flatten out after all the classes are loaded and the methods have been called (code generation is over). This usually happens within the first few hours for most of the applications. After that, the JVM uses only little native memory that may be due to run time class loading, code generation due to optimization etc.In order to narrow down the problem, try disabling run time optimizations and check whether that makes any difference.
   • In case of Jrockit, -Xnoopt flag can be used to disable run time optimizations.
   • In case of SUN hotspot JVM, -Xint flag will force the JVM to run in interpreted mode (no code generation).

   If the native memory usage continues to grow constantly throughout the run, then this could be a memory leak in the native code.

5. Third party native modules or JNI code in the applicationCheck whether you are using any third party native module like database drivers. These native modules could also allocate native memory and the leak may be from these modules. In order to narrow down the problem, you should attempt to reproduce the problem without these third party modules. For example, you can use pure java drivers instead of native database drivers.Check whether your application uses some JNI code. This could also be causing native memory leak and you can try to run the application without the JNI code if possible.
6. If the source of native memory cannot be found after the above steps, then you need to work with the JVM vendor to get a special build which can trace the native memory allocation calls and give more information about the leak.

Top of Page

The following URL gives some tools and tips specific to OOM situations with HP JVM: HP JVM tools/tips

Jrockit 8.1 SP1 and above supports JRA recording (Java Runtime Analyzer). This is useful to gather information at JVM run time which will give information about the application like, number of GCs running, number of soft/weak/phantom references, hot methods etc. It is useful to make a recording for few minutes and analyze the data if the JVM has performance problems or hang problems. More details on this can be found in Jrockit docs.

http://edocs.bea.com/wljrockit/docs142/usingJRA/jra.html

• Sun Java Web Site
• Sun Java Newsgroups
• JRockit Documentation
• dev2dev (JRockit)
• JRockit newsgroup

Linux System and Performance Monitoring(Memory篇)
Date: 2009.07.21
Author: Darren Hoch
译: Tonnyom[AT]hotmail.com

5.0 Virtual Memory介绍

虚拟内存就是采用硬盘对物理内存进行扩展,所以对可用内存的增加是要相对在一个有效范围内的.内核会写当前未使用内存块的内容到硬盘上,此时这部分内存被用于其它用途.当再一次需要原始内容时,此时再读回到内存中.这对于用户来说,是完全透明的;在Linux 下运行的程序能够看到,也仅仅是大量的可用内存,同时也不会留意到,偶尔还有部分是驻留在磁盘上的.当然,在硬盘上进行读和写,都是很慢的(大约会慢上千倍),相对于使用真实内存的话,因此程序无法运行的更快.用硬盘的一部分作为Virtual Memory,这就被称为”swap space”(译注:交换空间).

5.1 Virtual Memory Pages

虚拟内存被分为很多 pages(译注:页),在X86架构中,每个虚拟内存页为 4KB.当内核写内存到磁盘或者读磁盘到内存,这就是一次写内存到页的过程.内核通常是在swap 分区和文件系统之间进行这样的操作.

5.2 Kernel Memory Paging

内存分页在正常情况下总是活跃的,与memory swapping(译注:内存交换)之间不要搞错了.内存分页是指内核会定期将内存中的数据同步到硬盘,这个过程就是Memory Paging.日复一日,应用最终将会消耗掉所有的内存空间.考虑到这点,内核就必须经常扫描内存空间并且收回其中未被使用的内存页,然后再重新分配内存空间给其他应用使用.

5.3 The Page Frame Reclaim Algorithm(PFRA)(译注:页框回收算法)

PFRA 就是OS 内核用来回收并释放内存空间的算法.PFRA 选择哪个内存页被释放是基于内存页类型的.页类型有以下几种:

Unreclaimable –锁定的，内核保留的页面
Swappable –匿名的内存页
Syncable –通过硬盘文件备份的内存页
Discardable –静态页和被丢弃的页

除了第一种（Unreclaimable）之外其余的都可以被PFRA进行回收.

与PFRA 相关的,还包括kswapd 内核线程以及Low On Memory Reclaiming(LMR算法) 这2种进程和实现.

5.4 kswapd

kswapd 进程负责确保内存空间总是在被释放中.它监控内核中的pages_high和pages_low阀值.如果空闲内存的数值低于 pages_low,则每次 kswapd 进程启动扫描并尝试释放32个free pages.并一直重复这个过程,直到空闲内存的数值高于 pages_high.

kswapd 进程完成以下几个操作:

1,如果该页处于未修改状态,则将该页放置回空闲列表中.
2,如果该页处于已修改状态并可备份回文件系统,则将页内容写入到磁盘.
3,如果该页处于已修改状态但没有任何磁盘备份,则将页内容写入到swap device.

# ps -ef | grep kswapd
root 30 1 0 23:01 ? 00:00:00 [kswapd0]

5.5 Kernel Paging with pdflush

pdflush 进程负责将内存中的内容和文件系统进行同步操作.也就是说,当一个文件在内存中进行修改后, pdflush 将负责写回到磁盘上.

# ps -ef | grep pdflush
root 28 3 0 23:01 ? 00:00:00 [pdflush]
root 29 3 0 23:01 ? 00:00:00 [pdflush]

当内存中存在10% 的脏页,pdflush 将被启动同步脏页回文件系统里.这个参数值可以通过 vm.dirty_background_ratio 来进行调整.

(译注:
Q:什么是脏页?
A:由于内存中页缓存的缓存作用,写操作实际上都是延迟的.当页缓存中的数据比磁盘存储的数据还要更新时,那么该数据就被称做脏页.)

# sysctl -n vm.dirty_background_ratio
10

在多数环境下,Pdflush与PFRA是独立运行的,当内核调用LMR时,LMR 就触发pdflush将脏页写入到磁盘里.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
在2.4 内核下,一个高负荷的内存环境中,系统将遇到交换过程中不断的崩溃.这是因为PFRA 从一个运行进程中,偷取其中一个内存页并尝试使用.导致结果就是,这个进程如果要回收那个页时,要是没有就会尝试再去偷取这个页,这样一来,就越来越糟糕了.在2.6 内核下,使用”Swap token”修复了这个BUG,用来防止PFRA 不断从一个进程获取同一个页.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

5.6 案例学习:大量的入口I/O

vmstat 工具报告里除了CPU 使用情况,还包括了虚拟内存.以下就是vmstat 输出中关于虚拟内存的部分:

Table 2: The vmstat Memory Statistics
Field Description
Swapd The amount of virtual memory in KB currently in use. As free memory reaches low thresholds, more data is paged to the swap device.
当前虚拟内存使用的总额(单位:KB).空闲内存达到最低的阀值时,更多的数据被转换成页到交换设备中.
Free The amount of physical RAM in kilobytes currently available to running applications.
当前内存中可用空间字节数.
Buff The amount of physical memory in kilobytes in the buffer cache as a result of read() and write() operations.
当前内存中用于read()和write()操作的缓冲区中缓存字节数
Cache The amount of physical memory in kilobytes mapped into process address space.
当前内存中映射到进程地址空间字节数
So The amount of data in kilobytes written to the swap disk.
写入交换空间的字节数总额
Si The amount of data in kilobytes written from the swap disk back into RAM.
从交换空间写回内存的字节数总额
Bo The amount of disk blocks paged out from the RAM to the filesystem or swap device.
磁盘块页面从内存到文件或交换设备的总额
Bi The amount of disk blocks paged into RAM from the filesystem or swap device.
磁盘块页面从文件或交换设备到内存的总额

以下 vmstat 的输出结果,就是演示一个在I/O 应用中,虚拟内存在高负荷情况下的环境

# vmstat 3
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
3 2 809192 261556 79760 886880 416 0 8244 751 426 863 17 3 6 75
0 3 809188 194916 79820 952900 307 0 21745 1005 1189 2590 34 6 12 48
0 3 809188 162212 79840 988920 95 0 12107 0 1801 2633 2 2 3 94
1 3 809268 88756 79924 1061424 260 28 18377 113 1142 1694 3 5 3 88
1 2 826284 17608 71240 1144180 100 6140 25839 16380 1528 1179 19 9 12 61
2 1 854780 17688 34140 1208980 1 9535 25557 30967 1764 2238 43 13 16 28
0 8 867528 17588 32332 1226392 31 4384 16524 27808 1490 1634 41 10 7 43
4 2 877372 17596 32372 1227532 213 3281 10912 3337 678 932 33 7 3 57
1 2 885980 17800 32408 1239160 204 2892 12347 12681 1033 982 40 12 2 46
5 2 900472 17980 32440 1253884 24 4851 17521 4856 934 1730 48 12 13 26
1 1 904404 17620 32492 1258928 15 1316 7647 15804 919 978 49 9 17 25
4 1 911192 17944 32540 1266724 37 2263 12907 3547 834 1421 47 14 20 20
1 1 919292 17876 31824 1275832 1 2745 16327 2747 617 1421 52 11 23 14
5 0 925216 17812 25008 1289320 12 1975 12760 3181 772 1254 50 10 21 19
0 5 932860 17736 21760 1300280 8 2556 15469 3873 825 1258 49 13 24 15

根据观察值,我们可以得到以下结论:

1,大量的disk pages(bi)被写入内存,很明显在进程地址空间里,数据缓存(cache)也在不断的增长.

2,在这个时间点上,空闲内存(free) 始终保持在17MB,即使数据从硬盘读入而在消耗RAM.

3,为了维护空闲列表, kswapd 从读/写缓存区(buff)中获取内存并分配到空闲列表里.很明显可以看到buffer cache(buff) 在逐渐的减少中.

4, 同时kswapd 进程不断的写脏页到swap device(so)时,很明显虚拟内存的利用率是在逐渐的增加中(swpd).

5.7 结论

监控虚拟内存性能由以下几个部分组成:

1,当系统中出现较少的页错误,获得最好的响应时间,是因为memory caches(译注:内存高速缓存)比disk caches更快(译注:磁盘高速缓存).

2,较少的空闲内存,是件好事情,那意味着缓存的使用更有效率.除非在不断的写入swap device和disk.

3,如果系统不断报告,swap device总是繁忙中,那就意味着内存已经不足,需要升级了.