Most modern operating systems utilize the paged virtual memory model. In this model each process has its own "virtual" address space and special page table is used to map each virtual memory address to the corresponding physical memory address.
Mapping of each byte of virtual memory to physical address would be very inefficient because for each allocated byte of we need to keep 4+ bytes entry in the page table. In practice memory is allocated by bigger chuncks called pages. Most common page size is 4096 bytes.
There is a trade-off between page size and page table size. For small pages we are wasting memory to keep huge page table. For big pages we're wasting memory on partially-used pages, because it is very rare when a memory allocation request corresponds to exact number of pages and last allocated page is only partially used.
Huge pages is a memory allocation mode when page size is bigger than 4kB. Actual page size depends on the OS and hardware platform and for most of x86 systems it can be either 2MB or 4MB. If a process operates with big blocks of memory and a computer has enough physical memory installed, then "huge pages" mode can significantly improve performance (up to 3x, according to some syntetic tests).
On Linux, if a process wants use "huge pages" support, it should use the libhugetlbfs OS API funtions to request it, i.e. program should be written and compiled with "huge pages" support in mind.
In order to bring "huge page" performance benefits to legacy software and software written without "libhugetlbfs" support, RedHat has implemented custom Linux kernel extension called "redhat_transparent_hugepage".
As I will show below, transparent huge pages is not always a good idea, even if you have more then enough physical memory installed.
To perform automatic Continious Integration builds of our Java software we're using a quite powerful server machine with 2 x Intel(R) Xeon(R) CPU X5660 @ 2.80GHz (2 CPU * 6 cores * 2[HT] = 24 threads), 36GB of memory and RHEL-based Scientific Linux with 2.6.32-279.9.1.el6.x86_64 kernel.
Despite the reasonable number of processors and only 4 concurrent builds running in parallel, the machine was showing a very high CPU average of 35% or more and was plagued with random freezes when almost everyting got blocked for 3-5 seconds.
After throughfull testing and analysis we noticed two interesting facts. First, CPU spends most of its time in the kernel mode. And second, for very short intervals of time "hugememd" daemon eats all CPU power.
Based on this symptoms, it was easy assumption that "hugememd" is strongly correlated with the Java performance degradation and system freezes, what led us to the existing bug report on the CentOS bugtracker.
And indeed, disabling "redhat_transparent_hugepage" extension eliminates the problem completely! Now average CPU rarely goes higher than 5% and server can keep up to 8 parallel Java builds.
It is still not clear to me if the problem is the way JVM manages memory what leads to huge "memory defragmentation" costs or if there is a bug in "redhat_transparent_hugepage" implementation, but the fact is that at the moment "redhat_transparent_hugepages" doesn't work well with Java.
This means that if you are experiencing surprisingly bad perfomance of your Java web server or database with accidental freezes or slowdowns, don't rush to blame Java. It may be your server OS doing some nasty "optimizations" behind you.
