VMFS3HeapSize Problems
A Customer called me yesterday and told me that he had certain problems with his VMFS datastores.
He explained the following behavior:
- Cannot migrate VMs from one host to another host
- Cannot power on or off VMs
- VMDK files shows 0gb
We done a remote session and I went through the vmkernel.log. I found the following error messages:
WARNING: Heap: 2900: Heap_Align(vmfs3, 524288/524288 bytes, 8 align) failed. caller: 0x41801ccfae74 WARNING: Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand. WARNING: Heap: 2900: Heap_Align(vmfs3, 524288/524288 bytes, 8 align) failed. caller: 0x41801ccfae74 WARNING: Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
I checked the datastores and saw that there are several stores with more than 2tb (1 with 10tb, 1 with 3tb and 2-4 with 2 tb).
He told me that the complete storage box is about 20tb.
I found a KB article which describes the problem. After reading Heap Size in this article I could remember that I read about this problem some month ago when Michael Webster blogged about the topic The Case for Larger Than 2TB Virtual Disks and The Gotcha with VMFS.
After I know the problem and how to fix it, I wanted to know which impact it would have when Heap Size runs out of space. I found a blog post at VirtuallyHyper from Karim Elatov which describes how to get the currently used Heap Size.
There are two ways to get the used and configured Heap Size. According to VirtuallyHyper you can use the following:
At the ESXi Console type:
vsish cat /system/heaps/vmfs3-xxxxxxxxxx/stats
After this inputs you get a similar output:
Heap stats {
Name:vmfs3
dynamically growable:1
physical contiguity:MM_PhysContigType: 1 -> Any Physical Contiguity
lower memory PA limit:0
upper memory PA limit:-1
may use reserved memory:0
memory pool:1055
# of ranges allocated:2
dlmalloc overhead:1008
current heap size:37311424
initial heap size:2097152
current bytes allocated:10889344
current bytes available:26422080
current bytes releasable:24727456
percent free of current size:70
percent releasable of current size:66
maximum heap size:671088640
maximum bytes available:660199296
percent free of max size:98
lowest percent free of max size ever encountered:98
The output shows that the configured maximum Heap Size is 640MB, which can also be checked by typing esxcfg-advcfg -g /VMFS3/MaxHeapSizeMB at the ESXi console. I’m using a fresh installed ESXi and as you can see VMware changed the default MaxHeapSizeMB to it’s maximum of 640MB. In the output you can also see the utilization of the heap size.
There are a second methode of showing the heap size. You can type
memstats -r heap-stats -umb | grep "\(vmfs\)\|\(size\)"
and the the following output
Now I know how much Heap Size the ESX is using but I still didn’t now what will happen if the Heap Size runs out of memory.
I decided to twitter to Cormag Hogan (@vmwarestorage) to ask for an advice and only few seconds later he sent me a post from his blog with the required information.
Shame on me I haven’t read this post before. 😉