VMFS3HeapSize Problems

A Customer called me yesterday and told me that he had certain problems with his VMFS datastores.
He explained the following behavior:

  • Cannot migrate VMs from one host to another host
  • Cannot power on or off VMs
  • VMDK files shows 0gb

We done a remote session and I went through the vmkernel.log. I found the following error messages:

WARNING: Heap: 2900: Heap_Align(vmfs3, 524288/524288 bytes, 8 align) failed.  caller: 0x41801ccfae74
WARNING: Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
WARNING: Heap: 2900: Heap_Align(vmfs3, 524288/524288 bytes, 8 align) failed.  caller: 0x41801ccfae74
WARNING: Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.

I checked the datastores and saw that there are several stores with more than 2tb (1 with 10tb, 1 with 3tb and 2-4 with 2 tb).
He told me that the complete storage box is about 20tb.
I found a KB article which describes the problem. After reading Heap Size in this article I could remember that I read about this problem some month ago when Michael Webster blogged about the topic The Case for Larger Than 2TB Virtual Disks and The Gotcha with VMFS.
After I know the problem and how to fix it, I wanted to know which impact it would have when Heap Size runs out of space. I found a blog post at VirtuallyHyper from Karim Elatov which describes how to get the currently used Heap Size.
There are two ways to get the used and configured Heap Size. According to VirtuallyHyper you can use the following:
At the ESXi Console type:

cat /system/heaps/vmfs3-xxxxxxxxxx/stats

After this inputs you get a similar output:

Heap stats {
 dynamically growable:1
 physical contiguity:MM_PhysContigType: 1 -> Any Physical Contiguity
 lower memory PA limit:0
 upper memory PA limit:-1
 may use reserved memory:0
 memory pool:1055
 # of ranges allocated:2
 dlmalloc overhead:1008
 current heap size:37311424
 initial heap size:2097152
 current bytes allocated:10889344
 current bytes available:26422080
 current bytes releasable:24727456
 percent free of current size:70
 percent releasable of current size:66
 maximum heap size:671088640
 maximum bytes available:660199296
 percent free of max size:98
 lowest percent free of max size ever encountered:98

The output shows that the configured maximum Heap Size is 640MB, which can also be checked by typing esxcfg-advcfg -g /VMFS3/MaxHeapSizeMB at the ESXi console. I’m using a fresh installed ESXi and as you can see VMware changed the default MaxHeapSizeMB to it’s maximum of 640MB. In the output you can also see the utilization of the heap size.
There are a second methode of showing the heap size. You can type

memstats -r heap-stats -umb | grep "\(vmfs\)\|\(size\)"

and the the following output
Now I know how much Heap Size the ESX is using but I still didn’t now what will happen if the Heap Size runs out of memory.
I decided to twitter to Cormag Hogan (@vmwarestorage) to ask for an advice and only few seconds later he sent me a post from his blog with the required information.
Shame on me I haven’t read this post before. 😉

Leave a Reply

Your email address will not be published. Required fields are marked *