VMware vSphere Issue : MAINSYS Full
June 23, 2011 
Sean Crookston 
I encountered a weird issue today that is documented well at http://www.vm-help.com/esx41/full_root_fs.php
It involves not being unable to make changes to a vSphere host as result of the MAINSYS filling up. I was surprised to find out this had filled up and when searching for the root cause found IPMI was causing the issue.
Looking on the web I found that deleting the sel and sel.raw files in the /var/log/ipmi/0 folder would clear up the space in the MAINSYS. Following that up with a restart of the management agents resulted in being able to delete the temporary vSwitch that was previously created.
The problem is likely not solved there though as I’ve read numerous reports of IPMI continuing to fill the MAINSYS. It seems this is a flaw in the way IPMI is enabled in that there is no mechanism to clear out older entries to avoid the MAINSYS from filling up. The only recourse I had to ensure this would not continue to fill up was to configure one of the advanced settings, vmkernel.boot.ipmi.enabled and set it to no.
I’d be interested to know if any one else out there has seen this first hand and if they have had the issue reoccur over time. This is something I would hope is fixed with the release of vSphere 5 this fall.
Tags: 
We had this issue a couple of weeks ago. Created a new resource pool, and DRS failed almost instantly (“unable to apply settings” error). Recommended fix was to restart the management agents, but the first host this was done on they didn’t come back. Support call raised, and turned out that MAINSYS was full. Hence no logs either. Cue painful command line re-inventorying of all VMs on the host. The only curveball in our situation is we engaged Operation Footbullet when we built these hosts with only a 6GB san-boot disk.
Issue is yet to reoccur.
Thanks for the comment. I think one important thing I should have mentioned above is that the MAINFS is 32MB, which really suprised me. With disk pretty much being underutilized on ESXi hosts why not go a little bigger on there? I’d also like to see a cleanup mechanism to prevent this type of situation from ever happening in the next update/release.
We had the same problem with a SuperMicro X8DTL board, disabling ipmi in VMware seems to have solved it for now.