ERROR: "The operation is not allowed in the current state of the host"

I experienced the above error while powering on virtual machines after a scheduled shutdown. Just so you can replicate my experience, I was administering vCenter v5.5 Update 2b and ESXi v5.5 Update 2 hosts. The cluster's HA was turned off, DRS was set to partially automated from its fully automated setting, and all ESXi hosts were put into maintenance mode prior to them shutting down.

Powering up there were no issues -- all VMFS datastores were mounted and there are no alarms nor alerts. So after checks and everything was all set, ESXi hosts were exited from maintenance mode and virtual machines were started.

Most of the virtual machines booted back. But some of those that were powered up did not proceed on booting but instead prompted the error: "The operation is not allowed in the current state of the host."

This was completely new to me. And VMware forums and public knowledge base had an almost similar error: "The operation is not allowed in the current connection state of the host." Which was not quite what I was facing.

I was using the vSphere Web Client, so I tried using the old graphical user interface. Things did not seem to change -- same error.

I restarted the VMware VirtualCenter Server service but I still ended up with the same error. So I started to scratch my head. Of the several virtual machines that manifested this error, one thing was common: they belonged to similar ESXi hosts.

I went back to reading the posts with the almost identical error and it suggested to restart the management agents on the ESXi hosts. So I tried that but instead of just the management agents, I executed a stop and start of the entire services.sh.

Below are the traces of the execution.

~ # services.sh stop
Running hp-ams.sh stop
Terminating hp-ams-watchdog process with PID 35513 50567 50568
Stopping process 35479 35477 ...
hpHelper process is now stopped.
Running vmware-fdm stop
Stopping vmware-fdm:success
Running hp-mst.init stop
Unable to unload module mst: Module not found
Running xorg stop
Running sfcbd stop
This operation is not supported.
Please use /etc/init.d/sfcbd-watchdog stop
Running wsman stop
Stopping openwsmand
Running snmpd stop
Running sfcbd-watchdog stop
Running hpcru.init stop
Running hpilo.init stop
Running hpsmx.init stop
Running vpxa stop
watchdog-vpxa: Terminating watchdog process with PID 34561
vpxa stopped.
Running vobd stop
watchdog-vobd: Terminating watchdog process with PID 33332
vobd stopped
Running lacp stop
watchdog-net-lacp: Terminating watchdog process with PID 33542
Running memscrubd stop
memscrubd is not running
Running smartd stop
watchdog-smartd: Terminating watchdog process with PID 34425
smartd stopped
Running dcbd stop
watchdog-dcbd: Terminating watchdog process with PID 34370
Running cdp stop
watchdog-cdp: Terminating watchdog process with PID 34328
Running nscd stop
watchdog-nscd: Terminating watchdog process with PID 34283
Running slpd stop
Stopping slpd
Running storageRM stop
watchdog-storageRM: Terminating watchdog process with PID 34231
storageRM stopped
Running hostd stop
watchdog-hostd: Terminating watchdog process with PID 34191
hostd stopped.
Running vmfstraced stop
watchdog-vmfstracegd: PID file /var/run/vmware/watchdog-vmfstracegd.PID does not exist
watchdog-vmfstracegd: Unable to terminate watchdog: No running watchdog process for vmfstracegd
vmfstracegd is not running
Failed to clear vmfstracegd memory reservation
Running lbtd stop
watchdog-net-lbt: Terminating watchdog process with PID 34130
net-lbt stopped
Running sdrsInjector stop
watchdog-sdrsInjector: Terminating watchdog process with PID 34094
sdrsInjector stopped
Running rhttpproxy stop
watchdog-rhttpproxy: Terminating watchdog process with PID 34053
rhttpproxy stopped.
Running sensord stop
sensord is not running
Running hp-oem.init stop
Running usbarbitrator stop
watchdog-usbarbitrator: Terminating watchdog process with PID 33941
usbarbitrator stopped
Running DCUI stop
Disabling DCUI logins
VobUserLib_Init failed with -1
Running ntpd stop
Stopping ntpd
watchdog-ntpd: Terminating watchdog process with PID 33876
Connect to localhost failed: Connection failure
Running vsantraced stop
watchdog-vsantraced: Terminating watchdog process with PID 33807
vsantraced stopped
watchdog-vsantracedUrgen: Terminating watchdog process with PID 33836
vsantracedUrgen stopped
Persisting traces to /locker/vsantraces
~ #
~ # services.sh start
Running vsantraced start
Scratch partition is not backed by persistent storage
Storing traces to /vmfs/volumes/4dc4866f-040de4b6-1f88-0017a477f825/vsantraces
vsantraced started
vsantracedUrgen started
Running ntpd start
Connect to localhost failed: Connection failure
Starting ntpd
Running DCUI start
Enabling DCUI login: runlevel = 
VobUserLib_Init failed with -1
Running usbarbitrator start
usbarbitrator started
Running hp-oem.init start
Running sensord start
sensord started
Running rhttpproxy start
rhttpproxy started.
Running sdrsInjector start
sdrsInjector started
Running lbtd start
net-lbt started
Running vmfstraced start
VMFS Global Tracing is not enabled.
Running hostd start
Ramdisk 'hostd' with estimated size of 803MB already exists
hostd started.
Running storageRM start
storageRM started
Running slpd start
Starting slpd
Running nscd start
nscd started
Running cdp start
cdp started
Running dcbd start
dcbd started
Running smartd start
smartd started
Running memscrubd start
The checkPages boot option is FALSE, hence memscrubd could not be started.
Running lacp start
LACP daemon started
Running vobd start
vobd started
Running vpxa start
Connect to localhost failed: Connection failure
Running hpsmx.init start
Running hpilo.init start
Running hpcru.init start
Running sfcbd-watchdog start
Connect to localhost failed: Connection failure
Connect to localhost failed: Connection failure
Running snmpd start
Running wsman start
Starting openwsmand
Running sfcbd start
This operation is not supported.
Please use /etc/init.d/sfcbd-watchdog start
Running xorg start
Running hp-mst.init start
Running vmware-fdm start
Starting vmware-fdm:success

Running hp-ams.sh start
Starting hpHelper service...
~ # 

One thing to note here is that during the stop and start of services.sh vCenter server will complain about the host being disconnected. This is normal as the daemons handshaking and talking to each other lose connectivity.

After the stop and start was done, powering up of the virtual machines that had errors were successfully executed. Another point chalked to experience.

You might also be interested in:

Feedback

We at pimp-my-rig strive to keep on improving, help us reach that goal by leaving comments or constructive criticisms. Don't miss out on our next feature -- subscribe via RSS (What is RSS?).

Share This

0 comments: