Browse Category


Shutting down a vSAN Cluster

I have the need to completely shut down some of my vSAN clusters for various clusters and ive been having a hard time finding the proper procedure. As of 2/16/2021 VMware released guidance here

Here are the steps i took to do it on my end. If you have the vCLS service enabled follow my other instructions here prior to starting the rest of this guide.

Disable cluster member updates from vCenter on each ESXi host in the cluster by running

esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates

After the above is completed run the below only on 1 of the ESXi hosts. Take note of the host

python /usr/lib/vmware/vsan/bin/ prepare

Place all ESXi hosts in maintenance mode with NoAction

esxcli system maintenanceMode set -e true -m noAction

Perform the necessary maintenance. Once the hosts are back up we will run the above in reverse

Remove the maintence mode on all ESXi hosts by running

esxcli system maintenanceMode set -e false

Run the below command on the same host it was originally run

python /usr/lib/vmware/vsan/bin/ recover

vSAN availability can be checked by running

esxcli vsan cluster get

Enable cluster member updates for vCenter

esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

Removing NSX stale packages from ESXi host

I recently ran in to a problem where i wanted to perform a clean configuration of one of my ESXi hosts from an NSX perspective, however i ran in to a problem where NSX was reporting that the packages are already installed. To fix the issue i had to run the following to list the packages installed:

esxcli software vib list | grep -i nsx

Once i had the list all i had to do is uninstall them using:

esxcli software vib remove -n packagename1 -n packagename2 ...

Once the uninstall was complete i was able to redeploy NSX from the NSX Manager

Patching\Upgrading ESXi 7 to ESXi7U1 via esxcli

With the latest release of ESXi7U1 i wanted to get my lab up to date. I dont have enough resources in my lab to migrate the vCenter to another ESXi server so i want to perform the upgrade via cli. More details about the release can found here

The first step was to open the firewall for outgoing traffic for http

esxcli network firewall ruleset set -e true -r httpClient

Second step was to list the updates by executing

esxcli software sources profile list -d | grep -i ESXi-7

This returned a list of updates that were available:

I reviewed the downloads website here to double check the version. Based on the information i found the ESXi-7.0.1-16850804-standard is the latest release

The next step was to run the update by executing

esxcli software profile update -p ESXi-7.0.1-16850804-standard -d

If you run in to the below error follow my other post here:

Once the upgrade was complete i was presented with this output:

The installation reports that a reboot is not required however the new version wasnt reflected in my vcenter so i went ahead and rebooted the server

After the server was back up, checking in vCenter, the server reports build 16850804

ESXi No space left on device error when upgrading

While i was trying to perform an upgrade to my ESXi installation i ran in to the below error:

The fix was pretty simple. The hypervisor doesnt have enough space to download and install the package so we need to enable one of the available datastores for the function.

For vCenter go to the host in question to Configure -> System -> System Swap

While reviewing the configuration i noticed that i had 2 options enabled

Clicking on the edit button on the right corner i noticed that i can add one of the datastores

After i Clicked ok i was able to continue with my upgrade

vCLS not starting with Insufficient resources message

With the release of vCenter 7 Update 1, VMware introuced the vCLS (vSphere Clustering Service). More information can be found here.

Looking at the error details it looks like it is looking for a feature called cpuid.mwait

Reviewing the VMX file it seems like EVC is enabled on the vCLS VMs. I didnt want to enable EVC on the whole cluster so i wanted to do it only on the specific VMs.

Doing some research i found that the VMs need to be at version 14. After upgrading the VM i was able to disable EVC on the specific VMs by following these steps:

In the vSphere Client, navigate to the virtual machine

Under the Actions -> Compatibility -> Upgrade VM compatibility

We can disable EVC on per VM level on version 14 and above, so in my case i chose ESXi 6.7 and later

Next go to the Configure Tab

Pick VMware EVC and click on Edit

Click on Yes

Click on Disable EVC and Click OK

The next time it tries to power on the VM it should go through.

Once the first VM starts up it will most likely deploy a few additional ones, follow the same steps as above again on the new VMs

Extracting SSL Thumbprint

I recently ran in to an issue where i had to re-register my NSX server with vIDM.

The ask was to extract the Thumbprint from vIDM. The command i ran to extract it was:

echo -n | openssl s_client -connect hostname:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256

This can be used across multiple products where the Thumbprint needs to be extracted

vCenter update from the command line

Ive ran recently in to an issue where i couldnt perform an update from the admin UI, in order to get up and running i executed the following from a ssh shell. I was luky enough to find the documentation here

/usr/lib/applmgmt/support/scripts/ install --url

Some of the additional options that could be used

software-packages stage --url
software-packages install --staged

I like to perform the update from the command line because it gives me additional information that i might not be able to see in the UI

Typically the patching requires a reboot and the installation can be completed by executing the following:

shutdown reboot -r "patch reboot"

vCenter Appliance Update installation is in progress

I recently ran in to an issue where the vCenter server appliance update timed out. Rebooting the vCenter server brought the server back up but i was unable to log in to the administration section due to an Update installation in progress error

Doing some research i found that there is a file that holds this information fount at /etc/applmgmt/appliance/software_update_state.conf At this point all i had to do was edit the file using vi

vi /etc/applmgmt/appliance/software_update_state.conf

When opening the file i noticed that the state was INSTALL_IN_PROGRESS

Tge version was also different from what the console was showing

So i updated the file to reflect the actual version of the appliance. The end result looked like this:

Upgrading ESXi via command line

I recently ran in to a problem where i couldn’t update my esx server from the UI so i wanted to to it from the command line. After doing some research this is the way i was able to do it:

Step 1 was to open the firewall by running:

esxcli network firewall ruleset set -e true -r httpClient

Step 2 was to list the profiles available:

esxcli software sources profile list -d | grep -i ESXi-7

or if i already have a bundle downloaded available:

esxcli software sources profile list -d /vmfs/volumes/datastore/ | grep -i ESXi-7

Step 3 was to pick one of the releases and apply the update

esxcli software profile update -p ESXi-7.0b-16324942-standard  -d

or from the local zip file

esxcli software profile update -p ESXi-7.0b-16324942-standard  -d /vmfs/volumes/datastore/

Step 4 dont forget to close the firewall

esxcli network firewall ruleset set -e false -r httpClient

Install/Upgrade ESXi 7.0 on unsupported hardware.

As i was installing\upgrading my lab environment to ESXi 7.0 i received an error that the CPU was unsupported. As per the release notes vSphere 7.0 release notes my CPU is no longer on the compatibility list.

Since this is a lab environment i wanted to continue using my current hardware. It goes without saying that this method should not be used in a production environment.

To allow the legacy CPU all i had to do is boot up from the ISO, on the boot Menu i pressed TAB or Shift + O and added allowLegacyCPU=True option as seen in the screenshot below