Patching\Upgrading ESXi 7 to ESXi7U1 via esxcli

With the latest release of ESXi7U1 i wanted to get my lab up to date. I dont have enough resources in my lab to migrate the vCenter to another ESXi server so i want to perform the upgrade via cli. More details about the release can found here

The first step was to open the firewall for outgoing traffic for http

esxcli network firewall ruleset set -e true -r httpClient

Second step was to list the updates by executing

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep -i ESXi-7

This returned a list of updates that were available:

I reviewed the downloads website here to double check the version. Based on the information i found the ESXi-7.0.1-16850804-standard is the latest release

The next step was to run the update by executing

esxcli software profile update -p ESXi-7.0.1-16850804-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

If you run in to the below error follow my other post here:

Once the upgrade was complete i was presented with this output:

The installation reports that a reboot is not required however the new version wasnt reflected in my vcenter so i went ahead and rebooted the server

After the server was back up, checking in vCenter, the server reports build 16850804

ESXi No space left on device error when upgrading

While i was trying to perform an upgrade to my ESXi installation i ran in to the below error:

The fix was pretty simple. The hypervisor doesnt have enough space to download and install the package so we need to enable one of the available datastores for the function.

For vCenter go to the host in question to Configure -> System -> System Swap

While reviewing the configuration i noticed that i had 2 options enabled

Clicking on the edit button on the right corner i noticed that i can add one of the datastores

After i Clicked ok i was able to continue with my upgrade

Upgrading VCF 4.0.1.1 to 4.1.0.0

With the release of of VCF 4.1 i wanted to get my lab upgraded. The release blog can be found here and the release notes are here

In my case i pre downloaded the VMware Cloud Foundation Update 4.1.0.0 by going to Repository -> Bundle management -> Download now

The next step is to upgrade VCF by going to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Update Now

Next we are taken to the Upgrade page where we can follow the upgrade for each one of the components

Once the upgrade is complete we can click Finish to be returned back to the main screen

Because we are changing the SDDC-Manager version i would strongly recommend to clear cache and log back in before going forward.

The next update is the configuration drift bundle. We can go to nventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Download now

Once the download is complete click on update now

Once the upgrade started i got redirected tot he Update status page. Considering the update is only 204 MB the upgrade went through pretty quickly. Once its completed we can click finish to get back to the main sddc manager page

Next step is to upgrade NSX-T installation to NSX-T 3.0.2. Click on Download now from the same page as above

Once the download is complete click on Update Now

We can view the status and the steps by clicking on View Status. Once the upgrade is complete we are redirected back to the available upates page showing that the vCenter server is next

Click on Download now and wait for the download to complete. Once the download is complete click on update now

We can view the task by clicking on View Status

Once the upgrade is complete we are taken back to the previous page where we can see that the ESXi servers are next. Click on Download Now

Once the download is complete we can click on Update now

If we have multiple clusters we can enable Cluster-level selection and select the specific luster(s) we want to upgrade.

We can also enable sequential cluster upgrade as well as quick boot

We get to review the options once again before we click finish to to submit the task

Once submitted we can view the status by clicking on View Status

And with that we are finished with the workload domain. We can get back to the Update/Patches page to see that there are no more updates available

Next is the Workload domains where we can follow the same instructions as above. The process will be allot quicker because the upgrades are already downloaded

Unable to Sync Lifecycle Manager ‘integrity.fault.HostPatchInvalidVendorCode’

After upgrading my vCenter Server to the latest vSphere 7.0 Update 1, I was unable to see ESXi 7.0 Update 1 Image. When I attempted to sync the updates I got the error “Download patch definitions task failed while syncing depots. Error: ‘integrity.fault.HostPatchInvalidVendorCode’.”

Seeing as the error mentioned the depots, navigate to Menu->Lifecycle Manager and then select the Settings tab and then Patch Setup.

Select the radio button for Partner provided Addons for ESXi and click Disable. Proceed to click on Action and then Sync Updates. The sync should now complete successfully.

vCLS not starting with Insufficient resources message

With the release of vCenter 7 Update 1, VMware introuced the vCLS (vSphere Clustering Service). More information can be found here.

Looking at the error details it looks like it is looking for a feature called cpuid.mwait

Reviewing the VMX file it seems like EVC is enabled on the vCLS VMs. I didnt want to enable EVC on the whole cluster so i wanted to do it only on the specific VMs.

Doing some research i found that the VMs need to be at version 14. After upgrading the VM i was able to disable EVC on the specific VMs by following these steps:

In the vSphere Client, navigate to the virtual machine

Under the Actions -> Compatibility -> Upgrade VM compatibility

We can disable EVC on per VM level on version 14 and above, so in my case i chose ESXi 6.7 and later

Next go to the Configure Tab

Pick VMware EVC and click on Edit

Click on Yes

Click on Disable EVC and Click OK

The next time it tries to power on the VM it should go through.

Once the first VM starts up it will most likely deploy a few additional ones, follow the same steps as above again on the new VMs

Extracting SSL Thumbprint

I recently ran in to an issue where i had to re-register my NSX server with vIDM.

The ask was to extract the Thumbprint from vIDM. The command i ran to extract it was:

echo -n | openssl s_client -connect hostname:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256

This can be used across multiple products where the Thumbprint needs to be extracted

vRealize Operation Endpoint Agent (EPops) Unable to register error

While i was doing the operations endpoint install on a new vROPS environment i experienced an strange error that i didnt see before 

- Unable to register the agent due to server error.

In order to find the issue i had to log in to my vROPS server in order to review the logs. I was able to find the End Point Adapter log file here:

/storage/log/vcops/log/adapters/EndPointAdapter/

While reviewing the log i found a strange error pointing me to the adapter

2020-09-27T12:12:01,514 ERROR [http-nio-127.0.0.1-8877-exec-4] (6) com.vmware.vcops.aim.agent.webserver.AgentAdapterController.handleCommand - Error when executing the agent command REGISTERAGENT Token:1601208564446-9168989700718889387-1917899764042627947. Failed to create agent instance. Reason:Failed to create resource: message=ResourceKind is not found: {adKind=EP Ops Adapter, resKind=EP Ops Agent}, localizedMessage=ResourceKind is not found: {adKind=EP Ops Adapter, resKind=EP Ops Agent}, code=0; message=resourceKind is null for resourceKind Key: EP Ops Agent and adapterKind Key: EP Ops Adapter, code=0
com.vmware.vcops.aim.exception.AgentAdapterException: Token:1601208564446-9168989700718889387-1917899764042627947. Failed to create agent instance. Reason:Failed to create resource: message=ResourceKind is not found: {adKind=EP Ops Adapter, resKind=EP Ops Agent}, localizedMessage=ResourceKind is not found: {adKind=EP Ops Adapter, resKind=EP Ops Agent}, code=0; message=resourceKind is null for resourceKind Key: EP Ops Agent and adapterKind Key: EP Ops Adapter, code=0

I would seem that i forgot to enable the adapter for remote monitoring… Fortunately the solution was very easy. All i had to do was log in as an administrator user in my vROPs instance and Activate the Operating System / Remote Service Monitoring

After the solution was activated the agent successfully registered

- Testing secure connection ...
- Connection successful.
Enter your server username: admin
Enter your server password: **Not echoing value**
- Registering the agent with server.
- The agent has received a client certificate from server.
- The agent has been successfully registered.

Adding keys in vRealize Lifecycle Manager in a VCF 4 Environment

From a lifecycle manager perspective we have a couple of ways to add keys to the environment

If the my vmware user that is added to lifecycle manager has access to they keys, the keys can automatically discovered and imported. For this we can go to the locker from Home -> Locker -> License. If the keys are not discovered we can click on the refresh button up top to perform a manual sync

If the keys havent been imported or if we need to add a key manually we can click on the add button and complete the fields

Click on validate and add. This will add the key to the catalog and it will be consumable by lifecycle manager when we install the products

Deploying vRA with vRealize Lifecycle Manager in a VCF 4 Environment

If you followed my previous posts Deploying the vRealize suite using VCF 4.0 and vRealize Lifeycycle Manager (vRSLCM) enable product dowloads you should now have vRA and vIDM already downloaded.

In my previous post i went over the installation of vIDM which is a pre requisite for vRA. You can find the link here

Next, we need to create a new environment that will be used to deploy vRA. For this we can go to Lifecycle Operations -> Create Environment. Fill in the necessary information and click next

Select vRealize Automation, select the version and install type then click next

Review and accept the EULA then click Next

Next screen is the key. Click on Select and select the vRA key from the inventory. If we need to add the key manually we can follow the steps in my post here

Verify the key selection click on Validate Association and click on next

Select the certificate and click next. If a new certificate needs to be create follow the instructions on my post here

Make the proper selections for where the vRA server will get deployed and click Next

Fill in the proper network configuration and click next

Specify the proper network configuration for the product and click next

Run the precheck and verify that everything is green then click Next

Verify the summary and click submit

Next we are taken to the request details where we can follow the process that LCM is performing for us automatically.

We can see that the task completed

If we go to environments we can see that the vRA Environment

Deploying vIDM with vRealize Lifecycle Manager in a VCF 4 Environment

If you followed my previous posts Deploying the vRealize suite using VCF 4.0 and vRealize Lifeycycle Manager (vRSLCM) enable product dowloads you should now have vRA and vIDM already downloaded.

Because im running on a deployment from VCF i can see that the datacenter was already provioned for me under under Home -> Lifecycle Operations -> Datacenters

However it seems like my Environment is not completely configured. In order to configure we can go to Home -> Lifecycle Operations -> Create Environment

Add a new password to the vault that will be used by our installation wizards. We can add a password by clicking on the + sign next to the Default Password

Enter the password details and click add

Now i can select the new password by clicking on Select Default Password. I can select the Datacenter from the drop down and click Next.

Select the VMware Identify Manager and click Next

Accept the EULA and click Next

Select the certificate. If you do not have a certificate you can follow the instructions i have here. Click next

Select the proper details to where the server will be deployed

Fill in the network information and click next

Fill in the product information

Run the precheck and verify that everything is valid and click next

Verify the details in the Summary and click Submit

We are taken to the request details page

Once the deployment is complete we can see the vIDM server under Environments