Patching\Upgrading ESXi 7 to ESXi7U1 via esxcli

With the latest release of ESXi7U1 i wanted to get my lab up to date. I dont have enough resources in my lab to migrate the vCenter to another ESXi server so i want to perform the upgrade via cli. More details about the release can found here

The first step was to open the firewall for outgoing traffic for http

esxcli network firewall ruleset set -e true -r httpClient

Second step was to list the updates by executing

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep -i ESXi-7

This returned a list of updates that were available:

I reviewed the downloads website here to double check the version. Based on the information I found the ESXi-7.0U3f-20036589-standard is the latest release

The next step was to run the update by executing

esxcli software profile update -p ESXi-7.0U3f-20036589-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

If you run in to the below error follow my other post here:

Once the upgrade was complete i was presented with this output:

The installation reports that a reboot is required so I went ahead and rebooted the server

After the server was back up, checking the ESXi, the server reports build 20036589

Don`t forget to add the firewall rule back in

esxcli network firewall ruleset set -e false -r httpClient

Upgrading vRA (vRealize Automation) to 8.8.2

In this post i will go over upgrading my 8.x vRA appliance to 8.8.2. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8.2 Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRA

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8.2 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

A new feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors

The next screen remind us of hardware requirements for vRA and vIDM which can be checked here for vRA and vIDM which can be checked here. As far as i can tell they haven’t changed since the 8.3 release. Check the I took care of the manual steps above and am ready to proceed check mark and click on Run Precheck

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8.2

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vRA version.

Upgrading vRLI (vRealize Log Insight) to 8.8.2 via vRSLCM

In this post i will go over upgrading my 8.x vRLI appliance to 8.8.2 using vRSLCM (vRealize Suite Lifecycle Manager). As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8.2. Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. Make sure we select the upgrade package not the install package. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRLI

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

A new feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors or issues.

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next then Finish. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8.2

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vRLI version.

Upgrading SSC (SaltStack Config) to 8.8.2 using vRSLCM

In this post i will go over upgrading my 8.x SSC appliance to 8.8.2. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8.2. Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes SSC

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8.2 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

We can automatically create and delete a snapshot part of the upgrade process

Run the Precheck to make sure there are no errors

Once the check is complete, click on Next. Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8.2

Here are a few additional blogs that might be useful post upgrade:

Error Code: LCMVSSC10018

SSC 8.8 sseapi_rpc_queue: could not connect to SSE server

SSC 8.8 urllib3 (1.25.11) or chardet (4.0.0) doesn’t match a supported version

Authentication failed: no Authorization header

Upgrading vRSLCM (vRealize Lifecycle Manager) to 8.8.2

In this guide i will go over the steps of getting an existing 8.x vRSLCM appliance upgraded to the latest 8.8.2 release. The release notes can be found here

The first step is to log in to vRealize Suite Lifecycle Manager under the Lifecycle Operations section

Go to settings -> System Upgrade

Click on Check for Upgrade

We can see that the check found a new version available for 8.8.2

Click on Upgrade

Verify that a snapshot or backup exists in case the process fails. Check the check mark for I took a snapshot of the vRealize Suite Lifecycle Manager before I performed this operation. Click Next

Click on Run Precheck

Verify that all check have passed and click on upgrade

This will fire up the upgrade process and start upgrading packages. The system will automatically reboot on 8.8.2 once completed. We can check the version by going to Settings -> System Details

If you get the below error clear the browser cache and try again

vRealize Lifecycle Manager (vRSLCM) 8 certificate management

vRealize Lifeycycle Manager (vRSLCM) come with a Certificate Management feature. We can access the certificate management from Home -> Locker -> Certificate

We can generate Self Signed certificates for products managed by Lifecycle Manager as well as certificate requests to be signed by a certification authority

Generate a new CSR

Download the CSR and take it to the certified authority, in my case im using a Microsoft server /certsrv/certrqxt.asp

Click on advanced certificate requests and complete the request details

Download the certificate in the Base 64 encoded format

Next we can import it in to the certificate vault

Click on import and complete the details. The private key can be found in the certificate request file

Make sure you create a file that includes the signed certificate + the private key + the intermediate and root certificates.

The certificate can now be used

Deploying vRA 8.8 using the Easy Installer

In this guide i will go over the deployment of vRA 8.8 using the Easy Installer. The reason i like the Easy Installer is because it deploys everything that i need. vIDM, Lifecycle Manager as well as vRA.

To get started we need to download the installer from here.

Once downloaded mount the iso as a drive using your favorite tool or extract it as a archive using something like 7-zip

Once mounted go to the CDROM in the mac directory if on mac or win32 if you are on windows. In here run the installer file

We will be presented with the following window:

The migration option allows us to migrate from from vRSLCM 2.1 to 8.x, it can migrate Datacenter and vCenters, all existing environments, DNS, SNMP, NTP, My VMware, proxy details, migration of vIDM installation as well as Blackstone Content endpoints. For the purpose of this guide we will process with the Install option

Once we click Install we will be presented with an Introduction of what components will be included part on the installation. In my case i can see vRealize Automation 8.8, Lifecycle Manager 8.8, Identity Manager 3.3.6.

Click Next, Review, accept the agreement and click next

Provide the vCenter details and click next. In my case i used the administrator account. A detailed list of permissions needed for deployment can be found here

Sections 4, 5, 6 and 7 are straight forward so i will skip them

In section 8 select a password that will be used across all products for the local usernames. ex for LCM root and local admin, vIDM admin, sshuser, root etc

Select a VM name for the Lifecycle Manager appliance and its ip credentials. If you are expecting a large repository in lifecycle manager we can add additional disk under the Increase Disk Size in GB section.

The FIPS Mode compliance option enforces FIPS compliance, however keep in mind that with the FIPS mode enabled there are limitation on what the product can do. This can be disabled later and re enabled as needed.

Complete the fields and click next

The next portion is vIDM. In case there is an existing vIDM appliance in the environment we can also import and existing vIDM appliance. In my case i will deploy a new one.

One important option under vIDM is the Sync Group Members to the Directory When Adding Group. When enabled, members of the groups are synced when groups are added from Active Directory. When this is disabled, group names are synced to the directory, but members of the group are not synced until the group is entitled to an application or the group name is added to an access policy. Note: Post deployment this value cannot be changed from vRealize Suite Lifecycle Manager. To update this field post deployment, navigate to VMware Identity Manager

My configuration page looks like this:

The next section is the vRA Configuration. In here we have a couple of options. We can perform a standard 1 node deployment or a cluster deployment which includes 3 appliances. The FIPS Compliance mode enables FIPS compliance. Unlike LCM this mode cannot be disabled after the deployment. This disables a number of options in vRA from an LCM perspective. Please make sure that its only enabled if required.

The advanced configuration at the bottom of the page includes the option to change the internal Kubernetes cluster and Service ip range. This is useful if the default range is already in use on the internal network. We want to make sure we pick an ip range thats not used somewhere else in order to avoid routing issues. Once complete click next

The next page gives us a summary of our deployment and we can click submit to start the deployment.

Next we are presented with the installation process. We can follow it along, in my environment the full deployment took about 1.5 hrs

After the install is complete we are presented to links for the different services

Next i would recommend LCM certificate management found here

ccp-backups folder missing in NSX-T backup

If you like me tried to to cleanup the backups in NSX-T and ran into error Cleanup script works only in folders, that contains subfolders "cluster-node-backups", "ccp-backups" and "inventory-summary" this post is for you.

I was trying to cleanup the backups before going to the next major release of nsx and i kept getting an error running the nsx_backup_cleaner.py script.

It would seem that the ccp-backups folder has been removed from the backup job so it simply doesn’t exist. VMware did fix the script with the 3.2 release.

If you have anything preventing you from getting to 3.2 here is the updated script

#!/usr/bin/env python
# ***************************************************************************
# Copyright 2020-2021 VMware, Inc.  All rights reserved. VMware Confidential.
# ***************************************************************************
# The purpose of this script is to remove old NSX backup files. Typically, this script
# will be placed on the SFTP server where the NSX Manager is uploading backup files,
# and included into a scheduler, for example cron.  Before running this script, you
# should update the BACKUP_ROOT variable.  This script works on Linux and Windows with
# both Python 2 and Python 3.
#
# On Linux SFTP server:
# You can add this script in the crontab to automatically run this script once daily
# Edit the anacron at /etc/cron.d or use crontab -e and add following line to execute the script at 10am everyday
# 00 10 * * * /sbin/nsx_backup_cleaner.py
#
# On Windows SFTP server:
# schtasks /Create /SC DAILY /TN PythonTask /TR "PATH_TO_PYTHON_EXE PATH_TO_PYTHON_SCRIPT"
# or you can add the same in TaskScheduler



from stat import S_ISREG, ST_ATIME, ST_CTIME, ST_MODE, S_ISDIR, S_IWUSR
import os, sys, time, datetime, shutil, getopt

def delete_files(delete_path_list, count):
    deleted_files = []

    for file in delete_path_list:
        for root, dirs, files in os.walk(file):
            for fname in files:
                full_path = os.path.join(root, fname)
                os.chmod(full_path, S_IWUSR)
        if count > 0:
            deleted_files.append(file)
            if os.path.isdir(file):
                shutil.rmtree(file)
            else:
                os.remove(file)
            count = count - 1
    return deleted_files


def delete_old_backup_enteries(folder, keep_days, min_count):
    keep_files = []
    for elem in os.listdir(folder):
        paths_sorted = []
        entries1 = (os.path.join(folder, elem, fn) for fn in os.listdir(os.path.join(folder, elem)))
        entries2 = ((os.stat(path), path) for path in entries1)
        entries3 = ((stat[ST_CTIME], path) for stat, path in entries2)
        for cdate, path in sorted(entries3):
            paths_sorted.append(path)

        if (len(paths_sorted) <= min_count):
            for file in paths_sorted:
                keep_files.append(file)
            continue

        delete_path_list = []
        for path in paths_sorted:
            file_create_time = os.path.getmtime(path)
            time_now = time.time()
            if ((time_now - file_create_time) > (keep_days * 24 * 60 * 60)):
                delete_path_list.append(path)

        deleted_files = delete_files(delete_path_list, min(len(delete_path_list), len(paths_sorted) - min_count))
        for file in deleted_files:
            paths_sorted.remove(file)

        for file in paths_sorted:
            keep_files.append(file)

    print(("Keeping the following backup files for folder %s" % folder))
    for file in keep_files:
        print(file)

def usage():
    print("""\
    Usage: nsx_backup_cleaner.py -d backup_dir [-k 1] [-l 5] [-h]
           Or
           nsx_backup_cleaner.py --dir backup_dir [--retention-period 1] [--min-count 5] [--help]

           Required
               -d/--dir: Backup root directory
               -k/--retention-period: Number of days need to retain a backup file
           Optional
               -l/--min-count: Minimum number of backup files to be kept, default value is 100
               -h/--help: Display help message
           """)
def main():
    BACKUP_ROOT = None

    BACKUPS_KEEP_DAYS = None
    # Minimum allowed: 100
    BACKUPS_MINCOUNT = 100

    try:
        opts, args = getopt.getopt(sys.argv[1:], "hd:k:l:", ["dir=", "retention-period=", "min-count=", "help"])
    except getopt.GetoptError as err:
    # print help information and exit:
        print ((str(err))) # will print something like "option -a not recognized"
        usage()
        sys.exit()

    for opt, arg in opts:
        if opt in ("-h", "--help"):
            usage()
            sys.exit()
        elif opt in ("-d", "--dir"):
            BACKUP_ROOT = arg
        elif opt in("-k", "--retention-period"):
            BACKUPS_KEEP_DAYS = int(arg)
        elif opt in ("-l", "--min-count"):
            BACKUPS_MINCOUNT = int(arg)
        else:
            usage()
            sys.exit()

    if (BACKUP_ROOT == None):
        print("Missing Backup Root")
        usage()
        sys.exit()

    if (BACKUPS_KEEP_DAYS == None):
        print("Missing Backup Retention Period in number of days")
        usage()
        sys.exit()

    if (not os.path.isdir(BACKUP_ROOT)):
        print("Wrong backup root directory")
        usage()
        sys.exit()

    backup_dirs = os.listdir(BACKUP_ROOT)
    if (all(elem in ["cluster-node-backups", "inventory-summary"] for elem in backup_dirs)):
        for elem in backup_dirs:
            if (elem in ["cluster-node-backups"]):
                delete_old_backup_enteries(os.path.join(BACKUP_ROOT, 'cluster-node-backups'), BACKUPS_KEEP_DAYS, BACKUPS_MINCOUNT)
            if (elem in ["inventory-summary"]):
                delete_old_backup_enteries(os.path.join(BACKUP_ROOT, 'inventory-summary'), BACKUPS_KEEP_DAYS, BACKUPS_MINCOUNT)
    else:
        print ("Cleanup script works only in folders, that contains subfolders \"cluster-node-backups\" and \"inventory-summary\"")


if __name__ == "__main__":
    main()

Additionally for the script to work make sure there are no other folders within the backup folder. the backup folder should only have the cluster-node-backups and inventory-summary directories

vCLS not starting with Failed – Feature ‘cpuid.mwait’ was absent, but must be present error

With the release of vCenter 7 Update 1, VMware introduced the vCLS (vSphere Clustering Service). More information can be found here.

Looking at the error details it looks like it is looking for a feature called cpuid.mwait

Reviewing the VMX file it seems like EVC is enabled on the vCLS VMs. I didnt want to enable EVC on the whole cluster so i wanted to do it only on the specific VMs.

Doing some research i found that the VMs need to be at version 14. After upgrading the VM i was able to disable EVC on the specific VMs by following these steps:

In the vSphere Client, navigate to the virtual machine

Under the Actions -> Compatibility -> Upgrade VM compatibility

We can disable EVC on per VM level on version 14 and above, so in my case i chose ESXi 6.7U2 and later

Next go to the Configure Tab

Pick VMware EVC and click on Edit

Click on Yes

Click on Disable EVC and Click OK

The next time it tries to power on the VM it should go through.

Once the first VM starts up it will most likely deploy a few additional ones, follow the same steps as above again on the new VMs

Skip-Level Upgrading VCF 4.x to 4.4.1 Step by Step

With the release of of VCF 4.4.1 i wanted to get my lab upgraded. The release blog can be found here and the release notes are here

In order to get to 4.4.1.0 in the past we had to upgrade to each version incrementally. As per the documentation found here we can now upgrade from any release past 4.1.0.1 directly to 4.4.1 skipping the releases in between. We can do so by going to Repository -> Bundle management -> Download now.

Alternatively if we are on 4.4.1.0 it can be found under Lifecycle Management -> Bundle Management -> Download now

We can also download it directly from the domain by going to Inventory -> Workload Domains -> Select the domain -> Update/Patches -> Select the VCF version -> Download now

The next step is to upgrade VCF by going to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Select the cloud foundation version were upgrading to and click on Update Now for the VMware Cloud Foundation Update 4.4.1.0

Next we are taken to the Upgrade page where we can follow the upgrade for each one of the components

Once the upgrade is complete we can click Finish to be returned back to the main screen

Because we are changing the SDDC-Manager versions i would strongly recommend to clear cache and log back in before going forward.

Next step is to download and install the configuration drift package. We can find it under Lifecycle management -> Bundle management -> Download now

We can also download it directly from the domain by going to Inventory -> Workload Domains -> Select the domain -> Update/Patches -> Select the VCF version -> Download now

Once the download is complete go to Inventory -> Workload Domains -> Select the domain -> Update/Patches -> Select the VCF version and click on update now

We can view the upgrade process by clicking on view status

Next step is to upgrade NSX-T installation to NSX-T 3.1.3.7.4. The release notes can be found here. We can go to inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Under Available updates click on the drop down and select Cloud Foundation 4.4.1.0 -> Download now.

Once the download is complete we can proceed with updating the NSX components by clicking on the update now

Make the proper selection and click next

Make the proper selection and click next

Make the proper selection and click next

Review the options and click on Finish

The upgrade will go thought upgrading the NSX edges. We can view the upgrade status by clicking on view status

Once the edges are upgraded we an go back to Inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Under Available updates click on Update Now

Review the selection and click next

Review the host clusters and click next

Review the upgrade options and click next

Review the selection and click finish

We can view the status of the upgrade by selection view status

Once the upgrade is complete we can proceed with the vCenter Upgrade. VCF 4.4.1 comes with vCenter Server 7.0 Updated 3d. The release notes can be found here. We can go to inventory -> Workload Domains -> Select the workload domain -> Update/Patches -> Under Available updates click on the drop down and select Cloud Foundation 4.4.1 -> Download now.

Once the download is complete we can click on Update now

We can follow the status of the upgrade by clicking on the view status tab

Here we can see the different components that are getting upgraded

Once the upgrade is complete we are taken back to the previous page where we can see that the ESXi servers are next. The release notes can be found here. Click on Download Now

Once the download is complete we can click on Update now

If we have multiple clusters we can enable Cluster-level selection and select the specific cluster(s) we want to upgrade.

We can also enable sequential cluster upgrade as well as quick boot

We get to review the options once again before we click finish to to submit the task

Once submitted we can view the status by clicking on View Status

And with that we are finished with the workload domain. We can follow the same steps for the other domains

Don’t forget to clean up the download bundles by following the steps from my other blog here