vRSLCM (vRealize Lifecycle Manager) Product Support Pack

In this guide i will go over the steps of getting an existing 8.x vRSLCM appliance to support the latest product releases available. Here is a great blog that goes in to the details about what the Product Support Pack is https://blogs.vmware.com/management/2019/01/vrslcm-pspak.html. Typically the newer Product Support Pack is included part of the upgrade for LCM, however sometimes there are product releases in between releases where product support packs come in handy.

The first step is to log in to vRealize Suite Lifecycle Manager under the Lifecycle Operations section

Go to settings -> Product Support Pack

We can see that i recently upgraded to 8.8.2.0 however a new update is available 8.8.2.1. Based on what we can see in the details the new support pack adds support for vRA 8.9.0. If an update is not available click on the Check Support Packs Online button and refresh the screen within a few minutes

Click on Apply Version

Verify that a snapshot or a backup exists and click Submit

We can view the progress by clicking on the Click Here link after submitting the request

Once the process is complete the system will most likely reboot. To check the status we can go back to settings -> Product Support Pack. As we can see we are now at the updated patch level

If you get the below error clear the browser cache and try again

Why Automate Virtual Infrastructure, Why Do it Now, and Why Do it with vRealize Automation

Our IT culture has shifted from managing infrastructure to the management of services. We deliver a self-service catalog to our consumers who manage their environment. Providing our consumers with the self-service catalog reduced delivery of workloads from weeks to hours and it significantly increased their overall satisfaction. – Senior IT director and enterprise architect (Large Healthcare System in western Pennsylvania healthcare system)  

Automation is a journey. The primary reason to adopt automation is to streamline manual processes, enabling your information technology teams to focus on more valuable activities. The goal is to shift lifecycle management of workloads and day-to-day actions to the internal consumer. We look to the future to see where we want to end up and then plan how to get there.  

  • The first decision: your end-state – do you want a self-service or IT as a Service model?  
  • The second decision: which delivery method do you want to use, imperative or declarative code, and do you have the skillset to support the method of choice?  
  • Imperative is IT as a Service. It is programmatically based and relies on your IT coding team.  
  • Declarative can be consumer self-service or IT as a Service.  

Whether you are focused on a private or multi-cloud environment, either method can be employed. But only the declarative approach can deliver self-service capabilities to your internal customers. Here is an example of an organization using the declarative method and its impact.  

Examples:  

  • A well-known children’s hospital in Pennsylvania adopted lifecycle management/process automation.
    • They chose vRealize Automation due to their desire to build out a self-service catalog 
    • Once they deployed vRealize Automation they, they trained their consumers and turned over lifecycle management. 
    • This allowed the Automation Engineers to focus on delivering platform updates and new services for their consumers. 
    • They benefited by avoiding 13,000 helpdesk tickets in their first year by implementing day two actions. 
      • Based on a conservative estimate of a 20-minute resolution per ticket, they avoided 4400 hours of an FTE’s time or 110 weeks (about two years).  
      • Equally as important, this freed up the help desk to focus on level two, and level three issues 
  • Why Do it Now?
    • Why not do it now is the question to be asked? 
  • Information technologies’ role is to provide the infrastructure that supports business-critical applications.  
    • IT cannot be a bottleneck.  
    • The ability to deliver quickly and consistently gives your organization a competitive advantage.  
  • Process automation moves IT from a delivery arm of your business to a support arm. That means:  
    • Self Service: Internal application owners, Database Managers, and DevOps Engineers, lifecycle manage their environments.  
    • Risk Mitigation  
    • Automation of Day-to-Day tasks – You leave room for error when you repeatedly perform a task every day. Invariably it happens. Why is it because we are human, and human beings’ fat finger the keyboard?  
    • Creation of a process that ensures prompt delivery of your customer requests.  
    • Remove backlog of customer requests 
    • Increase internal customer Net Promoter Score (NPS)  
  • Why Do it with vRealize Automation? 
    • vRealize Automation uses the declarative code: build it, and they will come
  • Many organizations utilize commercial off-the-shelf applications.  
    • In this scenario, coders are less relevant.  
    • The declarative method enables IT to build out the relevant use cases for their customer base and make them available in the self-service catalog.  
  • Build use-case templates 
    • Day Zero- deployment of new workloads or services 
    • Day Two – manage the environment.  
    • Day N – retirement  
    • Place all use-cases in a self-service catalog.  
    • Let your Internal consumers consume  
  • The benefits of automating your virtual environment 
    • Deployment of workloads and services is consistent and reliable.  
    • IT or the internal consumer can deliver in hours versus weeks. 
    • IT can monitor compliance and remediate it as needed within minutes vs. hours. 
    • Support teams can recover time and focus on bringing more value to the organization 
    • vRealize Automation deploys and manages across the public, hybrid, and private clouds with the same processes 

Credit goes to Steve Lieberson, Tom Gillaspy and Cosmin Trif. You can find Steve on Twitter and LinkedIn, Tom on Twitter and LinkedIn, and Cosmin on Twitter and LinkedIn

Deploying an AVS cluster on Azure

In this post we will go over the steps for deploying an AVS cluster on Azure.

The first step was to log in to the azure portal at portal.azure.com. Once logged on we can search for “azure vmware solution”

Then I tried to create a cluster by clicking on the Create button on the top left

This opened a wizard for me with the Requirements. Trying to go forward without opening a ticket gave me this error:

Azure VMware Solution is available for all customers with an existing Microsoft Enterprise Agreement or those under a Cloud Solution Provider Azure plan. Prior to creating and deploying your Azure VMware Solution Private Cloud, please review and follow the process for node allocation to your subscription type here.

The instructions send me to the documentation on the steps required and I had to open a ticket to request a quota increase. Here is the direct link to open a ticket

  1. In your Azure portal, under Help + Support, create a New support request and provide the following information:
    • Issue type: Technical
    • Subscription: Select your subscription
    • Service: All services > Azure VMware Solution
    • Resource: General question
    • Summary: Need capacity
    • Problem type: Capacity Management Issues
    • Problem subtype: Customer Request for Additional Host Quota/Capacity
  2. In the Description of the support ticket, on the Details tab, provide information for:
    • Region Name
    • Number of hosts
    • Any other details NoteAzure VMware Solution requires a minimum of three hosts and recommends redundancy of N+1 hosts.
  3. Select Review + Create to submit the request.

It would look like this:

The next screens were pretty self explanatory so I won’t go through them. Once the ticket is created a Microsoft engineer will most likely reach out to verify the details and provision the capacity.

Once the capacity has been provisioned we have a few more steps to follow.

First is to go to subscriptions -> Select Subscription

-> Resource providers -> Search for ads -> Click on register

Before navigating away make sure the Resource shows as registered:

After completing the above going to the Azure VMware Solution allows me to register the go through the screen without errors. Please note that we can only provision resources where they were allocated in the ticket. For example we can’t use resources in West 2 region if the capacity was added to East 2. The ticket from Microsoft would include these details. Sample setup:

The last screen is the review and create. Once we click create the resources will get provisioned.

The deployment will go on for a while. In my case it was 4 hours. Once the deployment is complete we can go to the Azure VMware Solution

After completing the above I would recommend checking out the tutorials on the overview page

SSC 8.8 Authentication failed: no Authorization header

I recently upgraded my LCM deployed SSC server to 8.8.x. If you need a guide to go through the upgrade you can find my other post here.

After the upgrade was completed i was noticing strange behavior in the SSC UI so i checked the status the of the services. Here are the errors i found and how i fixed them

The first step was to check the status of the service

systemctl status salt-master

The return was this

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2022-07-16 20:30:29 UTC; 1 day 2h ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 801 (salt-master)
    Tasks: 40 (limit: 9830)
   Memory: 499.5M
   CGroup: /system.slice/salt-master.service
           |-  801 /bin/python3 /usr/bin/salt-master
           |- 1005 /bin/python3 /usr/bin/salt-master
           |- 1088 /bin/python3 /usr/bin/salt-master
           |- 1090 /bin/python3 /usr/bin/salt-master
           |- 1101 /bin/python3 /usr/bin/salt-master
           |- 1102 /bin/python3 /usr/bin/salt-master
           |- 1110 /bin/python3 /usr/bin/salt-master
           |- 1113 /bin/python3 /usr/bin/salt-master
           |- 1119 /bin/python3 /usr/bin/salt-master
           |- 1120 /bin/python3 /usr/bin/salt-master
           |- 1397 /bin/python3 /usr/bin/salt-master
           |- 1398 /bin/python3 /usr/bin/salt-master
           |- 1400 /bin/python3 /usr/bin/salt-master
           |- 1410 /bin/python3 /usr/bin/salt-master
           |- 1414 /bin/python3 /usr/bin/salt-master
           |- 1419 /bin/python3 /usr/bin/salt-master
           |- 1420 /bin/python3 /usr/bin/salt-master
           |- 1424 /bin/python3 /usr/bin/salt-master
           `-15430 /bin/python3 /usr/bin/salt-master

Jul 17 21:07:47 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:47 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to send minion key state to SSE: 401 Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to send minion cache to SSE: 401 Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:48 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to send master fileserver data to SSE: 401 Authentication failed: no Authorization header
Jul 17 21:07:50 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:50 ssc-01a.corp.local salt-master[801]: [ERROR   ] sseapi_event_queue: failed to send entries to SSE (will requeue): 401 Authentication failed: no Authorization header
Jul 17 21:07:55 ssc-01a.corp.local salt-master[801]: [ERROR   ] Failed to authenticate: Authentication failed: no Authorization header
Jul 17 21:07:55 ssc-01a.corp.local salt-master[801]: [ERROR   ] sseapi_event_queue: failed to send entries to SSE (will requeue): 401 Authentication failed: no Authorization header

The first step to resolve the error was to delete the master key from the UI by going to SSC UI -> Administration -> Master Keys -> Accepted -> Select the old key and click on delete ex:

Next we need to stop the salt master service by running

systemctl stop salt-master

Additionally on the cli we also need to delete the old key file located at:

/etc/salt/pki/master/sseapi_key.pub

We can delete it by running:

rm /etc/salt/pki/master/sseapi_key.pub

Once the above steps are complete we can get start the services again and accept the new key in the UI.

We can start the service back up by running:

systemctl start salt-master

We can now check the service and add the key back in the UI

systemctl status salt-master

Finally we can restart the saltstack service and verify that its running without errors:

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-05-15 20:02:56 UTC; 51s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 31309 (salt-master)
    Tasks: 39 (limit: 9830)
   Memory: 330.0M
   CGroup: /system.slice/salt-master.service
           |-31309 /bin/python3 /usr/bin/salt-master
           |-31315 /bin/python3 /usr/bin/salt-master
           |-31320 /bin/python3 /usr/bin/salt-master
           |-31323 /bin/python3 /usr/bin/salt-master
           |-31325 /bin/python3 /usr/bin/salt-master
           |-31326 /bin/python3 /usr/bin/salt-master
           |-31327 /bin/python3 /usr/bin/salt-master
           |-31328 /bin/python3 /usr/bin/salt-master
           |-31330 /bin/python3 /usr/bin/salt-master
           |-31397 /bin/python3 /usr/bin/salt-master
           |-31398 /bin/python3 /usr/bin/salt-master
           |-31400 /bin/python3 /usr/bin/salt-master
           |-31411 /bin/python3 /usr/bin/salt-master
           |-31412 /bin/python3 /usr/bin/salt-master
           |-31413 /bin/python3 /usr/bin/salt-master
           |-31414 /bin/python3 /usr/bin/salt-master
           |-31415 /bin/python3 /usr/bin/salt-master
           `-31416 /bin/python3 /usr/bin/salt-master

May 15 20:02:54 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
May 15 20:02:56 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

If the status page a return similar to this

* salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2022-07-17 23:19:35 UTC; 3min 24s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltproject.io/en/latest/contents.html
 Main PID: 21532 (salt-master)
    Tasks: 40 (limit: 9830)
   Memory: 346.7M
   CGroup: /system.slice/salt-master.service
           |-21532 /bin/python3 /usr/bin/salt-master
           |-21538 /bin/python3 /usr/bin/salt-master
           |-21546 /bin/python3 /usr/bin/salt-master
           |-21550 /bin/python3 /usr/bin/salt-master
           |-21552 /bin/python3 /usr/bin/salt-master
           |-21553 /bin/python3 /usr/bin/salt-master
           |-21554 /bin/python3 /usr/bin/salt-master
           |-21555 /bin/python3 /usr/bin/salt-master
           |-21556 /bin/python3 /usr/bin/salt-master
           |-21557 /bin/python3 /usr/bin/salt-master
           |-21628 /bin/python3 /usr/bin/salt-master
           |-21629 /bin/python3 /usr/bin/salt-master
           |-21631 /bin/python3 /usr/bin/salt-master
           |-21641 /bin/python3 /usr/bin/salt-master
           |-21644 /bin/python3 /usr/bin/salt-master
           |-21645 /bin/python3 /usr/bin/salt-master
           |-21646 /bin/python3 /usr/bin/salt-master
           |-21647 /bin/python3 /usr/bin/salt-master
           `-21648 /bin/python3 /usr/bin/salt-master

Jul 17 23:19:33 ssc-01a.corp.local systemd[1]: Starting The Salt Master Server...
Jul 17 23:19:35 ssc-01a.corp.local systemd[1]: Started The Salt Master Server.

Next we need to go back to the UI and accept the new master key. SSC UI -> Administration -> Master Keys -> Pending Select the new key and click on Accept Key

And with that the issue should be resolved.

Here are a few additional blogs that might be useful post upgrade:

Error Code: LCMVSSC10018

SSC 8.8 sseapi_rpc_queue: could not connect to SSE server

SSC 8.8 urllib3 (1.25.11) or chardet (4.0.0) doesn’t match a supported version

Patching\Upgrading ESXi 7 to ESXi7U1 via esxcli

With the latest release of ESXi7U1 i wanted to get my lab up to date. I dont have enough resources in my lab to migrate the vCenter to another ESXi server so i want to perform the upgrade via cli. More details about the release can found here

The first step was to open the firewall for outgoing traffic for http

esxcli network firewall ruleset set -e true -r httpClient

Second step was to list the updates by executing

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep -i ESXi-7

This returned a list of updates that were available:

I reviewed the downloads website here to double check the version. Based on the information I found the ESXi-7.0U3f-20036589-standard is the latest release

The next step was to run the update by executing

esxcli software profile update -p ESXi-7.0U3f-20036589-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

If you run in to the below error follow my other post here:

Once the upgrade was complete i was presented with this output:

The installation reports that a reboot is required so I went ahead and rebooted the server

After the server was back up, checking the ESXi, the server reports build 20036589

Don`t forget to add the firewall rule back in

esxcli network firewall ruleset set -e false -r httpClient

Upgrading vRA (vRealize Automation) to 8.8.2

In this post i will go over upgrading my 8.x vRA appliance to 8.8.2. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8.2 Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRA

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8.2 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

A new feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors

The next screen remind us of hardware requirements for vRA and vIDM which can be checked here for vRA and vIDM which can be checked here. As far as i can tell they haven’t changed since the 8.3 release. Check the I took care of the manual steps above and am ready to proceed check mark and click on Run Precheck

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8.2

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vRA version.

Upgrading vRLI (vRealize Log Insight) to 8.8.2 via vRSLCM

In this post i will go over upgrading my 8.x vRLI appliance to 8.8.2 using vRSLCM (vRealize Suite Lifecycle Manager). As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8.2. Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. Make sure we select the upgrade package not the install package. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes vRLI

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

A new feature that was added was the capability to automatically create a snapshot prior to the upgrade and remove it after the upgrade. On this screen we also have the ability to chose if we want to keep the snapshots post upgrade for validation testing for example. Click next

Run the Precheck to make sure there are no errors or issues.

Once the check is complete we can review the checks that were performed and we can continue by clicking Next.

Review the upgrade details and click on Next then Finish. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8.2

Since we are doing a major upgrade i strongly recommend to clean the cache before using the new vRLI version.

Upgrading SSC (SaltStack Config) to 8.8.2 using vRSLCM

In this post i will go over upgrading my 8.x SSC appliance to 8.8.2. As a pre requirement we do need to have vRSLCM (vRealize Lifecycle Manager) upgraded to 8.8.2. Instructions can be found here. The upgrade already includes the latest Product Support Pack so an update to the Product Support Pack is not required.

To get started we can go to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> Binary Mapping. (If you haven’t added your My VMware credentials you will need to do that first by going to vRealize Lifecycle Manager -> Lifecycle Operations -> Settings -> My VMware)

Click on Add Binaries under Product Binaries

Select My VMware and click on Discover

We can see a list of binaries that have been discovered. We can select what we need and click on Add

This will create a request and start downloading the package. To view the progress we can click on the Click Here hyperlink

Click on the in Progress button to view the details

We now have to wait for the download to complete

After the download is complete we can go to Environments -> View Details on the environment that includes SSC

Click on Upgrade

An Inventory sync is recommended if the environment has changed since LCM performed the last sync. We trigger the sync from the UI or click on Proceed to continue

Select product Version 8.8.2 and click Next. We can also review the compatibility matrix to make sure the environment is compatible.

We can automatically create and delete a snapshot part of the upgrade process

Run the Precheck to make sure there are no errors

Once the check is complete, click on Next. Review the upgrade details and click on Next. We are taken to the progress screen where we can follow the progress.

The system will get rebooted and once its back up we will be on 8.8.2

Here are a few additional blogs that might be useful post upgrade:

Error Code: LCMVSSC10018

SSC 8.8 sseapi_rpc_queue: could not connect to SSE server

SSC 8.8 urllib3 (1.25.11) or chardet (4.0.0) doesn’t match a supported version

Authentication failed: no Authorization header

Upgrading vRSLCM (vRealize Lifecycle Manager) to 8.8.2

In this guide i will go over the steps of getting an existing 8.x vRSLCM appliance upgraded to the latest 8.8.2 release. The release notes can be found here

The first step is to log in to vRealize Suite Lifecycle Manager under the Lifecycle Operations section

Go to settings -> System Upgrade

Click on Check for Upgrade

We can see that the check found a new version available for 8.8.2

Click on Upgrade

Verify that a snapshot or backup exists in case the process fails. Check the check mark for I took a snapshot of the vRealize Suite Lifecycle Manager before I performed this operation. Click Next

Click on Run Precheck

Verify that all check have passed and click on upgrade

This will fire up the upgrade process and start upgrading packages. The system will automatically reboot on 8.8.2 once completed. We can check the version by going to Settings -> System Details

If you get the below error clear the browser cache and try again

vRealize Lifecycle Manager (vRSLCM) 8 certificate management

vRealize Lifeycycle Manager (vRSLCM) come with a Certificate Management feature. We can access the certificate management from Home -> Locker -> Certificate

We can generate Self Signed certificates for products managed by Lifecycle Manager as well as certificate requests to be signed by a certification authority

Generate a new CSR

Download the CSR and take it to the certified authority, in my case im using a Microsoft server /certsrv/certrqxt.asp

Click on advanced certificate requests and complete the request details

Download the certificate in the Base 64 encoded format

Next we can import it in to the certificate vault

Click on import and complete the details. The private key can be found in the certificate request file

Make sure you create a file that includes the signed certificate + the private key + the intermediate and root certificates.

The certificate can now be used