I recently ran through an routing issue where the Kubernetes IP range in vRO 8.6 was used somewhere else on the network. I didn’t want to redeploy the appliance so i went through the below to get it updated

First i identified what the ip range in use is by running

vracli network k8s-subnets

Lets check the status of the pods to make sure everything is running as it should. If any of the pods experience issues the change wont go through and it will cause additional issues

kubectl get pods -n prelude

The expectation is that under the Ready tab we have something similar

NAME                               READY   STATUS    RESTARTS   AGE
docker-registry-695f9b8b45-d8gqr   1/1     Running   0          53m
postgres-0                         1/1     Running   0          53m
proxy-service-5d8f64b54-lmxg5      1/1     Running   0          54m
vco-app-78499d8cbd-4mcnk           3/3     Running   0          54m

To set a new internal Kubernetes ip range i ran

vracli network k8s-subnets --cluster-cidr 192.168.0.0/22 --service-cidr 192.168.4.0/22

Then in order to apply i changes i ran

vracli upgrade exec

I was prompted with a question

The services will be shut down while upgrade is in progress. Confirm you want to continue with the upgrade operation.[Y/n]

By pressing Y the system went ahead and reconfigured\redeployed the pods on the proper network

And lastly i wanted to check the status of the pods to make sure they all came back

kubectl get pods -n prelude

In this guide we will go over the vIDM 3.3.5 HA configuration. The official documentation can be found here

Im going to assume that the load balancer configuration is already completed, the vIDM appliance has a the required certificate in the LCM inventory. Please read the official documentation for the full requirements.

We will be using the scale out feature in Lifecycle Manager. To do so we can navigate to Lifecycle Operations -> Environments -> globalenvironment -> View Details -> Click on Add Components

It is recommended that an inventory sync is performed prior to starting the process. It can be triggered by pressing on Trigger Inventory sync button. In my case i don`t need one as i did it earlier so ill just click Proceed

Network configuration should be populated. Verify the config and click next

Towards the bottom of the Components page there will be a components section. Click on the Plus sign next to it and select VMware Identity Manager Secondary Node. Perform this task task 2 times so we can have 3 vidm nodes.

Complete the required fields Like network configuration and Cluster Virtual IP

On the next page run the precheck in order to execute the data validation

Verify the Manual Validation as described in the Pop Up Window and click on Run Precheck

Once all the check are complete click on next, Verify the Summary and click on Submit

This will take us to the Request Details Page where we can follow the steps taken

Once the additional nodes are installed validate that everything is working as expected.

In this guide we will go over Creating a Credential for MongoDB in vROPS for the True Visibility Management pack. The original documentation was posted here however it hasn’t yet been updated to vROPS 8.6 and the new vROPS cloud.

Procedure

1. From the Left navigation bar, select Data Sources -> Integrations -> Credentials

Click the Add button on the top left. In the Manage Credential window that appears, select whether or not authentication is required for your MongoDB environment, then enter the information indicated below:

If authentication is required we would just select Authentication required under Credential Kind

During VMworld 2021 we announced vRealize Operations 8.6 which now provides allot of new features and capabilities. You can see an Whats New overview here as well as a feature walk through here.

I this blog we will be focusing specifically on the Administration menu and how it maps to the new 8.6 release. I covered the UI Navigation on my blog other here.

As you might be aware the Administration has now been redesigned to make it easier to navigate and find items. Lets jump in to it.

The first item on the list is Data Sources. To make it easier to get started the Data Sources can now be found on the home page as we get started. The accounts are now grouped together by account type ex: vCenter servers, AWS, Azure and so on. The Other Accounts sub-menu has been consolidated under the same menu. Because we combined the Cloud Accounts with Other Accounts we renamed the menu as Integrations.

The repository is now another tab under the Data Sources -> Integrations -> Repository

The Inventory menu has been moved under the Environment -> Inventory

The Policies have been moved to Configure -> Policies

The Access Control and Authentication Sources can now be found under the Administration menu

Custom Profiles has been renamed to Custom VM Profiles and it can be found under Configure -> Custom VM Profile

End Point Operations functionality is now included part of the telegraf agent via the cloud proxy and this menu item has been retired as its no longer in use.

The Group Types can now be found with the Custom Groups under Environment -> Custom Groups -> Group Types

Icons is considered an Administration item and it can be found under Administration -> Icons

Maintenance Schedules are considered a Non Administration Option that an non vROPS admin should be able to configure so its been relocated to the Configure sub menu

Configuration Files are locate under Configure -> Configuration Files

The object Relationships has been removed from the UI however it can be accessed by going directly to https://vrops_instance/ui/index.action#configure/object-relationships. The URL is the same for the SaaS as well as on premise

Optimizations Schedules has been moved under the Workload Optimization. It can be found by navigating to Optimize -> Workload Placement -> Optimization Schedules

Super Metrics are under Configure -> Super Metrics

Cost Settings are found under Configure -> Cost Settings

As described earlier in this post the Integrations have been combined with Other accounts and Other Accounts under Data Sources -> Integrations

Certificates, Cluster Management, Collector Groups and Collection Status have all been moved under Administration

The credentials have been consolidated under Data Sources -> Integrations -> Credentials

Global Settings, Licensing, Log Forwarding, Content Management can all be found under Administration

Outbound Settings are found now under Configure -> Alerts -> Outbound Settings

The rest of the items like Audit, Recent Tasks, Dynamic Thresholds, Logs, Redescribe, Cost Reference Database and Support Bundles can all be found under the administration menu

For more information visit us at https://www.vmware.com/products/vrealize-operations.html

During VMworld 2021 we announced vRealize Operations 8.6 which provides a lot of new features and capabilities. You can see an Whats New overview here as well as a feature walk through here.

With the new UI release some of the content might not be where were used to find it in the previous versions. In this post we will navigate through some of the changes and point to “Where is it now”. I also covered the UI overview on my blog here

Lets jump in to it. As shown in the screenshot below in the old UI we used to have navigation items up top as well as on the side. Many of the menu options across the top were duplicated in the left-hand navigation menu and were consolidated in this release.

As we can see in the new UI we still have the 4 pillars that were used too however the top menu bar was removed in order to be more consistent with the rest of the products in our SaaS platform. The sidebar is also collapsible allowing us to use the space for other purposes.

The Home button can be easily be found as the first item in the menu

The Dashboards are now conveniently located under the Visualize menu. This allows us now to go directly to dashboards, views as well as reports.

The Alerts tab has been relocated under Troubleshooting. We can now easily navigate between different troubleshooting options like Alerts, the troubleshooting Workbench and log analysis all under the same menu.

The environment tab can be easily be spotted on the side menu. Once we click on it we will notice a number of additional items like the new Object Browser which allows us to navigate through the environment more easily as well as the older Inventory browser. Some of the other options that are included are the new Business Applications, the previously Applications, Custom Groups, Custom Datacenters as well as Cloud Zones

Lastly the Administration menu. The administration menu has been broken into 2 different menus. The first one is under Configure where we can make configuration changes for our monitoring options things like defining Policies, Alerts configuration, Super Metrics, Application Discovery, Application Monitoring, Cost Settings, Custom VM Profiles, Configuration Files as well as Maintenance Schedules

The second administration menu goes specifically in to overall product administration like Access Control, Certificates, Licensing, Collector Groups, Content Management, Orphaned Content, Global Settings, Icons, Audit, Recent Tasks, Dynamic Thresholds and Cost Calculation

Now that we have the top menu out of the way lets take a look at where the side Menu items are in the new UI.

The quick Start menu item is now our default Home page

The operations overview is now conveniently placed as another tab on the home screen

Workload optimization is now conveniently placed with other optimization items under the optimize menu and it has been renamed to Workload Placement

From a rightsizing perspective we can see that rightsizing is right under Workload Placement in the Optimize sub menu and it has been renamed to Rightsize

The Recommended Actions menu item has been deprecated however i did create the menu item as a dashboard and it can be downloaded from the Sample Exchange on code.vmware.com or by clicking here

The Optimize Capacity Overview page can be found under Capacity in the Optimize Menu

The Reclaim menu can be found under the Optimize Menu

What-If Analysis has been renamed as Capacity and it can be found under the Plan Menu

We can find the Troubleshooting Workbench under the Troubleshoot menu

The Virtual Machine menu was pointing to the VM Contention Dashboard which can now be found under Visualize -> All -> Performance -> Consumer -> VM Contention. The Dashboard can also be found by performing a search in the dashboard menu

The vSAN menu was pointing to the vSAN Contention Dashboard which can now be found under Visualize -> All -> Performance -> Provider -> vSAN Contention. The Dashboard can also be found by performing a search in the dashboard menu

Using Logs can now be found under Troubleshoot -> Log Analysis

The compliance menu has been moved under Optimize -> Compliance

Discover Services has been renamed to Application Discovery and it can be found under Configure -> Application Discovery

Monitor Applications was renamed to Application Monitoring and it can be found under the Configure menu

Automation Central can still be found as a separate item on the side menu

For more information visit us at https://www.vmware.com/products/vrealize-operations.html

During one of the recent upgrades in my VCF environment i accidentally upgraded the ESXi 7.x build number to a later build that was not in the validated design which caused all kinds of issues and errors in my sddc manager. In order to address this i wanted to see i could actually downgrade the packages to a lower build number. It goes without saying that this procedure should not be done in a production environment and you should contact VMware customer support for an supported downgrade method. We will be working in the ESXi cli on this guide

The following command brings all of the version currently available in the depot

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

I was specifically looking for build 18426014 so i leveraged grep to get the proper version

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep 18426014

The command above returned 2 versions for me. One with VMware tools and one without

ESXi-7.0U2c-18426014-standard     VMware, Inc.  PartnerSupported  2021-08-24T00:00:00  2021-08-24T00:00:00
ESXi-7.0U2c-18426014-no-tools     VMware, Inc.  PartnerSupported  2021-08-24T00:00:00  2021-08-04T11:40:25

We are specifically interested in the first part of each line

ESXi-7.0U2c-18426014-standard and ESXi-7.0U2c-18426014-no-tools

First i tried to install it using

esxcli software profile update -p ESXi-7.0U2c-18426014-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

However that only works if were actually upgrading to a later build. In order to downgrade i found an option –allow-downgrades that allows us to install a lower build. In the end i ended up with

esxcli software profile update -p ESXi-7.0U2c-18426014-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --allow-downgrades

This actually installed the lower versions of the packages associated with that build number and removed the later packages. The output in my case was

Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMware_bootbank_clusterstore_7.0.2-0.20.18426014, VMware_bootbank_cpu-microcode_7.0.2-0.20.18426014, VMware_bootbank_crx_7.0.2-0.20.18426014, VMware_bootbank_esx-base_7.0.2-0.20.18426014, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.20.18426014, VMware_bootbank_esx-update_7.0.2-0.20.18426014, VMware_bootbank_esx-xserver_7.0.2-0.20.18426014, VMware_bootbank_gc_7.0.2-0.20.18426014, VMware_bootbank_loadesx_7.0.2-0.20.18426014, VMware_bootbank_native-misc-drivers_7.0.2-0.20.18426014, VMware_bootbank_vdfs_7.0.2-0.20.18426014, VMware_bootbank_vsan_7.0.2-0.20.18426014, VMware_bootbank_vsanhealth_7.0.2-0.20.18426014
   VIBs Removed: VMware_bootbank_clusterstore_7.0.2-0.25.18538813, VMware_bootbank_cpu-microcode_7.0.2-0.25.18538813, VMware_bootbank_crx_7.0.2-0.25.18538813, VMware_bootbank_esx-base_7.0.2-0.25.18538813, VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.25.18538813, VMware_bootbank_esx-update_7.0.2-0.25.18538813, VMware_bootbank_esx-xserver_7.0.2-0.25.18538813, VMware_bootbank_gc_7.0.2-0.25.18538813, VMware_bootbank_loadesx_7.0.2-0.25.18538813, VMware_bootbank_native-misc-drivers_7.0.2-0.25.18538813, VMware_bootbank_vdfs_7.0.2-0.25.18538813, VMware_bootbank_vsan_7.0.2-0.25.18538813, VMware_bootbank_vsanhealth_7.0.2-0.25.18538813
   VIBs Skipped: VMW_bootbank_atlantic_1.0.3.0-8vmw.702.0.0.17867351, VMW_bootbank_bnxtnet_216.0.50.0-34vmw.702.0.20.18426014, VMW_bootbank_bnxtroce_216.0.58.0-20vmw.702.0.20.18426014, VMW_bootbank_brcmfcoe_12.0.1500.1-2vmw.702.0.0.17867351, VMW_bootbank_brcmnvmefc_12.8.298.1-1vmw.702.0.0.17867351, VMW_bootbank_elxiscsi_12.0.1200.0-8vmw.702.0.0.17867351, VMW_bootbank_elxnet_12.0.1250.0-5vmw.702.0.0.17867351, VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014, VMW_bootbank_iavmd_2.0.0.1152-1vmw.702.0.0.17867351, VMW_bootbank_icen_1.0.0.10-1vmw.702.0.0.17867351, VMW_bootbank_igbn_1.4.11.2-1vmw.702.0.0.17867351, VMW_bootbank_irdman_1.3.1.19-1vmw.702.0.0.17867351, VMW_bootbank_iser_1.1.0.1-1vmw.702.0.0.17867351, VMW_bootbank_ixgben_1.7.1.35-1vmw.702.0.0.17867351, VMW_bootbank_lpfc_12.8.298.3-2vmw.702.0.20.18426014, VMW_bootbank_lpnic_11.4.62.0-1vmw.702.0.0.17867351, VMW_bootbank_lsi-mr3_7.716.03.00-1vmw.702.0.0.17867351, VMW_bootbank_lsi-msgpt2_20.00.06.00-3vmw.702.0.0.17867351, VMW_bootbank_lsi-msgpt35_17.00.02.00-1vmw.702.0.0.17867351, VMW_bootbank_lsi-msgpt3_17.00.10.00-2vmw.702.0.0.17867351, VMW_bootbank_mtip32xx-native_3.9.8-1vmw.702.0.0.17867351, VMW_bootbank_ne1000_0.8.4-11vmw.702.0.0.17867351, VMW_bootbank_nenic_1.0.33.0-1vmw.702.0.0.17867351, VMW_bootbank_nfnic_4.0.0.63-1vmw.702.0.0.17867351, VMW_bootbank_nhpsa_70.0051.0.100-2vmw.702.0.0.17867351, VMW_bootbank_nmlx4-core_3.19.16.8-2vmw.702.0.0.17867351, VMW_bootbank_nmlx4-en_3.19.16.8-2vmw.702.0.0.17867351, VMW_bootbank_nmlx4-rdma_3.19.16.8-2vmw.702.0.0.17867351, VMW_bootbank_nmlx5-core_4.19.16.10-1vmw.702.0.0.17867351, VMW_bootbank_nmlx5-rdma_4.19.16.10-1vmw.702.0.0.17867351, VMW_bootbank_ntg3_4.1.5.0-0vmw.702.0.0.17867351, VMW_bootbank_nvme-pcie_1.2.3.11-1vmw.702.0.0.17867351, VMW_bootbank_nvmerdma_1.0.2.1-1vmw.702.0.0.17867351, VMW_bootbank_nvmxnet3-ens_2.0.0.22-1vmw.702.0.0.17867351, VMW_bootbank_nvmxnet3_2.0.0.30-1vmw.702.0.0.17867351, VMW_bootbank_pvscsi_0.1-2vmw.702.0.0.17867351, VMW_bootbank_qcnic_1.0.15.0-11vmw.702.0.0.17867351, VMW_bootbank_qedentv_3.40.5.53-20vmw.702.0.20.18426014, VMW_bootbank_qedrntv_3.40.5.53-17vmw.702.0.20.18426014, VMW_bootbank_qfle3_1.0.67.0-14vmw.702.0.0.17867351, VMW_bootbank_qfle3f_1.0.51.0-19vmw.702.0.0.17867351, VMW_bootbank_qfle3i_1.0.15.0-12vmw.702.0.0.17867351, VMW_bootbank_qflge_1.1.0.11-1vmw.702.0.0.17867351, VMW_bootbank_rste_2.0.2.0088-7vmw.702.0.0.17867351, VMW_bootbank_sfvmk_2.4.0.2010-4vmw.702.0.0.17867351, VMW_bootbank_smartpqi_70.4000.0.100-6vmw.702.0.0.17867351, VMW_bootbank_vmkata_0.1-1vmw.702.0.0.17867351, VMW_bootbank_vmkfcoe_1.0.0.2-1vmw.702.0.0.17867351, VMW_bootbank_vmkusb_0.1-4vmw.702.0.20.18426014, VMW_bootbank_vmw-ahci_2.0.9-1vmw.702.0.0.17867351, VMware_bootbank_elx-esx-libelxima.so_12.0.1200.0-4vmw.702.0.0.17867351, VMware_bootbank_esx-ui_1.34.8-17417756, VMware_bootbank_lsuv2-hpv2-hpsa-plugin_1.0.0-3vmw.702.0.0.17867351, VMware_bootbank_lsuv2-intelv2-nvme-vmd-plugin_2.0.0-2vmw.702.0.0.17867351, VMware_bootbank_lsuv2-lsiv2-drivers-plugin_1.0.0-5vmw.702.0.0.17867351, VMware_bootbank_lsuv2-nvme-pcie-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-oem-dell-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-oem-hp-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-oem-lenovo-plugin_1.0.0-1vmw.702.0.0.17867351, VMware_bootbank_lsuv2-smartpqiv2-plugin_1.0.0-6vmw.702.0.0.17867351, VMware_bootbank_qlnativefc_4.1.14.0-5vmw.702.0.0.17867351, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.42-1vmw.702.0.0.17867351, VMware_locker_tools-light_11.2.6.17901274-18295176

And with that all i had to do was reboot the server and it came back at the version i needed.

In this guide we will go over upgrading vROPS (vRealize Operations Manager) to the latest 8.6 releases using the .pak file. If you you vRSLCM (vRealize Suite Lifecycle Manager) i would strongly recommend upgrading using the vRSLCM method found here as it does additional checks of the environment like compatibility with the rest of the vRealize Products we might have installed. The release notes for vROPS 8.6 can be found here

To get started we need to download the .pak file from the new customerconnect portal (previously known as my vmware). The direct link to the product download page can be found here. Make sure we have 8.6 selected under the Select Version and click on any of the Go To Downloads links

Once were at the Product Download Page we can download the .pak file under vRealize Operations Manager – Virtual Appliance upgrade .pak file with Cloud Proxy. I also strongly recommend to download the vRealize Operations 8.6.0 – Upgrade Assessment Tool in order to make sure that everything is in order to get a successful upgrade. The Upgrade assessment tool checks for a number of items like dashboards, views, reports etc which can be impacted by metric changes (if any), however it is optional.

Once we click on the download link if were not logged on we will get asked for the login information. This login information is the same as the login to open tickets. Accept the EULA and the download should start

Once we have the downloads completed we can go to https://vrops_url/admin and log in with the admin username\password. Make sure the System Status shows Online and everything is running.

On the left side menu click on Software Update -> Install a software update button

Next we can upload the Upgrade assessment tool in order to run the upgrade assessment (optional) by clicking browse -> select the downloaded .pak file in my case APUAT-8.6.0.18696447.pak and click on Upload

The upload button will stage the package on the vROPS server. We can review the details and click on Next

Accept the terms of agreement and click Next

Review the important notes and click Next

Click Install

We can now follow the installation progress

Once the install is complete we will see the status as completed.

Next lets go and review the tool`s output by going to Support -> Support bundles -> Select the bundle that was just generated and click on the download button

Extract the content of the zip file and go to the apuat-data\report folder and open the index.html. Here we can review Removed\Discontinued Metrics, Dashboards that might be effected as well as a number of system validation checks.

If everything looks good we can return to the admin UI and upload the product upgrade package under Software Update -> Install a software update -> Brose to the downloaded .pak file -> Click on Upload to stage the package -> Click on Next. Follow the rest of the screens till we click submit.

Once the the system is finished the server will get rebooted on the new 8.6 version

During VMworld 2021 we announced vRealize Operations 8.6 which now provides allot of new features and capabilities. You can see an Whats New overview here as well as a feature walk through here.

I this post we will focus specifically on the UI enhancements. The UI changes are the same for the SaaS version as well as the on premise release. Lets dive in!

When we used to log on to vRealize Operations in the past we used to have a list of menu items up top as well as the left side example:

The new simplified navigation interface creates a smoother navigation with enhancements like favorites for dashboards, a new administration page, the ability to pin dashboards and views as well as support for expressions in custom groups. With the UI enhancements we now also have the ability to customize the menu for Non-admins in order to provide the end users of the product with only the access they require.

Here is a quick glimpse into favorites as well as the pinning:

From a navigation perspective the side menu automatically collapses while navigating in order to give us more space to navigate through the menus. With the redesigned UI we now have secondary navigation allowing us to navigate through the menu`s allot easier without having to go back and forward to get where we need to

The object browser also allows us to switch to the correlation mapping by just pressing the toggle switch up top. The correlation mapping is allot easier to navigate as well. In the example below i have an example of a correlation mapping spanning from the PDU system which monitors the power as well as the cabinet that can monitor the fan speeds via the Flowgate management pack that can be downloaded from the Marketplace

The home page has received a couple of enhancements as well with a couple of new additions

I was recently going through a vCenter upgrade and got stuck at an error Exception occurred in postinstallHook. I tried reverting from snapshot and trying again but i ran through the same issue.

The issue in my case was that the catalina.properties had the wrong permissions. The fix was to change the permissions to perfcharts:cis and then resuming the update in the VAMI interface. The catalina.properties is located /usr/lib/vmware-perfcharts/tc-instance/conf/catalina.properties. The command to fix it is

chown perfcharts:cis /usr/lib/vmware-perfcharts/tc-instance/conf/catalina.properties

Then restart the vmware-perfcharts service by running:

service-control --stop vmware-perfcharts
service-control --start vmware-perfcharts

I recently upgraded my vCenter to 7.0.3 18700403 and i noticed that when i went to Lifecycle Manager i was welcomed with this error. “Authentication failed, Lifecycle Manager server could not be contacted.“

Im not aware of a fix for this issue however using the [email protected] seems to be working.