vRealize Orchestrator - useful tips and commands

This week I have worked with one my customer on vRealize Orchestrator (vRO) Proof of Concept. vRealize Orchestrator is a pretty good tool for data center orchestration but it is a very hidden tool and customers usually do not know they are entitled to use such great way how to automate and orchestrate not only their infrastructure but almost anything.

Here are some good vRO resources

• vRO Video Tutorials
• vRO 3rd party Plugins - link1, link2, link3.

And here are some of my technical notes from the PoC.

TCP Ports

• VRO Server Service is running on port 8281. This is the port where vRO Client has to connect. Direct access is https://[VRO-HOSTNAME]:8281/vco/

vRO Client login

• VRO Server Control is running on port 8283. This is the port where vRO Orchestrator Control Center is running.  Direct access is https://[VRO-HOSTNAME]:8283/vco-controlcenter/

vRealize Orchestrator Control Center

• VRO VAMI (Virtual Appliance Management Interface) is running on port 5480

Network configuration from the console or over ssh session
/opt/vmware/share/vami/vami_config_net

Reset vRO Authentication
/var/lib/vco/tools/configuration-cli/bin/vro-configure.sh reset-authentication

Restart vRO Configurator service
service vco-configurator restart

Restart vRO Server service
service vco-server restart

vRO 7.4 configuration synchronization across cluster nodes
(source)
In vRO 7.4 there are changes in regards to the way the configuration synchronization is done. Configuration changes done through the control center are going to be "permanent" and going to be replicated to the other nodes as well. Changes done manually like through editing a file by a text editor are going to be overwritten with the latest configuration.
Therefore if you make manual changes like editing the js-io-rights file you need to execute the following CLI command to apply the changes to the latest configuration which going to be replicated.
you should do the following

1. stop the control center
2. make manual changes
3. execute /var/lib/vco/tools/configuration-cli/bin/vro-configure.sh sync-local
4. start control center

Time considerations
Time on vRO and Authentication Provider (vCenter/PSC, vRealize Automation/vIDM) must be in sync otherwise you will see error message like "Server returned 'request expired' less than 0 seconds after request was issued". This issue occurs due to a time skew between vCenter Server and the vSphere Replication Appliance.

You should use NTP or sync time with the host. If you want to synchronize vRO time with host use this command from Guest OS
vmware-toolbox-cmd timesync enable
to disable time sync with host use this command
vmware-toolbox-cmd timesync disable

vRealize Orchestrator 7.x - Unlocking vRO Root Account after too many failed login attempts

When you did too many failed login attempts as root account, your vRO root account will be locked. As SSH does not work, you need console access to the vRO server.

Step 1 - Gain access vRO server root shell via Console

Step 2 - Reboot server

Step 3 - When the GRUB bootloaders appear, press spacebar to disable autoboot.

Step 4 - Select VMware vRealize Orchestrator Appliance and type “e” to edit the boot commands. Then move down to the second line showing kernel boot parameter and type “e” again.

Step 5 - Append the init=/bin/bash to the kernel options.

Step 6 - Hit Enter and the GRUB menu will appear again. This time hit “b” to start the boot process.

Step 7 - Now you should be in the shell - ready to issue commands to unlock or reset the password.

Step 8 - To unlock account use type following command:
# pam_tally2 --user root --reset

Optional Step 9 - If you cannot remember the password change password by using passwd command:
# passwd root 

Optional Step 10 - Disabling the lockout possible can come in handy. To do so modify the /etc/pam.d/common-auth file. Use vi or any preferred editor to modify the common-auth file. Comment out the line where “pam_tally2.so deny=3….” 

Undocumented SDRS Advanced Options

Almost two years ago, I was challenged by one VMware customer who has experienced failed VM's provisioning in case of parallel VM deployments. SDRS default behavior is not optimized for fast multiple parallel deployments because it returns just SDRS recommendations (step 1) and later (step 2) these recommendations are applied by someone else who is executing VM provisioning. Back in the days, when SDRS was designed and developed, it was optimized for few VM provisioning, however nowadays when we are in a cloud era, more parallel provisionings are common.

Disclaimer: Please note that default behavior works best for most of the cases, do use advanced options only when it is required. Below are some of the Storage DRS advanced options user can configure on Storage DRS cluster. These options may or may not have been properly documented from my research/digging and it is most likely not supported by VMware. Please take caution if you decide to play with this advanced settings.

SDRS Advanced settings

VraInitPlacement
The reason behind this option is explained in this video - "VMware vSphere SDRS VM provisioning process". Based on my customer feedback, VMware engineering has introduced 'VraInitialPlacement' - SDRS advanced setting - which reconfigures SDRS to generate only one recommendation and reserve the resource till "lease" expires. Lease time is defined in API call as property resourceLeaseDurationSec. It is by default 0 (no lease) but it can be reconfigured on VRA side. It is used in conjunction with VRA custom property "VirtualMachine.Admin.Datastore.Cluster.ResourceLeaseDurationSec" at VMware VRA Documentation here - https://docs.vmware.com/en/vRealize-Automation/7.4/com.vmware.vra.prepare.use.doc/GUID-FA2ED665-4973-435C-A93B-8E4EAB5D1F8A.html

Custom property description: When provisioning to multiple VMs and using SDRS, specifies a value in seconds, in the range of 30 to 3600, for reserving storage resources during the RecommendDataStore API call. You can add this property to a business group or blueprint or when you request provisioning. The lease lock is only applied to the datastore that is used by the deployment, not all datastores in the storage cluster. The lease lock is released when provisioning either completes or fails. If not specified, no lock is applied to the storage resources at provisioning time.

Long story short, when 'VraInitPlacement' is set to "1", it generates only one recommendation. This is VMware internal undocumented option but in theory, it can be done by vSphere admin. But Lease Time is not configurable by vSphere Admin and it is the parameter (property) during vSphere SOAP API call asking for SDRS recommendations.

To set this advanced parameter choose Datastore Cluster > Edit Settings > SDRS Automation > Advanced Options (SDRS) > insert property  'VraInitialPlacement' with value 1

VraExpandDisk 
This is another SDRS Advanced option. When this is enabled and an expand disk will exceed the size of the datastore the disk will be storage vMotioned to another datastore which can service the request.

To set this advanced parameter choose Datastore Cluster > Edit Settings > SDRS Automation > Advanced Options (SDRS) > insert property  'VraExpandDisk' with value 1

EnforceStorageProfiles
To configure Storage DRS interop with SPBM, below options need to be set.

• 0 – disabled (default)
• 1 – soft enforcement
• 2 – hard enforcement

percentIdleMBinSpaceDemand
The PercentIdleMBinSpaceDemand setting defines the percentage of IdleMB that is added to the allocated space of a VMDK during free space calculation of the datastore. The default value is set to 25%. This value can range from 0 to 100.

For more info look at http://frankdenneman.nl/2012/10/01/avoiding-vmdk-level-over-commitment-while-using-thin-disks-and-storage-drs/

EnforceCorrelationForAffinity
Use datastore correlation while enforcing/fixing anti-affinity rules

• 0 – disabled (default)
• 1 – soft enforcement
• 2 – hard enforcement

OTHER VMWARE VSPHERE ADVANCED SETTINGS
Back in the days, William Lam published the blog post "New vSphere 5 HA, DRS and SDRS Advanced/Hidden Options". It is worth to read it.