I needed to do a massive rip and replace on some IPv6 IP’s and so a regex seemed the best way to go.
What I was using: Link-local
fe80::([0-9a-f])*:([0-9a-f])*:([0-9a-f])*:([0-9a-f])*
All IPv6 IP’s.
([0-9a-f])*::([0-9a-f])*:([0-9a-f])*:([0-9a-f])*:([0-9a-f])*
It’s funny, I’ve edited this post 4 times because I ran into little catch-22 situations as I continued to work using my test vCO instance. Hopefully this post will save somebody else some time when configuring vCO 5.5 in a cluster. Let’s get started!
Deploy a new VM that will host the PostgreSQL Database. I’m using CentOS just in case you are curious.
You can find the latest version of PostgreSQL with:
yum list postgres*
Now install PostgreSQL-server:
yum install postgresql-server
Once it is done installing then we need to configure Postgres:
chkconfig --level 2345 postgresql on
service postgresql initdb
vim /var/lib/pgsql/data/postgresql.conf
Un-comment and modify the listen_addresses and port:
Next modify the what servers are allowed to talk to the Postgres database and how. The database and user have not been created yet but we are going to call them vco and vcouser. The method is an md5 hash of the password for authentication.
vim /var/lib/pgsql/data/pg_hba.conf
Now start Postgres and create the vCO user and database:
service postgresql start
su postgres
psql
CREATE USER vcouser with PASSWORD '$uperG00dP@ss!';
CREATE DATABASE vco;
GRANT ALL PRIVILEGES on DATABASE vco to vcouser;
\q
Now I’m going to deploy 3 vCO appliances from OVA. Near the end of the process to deploy the OVA you will be prompted to create a password. The second password's username is vmware and it is for the web interface that you will use in a minute to configure vCO.
Go to to http://<primary vCO node IP> and click on Orchestrator Configuration. From here login using vmware for the username and the password that you specified when you deployed the OVA.
Next go to the Database section and fill in the information for the PostgreSQL database server we just built. It should fail with an error that the database needs tables created. Click to create.
Once you click to create the tables then it's time to generate self-signed certificates. Navigate to the Server Certificate section and chose the self-signed option. Give it the FQDN of your VIP. (Example: I have cos-test-vco1, cos-test-vco2 and cos-test-vco3 but the VIP is cos-test-vco)
Now we need to grab our vCenter Server's SSL Certificate. Click on Licenses and SSL Certificates
Give it https://<IP of vC Server> and verify the import.
Add any plugins that you want (they can be found at https://solutionexchange.vmware.com).
Next navigate to the Licenses section and give it the IP of your vCenter. Once this succeeds you should have all green statuses.
It's now time to configure vCO to work in a cluster. Go to Server Availability and change it to have 2 active nodes:
Next navigate to vCenter Server and add a new vC. BEWARE: the default setting is "Session per user" and this will appear to succeed on this screen but will be broken later on down the road if you don't change it. The only reason this should be left at the default is if you are using SSO and the same user has rights on both vCO and vCenter Server.
Now because of the change going from the internal to the external Postgres database we need to reinstall all the plugins even though they show up as green on the configuration screen. Don't believe me? If you continue as-is you will run into the below screenshot where your workflow elements are gone.
To do that click "Reset Current Version" and reboot the VM.
Lastly we need to modify the network binding to the correct IP address:
Once you have reached this point it's time to export the vCO Configuration. This is critical because all of your vCO servers in the cluster must be identical with the exception of the network binding. Copy this file via SCP to the other Orchestrator VMs.
Next repeat the below steps for each additional vCO server.
Import the vCO Master Node's configuration file making sure to UNCHECK the override box.
Configure networking to the correct IP address for each node.
At this point you should get an error that says that vCenter is not configured correctly. Follow the prompts and re-enter your credentials to connect to the vC Server.
Once this is completed start your Primary Node and wait for it to start, then start all the other nodes.
Repeat for all nodes until they show as online.
Congratulations, you have now configured a vCO Cluster with a standby node.
Here’s a little preview of one of our two OneCloud datacenters that is running VMworld 2013 Hands On Labs. We’re using Hyperic, vC Ops and VMware Log Insight to make sure that you guys have the best labs possible! Enjoy!
Check out these awesome custom vC Ops Dashboards!
This one displays vCD information such as the current VM Consoles that each cell server is serving to clients. Most of these metrics are gathered via a custom Hyperic vCD Plugin that we built (to be posted later).
Simple but effective Lab and VM deployment stats and trending.
Shout out to my co-worker Jacob Ross who was my teammate in designing and building the monitoring for VMworld 2013 HOL. Hope you all enjoy the show!
Let’s assume that one of your vCD cells is using a bit of CPU and you are curious what actual task inside of vCD is requiring all of those resources. Here’s how you can find out.
Get the PID from “top –H –u vcloud”; this is the specific task in vCD as opposed to the vCD general process you will find later. In my case the PID = 16225
Convert this PID to Hex = 3F61 or 0x3F61
Find the vCD Java Process; “ps –auxf |grep vcloud” should return at least 2 results. You want the process that has /opt/vmware/vcloud-director/jre/bin/java.
Get a java thread dump with “kill –3 < vCD Java Process PID>”.
Go search the cell.log (/opt/vmware/vcloud-director/logs) for the Hex value of the process that you are trying to identify. In our case the process we want to investigate is 0x3F61. A quick search for that value brings up the java trace of what vCD is attempting to do.
