vCloud Director Logging

I was recently asked how to go about configuring the Log Insight Agent with VMware vCloud Director and thought that I would take the time to document it here for anyone else who is interested.

Logging in vCD is normally handled by log4j and configured by $VCLOUD_HOME/etc/log4j.properties.with the official KB located here. You should either use log4j OR the Log Insight Agent, but not both or you will have event duplication.

Log4j Configuration
First a quick overview of the log4j configuration.
1. Open $VCLOUD_HOME/etc/log4j.properties
2. Append "vcloud.system.syslog" to the rootLogger and make sure to not forget the comma before it

 3. At the bottom of the file go ahead and append the below 6 lines outlined in the KB making sure to change your target FQDN.

4. Unfortunately with vCD 5.x you also have to restart the vmware-vcd service for the changes to take effect. Hint: if you don't want to restart the services and take an outage you can continue reading and use the Log Insight Agent instead :)

Log Insight Agent
vCloud Director supports RHEL and CentOS so you only need to worry about the RPM install of the Log Insight Agent. First though, we need to do some prep work on the Log Insight Server.

1. Install the vCD Content Pack - On the Log Insight Server that you will be pointing your LI Agent at you will need to have the vCD Content Pack installed so the Agent Group is available. This is easily done via the Marketplace

2. Create your Agent Group - From the Administration window select Agents and then highlight the vCloud Director Cell Servers pre-defined Agent Group.

 

Next scroll to the bottom of the page and select Copy Template

3. Next you will need to define a filter that limits this collection to only vCD Cells. My test example here is very basic and limiting to hosts with a certain hostname prefix.

You can see in the bottom section of the agent group the actual files that will be collected by the agent.

By default the agent only collects info level logs but you can easily switch that to debug level logs if you desire. Feel free to check out my very basic sizing calculator on Github if you are curious of the impact of the additional logs. For now, just hit Save Agent Group to continue.

4. Now you are ready for the actual agent installation! You will need to copy the RPM to your vCD cells /tmp directory. The LI Agent will need to be installed and configured on every vCD Server.

Note: At some point after this step you will need to decide when to remove the log4j configuration and when to enable the Agent. I would personally recommend disabling log4j before installing the agent as short term you won't lose any events since the LI Agent will go through all the log files on the server and forward them on.

5. Install the agent via RPM

6.  If you downloaded the agent from the Log Insight server it is supposed to be forwarding to then you don't need to modify the liagent.ini file but if you downloaded it from my.vmware.com or another Log Insight Server you will need to update the target hostname.

 

If you want to be secure you can enable ssl and your /etc/liagent.ini file will look more like the below

Don't forget that you'll need certificates for SSL so follow the full official documentation available here 

 

At this point you should see that your agents are alive and sending data to your Log Insight Server

 

Log Insight Content Pack for vCD

Hey all,
I just released the official GA version of the vCD 5.5 Log Insight Content Pack which is now available on the VMware Solutions Exchange at https://solutionexchange.vmware.com/store/products/vcd-log-insight-content-pack#.U77ZHPldV8E. Take a look and let me know of any changes that you think would be helpful to you.

Cloud Content Pack (vCD) for Log Insight

For those of you out there who use VMware vCloud Director and Log Insight you may be interested in a content pack that we have built for use by the OneCloud team to help make our cloud run smoother and to give us a ton of (wait for it) Insight into our environment. It's been a work in progress for about 9 months off and on but has served us very well. I hope that it serves you just as well.

Here are some screenshots:

 

I hope that this content pack is able to help you better manage your VMware vCloud Director environment. You can download the Content Pack here just be aware that this is not released by VMware and is not supported by them. Like everything else on my blog it just came from a random blogger on the internet :)

 

 

 

VMworld 2013 Hands On Labs!

Here’s a little preview of one of our two OneCloud datacenters that is running VMworld 2013 Hands On Labs. We’re using Hyperic, vC Ops and VMware Log Insight to make sure that you guys have the best labs possible! Enjoy!

Check out these awesome custom vC Ops Dashboards!

This one displays vCD information such as the current VM Consoles that each cell server is serving to clients. Most of these metrics are gathered via a custom Hyperic vCD Plugin that we built (to be posted later).

Simple but effective Lab and VM deployment stats and trending.

Shout out to my co-worker Jacob Ross who was my teammate in designing and building the monitoring for VMworld 2013 HOL. Hope you all enjoy the show!

Isolating Java Process to vCD Job

Let’s assume that one of your vCD cells is using a bit of CPU and you are curious what actual task inside of vCD is requiring all of those resources. Here’s how you can find out.

Get the PID from “top –H –u vcloud”; this is the specific task in vCD as opposed to the vCD general process you will find later. In my case the PID = 16225

Convert this PID to Hex = 3F61 or 0x3F61

Find the vCD Java Process; “ps –auxf |grep vcloud” should return at least 2 results. You want the process that has /opt/vmware/vcloud-director/jre/bin/java.

Get a java thread dump with “kill –3 < vCD Java Process PID>”.

Go search the cell.log (/opt/vmware/vcloud-director/logs) for the Hex value of the process that you are trying to identify. In our case the process we want to investigate is 0x3F61. A quick search for that value brings up the java trace of what vCD is attempting to do.

Writing PowerCLI for Performance

I was building a piece of PowerCLU code to check on my vCD Deployed vAPPs for certain errors and because it’s me I was curious what performance difference there was between using array based operations verses piped operations in powershell. Here’s what I found out.

Here is the array based operation – time to completion 2 minutes 30 seconds.

$myVAPP = @(Get-CIVApp | Select-Object Org, Name, Status)
        foreach ($myVAPP.Name in $myVAPP)
            {
                IF ($myVAPP.Status -eq "FailedCreation")
                    {
                        Write-Host 'vAPP' $myVAPP.Name' - Status is' $myVAPP.Status
                    }
                ELSE
                    {
                            IF ($myVAPP.Status -eq "Unresolved")
                            {
                                Write-Host 'vAPP' $myVAPP.Name' - Status is' $myVAPP.Status
                            }
                            else
                            {
                                IF ($myVAPP.Status -eq "Unknown")
                                {
                                    Write-Host 'vAPP' $myVAPP.Name' - Status is' $myVAPP.Status
                                }
                                else
                                {
                                }
                            }                           
                    }
            }
           

Here is the pipe based operation which is recommended for reasons that I now can testify for first hand.

Get-CIVApp -Status "Unresolved", "FailedCreation", "Unknown", "Unrecognized" | Select-Object Org, Name, Status

This operation returned the same results as the array based operation in 14 seconds, that’s 9% of the original time of using an array…

Using ESXi Autodeploy

I’m new to the ESXi Autodeploy bandwagon but I’ve got to say after using it for the first time I’m sold. If you are interested in what Autodeploy is a quick summary is that it allows you to dynamically install ESXi via PXE on servers and add them to your vSphere Infrastructure in an intelligent manner, all automatically.

For this real life example I have 2 Clusters of 7 HP BL490c G7 Blades that I need to dynamically provision with ESXi (complete with a custom NIC driver) and add them to the correct cluster based on IP ranges. Here is how I went about doing so.

Assumptions: I am assuming that you are not new to vSphere and that your environment already has a functioning vCenter Server (ver5), PowerCLI is installed, ESXi Coredump Collector and Autodeploy is installed. If you are missing any of these please follow the below link to get them installed before continuing: http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-CAB84194-3D8E-45F0-ABF9-0277710C8F98.html

A couple other things are also needed:

1. You need to set the hardware clock on all your servers to have the correct date and time. If you don’t do this there is a chance that you will run into an issue where the blades will not add to the cluster and you will find that they are trying to use an evaluation license, even with bulk licensing correctly configured.

2. A TFTP Server must be installed. I used the free Solarwinds TFTP server which I installed on my vCenter Server.

a. Create a folder called TFTP-Root

b. Create a folder called ESX_Depots

c. Open you vCenter Client and on the home screen select the “Auto Deploy” option. From there click on the “Download TFTP Boot Zip” and extract the contents into the TFTP-Root folder you created earlier.

3. Create DHCP Reservations for all of your blade servers and give them the following options:

a. All normal networking options (003 – Router, 006 – DNS, 015 – DNS Domain Name, etc…)

b. Add Option 066 “Boot Server Host Name” and give it the value of your TFTP Server, in my case this is also my vCenter Server.

c. Add Option 067 “Bootfile Name” with the value of “undionly.kpxe.vmw-hardwired”. You will notice that this is a file that is now located in your TFTP-Root directory from the zip file you extracted there.

4. You should be able to boot one of your servers at this point and see a screen that informs you that it booted from PXE and no ESX image is assigned and your server will reboot in 5 minutes… So far, so good…

At this point we have got the infrastructure in place to support ESXi Autodeploy and we have to create the actual ESX images that we want to install. Also we need to create the Host Profile(s) that we want to use and add a custom networking driver because the one included by default does not work with this specific model and generation blade. Let’s get started:

1. Download ESX Image – At this point you need to download a couple things from the VMware website. The first is the .zip Depot files for the version of ESXi that you want to install. The second is a package called vSphere-HA-Depot or vmware-fdm that you will need if you are going to add your ESX hosts to a HA enabled cluster.

2. In my specific case the driver that is included with ESX for our NIC on the BL490c G7 blade has a bug that was fixed by HP; that means that I will need to download the additional driver as well. Most people will not need to do this step unless you are using HP G7 blades with the NIC issues.

Once you have downloaded the software depots that are needed you are ready to create the ESXi image. To do that we are going to use the versatile PowerCLI. (Hint: If you are ever wondering about the syntax of a command or what options you have there is a great list of commands and syntax located at http://pubs.vmware.com/vsphere-50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID-16E1D78F-2466-4794-8D12-BE5EC7AA41D3.html)

Step 1: Adding software depots: We downloaded all these software depots (ESX, drivers and vmware-fdm) but now we need to make them accessible for use in the PowerCLI. To do that we add all of them via the syntax: Add-EsxSoftwareDepot –DepotUrl “c:\esx_depot\<filename>

Now we should be able to see all of our available software packages (VIBs) by issuing the command get-EsxSoftwarePackage:

On this screen you can see that we have 2 versions of the net-be2net, the newer one being from Emulex as well as a new file called vmware-fdm. Cool, we are on a roll….

Next we need to clone the original ESXi image so that we can make changes to it. First let me show you what the default looks like: Execute Get-EsxImageProfile and you should see the 2 entries for the version of ESX that was in your depot that you added. Now using New-EsxImageProfile –CloneProfile <original name> -Name <New Name> create a copy of the ESXi image that we will be editing with our driver and vmware HA Depot (vmware-fdm).

Now that we have our ESXi Image created we need to add the new driver and the vmware-fdm VIBs. Here’s how I did it:

1. For demonstration purposes I tried to remove the vmware-fdm and as you can see it failed because it is not included by default. Next I removed the Broadcom network driver from our Image. That one succeeded as expected.

2. Now I need to add the new network driver. In theory I could just specify the driver by name and it would install the new one because the version number is higher but just for grins I want to manually specify the entire driver name and version… You can see from the below screenshot that you can specify the short name or the name and version.

Now that we have this nice new shiny ESXi image with custom drivers, now what… Well, we need to create a Deployment Rule that has the logic that tells vCenter what servers get this image as well as what Host Profile to use but before we go there I need to throw out a couple hints.

Hint #1: On your first round you will want to create a deployment rule that does not use a host profile. Once that server is provisioned you will want to configure it and then use it as a template for creating your Host Profile that will be used to build all the other servers.

Hint #2: You will get a message that the password was not changed on the ESXi servers even if you create a host profile using a server as a template that has a valid root password. You need to manually edit your Host Profile after creating it to have the below information to get a new root password applied to your hosts as they come online.

Hint #3: You will most likely get a couple errors when you try and apply your host profile that say the Path Selection Policy needs changed and that the “device mpx.vmhba32:xx:xx:xx needs to be reset.” These errors are sometimes caused by virtual CDROMs or local storage and will cause your servers to fail compliance. This is easily avoided by disabling those options in the host profile if you so choose. A great article on doing so is located at http://bsmith9999.blogspot.com/2012/01/vmware-host-profiles-giving-errors.html

Hint #4: You can configure the ESXi Dump Collector via Host Profiles but for some reason I had to do it manually on my first host (the one that you create the host profile against) using the below command for it to work properly. Maybe just something goofy that I did, or maybe not… anyways here are the commands to do it manually:

Esxcli system coredump network set –interface-name vmk0 –server-ipv4 <ip of coredump server> --server-port 6500

Esxcli system coredump network set –enable true

And the location in the host profile:

Ok, now that you have created your host profile using the first host now it is time to refine your deployment rules. First off go ahead and delete your original deployment rule that did not include any host profiles or install patterns since it is no longer needed.

Remove-DeployRule –Name <Whatever you named it>

Now we create a new one that uses IPs to limit what servers get the ESXi image installed and assigns them the new Host Profile we just created.

As you can see from the above screenshot we do a couple things:

Create a new Deployment Rule called vCD_Cluster_4 that uses:

i. “Cluster_4_vDS Ready” Host Profile

ii. 5.0-GA ESXi Image Profile we created earlier

iii. Adds the servers that fall into the 10.25.96.29-35 to the “Cluster 4” Cluster in my vCenter. All servers that are outside of this range will not get this install.

Last of all we need to enable that Deployment Rule by issuing the below commands:

One last hint: If you have a Host Profile assigned to the cluster in vCenter that Host Profile takes precedence over the one that you specified in the Deployment Rules.

There you go, now you have a good idea on how to get your environment up and running via ESXi Autodeploy complete with HA VIBs and a custom driver. It’s really cool technology and now I don’t know why I’ll ever need an ISO of ESXi again…