vCloud Director Logging

I was recently asked how to go about configuring the Log Insight Agent with VMware vCloud Director and thought that I would take the time to document it here for anyone else who is interested.

Logging in vCD is normally handled by log4j and configured by $VCLOUD_HOME/etc/log4j.properties.with the official KB located here. You should either use log4j OR the Log Insight Agent, but not both or you will have event duplication.

Log4j Configuration
First a quick overview of the log4j configuration.
1. Open $VCLOUD_HOME/etc/log4j.properties
2. Append "vcloud.system.syslog" to the rootLogger and make sure to not forget the comma before it

 3. At the bottom of the file go ahead and append the below 6 lines outlined in the KB making sure to change your target FQDN.

4. Unfortunately with vCD 5.x you also have to restart the vmware-vcd service for the changes to take effect. Hint: if you don't want to restart the services and take an outage you can continue reading and use the Log Insight Agent instead :)

Log Insight Agent
vCloud Director supports RHEL and CentOS so you only need to worry about the RPM install of the Log Insight Agent. First though, we need to do some prep work on the Log Insight Server.

1. Install the vCD Content Pack - On the Log Insight Server that you will be pointing your LI Agent at you will need to have the vCD Content Pack installed so the Agent Group is available. This is easily done via the Marketplace

2. Create your Agent Group - From the Administration window select Agents and then highlight the vCloud Director Cell Servers pre-defined Agent Group.

 

Next scroll to the bottom of the page and select Copy Template

3. Next you will need to define a filter that limits this collection to only vCD Cells. My test example here is very basic and limiting to hosts with a certain hostname prefix.

You can see in the bottom section of the agent group the actual files that will be collected by the agent.

By default the agent only collects info level logs but you can easily switch that to debug level logs if you desire. Feel free to check out my very basic sizing calculator on Github if you are curious of the impact of the additional logs. For now, just hit Save Agent Group to continue.

4. Now you are ready for the actual agent installation! You will need to copy the RPM to your vCD cells /tmp directory. The LI Agent will need to be installed and configured on every vCD Server.

Note: At some point after this step you will need to decide when to remove the log4j configuration and when to enable the Agent. I would personally recommend disabling log4j before installing the agent as short term you won't lose any events since the LI Agent will go through all the log files on the server and forward them on.

5. Install the agent via RPM

6.  If you downloaded the agent from the Log Insight server it is supposed to be forwarding to then you don't need to modify the liagent.ini file but if you downloaded it from my.vmware.com or another Log Insight Server you will need to update the target hostname.

 

If you want to be secure you can enable ssl and your /etc/liagent.ini file will look more like the below

Don't forget that you'll need certificates for SSL so follow the full official documentation available here 

 

At this point you should see that your agents are alive and sending data to your Log Insight Server

 

Cloud Content Pack (vCD) for Log Insight

For those of you out there who use VMware vCloud Director and Log Insight you may be interested in a content pack that we have built for use by the OneCloud team to help make our cloud run smoother and to give us a ton of (wait for it) Insight into our environment. It's been a work in progress for about 9 months off and on but has served us very well. I hope that it serves you just as well.

Here are some screenshots:

 

I hope that this content pack is able to help you better manage your VMware vCloud Director environment. You can download the Content Pack here just be aware that this is not released by VMware and is not supported by them. Like everything else on my blog it just came from a random blogger on the internet :)

 

 

 

VMworld 2013 Hands On Labs!

Here’s a little preview of one of our two OneCloud datacenters that is running VMworld 2013 Hands On Labs. We’re using Hyperic, vC Ops and VMware Log Insight to make sure that you guys have the best labs possible! Enjoy!

Check out these awesome custom vC Ops Dashboards!

This one displays vCD information such as the current VM Consoles that each cell server is serving to clients. Most of these metrics are gathered via a custom Hyperic vCD Plugin that we built (to be posted later).

Simple but effective Lab and VM deployment stats and trending.

Shout out to my co-worker Jacob Ross who was my teammate in designing and building the monitoring for VMworld 2013 HOL. Hope you all enjoy the show!

Isolating Java Process to vCD Job

Let’s assume that one of your vCD cells is using a bit of CPU and you are curious what actual task inside of vCD is requiring all of those resources. Here’s how you can find out.

Get the PID from “top –H –u vcloud”; this is the specific task in vCD as opposed to the vCD general process you will find later. In my case the PID = 16225

Convert this PID to Hex = 3F61 or 0x3F61

Find the vCD Java Process; “ps –auxf |grep vcloud” should return at least 2 results. You want the process that has /opt/vmware/vcloud-director/jre/bin/java.

Get a java thread dump with “kill –3 < vCD Java Process PID>”.

Go search the cell.log (/opt/vmware/vcloud-director/logs) for the Hex value of the process that you are trying to identify. In our case the process we want to investigate is 0x3F61. A quick search for that value brings up the java trace of what vCD is attempting to do.