Log Insight Configuration APIs

For those of you who have followed my blog you will know that I deal with Log Insight quite a bit in our production environments. Because of this I was excited that in the latest release of Log Insight 3.3 there are several new Configuration API's released under Tech Preview status. That said, the documentation around these APIs is very difficult to nail down. The exciting part is that I've just uploaded a new and unofficial standalone audit and remediation tool to my github repo! As always this code is my personal code and not supported or officially recognized by VMware.

Here's how it works: 
The tool reads the desired state of your Log Insight Server from a JSON file that you define. It can use that file to then connect to the Log Insight Server and audit it to see if it matches your desired state. If you wish you can throw in the -r switch and the script will make the Log Insight Server match your desired state.

Let's see it in action:
First up, let's pull up the embedded documentation by running the script with the -d switch to see what the JSON file needs to look like. I've taken pains to try and include complex examples so that you won't be left in the dark on anything.

  

After creating a new JSON file with our desired state it's time to run the tool in audit only mode by just specifying the -f flag and the name of our JSON file. The results that come back are that we have several areas that need remediation (email, event forwarders) and 1 (content packs) that cannot be remediated yet (hopefully in a later version).

 

That's all good but we want the tool to fix those issues so we append the -r flag

 

If you run the tool again the output comes back as all objects matching desired state but the nice thing is that you don't need to run it again. Once the remediation HTTP POST is sent to the server the tool will automatically go back and query the server for the configuration to verify that your changes have been implemented and the server is now set correctly. It will then show you success in the message immediately following the remediation step. 

The portions of Log Insight that the tool has the ability to configure are:

License Key

NTP Configuration

SMTP Configuration

Event Forwarder Configuration

Active Directory Configuration

RBAC Configuration

Content Packs (audit only right now)

Stay tuned as I plan on updating the tool over time as more APIs are released and as my python knowledge increases. In the meantime happy auditing and automatic remediation!

 

SELinux Hangs on Relabel Attempt - RHEL 7

I've recently run into an issue where during a recovery scenario it was necessary to relabel SELinux contexts on RHEL7.1. This is normally done by either placing an empty file at the root of the file system "touch /.autorelabel" or using the "fixfiles onboot" command; both of which I tried in this case. However in this case upon reboot the machine just hung at "Reached Target initrd Default Target" with no sign of even attempting to relabel the filesystem. Doing a bit of troubleshooting isolated the issue to SELinux as adding the "enforcing=0" parameter to GRUB allowed the machine to boot without issue. I tried quite a few different things including setting SELinux to "Permissive" in /etc/selinux/config and then back again, as well as a failed attempt to use "fixfiles restore /" and "restorecon -Rv /" which I'm assuming failed because SELinux was in permissive mode. Until today I've never seen a machine that won't respect the "touch /.autorelabel" nuclear option.

Ok, here's the fix that I found, odd as it is:
1. Modify GRUB to include "enforcing=0" to allow the OS to boot this first time without SELinux
2. Once inside the OS make sure that /etc/selinux/config is set to enforcing
3. Change the default runlevel from graphical to multi-user (think runlevel 3) with "systemctl set-default multi-user.target"
4. Reboot without modifying GRUB so that selinux is properly enabled on this boot

On the next reboot oddly enough the system recognized that a relabel had been ordered and proceeded as it should have the whole time. After another reboot and setting the default target back to graphical "systemctl set-default graphical.target" and another reboot as a sanity check it's working as expected again. Very odd problem and I must admit a very odd solution....

Log Insight - Migration to debug level logs

One of the projects that I am working on is enabling the forwarding of debug logs on all of our VMware vCloud Director Cells to our global Log Insight instance. To do this however we need a fairly accurate appraisal of what the increased overhead is going to look like. As part of this process I'm starting to create a python program that will allow me to quickly find what the current Events per Second (EPS) and log size in KBps are.

As you can see the script can be run locally or be pointed at a remote host and looks for the latest fully committed debug log. If you don't want to use that one, no worries, you can easily specify a different log file to use. If the target is a remote server the script will copy the appropiate log file to the machine running the script and then do the analytics locally to remove any possibility of unnecessary overhead from the cell server.

The script is still in active development as a side project but I hope to add the ability to query vCenter Servers as well in the near future. If you're curious the code is hosted on my Github repo and as always is not supported or affiliated with VMware in any way....

Zenoss Monitoring WinRM Error (Server not found in Kerberos database: HTTP@XX.XX.XX.XX

We recently rolled out Zenoss 5 in an environment and when we attempted to connect to Windows Servers over WinRM using a domain user received the below error:

Error on : Server not found in Kerberos database: HTTP@10.xx.xx.xx

  

After doing a tcpdump on the Zenoss server using "tcpdump -s 65535 -w filename.dmp"and loading the results into wireshark I found this in the traffic between Zenoss and the Domain Controller:

 

Turns out the Zenoss server was looking for an SPN of the FQDN of my Windows host and not the hostname. We went ahead and added a new SPN for the FQDN using "setspn -S HTTP/ < FQDN > < hostname >" and that resolved the issues!

One other side note: In one of the Zenoss documents that I saw it mentioned the need to install the krb5-workstation package on your remote collectors. This is easily accomplished by "yum -y install krb5-workstation" but due to the sequence of of my troubleshooting I cannot prove/disprove that it made a difference. If anyone finds this and can sort out that detail I'd be happy to hear the results.

Presenting at VMworld 2015

I will be presenting at VMworld 2015 in session MGT4579 on "Data In-Sight!! Experiences Running VMware's Private Cloud with Log Insight". If you are at VMworld feel free to attend and say hi as I'd love to get to meet you in person!

Log Insight Content Pack for vCD

Hey all,
I just released the official GA version of the vCD 5.5 Log Insight Content Pack which is now available on the VMware Solutions Exchange at https://solutionexchange.vmware.com/store/products/vcd-log-insight-content-pack#.U77ZHPldV8E. Take a look and let me know of any changes that you think would be helpful to you.

Setting vShield Edge Device Syslog via API

If you need to update your vSE devices to send traffic to a syslog server then you might be slightly disappointed to see that there are no instructions in the vCNS API guide to do this, especially if you have a bunch of edges. I experienced this same feeling today and am happy to say that now you don’t have to. Below are a quick couple scripts that will allow you to update single edges or a whole bunch at once using curl!

1. First off we need to get a list of our edge devices from the vCNS Manager. We will be using the edgeID acquired here to configure the syslog settings in a minute.

2. This list is a bit too much for our use so I’m going to parse it down to just the edgeID of all the devices.

3. But I’m going to add them all to a text file (edges_test.txt) that I can parse later (code below):

curl -k -H "Authorization: Basic XXXXXXXXXXXXX" -X GET https://vsm.sub.domain.com/api/3.0/edges | xmllint --format - | grep "<id>edge-[0-9]*" | sed -n 's/<id>//p' | sed -n 's/<\/id>//p' > edges_test.txt

Now you have to make a decision, modify individual edges or all of them?

a. Let’s just edit one (MAKE SURE to set the edgeID in the below statement):

curl -k -H "Authorization: Basic XXXXXXXXXXXXX" -H "Content-Type: application/xml" -d '<?xml version="1.0" encoding="UTF-8"?><syslog><enabled>true</enabled><protocol>udp</protocol><serverAddresses><ipAddress>XX.XX.XX.XX</ipAddress></serverAddresses></syslog>' -X PUT https://vsm.sub.domain.com/api/3.0/edges/edge-282/syslog/config

b. Let’s edit them all! For this one I have a simple bash script that loops through the text file with all the edge devices and runs the curl statement against them.

Here’s the script:

while read edge; do
echo "Beginning Update on $edge"
curl -k -H "Authorization: Basic XXXXXXXXXXXXX" -H "Content-Type: application/xml" -d '<?xml version="1.0" encoding="UTF-8"?><syslog><enabled>true</enabled><protocol>udp</protocol><serverAddresses><ipAddress>XX.XX.XX.XX</ipAddress></serverAddresses></syslog>' -X PUT https://vsm.sub.domain.com/api/3.0/edges/$edge/syslog/config
echo "Ending Update on $edge"
sleep 5s
done < edges_test.txt

Really simple but very effective!

Now all that is left is to verify the results:

curl -k -H "Authorization: Basic XXXXXXXXXXXXX" -X GET https://vsm.sub.domain.com/api/3.0/edges/edge-282/syslog/config | xmllint --format -

Have fun not having to use the UI :)