Early Boot Windows Debugging - Part 1 - Basics

I have a Windows Server 2012 VM that will not boot past the Windows splash screen but throws a BSOD with the error "SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (NETIO.SYS). It's been a long while since working on troubleshooting Windows (I primarily use CentOS) but here's what I've found. I don't have the solution yet but I'm recording some tidbits that I found so I will have them later.

First a bit of preamble:

1. Advanced Boot Options - When you select "Enable Boot Logging" this is supposed to write a log file named ntbtlog.txt. However, in this particular case that never happens. This is presumably because it is before the appropriate driver is loaded to write log files. However with 2012 this is conjecture since the latest Microsoft documentation that I can find applies to Server 2000. Regardless of reason, it isn't captured in this instance.
2. This VM was originally running on ESXi but I have exported and OVF to my local VMware Workstation for my troubleshooting.
3. In the below operations I will be referencing "d:\" which is actually the c:\ of the server. It is referenced from the rescue command prompt as d:\ on my system.

Step 1: Boot to the command prompt from the troubleshooting menu in the Automatic Repair wizard
Step 2: Run a chkdsk to verify the filesystem is in working order. My scan came back with required repairs which it corrected. Subsequent scans come back clean.

Command: chkdsk d: /f

Step 2: Run sfc to verify that Windows is ok. This returns that everything is ok

Command: sfc /offbootdir=d:\ /offwindir=d:\Windows /scannow

Step 3 - Just for grins I also ran DISM (Deployment Image Servicing and Management) to check the integrity. It will throw a warning if you don't give it a scratch directory so I just created a temporary one on my drive. This also returns no found corruption.

Command: dism /image=d:\ /cleanup-image /scan-health /scratchdir=d:\temp

So far, so good... except it still won't boot up. I have an existing "twin" of this machine that should match it in most regards so just to be super certain I also run a manual hashing check on netio.sys and sacdrv.sys (more on that file later). The syntax for that is:

certutil.exe -hashfile drivers\netio.sys md5 (or sha1)

The number 1 cause of netio.sys BSOD are driver conflicts according to googling so I start down that road next. An export of all the drivers between the 2 systems shows that they are absolutely identical. Because that doesn't help me I start yanking out drivers to see if it will make a difference.

To get a list of non-Microsoft drivers I again use DISM and find that there are fortunately only 8 to worry about.

Command: dism /image:d:\ /scratchdir:d:\temp /get-drivers

I'm going to start removing drivers to see if that makes any difference. Again, using DISM I start by removing the vmxnet3 driver since it makes the most sense considering a netio.sys error.

Command: dism /image:d:\ /scratchdir:d:\tetmp /remove-driver:oem4.inf

After a reboot, no change. In 1 of my tests I also then proceed to remove the 7 remaining drivers, that also did nothing. Time to get more information.... Queue next post.... 

Log Insight Configuration API Audit and Standalone Remediation Tool - Updated!

For those of you who are interested I have updated the API based audit and remediation tool with a couple new features. After all, what is the use of automation if it isn't user friendly?

1. Better error handling of remediation errors: In the past you would just get a message to the effect of "Something went wrong" but now the tool will pass the HTTP status code and Error Details from the Log Insight Server's response to your remediation request. In the below example you can see this in action.

2. Now includes a wizard to help build a simplified JSON configuration file! Now, without having to create a single bit of JSON you can quickly get value from the tool. The wizard is simplified because let's be honest, if you want the wizard you don't want to answer 250 questions. Because of this some things are assumed/disabled. If you want them then you can simply add it to the code or use the template in the included docs (use the -d switch).

I hope that this helps you get started in seeing the value of using Configuration APIs to manage your Log Insight Servers!

Log Insight Configuration APIs

For those of you who have followed my blog you will know that I deal with Log Insight quite a bit in our production environments. Because of this I was excited that in the latest release of Log Insight 3.3 there are several new Configuration API's released under Tech Preview status. That said, the documentation around these APIs is very difficult to nail down. The exciting part is that I've just uploaded a new and unofficial standalone audit and remediation tool to my github repo! As always this code is my personal code and not supported or officially recognized by VMware.

Here's how it works: 
The tool reads the desired state of your Log Insight Server from a JSON file that you define. It can use that file to then connect to the Log Insight Server and audit it to see if it matches your desired state. If you wish you can throw in the -r switch and the script will make the Log Insight Server match your desired state.

Let's see it in action:
First up, let's pull up the embedded documentation by running the script with the -d switch to see what the JSON file needs to look like. I've taken pains to try and include complex examples so that you won't be left in the dark on anything.

  

After creating a new JSON file with our desired state it's time to run the tool in audit only mode by just specifying the -f flag and the name of our JSON file. The results that come back are that we have several areas that need remediation (email, event forwarders) and 1 (content packs) that cannot be remediated yet (hopefully in a later version).

 

That's all good but we want the tool to fix those issues so we append the -r flag

 

If you run the tool again the output comes back as all objects matching desired state but the nice thing is that you don't need to run it again. Once the remediation HTTP POST is sent to the server the tool will automatically go back and query the server for the configuration to verify that your changes have been implemented and the server is now set correctly. It will then show you success in the message immediately following the remediation step. 

The portions of Log Insight that the tool has the ability to configure are:

License Key

NTP Configuration

SMTP Configuration

Event Forwarder Configuration

Active Directory Configuration

RBAC Configuration

Content Packs (audit only right now)

Stay tuned as I plan on updating the tool over time as more APIs are released and as my python knowledge increases. In the meantime happy auditing and automatic remediation!

 

SELinux Hangs on Relabel Attempt - RHEL 7

I've recently run into an issue where during a recovery scenario it was necessary to relabel SELinux contexts on RHEL7.1. This is normally done by either placing an empty file at the root of the file system "touch /.autorelabel" or using the "fixfiles onboot" command; both of which I tried in this case. However in this case upon reboot the machine just hung at "Reached Target initrd Default Target" with no sign of even attempting to relabel the filesystem. Doing a bit of troubleshooting isolated the issue to SELinux as adding the "enforcing=0" parameter to GRUB allowed the machine to boot without issue. I tried quite a few different things including setting SELinux to "Permissive" in /etc/selinux/config and then back again, as well as a failed attempt to use "fixfiles restore /" and "restorecon -Rv /" which I'm assuming failed because SELinux was in permissive mode. Until today I've never seen a machine that won't respect the "touch /.autorelabel" nuclear option.

Ok, here's the fix that I found, odd as it is:
1. Modify GRUB to include "enforcing=0" to allow the OS to boot this first time without SELinux
2. Once inside the OS make sure that /etc/selinux/config is set to enforcing
3. Change the default runlevel from graphical to multi-user (think runlevel 3) with "systemctl set-default multi-user.target"
4. Reboot without modifying GRUB so that selinux is properly enabled on this boot

On the next reboot oddly enough the system recognized that a relabel had been ordered and proceeded as it should have the whole time. After another reboot and setting the default target back to graphical "systemctl set-default graphical.target" and another reboot as a sanity check it's working as expected again. Very odd problem and I must admit a very odd solution....

Log Insight - Migration to debug level logs

One of the projects that I am working on is enabling the forwarding of debug logs on all of our VMware vCloud Director Cells to our global Log Insight instance. To do this however we need a fairly accurate appraisal of what the increased overhead is going to look like. As part of this process I'm starting to create a python program that will allow me to quickly find what the current Events per Second (EPS) and log size in KBps are.

As you can see the script can be run locally or be pointed at a remote host and looks for the latest fully committed debug log. If you don't want to use that one, no worries, you can easily specify a different log file to use. If the target is a remote server the script will copy the appropiate log file to the machine running the script and then do the analytics locally to remove any possibility of unnecessary overhead from the cell server.

The script is still in active development as a side project but I hope to add the ability to query vCenter Servers as well in the near future. If you're curious the code is hosted on my Github repo and as always is not supported or affiliated with VMware in any way....

Zenoss Monitoring WinRM Error (Server not found in Kerberos database: HTTP@XX.XX.XX.XX

We recently rolled out Zenoss 5 in an environment and when we attempted to connect to Windows Servers over WinRM using a domain user received the below error:

Error on : Server not found in Kerberos database: HTTP@10.xx.xx.xx

  

After doing a tcpdump on the Zenoss server using "tcpdump -s 65535 -w filename.dmp"and loading the results into wireshark I found this in the traffic between Zenoss and the Domain Controller:

 

Turns out the Zenoss server was looking for an SPN of the FQDN of my Windows host and not the hostname. We went ahead and added a new SPN for the FQDN using "setspn -S HTTP/ < FQDN > < hostname >" and that resolved the issues!

One other side note: In one of the Zenoss documents that I saw it mentioned the need to install the krb5-workstation package on your remote collectors. This is easily accomplished by "yum -y install krb5-workstation" but due to the sequence of of my troubleshooting I cannot prove/disprove that it made a difference. If anyone finds this and can sort out that detail I'd be happy to hear the results.

Presenting at VMworld 2015

I will be presenting at VMworld 2015 in session MGT4579 on "Data In-Sight!! Experiences Running VMware's Private Cloud with Log Insight". If you are at VMworld feel free to attend and say hi as I'd love to get to meet you in person!