Exploring the Wonder

How to find where a VM is hosted

2012-02-17T14:16:00.000-08:00

A fairly common problem is if you know a machine is virtual but you don't know where it is hosted. A perfect example is if you get notified that a machine in your environment has a virus and needs powered off but nobody knows where the machine is hosted. This gets even more interesting when you have nested virtual ESX hosts with guests running under them. With a quick google search I found 2 PowerCLI scripts that got close to what I wanted and then tweaked them to make this process very easy.

1. Script #1 is a modified version of the script located on VMware's website on information about the CDP network information: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007069 and allows you to find what host is tied to a specific CDP enabled switchport. Change the port in bold and cut and paste into PowerCLI

------ Start Script -----
Get-VMHost Where-Object {$_.State -eq "Connected"}
%{Get-View $_.ID}
%{$esxname = $_.Name; Get-View $_.ConfigManager.NetworkSystem}
%{ foreach($physnic in $_.NetworkInfo.Pnic){
$pnicInfo = $_.QueryNetworkHint($physnic.Device)
foreach($hint in $pnicInfo)
{

if( $hint.ConnectedSwitchPort.PortID -eq "GigabitEthernet3/13" )
{
Write-Host We have a match... $esxname is connected to remote switch port $hint.ConnectedSwitchPort.PortID on $physnic.Device -fore green
$hint.ConnectedSwitchPort
}
else
{
Write-Host "Not guilty..."; Write-Host
}
}
}
}
----- End Script ----

Script #2 is a very slightly modified script that I got from one of the comments at http://www.virtu-al.net/2009/07/07/powercli-more-one-liner-power/ and modified ever so slightly to show current power state.

------ Start Script -----
Get-VM where { (Get-NetworkAdapter $_).MacAddress -eq "00:50:56:86:01:f1" } Format-Table Name,Host, PowerState
----- End Script ----

Nothing terribly original but hopefully it will help somebody else.

Disk Space Woes

2012-02-13T10:20:00.002-08:00

So I have a 160GB hard drive in a laptop and I am using 155 GB of it but I can only account for about 60 GB... how do I find what is eating all my space...?

Try this really cool little program, Space Sniffer.

http://download.cnet.com/SpaceSniffer/3000-18512_4-10913555.html

Configuring Openfiler iSCSI and VMware Data Recover (vDR)

2012-02-02T14:31:00.000-08:00

If you have some extra hardware floating around with reasonable disk space you can use it as a cheap iSCSI device to store non-critical VM backups using VDR. The process is pretty easy and works really well. I have several of these running in our lab, including some that are Openfiler running as a VM on ESX 5 and even then performance is pretty good. This article assumes that you already have Openfiler (http://www.openfiler.com/community/download) installed on a piece of hardware and the networking configured. For sure I recommend running 2 nics, one for management and the second as a private iSCSI network. Once all that is done all you need to do is configure the storage and iSCSI on the Openfiler box as well as your ESX host. Let's see how we acomplish that:

1. Login to the Openfiler web interface and click on the Volumes tab and then on the right sidebar select Block Devices.

2. Select the section that says "Create a partition in /dev/sd(b)" The settings you use are: "Mode = Primary", "Partition Type = Physical Volume". Starting and Ending cylinders are set automatically to the largest setting and do not need modified unless you so choose. Then hit "Create"

3. Now click on the "Volume Groups" option on the right sidebar and you will see an option to "Create a new volume group". Check the box for the physical volume you just created and give it a name like "VDR".

4. Once that is complete click on the "Add Volume" option on the right sidebar and select your new "VDR" volume group. Under the section "Create a volume in 'VDR'" you can create a new volume. Name it something like "VDR" and give it all the available space. MAKE SURE that you change the "Filesystem / Volume Type" to iSCSI.

5. Your storage is ready, now we need to get iSCSI running. To do this you need to start the iSCSI Target Service and allow network access. To start the service click on the "Services" tab on the top toolbar and start the service. To get the iSCSI Network up and running go back to the "Volumes" tab and "iSCSI Targets". Now you will notice 4 grey tabs under your top toolbar. Using these tabs create a new IQN under the "Target Configuration" tab and give your private iSCSI LAN access via the "Network ACL" tab.

6. Now that iSCSI is running we need to map the IQN to the LUN that we created earlier. It is as easy as going to "LUN Mapping" and selecting the LUN and hitting "Map". Your job in the Openfiler webconsole is complete. Now off to the ESX host.

7. Open your vSphere Client and connect to your vCenter Server and your ESX host that is connected to the Openfiler box. I am assuming that you have already configured a NIC on the ESX host and it can ping the Openfiler private IP. If not you need to get that working before continuing.

8. In the vSphere client select the ESX host and go to the "Configuration" tab and select "Storage Adapters" from the left sidebar. Now select the "iSCSI Software Adapter" and click "Properties...". Click on the "Configure..." button and enable iSCSI. Once iSCSI is enabled click on the "Dynamic Discovery" tab and add the IP of your Openfiler box. Once that is done and you Ok out of the menu it will prompt you to do a rescan which you want to do. If it all went well then once the Rescan is complete you should see a new device. If not then make sure your 2 boxes are talking to each other.

9. Now that our iSCSI Adapter can talk to our shiny Openfiler box we need to actually use that space to store our VDR backups. To do this click on the "Storage" option and select "Add Storage" option. This should bring up a wizard explaining how to add the storage. Name the new Datastore something like "VDR Backups".

10. Right click on your VDR Appliance and go to "Edit Settings" and "Add.." a new hard drive making sure to select the new Datastore you just created. Don't give the drive all the space on the Datastore. Make sure to leave several GB free for ESX to use for file operations.

11. Congratulations, now all you need to do is add the Datastore to your VDR Appliance as a backup destination!

Unable to Delete File - Could Not Find this Item Error

2011-11-15T08:46:00.000-08:00

So apparently Windows has difficulty deleting 0 byte files. If you ever try to delete a folder and get the following error "Could not find this item. This is no longer located in \\. Verify the item's location and try again" check the size.

If the file size is 0 bytes you might need to do the following:

1. Open an elevated command prompt (Right click on Command Prompt > Run As Administrator)

2. Use rmdir /S to delete the entire folder

3. There is probably a way to delete the files individually but I tried a couple and none of them worked.

Changing VMware ESX Path Selection Plugin with PowerCLI

2011-10-04T13:05:00.001-07:00

It is really quite easy to modify your Path Selection Plugin on VMware ESX using the vSphere PowerCLI which is very useful if you have to modify several hosts. In the below example I am just connecting to a single ESX 5.0 host but you can connect to a vSphere server instead.

1. Connect to the ESX server (Connect-VIServer 10.25.62.65)
2. Check your current PSP (Get-ScsiLun)
3. Change PSP to RoundRobin, MostRecentlyUsed or Fixed (Get-ScsiLun Set_ScsiLun -MultipathPolicy RoundRobin)

If you are interested in what policy to use there is a great article by one of my co-workers about it at

http://bsmith9999.blogspot.com/2011/10/storage-path-selection-policy-choices.html

Unable to run Installer on Windows Server

2011-08-09T12:41:00.000-07:00

If you ever run an installer and get the following message "Windows cannot access the specified device, path, or file. You may not have the appropiate permission to access the item" try the below. Note: I am assuming that you already have rights...

1. Copy the file to your local machine

2. Right click on the file and go to Properties.

3. Down at the bottom of the General tab you might see a Security option that says "This file came from another computer and might be blocked to help protect this computer."

4. Click Unblock.

Creating Windows Event Log Events

2011-08-04T14:59:00.000-07:00

We use VMware vCenter Configuration Manager at our site to monitor our Windows servers for signs of future problems by searching the Windows Event Logs for a predefined list of events. Every once in a while (especially after a while without alerts) you want to test and make sure that everything is indeed being monitored properly. To do this I needed to inject an event into the Event Log and see if we received an alert in VCM. The question was this: how do I impersonate an existing service? The built in eventcreate.exe will not let you add events for a source that already exists, which is exactly what I want to do... Ah, I love it when people have already coded an application to do something that I need to do and save me from having to write it myself.

Meet eventcreate2.exe: http://code.google.com/p/eventcreate2/

Eventcreate2.exe has one flaw, you cannot specify a remote machine; as opposed to the built in eventcreate.exe which does let you specify remote machines. Other than that it is awesome. Using a command like the below I was able to successfully inject events impersonating an existing service and verify that our alerting was indeed working as advertised.

eventcreate2.exe /L Application /T Warning /SO dmboot /ID 2 /D "This is a test Windows RAID Failure Alert"

VMware VDR Service Did not Start Up

2011-08-01T09:49:00.000-07:00

I recently ran into an issue after replacing my VDR (now VCDR) Appliance from version 1.2 to 2.0 and ran into a very misleading error that can cause confusion. A week after installing the VCDR Appliance I tried to connect to it using the vSphere Client on my laptop and was greeted with the following error: "Error: The Data Recovery service did not start up. If the problem persists, please restart or redeploy the Data Recovery appliance."

Restarting the appliance does not change the error message and I was worried it was dead but thankfully after doing some digging in the log files I discovered a very interesting error: "UnixResponder: Respond: Rejecting non-SSL connection from client 10.25.43.72 (socket 40)".

Turns out the problem was something really quite simple. The reason for this behavior was I had the 1.2 version of the VDR Plugin installed on my laptop. Apparently it can sort of talk to the new version 2.0 appliance but not enough to work. All you need to do is uninstall the old 1.2 version of the VDR plugin via Add and Remove Programs and install the new version 2.0 plugin.

Problem with MacBook Pro and Seagate Momentus XT Hybrid Drive

2011-03-24T12:56:00.000-07:00

I just tried to drop a mac image onto a brand new Seagate Momentus XT hybrid drive using a process that has worked on other drives hundreds of times. This time however I ran into issues. After using the Restore feature and rebooting I got the dreaded blinking folder with a question mark. I tried 2 different MBP's as well as 2 different drives, same issue. Tried scanning and repairing the drive to no avail. Finally I dropped in the original Mac OS X DVD and tried to load it manually. When I did that I ran into a nice little error of "Mac OS X Cannot Start Up From This Disk". No additional information, gee thanks. Doing a bit of googling I ran across an article from Apple saying that this error is caused by a partitioning problem. The drive check reported that it was a-ok but at this point I was willing to give it a shot. So, I went back into the mac tools, deleted all partitions on the Seagate drive and created 1 new partition at 495GB. Then I attempted the restore to that partition I had manually created and it worked. I don't know what it is about these drives but it was the same issue for both and I have never seen that issue on other drives. After that however it seems to be working great.

Unable to Disable Bluetooth Radio

2011-03-02T07:55:00.000-08:00

I ran into an interesting issue yesterday where a user needed to disable his bluetooth radio on a bootcamped Apple MacBookPro but the disable option was greyed out in Mac OS X and under the device manager in Windows it was missing. The easy answer is that you cannot disable the radio if you have devices bound to it. Delete any existing device bindings and then you can disable the radio.

SQL Management Studio 2005 Express x64 Install Fails

2011-02-22T09:09:00.000-08:00

On one of my Windows Server 2008 x64 boxes running SQL 2005 Express I ran into a situation where I needed to install the Management Studio. After downloading and running the installer from Microsoft's site I got the following error:

"This installer has encountered an unexpected error installing this package. This may indicate a problem with the package. The error code is 29506."

I tried re-downloading the package and changing what was getting installed and even modifying the install directory to no avail. Finally I noticed that it was failing on the step "Setting File Security". After noticing that I went and set the UAC to the lowest possible settings and it installed correctly. Looks like the UAC does not like that installer for some reason...

XP Make Available Offline Fails with Access is Denied

2011-02-18T12:33:00.000-08:00

If you have permissions to the folders and files you probably need to give the user read access to the root directory of the share. Example:

\\server1.walker.dps\shares$\myshare

User will need Read permissions to the shares$ share. You can set it to Read and that folder only for security purposes.

RHEL 2.1 P2V Fails at 99%

2011-02-18T07:42:00.000-08:00

I have a physical RHEL 2.1 box that I am trying to P2V to an ESXi host using the VMware vCenter Converter Standalone 4.3 app and it is failing at 98% and 99%. The log is showing the following errors:

[#2] [2011-02-11 16:45:39.209 07840 info 'task-1'] Worker CloneTask updates, state: 1, percentage: 98, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:04.379 05412 info 'task-2'] Remote Helper VM is reconfiguring, data clone is finished
[#2] [2011-02-11 16:46:04.379 05412 info 'task-2'] Volume-based cloning dlinux5.XX.XXX--> dlinux5 updates, state: 1, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:04.379 05412 info 'task-2'] CloneTask updates, state: 1, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:04.379 07840 info 'task-1'] WorkerCloneTask: Remote Helper VM is reconfiguring, data clone is finished
[#2] [2011-02-11 16:46:04.379 07840 info 'task-1'] Worker CloneTask updates, state: 1, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:16.489 05412 info 'task-2'] Volume-based cloning dlinux5.XX.XXX--> dlinux5 updates, state: 4, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:16.489 05412 info 'task-2'] Generating helperVM task bundle for task with id="task-1".
[#2] [2011-02-11 16:46:16.519 05412 info 'task-2'] Retrieving helper VM log bundle to "C:\Windows\TEMP\vmware-temp\vmware-SYSTEM\helperVM-task-1-sljnzvkf.zip".
[#2] [2011-02-11 16:46:16.619 05412 info 'task-2'] Bundle successfully retrieved to "C:\Windows\TEMP\vmware-temp\vmware-SYSTEM\helperVM-task-1-sljnzvkf.zip".
[#2] [2011-02-11 16:46:16.619 05412 info 'task-2'] powering off vm after linux p2v ...
[#2] [2011-02-11 16:46:16.619 05412 info 'task-2'] Reusing existing VIM connection to XXXXXXesx2.XX.XXX
[#2] [2011-02-11 16:46:18.099 05412 info 'task-2'] power off vm succeeded
[#2] [2011-02-11 16:46:18.099 05412 info 'task-2'] Reusing existing VIM connection to XXXXXXesx2.XX.XXX
[#2] [2011-02-11 16:46:18.579 05412 info 'task-2'] successfully reconfigured target vm
[#2] [2011-02-11 16:46:18.579 05412 info 'task-2'] CloneTask updates, state: 4, percentage: 99, xfer rate (Bps): 8817664
[#2] [2011-02-11 16:46:18.579 05412 info 'task-2'] CloneTask failed
[#2] [2011-02-11 16:46:18.579 05412 error 'App'] Task failed:

And earlier in the log:
#2] [2011-02-11 15:44:34.036 05412 error 'App'] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1)

And again:
[#1] [2011-02-11 15:43:08.256 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.266 03284 info 'App'] [,0] Disk number 1 has been skipped because of errors while reading partition table
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Disk number 2 has been skipped because of errors while reading partition table
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Disk number 1 has been skipped because of errors while reading dynamic disks header or LDM database is corrupted
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.276 03284 info 'App'] [,0] Disk number 2 has been skipped because of errors while reading dynamic disks header or LDM database is corrupted
[#1] [2011-02-11 15:43:08.286 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.296 03284 info 'App'] [,0] Partition:Invalid sector magic number.
[#1] [2011-02-11 15:43:08.296 03284 warning 'App'] [MoveActiveDiskIfNeeded] GetFirstBootDisk failed, mntapi error: 176

The great thing about having a VM is that you can snapshot it. DO IT. I destroyed around a dozen copies of my VM playing with this trying to get it right.

I have already manually run an fsck against the physical machine before the P2V attempts so I am not sure what is wrong but when I attempt to power up the virtual machine I get a nice “Error Loading Operating System”. Looks like a bootloader issue, possibly when the VM Reconfiguration Job was going something bad happened. No worries, I just dropped in a Fedora 13 DVD and selected the “Rescue Installed System” option. Follow the prompts (networking not required) and then “chroot /mnt/sysimage”. From here I can see the data on my VM and it all looks good. Now I go to /boot/grub and notice that I only have 4 files:
1. Grub.conf
2. Device.map
3. Menu.1sl
4. Splash.xpm.gz
Looks like the boot loader files are gone. Now I need to somehow recreate them. To do this you need to do the following. First, figure out what your partitions look like using the “fdisk –l” command. Then using that information open grub by typing “grub”.

Once inside the grub shell type in “root” and if you have the same issue that I had you should see the following returned value: “(fd0) : filesystem type unknown, partition type 0x0” We need to change that to our existing boot partition which in most cases is ‘hd0,0’. To do that type in “root (hd0,0)” and hit enter. If you get lost you can hit tab to see your options. From there you need to execute the changes by typing “setup (hd0,0)”. You should see it execute and give some output. After it is complete you can “quit” and then do an “ls” on the /boot/grub directory and you should see that grub has added several files.

Now you might need to convert the linux configuration from IDE to SCSI based. To do this you need to modify all entries that are “hdXY” where X is a letter starting with a and Y is a number starting with 1. You need to modify these to become sdXY. As an example hda1 becomes sda1. You need to do this in the following files:
1. /boot/grub.conf
2. /boot/device.map
3. /etc/fstab
Note: for some weird reason your 2nd drive’s partitions may show up as in the /etc/fstab as hdc1 instead on hdb1 as I would have expected. This will cause errors upon boot if you don’t change that to sdb1 instead of sdc1. Not sure what causes that initial value to be wrong.

Now you need to add scsi support. Open the /etc/modules.conf file and edit the following things:
Find ‘alias ethX module’ entires and replace the value of the module to “pcnet32”. This is the network adapter that you will be using in ESX(i).
Now find ‘alias scsi_hostadapter’ and if it does not exist add the following:
“alias scsi_hostadapter BusLogic”
For multiple controllers I assume you would have to add another line per controller like:
“alias scsi_hostadapter1 BusLogic”

After this you need to rebuild the ramdisk image. Locate the .img file in the /boot directory and make a note of the file name. Now you want to run the following command:
Mkinitrd –v –f /boot/initrd-X.X.X-Y.X.X.img X.X.X-Y.X.X where the X and Y symbolize the actual values on your system. For mine the actual command was
“mkinitrd –v –f /boot/initrd-2.4.9.0-e.34.img 2.4.9.0-e.34”
After that runs you should be able to reboot and it should work.

Thanks to everybody who posted bits and pieces of the fixes on various blogs and papers. Hope this helps somebody else so you don't have to do all the research and tweaking that I did.

Bind DNS on CentOS 5

2011-02-07T13:18:00.000-08:00

Here is my quick and dirty cheatsheet on how to get bind working on CentOS5.
1. Make sure that iptables is not blocking your DNS queries ("service iptables stop" will disable the firewall)
2. Install Bind "yum install bind*"
3. Copy example configs to working directories:
a. "cp /usr/share/doc/bind-x.x.x/sample/etc /var/named/chroot/etc"
b. "cp /usr/share/doc/bind-x.x.x/sample/var /var/named/chroot/var/named"
4. Modify the named.conf in /var/named/chroot/etc and remove the un-needed views (I only need the internal view)
5. Modify the named.conf to have the correct zone information
6. Run /usr/sbin/dns-keygen and copy the generated key to the named.conf file under the ddns_key section.
7. Copy your my.internal.zone.db from /var/named/chroot/var/named or rename it to your correct zone name
8. Add your A records to the new zone.db file
9. Automatically start bind on boot
a. chkconfig named --add
b. chkconfig on level 2,3,4,5

Transparent Windows Command Prompt

2011-02-07T10:35:00.000-08:00

So one of the things that I really wished Windows 7 had was a transparent command prompt. I have been using it on Linux and gotten spoiled having it. Today I found one at http://www.askvg.com/get-glass-cmd-command-prompt-in-windows-vista-and-7/ that seems to work pretty well on Windows 7 x64.

VMware HA

2010-11-15T15:01:00.000-08:00

You will need this if you have environments with both ESXi and ESX.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006541

Collecting from ESX/ESXi with VMware vCM 5.4

2010-11-12T07:49:00.000-08:00

If you have VMware vCM (Configuresoft ECM, EMC Ionix SCM) and want to collect from ESX/ESXi you have to create a local user on the ESXi host and give that user Administrator permissions. That can be quite a daunting task if you have more than a handful of machines so I wrote the below powershell program that allows interactive use or automated use with a .CSV file.

---Start Script
Function NextUser
{
if ($newUser)
{
Write-host "Do you want to specify a new service account username or password? [y],[n]" -Fore Green
$continueUser = read-host
if ($continueUser -eq "y")
{
DoIt
}
if ($continueUser -eq "n")
{
DoItUsers
}
}
}

Function ExistingUsers
{
if ($newUser)
{
Write-host "Do you want to specify a new service account username or password? [y],[n]" -Fore Green
$continueUser = read-host
if ($continueUser -eq "n")
{
DoItUsers
}
if ($continueUser -eq "y")
{
DoIt
}
}
if (!$newUser)
{
DoIt
}
}

Function DoIt
{
write-host "Please enter the username that you would like to use for the vCM Service Account:" -Fore Green
$newUser = read-host
write-host "Please enter the password that will be used for the service account:" -Fore Green
$newPass = read-host
write-host "Ok, I am going to add $newUser with a password of $newPass to $connectedServer" -Fore Green
Start-Sleep -seconds 3
New-VMHostAccount -ID $newUser -password $newPass -Description "vCM Service Account"
write-host "Ok, I am now adding administrator permissions to the account we just created" -Fore Green
New-VIPermission -Role admin -Principal $newUser -Entity $connectedServer
write-host "Congratulations, you have added the new vCM service account successfully." -Fore Green
write-host "Would you like to configure another host? [y],[n]" -Fore Green
$nextHost = read-host
if ($nextHost -eq "y")
{
NextHost
}
if ($nextHost -eq "n")
{
write-host "Thanks, have a great day!" -Fore Green
exit
}
}

Function DoItUsers
{
write-host "Ok, I am going to add $newUser with a password of $newPass to $connectedServer" -Fore Green
Start-Sleep -seconds 3
New-VMHostAccount -ID $newUser -password $newPass -Description "vCM Service Account"
write-host "Ok, I am now adding admin permissions to the account we just created" -Fore Green
New-VIPermission -Role admin -Principal $newUser -Entity $connectedServer
write-host "Congratulations, you have added the new vCM service account successfully." -Fore Green
write-host "Would you like to configure another host? [y],[n]" -Fore Green
$nextHost = read-host
if ($nextHost -eq "y")
{
NextHost
}
if ($nextHost -eq "n")
{
write-host "Thanks, have a great day!" -Fore Green
}
}

Function NextHost
{
write-host "Please enter the FQDN of the server that you would like to connect to:" -Fore Green
$server = read-host
write-host "Connecting to $server using the credentials that you specified..." -Fore Green
Connect-VIServer -Server $server -User $existingUser -Password $existingPass
$connectedServer = Get-VMHost
write-host "Now using $connectedServer..." -Fore Green
if (!$connectedServer)
{
write-host ""
write-host "That host does not appear to be valid, please try again." Fore Yellow
NextHost
}

ExistingUsers
}

write-host ""
write-host ""
write-host "-- Welcome to my VMware vCM ESX/ESXi Service Account user deployment tool. This tool will allow you to easily deploy a vCM Service Account to your ESX/ESXi hosts -calebs71.blogspot.com --" -Fore Cyan
write-host ""
if (!$args[0])
{
write-host "Did you know that you can provide a CSV file with your server FQDN, ESX user, ESX Password, New Service User, New Service Password and automate the whole process?" -Fore Yellow
write-host ""
write-host "Example: vCM Service Acct on ESX.ps1 c:\list.csv"
write-host "FQDN,User,Password,SVCuser,SVCPass"
write-host "machine1.test,root,password,vcm_svc,password2"
write-host ""
write-host "Please enter a valid ESX/ESXi account with permissions to add a new user:" -Fore Green
$existingUser = read-host
write-host "Please enter the password:" -Fore Green
$existingPass = read-host
$connectedServer = Get-VMHost
if (!$connectedServer)
{
write-host ""
write-host "You are not connected to an ESX host..." -Fore Yellow
NextHost
}
write-host "You are currently connected to $connectedServer do you want to change servers? [y], [n]" -Fore Green
$response = read-host
if ($response -eq "y")
{
NextHost
}
if ($response -eq "n")
{
write-host "Continuing to use $connectedServer" -Fore Green
DoIt
}
}
if ($args[0])
{
$listMachines = import-csv $args[0]
if ($listMachines)
{
foreach ($MachineItem in $listMachines)
{
# write-host "FQDN : " $MachineItem.FQDN
# write-host "User : " $MachineItem.User
# write-host "Password : " $MachineItem.Password
# write-host "SVC User : " $MachineItem.SVCUser
# write-host "SVC Pass : " $MachineItem.SVCPass
# write-host "Connect-VIServer -Server "$MachineItem.FQDN " -User "$MachineItem.User " -Password "$MachineItem.Password
# write-host "New-VMHostAccount -ID "$MachineItem.SVCUser " -password "$MachineItem.SVCPass " -Description "vCM Service Account""
# write-host "New-VIPermission -Role admin -Principal " $MachineItem.SVCUser " -Entity " $MachineItem.FQDN

Connect-VIServer -Server $MachineItem.FQDN -User $MachineItem.User -Password $MachineItem.Password
New-VMHostAccount -ID $MachineItem.SVCUser -password $MachineItem.SVCPass -Description "vCM Service Account"
New-VIPermission -Role admin -Principal $MachineItem.SVCUser -Entity $MachineItem.FQDN
}

}

}

Error Removing Host from vSphere

2010-10-27T09:22:00.000-07:00

I have a host (ESX 4.0) that I need to remove from one of my clusters but after putting it into maintenance mode and shutting it down the Remove option is grayed out. I found some references online that mention using the PowerCLI to manually remove the host but ran into a caveat that none of the other posts mentioned. When I ran the remove-vmhost command I got the following error: "The method is disabled by 'com.vmware.vcintegrity'". To get around this I had to take the disconnected host and remove it from the cluster and then run the remove-vmhost command to remove it from vSphere. I imagine that this is related to capacity issues on our old cluster but we are trying to move our blades and accompanying guests to a new cluster so HA and DRS is not high on the priority list.

Syntax to install HP CIM Providers on ESX4i

2010-10-20T12:40:00.000-07:00

First, download and install the vCLI tools from vmware.com on your desktop.

Second, download the CIM Providers from HP.com

Third, Place host in Maintenance Mode and then scan the ESXi host for the Bulletin ID to be installed

c:\Program Files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server -username root - scan - bundle

should return a name like

hpq-esxi4.1uX-bundle-1.0 (note: this is case sensitive)

Fourth, install the bulletin

c:\program files (x86)\VMware\VMware vSphere CLI\bin>vihostupdate.pl -server -username root -bundle c:\hpq-esxi4.1uX-bundle-1.0.zip -install - bulletin hpq-esxi4.1uX-bundle-1.0

Fifth, reboot the host.

A Truly Virtual ESX Environment

2010-09-27T14:01:00.000-07:00

Here is a training environment that I am almost done building on my laptop. I am running Windows 7 x64 with 8GB RAM and VMware Workstation 7.
Guests are:
1. Openfiler NAS (iSCSI Target)
2. ESX 4.1 Host
3. ESX 4.1 Host
4. XP
5. vCenter Server

In the past I always put the vCenter box on the ESX hosts however 4.1 requires a 64 bit OS and that is not possible yet in a nested virtual environment.

Up next DRS needs configured and I will have a ready to roll ESX Cluster.

Issues Configuring VMware ESX HA

2010-09-27T12:03:00.000-07:00

I got the following error when attempting to add my ESX hosts to a vCenter Cluster: "Cannot complete the configuration of the HA agent on the host. Misconfiguration in the host network setup." This was caused because I don't have a default gateway since it is a small test environment. It appears that you have to be able to ping your default gateway to enable HA. For my test environment I now have a VM that is a guest under my test ESX host that is on the DG IP. Problem solved.

Side note: If you have a VMware ESX cluster and one of the nodes fails and your default gateway is unavailable your HA will fail...

SSRS 2008 Domain User Issue

2010-09-21T14:39:00.000-07:00

Scenario: I have a new SQL Server Reporting Services 2008 R2 x64 instance that works if you are the local administrator. Once I switch to another user who is an implied administrator via an AD group I get the following error when attempting to access SSRS in a browser:

"User 'TESTDOM\user' does not have required permissions. Verify that sufficient permissions have been granted and Windows User Account Control (UAC) restrictions have been addressed."

The user is a member of the domain admins group which is obviously a member of the local admins. Also if I right click on IE8 and say "Run As Administrator" it works. The quick fix is to disable the UAC. If you don't want to do that then just do the below:

1. Launch IE8 using "Run As Administrator" and in SSRS (http://servername/Reports) click on Site Settings. In Site Settings navigate to "Security" and add your user "TESTDOM\user" as a System Administrator.

2. Click on "Home" and then "Folder Settings". Now add your account again this time assigning yourself all the roles (Browser, Content Manager, My Reports, Publisher, Report Builder).

3. Close Administrative IE session and open a normal IE8 window and your SSRS should work.

Leaving our imaginary safety of isolation

2010-09-08T19:54:00.001-07:00

It is constantly both amazing and embarrassing to me how we as Christians are so into ourselves and our individual churches and accompanying traditions. We have gotten into rote "religion" and forgotten Jesus Christ, the reason for it all. It is in the little things like the kid who just came to church and wears a hat inside during services and has an earring and long hair. Oh how much we have forgotten that we are all in the same boat. We are all just sinners saved by grace, but how often we forget to extend that grace to others. This is just a little reminder that we should never forget the depth that we came from by the grace and mercy of the Lord Jesus Christ and to always strive to extend that same grace and mercy to others, even if they don't fit in your little "this is a person who I like in church" mold. We all ultimately answer to ourselves and eventually to the Lord, let us act like he has shown us.

Removing BitLocker on Windows 7

2010-09-01T10:14:00.000-07:00

When attempting to remove BitLocker drive encryption from your Windows 7 machine you would naturally right click on the drive in question and say "Manage BitLocker". This brings up a couple options but none of them will allow you to remove BitLocker and decrypt the drive. What you need to do is go Control Panel > System and Security > BitLocker Drive Encryption and select the option to remove BitLocker.

Killing a hung VM in LabManager (ESX)

2010-07-26T10:18:00.001-07:00

First find the machine name which in LabManager is usually proceeded by a 6 digit number. Then from the console type
"ps -auxwww | grep -i (######-machine name).vmx"

This will give you the PID which you can kill using
"kill -9 (PID)"

Note to self....

2010-07-21T07:55:00.001-07:00

Don't use Windows 7 with IE8 for the HP BladeSystem Onboard Administrator. Too many weird results and broken tasks.

Windows Update Issue

2010-07-14T12:32:00.000-07:00

I just won against a Windows Update issue on XP that had me quite puzzled. The error that I was getting with both Microsoft and Windows Update was "Error Number: 0x8024402C The website has encountered a problem and cannot display the page you are trying to view." and "Error Number: 0x8024402C The website has encountered a problem and cannot display the page you are trying to view."

Looking at the WindowsUpdate.log file under "c:\Windows\" I found a line that says

"WARNING: SendRequest failed with hr = 80072ee7. Proxy List Used <proxy.company.com:1234> Bypass List Used : <null>"

The proxy however in IE has been removed so that there is no proxy under Tools > Internet Options > Connections > LAN Settings. Here is the catch, if you open a command prompt and type in "proxycfg" you will see that the proxy listed in the WindowsUpdate.log error is still in the registry. To clear this type in the following

proxycfg -d
net stop wuauserv
net start wuauserv

Once you have cleared that cached proxy MS update and Windows Update will work again. MS has a document on this proceedure on their website.

BartPE and GetDataBack for NTFS

2010-06-15T12:42:00.000-07:00

I was running into an issue while building a BartPE image with "GetDataBack For NTFS" where the Bart CD would work fine until I attempted to launch GetDataBack then nothing would happen. I have tried different versions of the Windows build source (2k3, xp, different service packs) and different versions of the GetDataBack plugin to no avail, it just will not launch. Oddly enough this appears to be caused by the build machine which in my case is running Windows 7. It appears that if you build your ISO on a Windows 7 machine even though you are using XP build source the plugin will not work. I copied my PeBuilder directory to an XP machine that had BartPE installed and it works fine.

Cisco IPSec 64bit VPN

2010-06-08T10:08:00.000-07:00

Cisco finally released a version of the Cisco VPN Client that works on 64bit machines and is IPSec capable.

From their site:
"The new release VPN Client 5.0.07 supports the Windows Vista on both x86 (32-bit) and x64. Refer to the 5.0.07.0240 Release Notes for more information."

I just installed it on my Windows 7 x64 laptop and it works just fine.

Bad ESX Network Driver on BL490c G6

2010-05-14T07:33:00.000-07:00

Ok, so here is the scenario: HP Blade (BL490c G6) has a bad stick of RAM, pull blade from cabinet, swap RAM, re-insert blade, power on blade and receive the following ESX error:

"networking-drivers ...
No compatible network adapter found. Please consult the product's Hardware Compatibility Guide (HCG) for a list of supported adapters."

At first we were sure that the issue was firmware related however it appears that it just might be a bad VMware driver. Here's why: I went ahead and did two test cases to isolate the issue to the VMware driver on our BL490c G6 blades.

Blade 1: Updated to newest Firmware from HP Smart Update Firmware. This updated the NIC Bootcode from 5.0.11 to 5.2.7 but left the iSCSI at 3.1.5. After this the machine will boot most of the time after the blade is removed from the chassis, however after several reboots there are seemingly random times when ESX will not load the NIC and you get the "No compatible network adapter found" error when attempting to boot ESX. After installing the new drivers found at "http://www.vmware.com/support/vsphere4/doc/drivercd/esx40-net-bnx2x_400.1.48.107-1.0.4.html" using the esxupdate command it works fine. I have rebooted many times and removed the blade and am no longer able to get the error.

Blade 2: I left the firmware alone and just updated the VMware driver to the latest from the above URL. After several reboots and physically removing the blade from the cabinet twice I am unable to get the error.

Conclusion: Looks like a VMware driver issue; however updating the HP Firmware on the NIC from 5.0.11 to 5.2.7 does seem to help the issue taking it from happening every time to approximatly %50 of the time.

W32/Wecorl.a McAfee False Positive Workaround

2010-04-21T09:17:00.001-07:00

If you are running McAfee Antivirus on Windows XP SP3 and have an issue with your computer rebooting with a message of "Windows must now restart because the DCOM Service Process Launcher service terminated unexpectedly" and W32/Wecorl.a is reported on your machine in the Application Event Log then it is caused by a known issue with DAT 5958. To abort the automated shutdown open a command prompt and type in "shutdown -a" and then get yesterday's (5957) superDAT from McAfee at http://www.mcafee.com/apps/downloads/security_updates/superdat.asp?region=us&segment=enterprise and run the file from the command prompt with a /F switch to force the downgrade and reboot when prompted. That will work until McAfee fixes the issue.

*QUICK UPDATE*
After doing the above steps you may not be able to connect to the network and also will notice that many of your services that are set to automatic will not start. This is because your svchost.exe file was eaten by McAfee. Copying SVCHOST.exe from another good XP SP3 machine to your broken one's c:\windows\system32\ directory fixes that issue.

Outlook 2007 - Cannot open the Outlook Window

2010-03-19T09:46:00.000-07:00

I ran into an interesting issue this morning. User is running Windows 7 with Outlook 2007 and when he attempts to launch Outlook he gets an error that says "Cannot Start Microsoft Outlook. Cannot open the Outlook Window." I have attempted to repair Office which failed with the bootstrapper crashing but was able to uninstall Office 2007 and re-install it. The issue still remained even after a full removal and re-install. I deleted the mail profile and OST which also did nothing. The final fix is this: Open a command prompt and navigate to the Office12 directory and run this command "outlook.exe /resetnavpane" Odd that a re-install does not reset it enough...

Robocopy and Server 2008 Script Error

2009-12-14T13:14:00.000-08:00

If you are going to use Robocopy with Windows Server 2008 as a script you are going to run into a few errors. The first is that you have to allow the account you are using to execute the script to be allowed to log on as a batch job.

Local Security Policy > Local Policies > User Rights Assignment > Log On As A Batch Job > Add User or Group

Next add the user account to the Backup Administrator's group in Computer Management > Local Users and Groups

You will also need to tweak the job in Task Scheduler to have the following things: Select "Run whether user is logged on or not" and check "Run with Highest Privileges". If you do not select the "Run with Highest Privileges" check box then you will get the following error:
-------------------
ERROR : You do not have the Backup and Restore Files user rights.
***** You need these to perform Backup copies (/B or /ZB).

ERROR : Robocopy ran out of memory, exiting.
ERROR : Invalid Parameter #%d : "%s"

ERROR : Invalid Job File, Line #%d :"%s"
--------------------
That box is the equivalent of right click, "Run As Administrator" on the command prompt window.

Methodical Bible Study

2009-12-05T20:54:00.000-08:00

Want a good book to help you dig deeper in Bible study? I have been reading Methodical Bible Study by Robert A. Traina after seeing it used at DTS (Dallas Theological Seminary) for several days now and it is a very difficult book to read. However that difficulty is not due to poor authorship but a very detailed and in-depth look at the art of study. It has 4 chapters and 267 pages. I just finished Chapter 1 and if anyone is interested I would recommend taking a look at it. You can also get the associated classroom portion of the DTS class around the book from the DTS page at iTunes University.

Seattle Trip

2009-10-17T17:33:00.000-07:00

I just had the opportunity to go on a trip with my grandfather, father and 2 brothers to Seattle on vacation. It was a wonderful trip and I wanted to share some of the photos that I took. All are taken with a Nikon D100 and a 55-200 VR Lens with a UV shield. On the last 3 I also used a circular polarizer because I was shooting from a moving vehicle and needed to get rid of the reflections. And no, I was not the driver :)

BTW, the route between Wenatchee, WA and Seattle is one of the most beautiful places I have ever been. If you are ever going down I-90 to Seattle it is worth the extra drive time. Also, the Boeing Museum of Flight is well worth a visit.

Hobbies...

2009-09-26T11:53:00.000-07:00

For the most part my hobbies are few and far between; except photography. I love grabbing my Nikon DSLR camera and heading out to see what new shots I can find. Or, sometimes the shots come to me as was the case yesterday and today. Here are some photos I took both of the practice runs yesterday and the actual flyovers for the Air Force football game today verses San Diego State... Go Falcons!

Click on the photos for a High Res Version.

Weird Event Log Event

2009-07-08T11:04:00.001-07:00

This box has a dying systemboard and one of the error messages I saw in the Event Log I have never seen before...

Error Source: i8042prt
Type: Information
EventID: 12

Description: The ring buffer that stores incoming mouse data has overflowed (buffer size is configurable via the PS/2 mouse properties in device manager).

Bulk DNS Update via CSV File

2009-07-07T12:40:00.000-07:00

So here is the scenario:

All of my DNS static entries are going to get nuked and changed to a completely different IP space (yes, the computers too). However since this is such a massive task and we have minimal time to make the change we would like to complete as much ahead of time as we can. All of my DNS servers are DCs running AD-Integrated Zones. The only way I can think of to do this ahead of time is by creating a spreadsheet and filling in the hostnames and the new IPs and then on the day of the IP change import the new records into DNS. So, the question is how exactly can I update the DNS tables on an AD Integrated Zone using a script or file import?

Here is what I found:
The simple answer is you can't :) However there is a workaround...
1. Export your DNS information to a CSV File
2. Modify the DNS information to show the correct IP information. I acomplished this by using Excel Spreadsheets sent to users to be filled in with the new IPs. Then using SQL I imported the Spreadsheets into a database and then scripted a massive UPDATE statement on the master DNS list. Then exported the updated master DNS list as a TAB DELIMITED file.
3. Next I needed to remove all my DNS Servers except one. Pick one to keep, uninstall DNS on all others.
4. On the remaining DNS Server I changed all my zones from Active-Directory Integrated Zones to Primary using the following: (right click on each zone) > Properties > General > Change Type > (Uncheck) "Store the zone in Active Directory"
5. Repeat for all zones
6. Keep the "Load Zone Data on Startup" at "From Active Directory and registry"
7. Open the DNS zone files (.dns) located in c:\windows\system32\dns\
8. Modify the DNS zone file with your new information keeping the proper TAB Delimited format.
9. Reboot the DNS server, this is gonna take a while but if you don't you are going to get an error like "the specified directory partition does not exist".
10. Open DNS Manager again and move all of your zones back to AD-Integrated Zones. Steps are the reverse of Step 4.
11. Reinstall DNS on all the other DNS Servers that we uninstalled on Step 3.
12. Once DNS is installed on all the other DNS Servers check to make sure that they have the latest DNS entries. They should and at this point you are done.

Piece of cake :)

Good 'ole UPS

2009-06-19T07:29:00.000-07:00

This replacement laptop keyboard was shipped from HP to one of my Field Users via UPS Ground. It's one of those new "bendable after being run over by a truck" types :)

Hope this made you laugh; we need to more often...

Converting .WMV to .MP3

2009-06-05T12:16:00.000-07:00

Here is a great little tool that I just discovered after finding out the Audacity was not a good tool to convert .wmv files to .mp3. Very easy to use and fast...

WinFF
http://winff.org/html/

It appears to have quite a bit of conversion options so if you are doing any conversions check it out.

Exchange 2007 Merge Fails with "Operation was cancelled" error

2009-05-29T07:32:00.000-07:00

During an Exchange 2007 restore I was attempting to do a merge from the RSG into my target mailbox and ran into an error. The error occurs in the Application Event Log after the Exchange Wizard reports that the operation was successful. The error text is the following:

Source: Exchange Migration
Event ID: 1008
Category: Restore Mailbox
Description:
The restore-mailbox task for mailbox 'XXXX' failed.
Error: Failed to copy messages to the destination mailbox store with error:
The operation was cancelled.

As odd as this sounds the fix is to skip the wizard and run the same commands in the EMS (Exchange Management Shell) directly. Here is the syntax:

Restore-Mailbox -identity "Target Mailbox" -RSGDatabase "Recovery Storage Group\RSG Datastore" -RSGMailbox "Mailbox to be restored" -TargetFolder "Folder in -Identity to place data in" -BadItemLimit "Int32"

Serv-U FTP Server 8.0 Windows Authentication Changes

2009-05-27T08:09:00.000-07:00

It appears that there is a change in the new version of Serv-U FTP Server that affects Windows Authentication. This change is that the Windows AD user must be a member of the "Domain Users" AD Group or Serv-U FTP Server will give you an Access Denied error upon connection. In the past you just needed an AD User account and could assign it to a "No Permissions" group so that it could only be used for FTP, however that is no longer the case.

Adobe Download Location

2009-05-20T07:29:00.000-07:00

Need an .MSI of an Adobe product?
ftp://ftp.adobe.com/pub/adobe/

66th Annual Pepperdine Bible Lectures

2009-05-14T09:49:00.000-07:00

Last week I was privileged to visit Pepperdine University in Malibu, California for the 66th Annual Bible Lectures. It was a wonderful time well spent with some great people, listening to some fantastic speakers and enjoying hours of wonderful music. On top of all that goodness was the beautiful view of the Pacific Ocean (plus the quick trip to the beach). The Lord has truly blessed me...

The Internet is broken...

2009-05-14T09:24:00.000-07:00

It looks like there was a rather serious issue with the Internet earlier this morning. The most obvious symptom was that all the Google Apps as well as www.google.com were unavailable from 8:53am MST to 11:15am. At this point it appears that it might have been an issue with the pipe going to Google. Here is a tracert of what we were seeing:

C:\Users\xxxx>tracert www.google.com

Tracing route to www.l.google.com [74.125.93.104]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms xxxxxx.xxxxx.xxxx[xx.xx.0.1]
2 <1 ms <1 ms <1 ms xx.xx.0.1
3 <1 ms <1 ms <1 ms xx.xx.0.9
4 4 ms 4 ms 4 ms xxx.xxx-1-0.xxxx-cust1.dnvr.uswest.net [xx.224.x
x.17]
5 4 ms 4 ms 4 ms cls-core-02.inet.qwest.net [205.171.152.94]
6 30 ms 32 ms 31 ms dap-brdr-03.inet.qwest.net [67.14.2.85]
7 30 ms 30 ms 30 ms 192.205.36.45
8 49 ms 48 ms 49 ms cr2.dlstx.ip.att.net [12.122.87.54]
9 48 ms 48 ms 49 ms cr1.attga.ip.att.net [12.122.28.173]
10 48 ms 48 ms 48 ms 12.123.22.5
11 * * * Request timed out.
12 * * 293 ms 72.14.233.54
13 * * * Request timed out.
14 287 ms * * 209.85.254.239
15 * * * Request timed out.
16 * * * Request timed out.
17 287 ms * 288 ms qw-in-f104.google.com [74.125.93.104]

Trace complete.

Everything looks good at this point, but it really makes you appreciate having Google.

Also, here are some tools to help out when taking a look at outages like this:
1. http://www.internettrafficreport.com/
2. http://isc.sans.org/
3. http://www.internetpulse.net/

Random Reboots on HP Hardware - Error ID 57

2009-04-29T08:28:00.000-07:00

We have a server (BL460c) that we upgraded the drivers and firmware on and now it is rebooting every couple hours. The Event Log has a warning with an ID of 57. The description is "NetFN 0x36, command 0x2 timed out" This is solved by downgrading both iLO drivers.

1. HP ProLiant Integrated Lights-Out Management Interface Driver for Windows Server 2003/2008 x64 Editions to version 1.13.0.0
2. HP ProLiant iLO 2 Management Controller Driver for Windows Server 2003 x64 Editions to version 1.8.0.0

Chuck Swindoll on lust and controlling the mind

2009-04-16T14:52:00.000-07:00

I really appreciate Chuck Swindoll for several reasons. First and foremost is the fact that he preaches like it is, no matter how hard the subject and that he practices what he preaches. That is probably what I want most to be remembered as. Here is a lesson that he recently presented at DTS on lust and purity in the mind. A very important yet terribly difficult lesson that you won't hear even in most churches since it is so convicting. If this message does not effect you or you think that it does not apply to you then you really need to get help; it will make your life so much better.

http://www.dts.edu/media/play/?MediaItemID=7fd7e223-479d-4183-ac94-ec74649a6d7a

If you are looking for help feel free to contact them at 469-252-5200.

ARP Poisoning in a Production Environment

2009-04-15T23:27:00.000-07:00

Here is a sanatized email the I sent last night that has a very interesting problem that we ran into at work. Never ran into ARP Poisoning before...
-------------------------

Sent: Thursday, April 16, 2009 12:21 AM
To: XXXXX XXXXX
Subject: ARP Poisoning

Ok, so here is 12 hours of work boiled down into a couple sentences of “what happened” …

Basic Topography: 10.70.0.1(hop1) > 10.50.0.1(2) > 10.50.0.9(3) > Internet(4) > Destination(5)

MAC Address for 10.50.0.1 = XX:XX:XX:XX:b4:23

Scenario: We isolated the issue (mainly by completely replacing 10.70.0.1 (ISA 2006) & 10.50.0.1 (Core Router) to no avail) so that we knew that traffic was going from 10.70.0.1 out to the internet, hitting the destination and generating response traffic. This response traffic made it to the firewall but died before reaching back to the 10.70.0.1 ISA server. This leaves 10.50.0.1 and 10.50.0.9 as the only possible culprits for the missing traffic. After replacing 10.50.0.1 we discovered that the traffic still exhibited the same behavior and we realized that the chances of 2 Routers both being bad was really remote so we focused on the firewall. Taking a packet trace with the network up and another when the network was dead we found a very subtle difference in the packets. While the network was operating normally the packets were flowing from the firewall to the core router using the level 2 routing address’ of:

Src: XX:XX:XX:XX:ea:b0 - dst: XX:XX:XX:XX:b4:23
When the network was broken the level 2 flow of inbound packets was like:
Src: XX:XX:XX:XX:ea:b0 - dst: 00:0c:29:a9:d2:25

So what we have at this point is ARP Poisoning where another machine on the 10.50.x.x is impersonating 10.50.0.1 which is the Core Router; the result of this is that all traffic coming inbound from the internet (hop3 > hop2) was getting redirected to the mystery machine (hop3 > hop_blackhole) with the mac of 00:0c:29:a9:d2:25. Going from switch to switch we tracked the mac to MachineNameX. After unplugging the machine from the network and clearing the ARP cache on the firewall traffic immediately started working normally and the network is happy. Check out the cool wireshark screenshot attached… now you know what ARP poisoning looks like.

Bottom line: Wireshark is awesome, the 10.50.x.x switches have a command “show bridge address-table” which shows you the mac address’ that are associated with each port on the switch, 2 heads and 4 eyes are better than 1 head and 2 eyes… and sleep is overrated.. :)

Backing up a MAC Server automatically

2009-04-13T09:44:00.001-07:00

Scenario: You run a Windows environment and all of the sudden have a couple Apple XServes that get added and then need backed up.

You are the responder: I will be backing this XServe up to a Windows File Server where the files will then be picked up to go to tape.
Step 1: Create the appropriate file share on the Windows box and and assign the proper permissions to it. Give it a unique AD Service Account since the password is stored in plain text in the backup scripts.
Step 2: Create a folder on the Apple that will be the mount point for your smbfs to the Windows File Server
Step 3: Create a batch file on the Apple that looks like this:

#!/bin/bash
cd /Volume/FoldertoBackup
/sbin/mount_smbfs //ADuserName:ADPassword@WinFileServer/ShareToSaveFilesTo /MyAppleMountPoint
tar -cf backup-servername-`date '+%d-%B-%Y`.tar /Volume/FoldertoBackup
cp backup-servername-`date ' +%f-%B-%Y`.tar /MyAppleMountPoint
umount /MyAppleMountPoint

Step 4: Test the script and see if it works when launched manually. If you cut and paste it from here you will need to use dos2unix to fix the hidden EOL characters since they are different from standard unix EOL.

Step 5: Automate with Crontab
a. 00 04 * * 02 /var/backups/backup_script.sh
This will execute the backup_script.sh file every Tuesday at 4am.

Step 6: You're done... sort of... Now you need to create a script to clean up the backup files after X amount of days on the Apple so you don't lose too much disk space. You can also edit the tar command to do differential backups if you so choose...

Cisco ASA 5510 - Disable Specific Alerts

2009-04-08T14:09:00.000-07:00

We get a ton of false positives on one specific alert on our ASA. The alert is a "No Translation Group Found" that happens when somebody brings a laptop from home and attempts to create a connection before getting the proper IP settings from DHCP. The alert looks like this:

<163>%ASA-3-305005: No translation group found for tcp src (InterfaceName):(IPAddress/Port) dst (InterfaceName):(IPAddress/Port)

We wanted to disable just this one alert so that we do not get so many false positives. After trying several things to no avail I finally opened a support case with Cisco and got a quick and easy fix. To acomplish this all that you need to do is type in:

ciscoasa(config)# no logging message 305005

and to re-enable it all you need to do is type:

ciscoasa(config)# logging message 305005

That was easy!

Painful Truth

2009-03-19T06:36:00.000-07:00

"If Christians would really live according to the teachings of Christ, as found in the Bible, all of India would be Christian today." - Gandhi

You learn something new every day. Cisco ASA 5510 Lessons.

2009-03-18T09:53:00.000-07:00

1. The IPS module is not configured by default to bind to an interface. Found at (in the ASDM) Configuration > IPS > Policies > IPS Policies. I created a new policy that is now bound to the backplane interface using a new set of Event Action Rules (see below), now the IPS is dropping packets and creating alerts as it should.

2. To allow Instant Messenger you need to do two things:

Allow IM in the firewall class maps. Configuration > Firewall > Objects > Class Maps > IM > Add. From here you can allow Yahoo! or MSN IM if you use the default criterion. You can also use Services Criterion to block certain features of IM such as Chat, Conference, File Transfers, Games, Voice Chat and Web Cam.
Tweak the IM rules in the IPS module to allow and deny the traffic that you want.

3. The email alerting is configured using both the IPS and Device Management sliders. Make sure that you can reach the email server's IP from your device or put in a static route to your email server, otherwise you will never get your email alerts :)

4. Event Action Rules are important to your IPS. They define the levels of risk and what to do with the three different levels: HIGHRISK, MEDIUMRISK, LOWRISK. Create your Event Action Rule and then use it via your IPS Policy.

Adobe Vulnerability APSB09-01

2009-02-24T14:47:00.000-08:00

In Adobe's bulletin released last Friday there are 3 solutions to reduce your chances of getting exploited. The first is to disable JavaScript in Adobe Reader by going (in Adobe) Edit > Preferences > JavaScript and unchecking the "Enable Acrobat JavaScript" option. That is fine for the everyday user but not quite a good fix for an enterprise of any size. To accomplish this I found a script located on the internet and modified it to disable Acrobat's JavaScript.

I built the batch file to do a few things including:
a. Add a HKLM key at HKLM\Software\Adobe\Acrobat Reader\8.0\JSPrefs
b. Add a DWORD value under that key called "bEnableJS" with a value of 0
These two steps disable Acrobat JavaScript for all users except those people who have clicked the box manually and created other keys under HKCU.

Next I attempt to change the HKCU values (if generated) to 0 thus disabling the Acrobat JavaScript for all users.

Below is the script:

-------------------------Start Script ---------------------------------
setlocalset regpath=%SystemRoot%\system32\reg.exe
set keypath=Software\Adobe\Acrobat Reader\9.0\JSPrefs
set valuename=bEnableJS
:: update current user
set hive=HKCU
set key=%hive%\%keypath%
:: Add a Master Disable for all users by using HKLM
%regpath% add "HKLM\Software\Adobe\Acrobat Reader\9.0\JSPrefs" /f >nul
%regpath% add "HKLM\Software\Adobe\Acrobat Reader\9.0\JSPrefs" /v %valuename% /d 0x00000000 /t REG_DWORD /f >nul:: Remove Javascript for all individual who have explicitly enabled it.
%regpath% add "%key%" /v %valuename% /d 0x00000000 /t REG_DWORD /f >nul:: update all other users on the computer, using a temporary hive
set hive=HKLM\TempHive
set key=%hive%\%keypath%:: set current directory to "Documents and Settings"
cd /d %USERPROFILE%\..
:: enumerate all folders
for /f "tokens=*" %%i in ('dir /b /ad') do ( if exist ".\%%i\NTUSER.DAT" call :AddRegValue "%%i" ".\%%i\NTUSER.DAT")endlocalgoto :EOF:AddRegValue
set upd=Yif /I %1 equ "All Users" set upd=N
if /I %1 equ "LocalService" set upd=N
if /I %1 equ "NetworkService" set upd=Nif %upd% equ Y (
%regpath% load %hive% %2 >nul 2>&1
%regpath% add "%key%" /v %valuename% /d 0x00000000 /t REG_DWORD /f >nul 2>&1 %regpath% unload %hive% >nul 2>&1
)
-----------------------End Script--------------------------
Note: Change all instances of "8.0" to "9.0" in the script for it to work with Acrobat Reader 9.0

Feel free to modify and use the script but like everything else, test it before you put it into production. I take no responsibility for what you do with it and any results that it might cause.

Thanks to the guys at http://www.ureader.com/ for the original script that I modified to get this running.

Windows Server 2003 Cannot Execute Network File

2009-02-24T13:46:00.000-08:00

On Windows 2003 Servers by default you cannot double click on an exe that is located on a file share or you will get the following error:

"Windows cannot access the specified device, path, or file. You may not have the appropiate permissions to access this item."

To get around this issue you can copy the file to your local drive, right click on the file and select properties. On the bottom of the General screen you will see a warning that the file is from another computer (This file came from another computer and might be blocked to help protect this computer) click "Unblock" and you can now execute the file.

or Uninstall the "Internet Explorer Enhanced Security Configuration".

Windows Server 2003 VM Running Exchange with Random Bluescreening

2009-02-24T08:03:00.000-08:00

I have a Microsoft Exchange 2007 server (x64) that was originally on a physical piece of hardware. However we have P2V'd it into a guest on a VMware ESX host and it seems to be running normally except for one little detail: It bluescreens every 15 or 20 days with the following information:

Event Type: Error
Event Source: System Error
Event Category: (102)
Event ID: 1003
Date: 1/31/2008
Time: 8:34:36 AM
User: N/A
Computer: SMTP2
Description:Error code 00000000000000d1, parameter1 0000000000000019, parameter2 0000000000000002, parameter3 0000000000000001, parameter4 fffffadfd99f4e5b.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.Data:0000: 53 79 73 74 65 6d 20 45 System E0008: 72 72 6f 72 20 20 45 72 rror Er0010: 72 6f 72 20 63 6f 64 65 ror code0018: 20 30 30 30 30 30 30 30 00000000020: 30 30 30 30 30 30 30 64 0000000d0028: 31 20 20 50 61 72 61 6d 1 Param0030: 65 74 65 72 73 20 30 30 eters 000038: 30 30 30 30 30 30 30 30 000000000040: 30 30 30 30 31 39 2c 20 000019, 0048: 30 30 30 30 30 30 30 30 000000000050: 30 30 30 30 30 30 30 32 000000020058: 2c 20 30 30 30 30 30 30 , 0000000060: 30 30 30 30 30 30 30 30 000000000068: 30 31 2c 20 66 66 66 66 01, ffff0070: 66 61 64 66 64 39 39 66 fadfd99f0078: 34 65 35 62 4e5b

It seems fairly random and there is not a whole lot to go on as far as troubleshooting... I found a MS patch (kb950772) that is supposed to fix "A computer that is running an x64-based version of Windows Server 2003... randomly restarts and then generates a Stop Error" After applying the patch the box seems stable and has yet to throw a stop error.

Patch Details at: http://support.microsoft.com/kb/950772

Allowing PPTP Through Cisco ASA 5510

2009-02-23T20:50:00.000-08:00

Pretty simple fix for a Cisco ASA5510 that does not allow PPTP traffic.

>Config t
>class-map inspection_default
>Policy-map global_policy
>class inspection_default
>inspect pptp
>service-policy global_policy global
>write

Cannot Connect to Cisco ASA 5510 with ASDM

2009-02-23T20:35:00.000-08:00

So, we have a couple Cisco ASA 5510 security appliances that we are working on. Here is the catch, when you attempt to connect to them using the ASDM tool downloaded from the device (or the CD from Cisco, or a download from Cisco's website) you get a connection error "ASDM cannot read the configuration from the ASA". After ensuring that the networking was indeed correct and I could reach the device I discovered (after several hours) that the ASDM is java based and that the version of java that I was using (Java 6 Update 11) was simply too new for the ASDM. To fix this I had to downgrade my java to Java 5 Update 17 and bingo, the ASDM could now talk to the ASA.

Next I was able to upgrade the via the ASDM the ASA's Software and ASDM software. Once updated to the newest versions (8.01 and 6.1551 respectivly) I was able to re-download the ASDM client from the device and use the newest java version.

IT Security Top 10 Tips for 2009

2009-02-23T20:26:00.000-08:00

I got to help write the Top 10 Security Tips for 2009 for our employees and this is what we came up with. I think it is a good security overview for the normal everyday user.

#10 Wifi: As you travel around you will frequently see “Free Public Wifi” in your list of available wireless networks. This is almost always a VIRUS on someone’s computer trying to get you to connect so it can infect you also. Think of this as the “free public used gum” stuck under your desk. DO NOT ‘connect’ to it for any reason. Never connect to any Wi-Fi you do not fully trust; unless of course you like hackers using your identity or credit cards…

#9 Fake News Emails: Never click on any links in an email from CNN or MSNBC, or any other "news alerts" that you have never subscribed to. No matter how realistic it looks. Usually they start with a very absurd or weird story such as "Britney Spears killed in a car accident or Bigfoot found in new jersey, etc.." Even if you have subscribed to news alerts it is best to be cautious when following links.

#8 Fake “tracking number” Emails: If you get a "UPS tracking " attachment never ever open these attachments, they are virus's. They also appear to come from FedEx, USPS, etc… A valid tracking email will never have an attachment.

#7 Fake “Greeting Cards”: Never open a email postcard (Hallmark e-card is the most popular) unless it’s your birthday and it’s from someone you expect it from. This is the main delivery mechanism of most of our virus’s today. Also, an e-card will never have an attachment with a .exe extension.

#6 Lock your Desktop when not in use and have a screensaver password. Also lock your mobile devices (phone) with a password. If you don’t lock the doors then it does not make much sense to bar the windows. Don’t make it easy for hackers or others who would want to cause damage.

#5 Fake Instant Messages: Many people here use IM to communicate. It is a great tool but you need to be suspicious of hyperlinks; even if the link appears to be from your friends or coworkers. When a computer gets infected by a virus it is not uncommon for it to steal the address book and email/IM all of that persons contacts with the same virus. Best rule of thumb: Don’t follow hyperlinks

#4 Don’t put every CD you get mailed or USB key you find lying in the parking lot into your PC, they can “auto-install” a virus onto your PC or do many other nasty things. You didn’t just win a free prize, this is like the “free used gum”; besides it is a very well known technique for hackers and pen-testers alike. Again, don’t make it easy for the bad guys.

#3 Make sure you have Antivirus Installed and make sure that it has recent definitions, if you AV software is not updating, it is almost as good as not having it at all. In today’s day and age antivirus is a must…. well maybe not if you don’t have an internet connection…

#2 Keep your software up to date. Do your Microsoft Updates and software updates for all the products that you use. This includes software like Adobe, VMware and whatever else you use. As the famous ex-hacker Kevin Mitnick suggests “Update your OS religiously and be vigilant in applying all security patches released by the software manufacturer.”And the #1 thing Everyone should do in 2009 is:

#1 Backup everything you use. Make sure you have it somewhere else, on an external hard drive, a file share, somewhere. Don’t assume that anyone else (even IT) is backing that data up. If you have a question if a file share is being backed up please contact the IT Department, otherwise assume it is not. One Worm or Trojan or drive crash can wipe out 100% of your data forever, don’t let it happen to you.

A Great Listen (or should I say "A Great Lesson"?)

2009-02-23T08:06:00.000-08:00

As I am a bit under the weather and not at work today I have been trying to do something constructive with my time. I found an excellent message by Dr. Ron Rhodes recorded at the Dallas Theological Seminary Chapel Service last week. It is an excellent message on "How to Make a Point Without Impaling Someone Upon It." The message is located at: http://www.dts.edu/media/play/?MediaItemID=c0c46e57-1e9b-49e4-9bc5-d75977a35f91

I hope that it is as much a blessing to you as it is to me. Now, the difficult part, actually living it...