Friday, May 14, 2010

Bad ESX Network Driver on BL490c G6

Ok, so here is the scenario: HP Blade (BL490c G6) has a bad stick of RAM, pull blade from cabinet, swap RAM, re-insert blade, power on blade and receive the following ESX error:

"networking-drivers ...
No compatible network adapter found. Please consult the product's Hardware Compatibility Guide (HCG) for a list of supported adapters."

At first we were sure that the issue was firmware related however it appears that it just might be a bad VMware driver. Here's why: I went ahead and did two test cases to isolate the issue to the VMware driver on our BL490c G6 blades.

Blade 1: Updated to newest Firmware from HP Smart Update Firmware. This updated the NIC Bootcode from 5.0.11 to 5.2.7 but left the iSCSI at 3.1.5. After this the machine will boot most of the time after the blade is removed from the chassis, however after several reboots there are seemingly random times when ESX will not load the NIC and you get the "No compatible network adapter found" error when attempting to boot ESX. After installing the new drivers found at "http://www.vmware.com/support/vsphere4/doc/drivercd/esx40-net-bnx2x_400.1.48.107-1.0.4.html" using the esxupdate command it works fine. I have rebooted many times and removed the blade and am no longer able to get the error.

Blade 2: I left the firmware alone and just updated the VMware driver to the latest from the above URL. After several reboots and physically removing the blade from the cabinet twice I am unable to get the error.

Conclusion: Looks like a VMware driver issue; however updating the HP Firmware on the NIC from 5.0.11 to 5.2.7 does seem to help the issue taking it from happening every time to approximatly %50 of the time.