Natural Born Coder

LSI MegaRAID Firmware Upgrade under VMWare

—

by

doozer
in ,

I was recently faced with the task of upgrading the firmware on an LSI MegaRAID 9260-4i RAID card. Normally this would be a fairly simple process as LSI provide an easy to use command line utility called MegaCLI. The difficulty was using this utility on a machine that was running VMWare ESXi 5 Update 1. This article describes the steps I went through to perform the update.

Update

Yep, before I even finished writing the article I had to put in an update because I found myself having to take a radically different path to the one I expected.

It seems LSI provide two versions of the MegaCLI tool. One called CLI_VMWare_MN_8.04.07.zip the other called CLI_VMWare_8.04.07.zip. The first one contains a vib package which, I believe, should install the MegaCLI tool but I was never able to make it work. The closest I got was running

esxcli software vib install -v /path/to/vib/file

This appears to be the correct command but it dies with this complaint:

[InstallationError]
  ('LSI_bootbank_vmware-esx-MegaCli-8.04.07_8.04.07-01', 'Could not find a trusted signer.')
 vibs = LSI_bootbank_vmware-esx-MegaCli-8.04.07_8.04.07-01
 Please refer to the log file for more details.

I could have run the command with –no-sig-check but I didn’t fancy the idea of having vSphere Client complain about unsigned packages all the time so I’ve not gone down this route.

The second zip file contains a single file called MegaCli which weighs in at about 2.2MB (and another zip containing the same file oddly enough). To use this just copy this single file into a new directory on the host called tools or something obvious using the vSphere Client datastore browser.

Next, shell into the server and switch to the tools directory. The file won’t be executable so make it so by issuing this command:

chmod u+x MegaCli

Now try running this command to gather information about the adaptors fitted in your host:

./MegaCli AdpAllInfo aAll

If you get the following error message (I think you probably will)

./libstorelib.so: cannot open shared object file: No such file or directory

Then you need to also copy the libstore library into the same folder as the MegaCli file. This library, apparently, is found in ESXi 4 and 4.1 but not in 5 and later. Some friendly soul has provided it here. Before you download and install it though just keep in mind you’ve just installed a library from an unknown source – that’s a big risk!

If all goes to plan when you run the “all info” command you should now see several pages of information about you RAID cards. If you do, copy over the firmware image into the tools folder, the image I was installing was called: mr2108fw.rom, yours may differ.

Execture the command:

./MegaCli -adpfwflash -f mr2108fw.rom -a0

And as I should have guessed it wouldn’t go smoothly:

Adapter 0: LSI MegaRAID SAS 9260-4i
Vendor ID: 0x1000, Device ID: 0x0079

Package version on the controller: 12.7.0-0007
Package version of the image file: 12.12.0-0111 Download Completed.
Flashing image to adapter…
Firmware Failed to FLASH flash. Stop!!!

Exit Code: 0x11

This process has been a total mission from start to now and it doesn’t look like it’s getting any better. Fingers crossed the failure to flash hasn’t actually done any damage. A quick Googeling hasn’t turned anything up yet for that error code. I’m wondering if the existing firmware is too old for this new firmware.

Well a quick request to LSI support later and I was right the firmware on the card is apparently too old to upgrade straight to version 12.12.0-0111. Instead I have to upgrade the card to 12.12.0-0073 first, re-boot and then do the upgrade to the latest version. Not exactly ideal – this card is brand new after all – and I’m a bit miffed there are no instructions for this but hey-ho.

Step one therefore is to upgrade to version 12.12.0-0073. Since this old firmware is, as far as I can tell, impossible to get hold of other than by putting in a support request I’m going to take the unusual step of providing a download for it sas_fw_image_12.12.0-0073. If LSI ask me to take this down I will so it may vanish at any time.

./MegaCli -adpfwflash -f AF2108_FW_Image.rom -a0
Adapter 0: LSI MegaRAID SAS 9260-4i
Vendor ID: 0x1000, Device ID: 0x0079
Package version on the controller: 12.7.0-0007
Package version of the image file: 12.12.0-0073
Download Completed.
Flashing image to adapter...
Adapter 0: Flash Completed.
Please reboot the system for the changes to take effect
Exit Code: 0x00

Note that the package contains two firmware versions. The 8mb version wouldn’t flash into my device. The reboot, like with most firmware upgrades, is scary because it’s much longer than usual presumably because it’s unpacking and installing the new firmware. I could tell the machine was doing something from the increased processor fan speed.

Step two is to again log into the host and perform the upgrade to 12.12.0-0111.

./MegaCli -adpfwflash -f mr2108fw.rom -a0
Adapter 0: LSI MegaRAID SAS 9260-4i
Vendor ID: 0x1000, Device ID: 0x0079
Package version on the controller: 12.12.0-0073
Package version of the image file: 12.12.0-0111
Download Completed.
Flashing image to adapter...
Adapter 0: Flash Completed.
Please reboot the system for the changes to take effect
Exit Code: 0x00

Another reboot and you should have the latest firmware installed and ready to use.

Speed Testing

The whole reason for me doing these updates was because I wanted for fit the LSIiBBU08 battery backup unit. When I took delivery of the server I was quite disappointed in the speed of the drive array so I read around a bit and discovered that write cache makes a huge difference to the write speed when using RAID 5 or 6 (I’m using RAID 6). As a test I forced the card to use write back mode to see how fast it would be – it was much faster. Write back mode is only safe though when using a back up battery or UPS as a loss of power in this mode without a battery backup could easily corrupt the whole drive. I did quite a lot of performance testing but some example figures from CrystalMark are shown here (all figures are MB/s, 3 runs, 1000MB).

Desktop Server
Write Through		 Server
Always Write Back		 Server, Write Back, Latest Firmware
Read Write Read Write Read Write  Read  Write
Sequential 112.6 109 215 18.57 224.5 171.2  222.5 174.9
512K 47.55 61.5 59.12 14.76 69.6 84.83  64.01  70.49
4K 0.654 1.354 0.849 0.455 0.95 3.402  0.916  3.474
4K QD32 0.714 1.376 6.183 1.558 6.594 2.942  6.283  2.896

So it looks like the new firmware only brought bug fixes and a couple of new features. The throughput of the drives is essentially unchanged – the minor differences are most likely artefacts of the testing.

Well it’s been a long day but I’m pleased to say it’s finally flashed and I’m off to watch a movie…

References