Using OFM to update firmware on rx7640/rx8640 series

In the past, updating firmware on cell-based servers was a daunting task, requiring an FTP server usually piggybacked directly on the MP, and lots of manual commands to flash each part independently. Not anymore. HP wanted to charge me to come in and flash a bunch of servers so this gave me the opportunity to do it myself. I flashed among these a two-cell rx7640 using OFM, and it's now dead easy: simply download a .iso file, burn it on a CD, and boot on it. It uses OFM which has been available for low-end Integrity servers for a while. We're still far from the Proliant Firmware Maintenance CD, but nevertheless it's still much better than nothing! One detail: the MP has to be configured to "allow upload of firmware updates from the OS", which is enabled in CM>SO. You still need to cut off AC power though at some point, so an onsite update is still mandatory.

And for those who still have rx7620s, there's no OFM version of the latest firmware and you still have to do it the long way. While it's not as trivial, it is at least well documented.

Apache.org hacked. What the hell were they thinking?

As many will know already, apache.org has been hacked yesterday. While events like these are rare, and sometimes look like science fiction, the path taken to exploit their servers was a relatively easy one that, if I understand it correctly, shows gross negligence from their part.

Here is my analysis of what the apache team posted today:

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org. The account was accessed using SSH key authentication from this host.

Having your SSH keys stolen is a possibility. With automated tasks, keys are not protected by a passphrase, so anyone who gains access to them can easily use them for their own purpose. The first line of defense is to protect file access to your private keys as much as possible, and use a dedicated user to own it. How were these keys stolen? That could possibly be inside job, and you can probably bet that it didn't require root privileges to grab it. If it did require root, then that should narrow down the culprit unlesss the provider got highjacked, too.

Or perhaps that "3rd party hosting provider" didn't bother protecting the key at all, leaving it world-readable, and didn't chroot its inbound data transfer accounts, so anyone who has FTP access to the server to upload his own stuff could have stumbled upon it by snooping on the server. For that part, I don't know.

But the next part is particularly interesting:

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts. These files were then rsynced to our production webservers by automated processes. At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

Now get this. From what I can see here, there a few problems here:

Whoever owns the SSH key can upload stuff on minotaur.apache.org. That's not really a problem per se. But there is probably no filtering done on the IP address to limit inbound connections to the provider's netblock, so he can come from possibly anywhere. That might be a usability requirement; in that case, you can bet they'll probably think about requiring port knocking from now on to at least mitigate the possibility of this happening again (and I insist on the verb mitigating, as knocking is more obscurity than real security). And for this to be "safe", the hosting provider will have to keep the knocking sequence as safe as the key.

Whether the account is kept under a tight leash on on minotaur, such as using a chroot jail or whatever else, doesn't make a difference! Why? Because data uploaded to this account is rsynced automatically from that account to the servers running www.apache.org, unverified. So you can possibly upload any nasty code you would like to compromise anyone reading a page on www.apache.org using an exploitable browser.

Hell, who knows, maybe these hackers have been injecting compromised pages for a few days as what seems to have tipped the apache admin off are rogue processes on their servers. They were launched remotely quite easily, as data can be rsync'ed straight in cgi-bin/ ! Now how good is that?

While not thinking about details like that inside a corporation is standard practice, and tolerable in many cases as there is an implicit trust within the organization, as soon as you have a server with a gateway exposed publicly on the net you need to take precautions to isolate it from your production. In this case, it's clear to me that the Apache group didn't think this completely through. Being the authors of a secure and great web server, being hacked like this will probably go down in as one of the shameful events in the Apache group's history.

August update

Please note that I'm on vacation thus there will be no updates for a while.

But what's coming for the next fall?

Well dear readers, we've had a security audit recently. While I've invested lots of time into hardening the server perimeter with IP Filter over the years, some adjustements will be needed to enhance security and compliance inside the OS itself.

There are especially discoveries and experiments to be made with the new 11iv3 auditing subsystem which is not well documented, and for which there is currently no whitepaper available at HP Docs. Auditing is now it is way better than what we had before with Trusted Mode, and you can bet I'll use it. I just hope HP did their homework so I won't need to write a hack like audenable to have it work correctly this time, having to rely on audenable in the 11.11 days sucked.

Furthermore, I don't forward all my logs to an external, secure server, except everything related to AUTH_LOG. More needs to be done to be compliant. An intern has worked hard to make this work under many scenarios a few months ago and this will be implemented soon. I'm just waiting for the official mandate. I'll keep you posted on what we'll be doing.

A tale of trying to run the HP BladeSystem Power Sizer

Normally when I want to evaluate the power that a server will pull from the grid, I just check its specifications and I'm done with it. Yesterday I needed to measure the power requirements of a C3000 chassis we're going to install in a data center downtown. It hasn't been that trivial.

Here is the scoop: I'll tell you how right away how much power a C3000 consumes.

• It has a maximum of 6 power supplies, each rated at 1200W.
• Since this is a 3+3 configuration of redundant power, this means that the total it can consume is 1200W * 3, which makes 3600W.
• At 240V, 3600W gives out 15 amperes.
• Done.

While this is possibly not the best way to calculate how much power it requires, doing otherwise is not very easy. The quickspecs detail every possible configuration but are shy on the exact consumption. 6.2A per power supply is mentioned at the end. So okay, we're at 18.6 amperes then.

Then there is the tool named "HP BladeSystem Power Sizer" that looked promising. I could not have been more wrong.

I expected a nice web-based application or, at worst, an Excel spreadsheet (Google docs would have been way cool). But of course, no, no, no, no, no, no! HP made a stand-alone software to do this. I don't see that much anymore, especially for such a purpose. Maybe it has nice bells and whistles that the programmers didn't know how to develop using AJAX, because they're stuck in limbo in a brave .NET world? Perhaps. How could I know? I haven't even been able to make it run!

For a vendor to release stand-alone software like this in 2009 is an excellent way to loose sales. Writing short-sighted Javascript too, by the way. Here's why:

1. Many corporate PCs are locked and users do not have administrator privileges, it's my case here. The Power Sizer is a full-blown InstallShield-packaged setup program, and it fails with an error if you're not an admin. Okay, so I ask an admin to install it.

2. Then I'm welcomed with a dialog that asks me to input my name, company, and e-mail address. It's definitely going to send this back to HP. Why not doing a web-based interface then? No privacy policy info can be seen anywhere (looks like it was at the end of the long EULA I blood-signed when I installed earlier).

3. And, at the end of it all, I get greeted by this:


As you see, this is a white window, with useless menus at the top which don't do anything except offer me the choice of leaving or changing my profile information and an about box.

Boy, does a tool like this suck. Not only does it run only on Windows (GNU/Linux users, scram!), but it requires administrator privileges to install (users with locked PCs, scram!), and according to the EULA, it sends back to HP what you're doing with it (privacy advocates, scram!). And finally, it doesn't even work (customers who made it here, scram!).

Back to the calculator to chew up the power requirements, I guess.

vgimport with persistent DSFs

When upgrading a server from 11.23 to 11.31 using a cold-install, the procedure to migrate your VGs and filesystems boils down to this:

1. vgexport -s -p -m vg_name.map vg_name
2. Keep a backup copy of the fstab
3. Reinstall
4. vgimport -N -s -m vg_name.map vg_name
5. Put back the filesystems in fstab, create the directories and mount

Pretty straightforward, huh?

System administrators familiar with vgexport/vgimport will notice the new -N option which, according to the man page does this:

 -N             Configure the volume group by populating
                persistent device special files in the /etc/lvmtab
                or /etc/lvmtab_p file, corresponding to the volume
                group, vg_name.  (See intro(7) for information
                about device special files.) This option can only
                be used with -s option.  If vgimport is invoked
                without a -N option, legacy device special files
                will be used to populate the /etc/lvmtab or
                /etc/lvmtab_p file.

                This option may become obsolete in future
                releases.

What this option does, is importing your imported VGs with persisten DSFs. If you don't do this, they will be imported using the legacy DSFs, and if you're on a SAN this means that PVLinks will still be used. I think that the storage stack on 11.31 is smart enough to apply multipathing in the background to legacy DSFs but it's much cleaner to use the persistent DSFs right away.
O./

Even though I'm not fond of RSP, it *does* work

This morning, a cache battery in one of our EVAs failed at 10h58 AM. At 11h20, the response center called me back to confirm the event (I hadn't read all my mails yet and was not even aware of it by then). I didn't even have mention the part number or the site address. We got in touch with a field engineer by noon.

That's the kind of event where of having invested so much time, and so much energy, in making RSP work pays off.

The ISEE EOL e-mail. Or is is AOL? Or LOL?

I got this e-mail today from the Insight Remote Support Team announcing the demise of ISEE:


Did you notice the image has been scaled down? And that the text in it is more readable than this blog post? This is because, my friends, the desk jockey who typed up that e-mail thought that using a 36 point font would gather my attention. How pathetic. It looks like a junk mail sent from @aol.com in the early 2000s. It's so unreadable that I threw it away, laughing. As my blog readers know already, I'm fully aware that ISEE is going away.

To help me appreciate HP further, did you know that I've been trying for 6 weeks now to get Licenses in Command View 7 format? This is becoming an interesting developing story. I'll soon write about this, and will give out how many people up the HP ladder had to be bothered so that I could get these fucking licenses. Help rebuild the U.S. economy: Ask for CV 7 licenses!