Using OFM to update firmware on rx7640/rx8640 series

In the past, updating firmware on cell-based servers was a daunting task, requiring an FTP server usually piggybacked directly on the MP, and lots of manual commands to flash each part independently. Not anymore. HP wanted to charge me to come in and flash a bunch of servers so this gave me the opportunity to do it myself. I flashed among these a two-cell rx7640 using OFM, and it's now dead easy: simply download a .iso file, burn it on a CD, and boot on it. It uses OFM which has been available for low-end Integrity servers for a while. We're still far from the Proliant Firmware Maintenance CD, but nevertheless it's still much better than nothing! One detail: the MP has to be configured to "allow upload of firmware updates from the OS", which is enabled in CM>SO. You still need to cut off AC power though at some point, so an onsite update is still mandatory.

And for those who still have rx7620s, there's no OFM version of the latest firmware and you still have to do it the long way. While it's not as trivial, it is at least well documented.

Apache.org hacked. What the hell were they thinking?

As many will know already, apache.org has been hacked yesterday. While events like these are rare, and sometimes look like science fiction, the path taken to exploit their servers was a relatively easy one that, if I understand it correctly, shows gross negligence from their part.

Here is my analysis of what the apache team posted today:

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org. The account was accessed using SSH key authentication from this host.

Having your SSH keys stolen is a possibility. With automated tasks, keys are not protected by a passphrase, so anyone who gains access to them can easily use them for their own purpose. The first line of defense is to protect file access to your private keys as much as possible, and use a dedicated user to own it. How were these keys stolen? That could possibly be inside job, and you can probably bet that it didn't require root privileges to grab it. If it did require root, then that should narrow down the culprit unlesss the provider got highjacked, too.

Or perhaps that "3rd party hosting provider" didn't bother protecting the key at all, leaving it world-readable, and didn't chroot its inbound data transfer accounts, so anyone who has FTP access to the server to upload his own stuff could have stumbled upon it by snooping on the server. For that part, I don't know.

But the next part is particularly interesting:

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts. These files were then rsynced to our production webservers by automated processes. At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

Now get this. From what I can see here, there a few problems here:

Whoever owns the SSH key can upload stuff on minotaur.apache.org. That's not really a problem per se. But there is probably no filtering done on the IP address to limit inbound connections to the provider's netblock, so he can come from possibly anywhere. That might be a usability requirement; in that case, you can bet they'll probably think about requiring port knocking from now on to at least mitigate the possibility of this happening again (and I insist on the verb mitigating, as knocking is more obscurity than real security). And for this to be "safe", the hosting provider will have to keep the knocking sequence as safe as the key.

Whether the account is kept under a tight leash on on minotaur, such as using a chroot jail or whatever else, doesn't make a difference! Why? Because data uploaded to this account is rsynced automatically from that account to the servers running www.apache.org, unverified. So you can possibly upload any nasty code you would like to compromise anyone reading a page on www.apache.org using an exploitable browser.

Hell, who knows, maybe these hackers have been injecting compromised pages for a few days as what seems to have tipped the apache admin off are rogue processes on their servers. They were launched remotely quite easily, as data can be rsync'ed straight in cgi-bin/ ! Now how good is that?

While not thinking about details like that inside a corporation is standard practice, and tolerable in many cases as there is an implicit trust within the organization, as soon as you have a server with a gateway exposed publicly on the net you need to take precautions to isolate it from your production. In this case, it's clear to me that the Apache group didn't think this completely through. Being the authors of a secure and great web server, being hacked like this will probably go down in as one of the shameful events in the Apache group's history.

August update

Please note that I'm on vacation thus there will be no updates for a while.

But what's coming for the next fall?

Well dear readers, we've had a security audit recently. While I've invested lots of time into hardening the server perimeter with IP Filter over the years, some adjustements will be needed to enhance security and compliance inside the OS itself.

There are especially discoveries and experiments to be made with the new 11iv3 auditing subsystem which is not well documented, and for which there is currently no whitepaper available at HP Docs. Auditing is now it is way better than what we had before with Trusted Mode, and you can bet I'll use it. I just hope HP did their homework so I won't need to write a hack like audenable to have it work correctly this time, having to rely on audenable in the 11.11 days sucked.

Furthermore, I don't forward all my logs to an external, secure server, except everything related to AUTH_LOG. More needs to be done to be compliant. An intern has worked hard to make this work under many scenarios a few months ago and this will be implemented soon. I'm just waiting for the official mandate. I'll keep you posted on what we'll be doing.