Apache.org hacked. What the hell were they thinking?

As many will know already, apache.org has been hacked yesterday. While events like these are rare, and sometimes look like science fiction, the path taken to exploit their servers was a relatively easy one that, if I understand it correctly, shows gross negligence from their part.

Here is my analysis of what the apache team posted today:

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org. The account was accessed using SSH key authentication from this host.

Having your SSH keys stolen is a possibility. With automated tasks, keys are not protected by a passphrase, so anyone who gains access to them can easily use them for their own purpose. The first line of defense is to protect file access to your private keys as much as possible, and use a dedicated user to own it. How were these keys stolen? That could possibly be inside job, and you can probably bet that it didn't require root privileges to grab it. If it did require root, then that should narrow down the culprit unlesss the provider got highjacked, too.

Or perhaps that "3rd party hosting provider" didn't bother protecting the key at all, leaving it world-readable, and didn't chroot its inbound data transfer accounts, so anyone who has FTP access to the server to upload his own stuff could have stumbled upon it by snooping on the server. For that part, I don't know.

But the next part is particularly interesting:

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts. These files were then rsynced to our production webservers by automated processes. At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

Now get this. From what I can see here, there a few problems here:

Whoever owns the SSH key can upload stuff on minotaur.apache.org. That's not really a problem per se. But there is probably no filtering done on the IP address to limit inbound connections to the provider's netblock, so he can come from possibly anywhere. That might be a usability requirement; in that case, you can bet they'll probably think about requiring port knocking from now on to at least mitigate the possibility of this happening again (and I insist on the verb mitigating, as knocking is more obscurity than real security). And for this to be "safe", the hosting provider will have to keep the knocking sequence as safe as the key.

Whether the account is kept under a tight leash on on minotaur, such as using a chroot jail or whatever else, doesn't make a difference! Why? Because data uploaded to this account is rsynced automatically from that account to the servers running www.apache.org, unverified. So you can possibly upload any nasty code you would like to compromise anyone reading a page on www.apache.org using an exploitable browser.

Hell, who knows, maybe these hackers have been injecting compromised pages for a few days as what seems to have tipped the apache admin off are rogue processes on their servers. They were launched remotely quite easily, as data can be rsync'ed straight in cgi-bin/ ! Now how good is that?

While not thinking about details like that inside a corporation is standard practice, and tolerable in many cases as there is an implicit trust within the organization, as soon as you have a server with a gateway exposed publicly on the net you need to take precautions to isolate it from your production. In this case, it's clear to me that the Apache group didn't think this completely through. Being the authors of a secure and great web server, being hacked like this will probably go down in as one of the shameful events in the Apache group's history.