Integrity fibre channel card firmware quick dive

I've had a fixation on the firmware of these cards for a while. Why? Because while I never update firmware on LAN cards, I still have many 2Gb fibre cards in my environment and they are based on designs probably made sometime around 2005-2006. Their firmware took a while to become mature and support specifics on the Integrity platform such as vPars, so there have been a few firmware releases in their lifetime.

There are two official and documented ways to update the firmware on QLogic-based fibre cards shipped with Integrity servers running HP-UX. The first one, which is also the easiest, consists of putting in the server or through Virtual Media a recent IA Offline Diagnostics CD (which comes with up to date drivers for many cards) and run fcd_update at the EFI Shell. The other one, which I personally prefer, requires you to copy the firmware files on the EFI partition, reboot to the EFI shell, and run fcd_update from there.

However, few administrators actually need to update this firmware as each release of the HP-UX fcd driver comes with the required RISC firmware and it updates the card automatically if required. The only situation where one might need to update the firmware manually is when booting on SAN as it might require updating the EFI driver. More on this below.

That's where being able to do it online can come handy if you want to save time. Why? Because flashing the firmware offline requires, from my experience, around 10 minutes per port and that can become very cumbersome if you're dealing with a rx8640 with multiple vPars or a Superdome. That online process is not well documented but I found this document here which explains how to do it using fcmsutil. It is actually easy: Simply run fcmsutil once to update the RISC firmware, and run it again for the EFI firmware.

So what is the difference between that RISC and EFI firmware? The same document linked above, although not well written, provides some definite answers. The RISC firmware is the storage processor on the card, which, not surprisingly, is based on a RISC chip; it it used to implement the fibre channel protocol. On the other hand, the EFI "firmware" is in fact an EFI driver embedded in a 2nd flash ROM on the card which is loaded by the EFI when booting the server. vPars themselves also go a bit deeper and I won't go too much in details here but they require an additional layer named fPars, or firmware partitions (Alan Hymes from HP has good slides on this), and that EFI firmware must support them if you're running vPars.

I hope this clears things up for you. Good luck with your firmware update endeavours!

O.

Moving a C3000 and C7000

Today, I had to move two blade chassis, a C7000 and C3000, to two different locations downtown. We no longer had the original packaging and this being sensible equipment, my fellow sysadmins and I didn't want to hire movers and risk having some parts broken. HP does offer an official moving service, and they will cover anything that breaks once at your destination, but it can be costly. As two blade chassis can fit quite well in a minivan, they can be moved around easily as long as you're cautious.

You know you're dealing with true geeks when you see a bunch of guys shoveling on a hand truck a naked C7000, tied to it with old orange fiber optic cable because they didn't find anything else. That image was so cool, I should have taken a picture. But man, these suckers are heavy. Even with all the blades, power supplies, fans and interconnects removed, you'll still need to be two to hold them up. And whatever yo do, don't drop'em, especially if you have off-the-shelf Hush Puppies right underneath.

Looks like things are still like what they used to be!

Last Wednesday, I tried to renew my subscription to a consumer protection magazine I've been reading for the last 11 years. I don't know how many subscribers it has, but it must not be beyond 100 or 200K so it's fair to expect their web services to be limited. Yet, they offered the possibility of renewing over the internet, so sure, I decided to save some carbon dioxide and use their web interface instead of snail mail.

Wrong idea. By following their subscription process, I ended up in the profile of another customer and saw his personal info. I didn't do any effort to do get there, and by that I really mean NONE. It just popped up in my browser. Looks like our sessions got mixed up. Man, even something using the infamous formmail would have given me a better sense of security! Looks like things are still like what they used to be.

There was no credit card info, but enough data to try doing a fraudulent phone call since I not only knew the guy's birth year, but also his address, phone number, and the pinnacle of it all: that he was subscribed to a highly respected magazine, along with the expiration date of his subscription... Social engineering anybody? Sure, many people put all this on display on facebook, but I'm note sure that customer would have liked me calling him up.

While a mom-and-pop operation could be a little more excusable, I'm surprised considering the nature of that publication that such a thing could happen. I left them an e-mail with a screenshot and sure hope they'll fix this soon. We're not in 1995; we're in 2009, and a bug like this shouldn't have gone unnoticed. And no, two business days later, I didn't get any reply to my mail whatsoever.

Needless to say I decided to delete all information in my profile... and send everything through the mail.

O.

Comparing log management products

In the last few weeks, I've been looking into SIEMs and log management products. Yes, you know it already, I've blogged extensively on how I was upset that I had to go through a sales channel to get a bit of info, but promised I would give out details on what I preferred between ArcSight and Splunk.

It turns out that doing a public comparison of these products won't be easy as ArcSight gives out technical info only under NDA. While I can probably announce loudly that "their appliances log stuff", I can probably say no more. So technical details will remain sealed to my business documents. Sorry. One thing I can say, however, it that their range of products seem to be the Cadillac of log management, and everything I could possibly think of needing to better score at our next audit will be in it.

Concerning Splunk, I inquired about ESS using the "contact sales" button as I didn't find much details on that application. They left me a VM some 4 business days after my initial request for info although I said in it I preferred e-mail, and that didn't rub me the right way (I hate voicemail but that subject is more fitting for a future blog post). No follow-up e-mail. I'll try to call them back when I'll be near a phone when it's California time, and with all these governance'n'compliance-related meetings I'm assisting to these days, it might turn out to be never.

Q1labs read my blog, knew I was looking for log management products, and gave themselves the trouble to track me down and find me at my workplace. I normally would have turned them away, but they showed some good will by having someone call me up in french, and their products being designed in Fredericton N.B., I just had to give them a chance. I saw what they make and it's similar in spirit to what ArcSight does, and their selling point is that their technology is simpler and quicker to deploy than ArcSight's. It sure looks interesting.

I'll see what political pressures I'll face internally but compared to some other cost centers in our company, for us IT is an expense, not a revenue. What will determine whoever wins might come down to be strictly business... as long as the tool does the job and has the feature set we're looking for, the financial aspect might end up having the most weight.

I'm all new to pleasing this IT Governance gestapo that came out of nowhere to bully our small, under-the-radar-IT dream team. But from what I understand until now, I first need to submit a "business opportunity" document to them to justify my funding, giving ball park figures and a few vendors, THEN I can make another "business case" document to explain which one I've chosen. Such a process takes time, and when I cannot give any clear timeframe, it's no wonder that these sales people get their hopes down.

Want to know why I prefer Open Source software? Because since it costs nothing, I've been able to pull it off for years without having to go through this shit. Now I'm knee-deep in it.

O.

HPTF 2010:if it happens, what would you like to see?

HPTF 2010 has not been confirmed yet. But should it happen for a fifth year, I sure hope to be able to make it again as I enjoy presenting to my peers very much. As abstracts must usually be submitted in January, I started thinking about what I would like to talk about in 2010.

I'll keep it to a technical presentation on what I know most and like the most, and that is - what a surprise - HP-UX.


The year 2008 was spent on increasing the availability and resilience of all mission critical systems under my responsibility. In 2009, my research and efforts have been increasingly towards manageability and security. The security aspect is totally not under my control, and I should rather talk about compliance rather than security. The two might be complementary but they're totally different. And I don't find that subject interesting.

So I think my 2010 paper will be in the manageability area. This being said, my current ideas for subjects are:

• Integrating HP-UX systems in a Nagios Core monitoring environment
• Easy and secure monitoring of HP-UX servers with SIM and Remote Support
• How I manage my HP-UX environment without getting paged

You're welcome to cast your vote on what you would like the most.

Are enterprise software details accessible to the average joe?

The post where I bashed an enterprise security software vendor because it wasn't possible to obtain technical information on their products without leaving personal information, and going through the sales channel, got me a lot of e-mails. Well, I wasn't exactly right. I discovered that other vendors in the SIEM industry follow similar standards and don't provide much information, except a feature list, without requiring visitors to register first. Even one product which is spun off from an open source project seems to do the same ! And no, I won't tell their names explicitly this time as I don't want this post to end up on Twitter and get blown out of proportion again. This blog is named Technocrat-UX, not Cranky-UX.

Having used lots of infrastructure security software over the years, where I never had any trouble getting an idea of what these product did exactly, all in a discipline where disclosure is paramount, I was surprised by the way SIEM products are presented. They're in their right to do it that way, but to me, a website is like a store, and if it makes me feel like I've just crossed the door of a very special car dealer instead of my corner Toyota dealership, my interest wanes quickly. Maybe it's just me. After all, I'm a Unix guy.

Perhaps companies that sell products based on business requirements, rather than technical requirements, have a modus operandi I'm not familiar with? Maybe they're, justifiably so, only targeting people with a business education instead of a scientific one? This is possible. So let's check. I've assembled a list of six "Enterprise" software products, and spent 15 minutes checking their websites to see if they have information relevant for a systems administrator. I've voluntarily excluded Open Source software since that wouldn't have been very fair. I also excluded HP software, as I've accumulated 10 years experience of searching through their web maze.

This is way, way, far from thorough. But here are my quick results.

Databases:

• Oracle 11g: Has lots of information freely available, and documentation is free to access.
• IBM DB2: Same as Oracle. Even better arranged than Oracle, with technical documentation easy to access.

Enterprise Content Management:

• Opentext Document Management: I need to register just to see a spec sheet. Yuck.
• EMC Documentum: I was curious about EMC, since they also make kickass hardware and own VMware, but for Documentum I also need to register to see info. DoubleYuck.

ITIL-related service request systems:

• CA Service Desk: I wasn't expecting much from CA but I was pleasantly surprised. They have lots of info, and access to manuals is free. I'll see CA differently from now on.
• BMC Remedy Service Desk: Information is passable, and manuals are not available.

O.

WEBES 5.6 just got released

WEBES 5.6 has just been released. There are no release notes on HP's web site but from what I've been able to gather, the two major changes are that it now uses PostgreSQL as its embedded database instead of relying on SQL Server, and it seems to replace OSEM outright.

I don't know if I will have time to try it out this week but I'll follow-up as soon as I can. For instance, I'll test if the old OSEM monitored devices I have will work out of the box (namely, Proliants and B-Series fibre switches). I'm also curious to see if the old SQL Server database will be deleted during the upgrade. I had some interface timeouts when checking HP-UX managed systems with 5.5 and I can't wait to see if they are resolved.

O.

Updating a server to 11iv3 while keeping it in SIM

Here is how to update a server to 11iv3 and keep everything working in SIM and RemoteSupport.

1. Update or re-install the server following your own procedure.
   
2. If necessary, configure all the requisites on the updated server so that it can be integrated correctly with SIM (there are too much to detail here, but this post can help)
3. Once the update is done, log into SIM and show the System Properties page of your server. Confirm that the two checkboxes "Prevent the discovery from changing these system properties" are unchecked.
4. Launch a discovery on your system. In 5.3, the process has changed: you need to create a discovery job and specify directly the target server in it.
5. Subscribe to WBEM events from your server from the Options->Events menu
   

If using RemoteSupport:

1. Redo an entitlement check to be sure that it your server is still entitled correctly.
   
2. This part is important, you need to restart WEBES (stop director, start director) or else I don't know if and when it will resubscribe to events. I waited 24 hours and it didn't subscribe, so screw it, I restarted it (I know that sucks, but I didn't find out how to force a resubscription besides restarting WEBES). Restarting the director results in WEBES subscribing to your server eventually, this might take a while depending on how many managed nodes you have.
   
3. Confirm with "evweb subscribe -b external -L" that there are SIM and WEBES subscriptions and run "sfmconfig -t -a" to test the delivery of events to SIM and the RemoteSupport back-end.

Good luck