Cold-Updating small ServiceGuard clusters -- FAST!

Here's my guerilla procedure to cold-update small ServiceGuard clusters without doing an official rolling upgrade.

I'm currently migrating many small two-node ServiceGuard clusters which are scattered in different sites from SG 11.18 / HP-UX 11.23 to SG 11.19 / HP-UX 11.31. I decided to upgrade not only the OS, but the clustering software too for the simple reason that I didn't want to stick with 11.18 and have to update SG later down the road... With 11.19, I should be good for a few years.

The "rolling upgrade" procedure documented in the Admin Guide doesn't work in such a scenario as last time I checked, it only supports running an update-ux on the nodes one after another. I don't do update-ux, I prefer cold-reinstalling my systems with my heavily customized Golden Image. And since I wanted to take advantage of the downtime to move to 11.19, I fell in the "unsupported" arena.

Here's how I'm pulling it off with a procedure that takes a mere 60 seconds more downtime than a straight failover:

1. Update the failover node
1a) reconfigure the packages to be runnable only on the main node
1b) reconfigure the cluster to remove the failover node (you'll end up with a one node cluster)
1c) dump the golden image on the failover node
1d) install and configure the requirements for SG 11.19 on the failover node (it takes maybe 10 minutes if you've documented the process correctly, I know it for fact)
1e) set up a configuration file for a brand new one-node cluster on the failover node. If using lock disks, you can either use new lock disks and start it right away, or prepare config files which you're sure will work and start the cluster at step 2b.
1f) bring in the package configuration files and volume groups on the failover node, and configure these packages to be runnable only on the failover node. Run a cmcheckconf on them but do NOT run cmapplyconf yet because they're still used on the other cluster!

2. Move the packages to the failover node
2a) stop the packages on the cluster running on the main node
2b) remove the cluster bit on the VGs (vgchange -c) to prevent SG from identifying the disks as part of a cluster
2c) cmapplyconf the packages on the failover node (you might need to run vgchange -c again)
2d) start the packages
Total downtime: maybe a few minutes more than a standard failover but not much. With a well-prepared scenario with pastable commands, it takes me less than 60 seconds to do 2b and 2c.

3. Upgrade the main node
3a) dump the golden image on the main node
3b) install SG on the main node
3c) have that node join the cluster running on the failover node
3d) configure the packages to be runnable on both nodes

4. Bring back the packages to the main node
Simply move back the packages as you would in a normal cluster. Downtime will be the same as during a standard failover.

O.

The Software Conspiracy, downloadable edition

I was going over my bookshelves and stumbled on a book I read some 10 years ago, named "The Software Conspiracy". The title pretty much says it all. Back then, I had just received by B.Sc. and was starting a career as a Unix admin; some thought I was crazy of not following the same path as others of that era which became part of an army of MCSEs. The fact is, I've always been very critical of the software industry, and back in these days of Microsoft in particular. UNIX just plain worked.

I still disklike a lot of "enterprise" software titles. However, I think software is actually better, for two reasons. First, it updates more easily, and unattendingly. Secondly, the move towards appliances, be it a smartphone or a DVD player, increased the quality of software overall as when an appliance doesn't work, you send it back.

I just found out the book was released for free in .pdf in 2005. It's a fascinating read.

O.

HP-UX's future is in question. Does it lie in Linux? Or not?

Juanma's blog made a follow-up to my december post where I expressed concerns on HP-UX and Integrity's future.

I've thought more about it over weeks and here is what I think of it.

First we have Brian Cox's (in)famous blog post where he openly states that Linux should be considered in the long run by enterprise customers. I'm not against what he said, au contraire. He had the courage to confirm what many were thinking. But what does this mean? Is HP abandonning HP-UX? Not necessarily, but I wouldn't be surprised if this announcement means between the lines that HP has plans for Linux in the future.

We first have to look at the market HP is pitching HP-UX to: enterprise customers. One could argue that as long as HP-UX keeps a resonable TCO and is well regarded by all the Gartners of this world, it should continue faring well in the enterprise. But the fact remains that even if the average enterprise architecture team usually chooses a platform based on who wins an RFP or what industry analysts have to say, the pressure to move to Linux will just intensify over time.

Gartner aside, HP-UX does have great qualities that are fit for the enterprise environment. For one thing, it doesn't change much, which can actually make it cheaper to run than operating systems that are constantly being revised. And for customers who are looking for a vertically integrated infrastructure where almost everything comes from one an only vendor, HP-UX, and AIX, for that matter, are a no brainer.

Yet, developing, supporting, and maintaining your own operating system is a costly venture which, unless you're a strong leader, doesn't have a good payback. What pays back is the support and services you'll sell.

The bread and butter for HP doesn't lie in HP-UX itself; it lies in their enterprise vertical platform which, in my opinion, consists not only of the hardware and the operating system, but all the services you can offer under the same umbrella. Could an HP-supported Linux be the answer to the slow decline of HP-UX? I sure think so. HP would bring to Linux a very strong support and services infrastructure, that would please enterprise customers. They also already have a bunch of manageability tools that are invaluable to the enterprise, many already available on Linux.

Furthermore, HP's all about converged infrastructure now. That's what they've been nailing their customers about and believe me, they want them to get the message. We all know that Oracle is becoming a menacing converged player: they have the software, the hardware, and the operating system. HP will probably not allow itself to become runner-up.

One has to wonder, though, what kind of deal they struck with Microsoft with their Frontline partnership. I'm sure Ballmer asked Hurd not to venture too much into the Linux business when signing the deal. This is speculation, of course, but a possibility no less. And if it is true, then we won't see HP release their own Linux for quite some time, and HP-UX will stay there as long as it can bring them money.

O.

Integrity iLO2 Advanced Pack trial license

HP offers a free 30 day trial of iLO2 Advanced Pack here:
http://h71028.www7.hp.com/enterprise/cache/279991-0-0-225-121.html

Scroll down to "Multi-OS" and you'll see licence keys that can be installed within seconds to enable the Advanced Pack features. The most useful one, in my opinion, is Virtual Media which lets you mount a remote .iso image and present it to the server as if it was a real DVD.

These keys could be extremely valuable if you ever need Virtual Media in a hurry on a remote server which doesn't have a licensed advanced pack. I can think of many situations where this could happen. I won't copy the keys here, so I suggest you save them in a text file and hold on to them in case the web page disappears.

Olivier.

ServiceGuard Manager once again has a map view!

Remember the days when ServiceGuard Manager's java interface was so cool it was prominently shown in HP's video where they blow up servers? We actually used this high-level GUI to demo ServiceGuard to upper Management, as it was the most effective way of explaining to non-technical people what a ServiceGuard cluster consisted of.

You can imagine my deception when, a few years ago, HP decided to ditch that client-based GUI in favor of a nasty, ugly web-based implementation where you lost all the ease-of-use that was a hallmark of ServiceGuard Manager.

Well, I just found out that the map view has finally been restored in the web version. ServiceGuard 11.19 patch PHSS_40152 was released a few months ago, and it introduces release 2.00.10 of ServiceGuard Manager which, finally, restores the much-needed map view. You get something like this:


This is much better and more intuitive than the older tabular view, and furthermore you're once again able to drag packages from one node to the other.

Usually, I would praise HP for doing something like this. But considering that they actually removed the map view and took years to bring it back, the only thing I can say is that it was about time.

O.

So, what is HP SIM good for?

Over time, HP Systems Insight Manager has grown to become an important part of our infrastructure. The path to success has not been easy; many man-hours have been invested by both me and our previous CMS administrator into making everything work reliably. The best advice I can give you is to install it from scratch on dedicated hardware.

So what do we use SIM for? Mainly for three things:

• System inventory
• Contract and warranty management
• Monitoring of hardware failures

We use SIM to monitor and manage a few hundred devices scattered in 7 data centers. It is used with almost every piece of HP hardware we have, from Proliant to Integrity, from MSAs to EVAs. Integrating SIM with Remote Support has been bumpy, to say the least, but once I made everything work, it has become a well-appreciated service. Support personnel like the fact that service calls are opened automatically at HP.

SIM and Remote Support helped us slowly move to a Preventive Maintenance way of thinking, instead of a Corrective Maintenance approach. We're now able to monitor the health of just about any piece of hardware, and have power supplies and disks replaced within a few hours when problems happen. I can't quantify how much our risk of downtime or data loss has decreased by using Remote Support, but it is certainly an appreciable amount.

On the downside, SIM has many other functions that we don't need, but they come bundled in anyway and cannot be hidden. Its interface is not very intuitive, with drop-down menus that tend to integrate many different concepts. The SIM UI has many performance and design problems which makes it a hard to learn and understand. Maybe 6.0 will be better.

It seems that HP is pushing SIM as the interface to to just about anything to manage servers -- from what I've seen in the recent years, they talk about it much more than OpenView and boast it as a component of the Converged Infrastructure ploy. However, Insight is not a brand I've grown to trust over time. SIM does the job, but it is clunky, slow, and I find its documentation to be rebarbative.

O.

New EVAs are coming in 2010

I didn't sign an NDA but since some of the information I've learned come from people who probably did, I cannot disclose precisely what's coming. But I can cite public sources that I found on the internet.

The Register found this out last fall and wrote a good description of what's coming.

A rumor I've seen on the internet is that Dot Hill might have something to do with the underlying technology. Which might no be so far-fetched after all. But I will believe it when I see it - something tells me this is a LeftHand design.

O.

Is a crossroad coming? You tell me.

Last new year's eve, my folks came over for dinner with my family. There were plenty for everyone but I unfortunately ran out of champagne glasses. Being the HP-UX groupie I am, I simply took out a promo HP-UX 25th glass I had in the cupboard and used it for the first time.

It sure looks nice, doesn't it?

But as we are starting 2010, it is also time for me to think about what HP-UX's future will become, and thus what my future career will consist of.

I'm a blogger; I'm no industry analyst. I might be making wild guesses, but the post where I raised concerns on HP's long time commitment toward HP-UX and the Integrity platform sure didn't close 2009 well for me. I've invested lots of time over the years on the HP 9000, then on Integrity, and all this started paying off only after years of dedication.

I don't have any sources inside Hewlett-Packard but on the surface, as a average customer, I've seen many things which have changed within the company in the last 5 years. But the debate must remain strictly technical: Is HP-UX still able to to the job, at a resonable TCO? The answer for me is still yes. For how long? Probably a while, but I'm still nervous anyway.

Over the years, HP-UX has never let me down, both as an enterprise-class operating system and as a career path. But nevertheless, there are times when we reach a crossroad, and in these times we must either continue on the same path, hoping for another crossroad further down the road, or simply switch our direction right away.

I think I'll be standing in a crossroad soon.

O.