Building Subversion on HP-UX

We've been running Subversion on HP-UX 11.23 since 2006, and it has been upgraded maybe once a year since then. I moved from 1.3.x to 1.4.x, then to 1.5.4. However, due to a nasty bug with 1.5.4 which put many merges to a halt, I had to put into production an emergency upgrade last week within hours, and decided to stick at with the 1.5 tree since 1.6 is still rather new.

In the past, building SVN on HP-UX was total chaos. Autoconf has problems with ia64 platform, and I had to fiddle with Makefiles a lot. I'm not a developper and that wasn't easy. At least things are better now, and it builds cleanly. Yet even if it's easier than it used to be, I have a strong desire to eventually migrate to RHEL to have SVN running on a more natural platform.

For those who might wonder, I don't use the Porting Center version of SVN because I want to have it run under Apache rather than svnserve which doesn't have many features.

If anyone is interested in building SVN from scratch, here are my build instructions:

http://www.mayoxide.com/toolbox/install-subversion-server-1.5.6.txt

The total time to build it is around 2-3 hours.

What's going on with RSP 15 days before ISEE shuts down?

ITRC: There's an increasing number of posts in the RSP ITRC Forum with people who have problems, and some people apparently internal to HP sometimes give out good answers (Andrew is one of them, good job!). At least it shows that many have awaken up and started to actually migrate.  

RSP in a VM: There are many angry posts about not being able to run RSP in a VM, and that's fully understandable. Someone found out how to change configuration files to fool RSP into installing into a VM, and posted a very detailed workaround, but the post got removed within three days. I prefer using physical hardware personally for the CMS, we purchased a dedicated server just for that. But come on: HP's argument that "RSP requires too much I/O to work reliably in a VM" is bullshit. I don't think VMware would like hearing such nonsense. Like we're talking about a real-time application here. Of course, SIM might generate a lot of IP traffic in large environments, but I wouldn't be surprised that there were so much bugs to purge out with the first releases, HP simply didn't want to have to support VMs as well.

CMS: After a weird problem I've decided to have my CMS reboot once per week. Besides that SIM is still as 5.2 SP2 and I haven't upgraded to 5.3 yet. When migration is 100% complete, then I'll update.

My own HP-UX migration: I've finished migrating the HP-UX 11.23 servers under my responsibility, everything works well. Gone into production early this week. I dispatched this to the field, the other sysadmins will take care of the servers in their own data centers. I scripted a lot of the requisites, such as creating a WBEM-specific unpriviledged user, uninstalling ISEE, pre-configuring SMH, opening IP Filter ports... the list goes on. I also uninstall and reinstall SysFaultMgmt each time, this re-registers the WBEM providers and has given good results.

ISEE: I read that ISEE will stay active until Oct 31st but that's for CS customers only. Non-CS customers, such as me, must be ready for June 1st. Some might see a conspiracy theory here so that less-important customers will crush all the bugs first and if I was a CS customer I'd be perfectly okay with that. :)  Humor aside, mission-critical shops take a lot of time to turn their boat around, and it is normal under these circumstances that they get more time. Not that my systems are not considered mission-critical... but CS was very expensive, and we decided against it. 

Two years with Integrity VMs

I've been running Integrity VMs for two years now. I started with HPVM 3.0, moved up to 3.5, then 4.0.

Here are the essentials of what has happened in that timeframe:

• There has been more and more demand for HP-UX VMs from my users, as they can be installed very quickly.
• The old "host consolidation" way of thinking is now dead and buried; instead of consolidating, let's just boot off a new VM! Hardware gets consolidated, but not operating systems. This requires management tools and procedures... as well as a lot more IP addresses.
• A few production, but not mission-critical, systems have been installed in HPVMs since I migrated to 4.0
• Performance with 3.0 was subpar, but with 3.5 came AVIO and this helped a lot. AVIO rocks. The performance with it is excellent.
• 4.0 introduced the new storage stack, with native multipathing and built-in APA
  
• I initially used a combination of the VxFS backend (slow) and LVM backend (painful to manage) but switched mostly to raw devices for increased performance and to benefit from my SAN features such as cloning and snapshots.
• I've had a few VM guest crashes. HP Support is good at troubleshooting the dumps quickly. Be sure to have a /var/adm/crash ready, or at least free space in /var, to be able to have dumps.
• Each release of HPVM seems to be rushed, as there is often an HPVM CORE patch available almost the same day as when the new revision was released to the general public. You have to search for it in the ITRC when you install the VM Host.
• We've had a small RHEL deployment and I asked an intern to install Red Hat Linux in an HPVM to evalutate it, it works, but we had to use an outdated version and it was deemed too exotic as a platform so I didn't pursue this project and had it deployed on Proliants and VMware VMs instead.

Here is what I would like to see in the future:

• NPIV support. VMware has had this since last year. This will make VMs truly transparent to the SAN administrator, and prevent potential mistakes on the VM host.
• A better VM Manager. Frankly, the bottom line is that most SMH-based tools truly suck and VM Manager goes into the lot. I've had so much problems with it that I rarely use it and became proficient with the CLI.
• Better integration with GlancePlus. The only way to have statistical data on VMs is to create an "Application" and view it from GlancePlus or Performance Manager. It works, but you have to think of configuring the Performance Agent each time you create or delete a VM. It would be nice for this to be done automatically. hpvmsar is a start, but how about a simple "esxtop" clone.
• Clustering of VMs with Online Guest Migration has to be easy to do. If it uses ServiceGuard, that is fine, but it has to be EASY. Virtual Center makes clustering VMs a two-minute job. I'd expect the same with HPVM.
• How about an "HPVM cluster in a big box", with a fully configured blade chassis full of clustered bl860c's... all that with a lean HP-UX host distribution that is completely flashed on SSDs, similar to ESXi, and for which we don't have to do anything under the hood. That would be really cool.

That's it for now.

Update to WEBES 5.5 : what's new?

The new features of WEBES 5.5 are documented here and the most important points that I can mention are support for Command View 9.x and the EVA 8400.

It gets auto-updated automatically when using the RSP Software Manager.

I noticed that this new version takes a lot of more time to discover and poll your systems. This seems to have reduced the CPU usage and memory footprint of DESTA_Service considerably, which is a good thing. However, after WEBES is upgraded from 5.4 to 5.5, it has to rediscover all your devices. This took over 4 hours at my site. Any event that happens during that time frame might get queued until it has finished discovering your systems.

Two months using RSP with EVAs and going strong

This blog post will be a recap of my painful migration to RSP, which is now almost finished for the systems I care for which are EVAs, Integrity Servers running HP-UX, and Proliant servers running ESX. This post will talk about EVAs. The other systems will be covered later.

When it came time to migrate from ISEE to RSP, I was very fearful of doing the migration for my EVAs. Why? Because server components rarely fail nowadays. This leaves us mostly with mechanical disks, power supplies and batteries. Since disk arrays such as EVAs have a bunch of them, they are the most prone to failure and require the most maintenance.

I started testing RSP as early as last October, with disastrous results. There's a price to pay for being bleeding edge. It took quite a while for me to understand all the RSP components, but now the puzzle is fairly complete. And now that many Jack Bauer-style customers like me have started migrating without the help of HP, I'm seeing an increasing number of disguised complaints in the ITRC forums of disgruntled users, even from HP employees.

I was so pissed with all the bugs I've had with my initial test run of RSP, I decided to go back to square one and put myself in the shoes of HP's QA people, thinking about how they must actually test their software; and I said to myself: "chances are they start with freshly installed environments"... thus I did the same. I reinstalled the CMS from scratch, even went as far as zapping my Storage Management Servers, and took great care to RTFM the boring prerequisites guide which is so clinical that it would make reading medical transcriptions a funny adventure. Every small dependency, from ELMC to MC3 components, to manually configuring WEBES to use the "CommandView" protocol have been taken care of.

And it works!

Up until now, I've had three events, on two different EVAs, and they were all forwarded to WEBES, then the ISEE client, then HP. HP then called back the contact person to schedule what needs to be done. What's fun now is that since everything is centralized rather than dispersed on multiple SMSes. Configuring the contacts for the EVAs is easier. We're spread out in multiple data centers across the province, and this proves much easier to manage.

Using a centralized CMS, however, has its drawbacks. For instance, what will happen if there's an event and the CMS is down? Will it be queued until it gets back up? I haven't tested this yet. Furthermore, should we invest in a highly-available CMS? That's quite a sum of money. And what will happen when my CMS or SMS are out of contract? Guess what, I wouldn't be surprised if it doesn't work anymore.

So many questions, so little time...

Building a lean'n'mean emacs for HP-UX

The port of emacs available in the Porting and Archiving Center for HP-UX is a helluva big build, with all options. It requires a total of 22 libraries, that's nonsense if you're only interested in running the textmode version of emacs. Here's how to build one for HP-UX.

You need the following to build it:

• The aCC compiler - I have not been able to compile it with gcc (both using the stock source or the one modified by the Porting Center). If you don't have it, there is an evaluation versionavailable on the HP Software Depot. Sorry, normally I prefer using gcc, too.
• GNU Make
• Libncurses (also required on the servers where you intend to run emacs. I haven't taken my chances with the stock curses library).

Here is the procedure to follow:

The porting center already did the grunt work of modifying the code to have it compile nicely under HP-UX -- thanks Richard. That is impossible if you take the stock emacs source. Grab that ported source directly from the porting center:

# wget http://hpux.connect.org.uk/ftp/hpux/Editors/emacs-22.3/emacs-22.3-hppa-11.11.depot.gz

Sidenote: if you don't have wget, now would be a good time to install it, it's very useful for doing this kind of stuff if your servers have an outbound connection to the internet.

Extract the tar file and run configure like this:
# ./configure --prefix=/opt/emacs --without-x

N.B. You don't need to add --prefix if you prefer having it under /usr/local. The report at the end of configure should look like this:

Should Emacs use the GNU version of malloc? yes
Should Emacs use a relocating allocator for buffers? yes
Should Emacs use mmap(2) for buffer allocation? no
What window system should Emacs use? none
What toolkit should Emacs use? none
Where do we find X Windows header files? NONE
Where do we find X Windows libraries? NONE
Does Emacs use -lXaw3d? no
Does Emacs use -lXpm? no
Does Emacs use -ljpeg? no
Does Emacs use -ltiff? no
Does Emacs use -lungif? no
Does Emacs use -lpng? no
Does Emacs use X toolkit scroll bars? no

Then run gmake to build emacs:

# gmake

Finally, install emacs:

# gmake install

And you're done.

The Three Way Carter System or: how I Learned to Stop Worrying And Love Rubber Gloves

Disclaimer: I don't work in the trade, and have no field experience. This blog post exists only for the purpose of informing what a Carter System is, and how to protect yourself if you have one -- any electrical work requires a qualified electrician.

The previous diagram shows a wacky (and dangerous) way to bring power from one end to another using three way switches with a light in between. It uses the travelers to carry one neutral and one hot. With this, the two commons can have 4 states:

1. Hot - Neutral (lamp turns on)
2. Neutral - Neutral (lamp turns off)
3. Neutral - Hot (lamp turns on, with reversed polarity)
4. Hot - Hot (lamp turns off, and the socket remains hot)

States #3 and #4 are dangerous, as anyone changing a lightbulb who happens to touch the socket gets an electric shock. An electrician can also have problems changing the light fixture thinking it is off, while it's not necessarily the case. There is no way to know. The only thing that can be of any indication is if the tradesman that initially installed this used both red wires to connect to the light as illustrated above.

I did some research and found out that this is called a "Carter System" or "Chicago Three-Way". I have no clue how local electricians call this over here, but I'm sure they have a name for this. The examples I've seen on the net show that it can be used to power a garage or shed with 3-conductor cable, with an outside light toggled at both ends. Oh yes, and I can't see any relation to President Carter since by the time he was in office, the NEC had outlawed the practice for over 50 years.

In my case, I've seen this inside my circa-1950 house and thus why I blog this here. There are many houses like mine in Montreal, with a long passageway using three-way switches, and I suspect that a cable routing like this must have been installed in more than one place around here.

Can it be fixed? Not without rewiring. You could remove one of the switches, replace the other one with a standard two-way switch, then bring the neutral to the light fixture from the other side of the circuit. But then you'll end up with a light that can be turned on only from one far end of the house, which is probably against some building code as well. Assuming this Carter System passed inspection and was allowed at the time of the installation, it can probably stay there as long as you don't modify the circuit. But if you remodel, it would be a good thing to have it fixed.

To my knowledge, a Carter System doesn't seem to pose any fire hazard, but something as trivial as changing a light bulb can result in an electric shock. And if you're really, really unlucky, the shock can be fatal... As an example, just imagine someone unlucky enough to have a weak heart, unlucky enough to be bare feet on an aluminium step ladder, unlucky enough to have that ladder grounded by one way or another, and unlucky enough to have the switches set to state #4 above. That's a really stupid death that could have been prevented, and a case for primetime news.

Lesson learned: If you live in an old house, wear rubber gloves when changing bulbs, or turn off your breakers first.