I've been a long time member of the ITRC forums but I don't read the HP-UX forums all the time, since there is a shortage of good, interesting posts. What's the point of reading through countless questions that are clearly RTFM, or the "how to get certification????" type, or even worse, questions from members who've been there a while, asking lots of questions, but never assign any points to those who take some time to answer. These are users to which I usually do not care to reply.

Yet sometimes, I stumble upon threads that are interesting, and fueled up by comments of people who know what they're doing, and these are candy for me.

Want an example? Click here.

HPTF2009: Coming soon

I don't know how many sysadmins follow this blog, but I do have a few subscribers, so it must mean something.

I should attend the HPTF mid-June this year and as usual, I'll spend most of my time following up on HP-UX sessions. The difference this year, is that over time I've met some of the key people who are involved in the design of the OS and its utilities, and I intend to participate as much as I can in the customer advisory process. While I'll be the first to say that not everyone at HP is receptive to suggestions, some are, and these good fellas are involved with Connect and the ITRC forums. There will be some roundtable discussions at the "connections cafe", where I expect small groups of users to be matched with these guys. So it will be time to ask questions and comment on some issues related to their field. 

The ONE question I would like to ask, but cannot ask, is what's going on at HP with all the loss of morale that seems to transpire from all the chopping Mark Hurd has been doing at the expense of long-time, dedicated employees. The only person this could be asked to is Hurd himself (like he actually cares). I'm sure strict measures will be taken this year to prevent any deception from taking place at the HPTF should he actually appear -- Anyone noticed he's not mentioned as a keynote speaker anywhere for the event? I wouldn't be surprised if he did a George-Bush-surprise-visit-in-Irak kind of thing.

All the internal problems at HP are concerning me not only as a customer, but as someone who have personally known many HP employees for years, and the fact remains that while it seemed like a great place to work at 10 years ago, it's no longer the case. With demotivated employees, what new products can I expect to see in 5 years time? Maybe crap! 

New features for usermod and useradd in 11iv3

I discovered that HP-UX 11iv3 has a few new options bundled into useradd and usermod , options that I was anticipating for quite some time.

The first one is -p which lets you specify an encrypted password directly on the command-line, without having to modify /etc/shadow directly. That one will be very useful to me as I've scripted the creation of all our generic accounts, such as the WBEM account, and I have to lock and modify the passwd and shadow files before touching them... yuck.

Example:
# useradd -p 7Q4UTWb5U7dCA hpwbem

The second one is -F which, when used with usermod, lets you force changes even though the login is in use. For instance, one can finally use usermod with the root account, without having to modify the passwd or shadow files directly (which is dangerous with /etc/shadow since there is no vipw equivalent that lets you edit shadow files with proper locking).

# usermod -F -p 7Q4UTWb5U7dCA hpwbem

How a Ubuntu live CD can help testing braindead Windows drivers

I've been using my work laptop as my personal music player for years. SInce I'm in IT, it's with me everywhere I go, has tons of disk space, and I don't need to travel with an ipod. An ipod would be nice, but it needs to be charged, and I already have enough gadgets with a laptop, cell phone, and pager. I don't want something else.

Yesterday I received from my employer a brand new HP 6730b laptop. I started testing it, and it had a very serious issue: sound output was atrocious. And I'm not being picky here; it was just abyssmal. This has probably been unnoticed by our staff's QA process since built-in laptop speakers are rarely good (unless you have an alienware or ferarri), and not everyone uses their laptops to play music. The problem was apparent with earphones.

I tried another identical laptop model, and it sounded as bad.

I tried the same earphones on other laptops, and they sounded fine.

So the problem had to be the laptop. I tried searching the forums with "6730b headphones bad sound" (and all possble combinations) and didn't find anyone complaining with a similar issue. So what was the problem? Possibly a bad batch of DSPs? Maybe.

Of course, having to endure bad sound quality is not the kind of reason you can invoke to return your business laptop. The only choices were:

1. Suck it up, buttercup, and don't listen to music anymore.
2. Wait a few months until a new model is available from IT, and try real hard to break my laptop to get a new one.
3. Buy an external USB sound card (such as the Turtle Beach Audio Advantage Micro) and use it to bypass the lame built-in Soundmax HD Audio
4. (see below)

Neither of these choices were interesting. 

I tried choice #4: Boot up the laptop with a Ubuntu Live CD and check it out. And it sounded fine! Darn. So the culprit was Windoze after all.

I'll spare the end of the story. I could have brought it back to the IT staff, put in my favorite CD, and brag on how Ubuntu sounds better than Windows. But let's just say that fiddling with drivers (illegaly, I must say, since our PCs are locked out) fixed the issue. It now sounds fine. 

So what should you do if you want to investigate some weird hardware issue on your work laptop, for which you are limited in the scope of how far exactly you can go to fix it? Boot Ubuntu!

Building Subversion on HP-UX

We've been running Subversion on HP-UX 11.23 since 2006, and it has been upgraded maybe once a year since then. I moved from 1.3.x to 1.4.x, then to 1.5.4. However, due to a nasty bug with 1.5.4 which put many merges to a halt, I had to put into production an emergency upgrade last week within hours, and decided to stick at with the 1.5 tree since 1.6 is still rather new.

In the past, building SVN on HP-UX was total chaos. Autoconf has problems with ia64 platform, and I had to fiddle with Makefiles a lot. I'm not a developper and that wasn't easy. At least things are better now, and it builds cleanly. Yet even if it's easier than it used to be, I have a strong desire to eventually migrate to RHEL to have SVN running on a more natural platform.

For those who might wonder, I don't use the Porting Center version of SVN because I want to have it run under Apache rather than svnserve which doesn't have many features.

If anyone is interested in building SVN from scratch, here are my build instructions:

http://www.mayoxide.com/toolbox/install-subversion-server-1.5.6.txt

The total time to build it is around 2-3 hours.

What's going on with RSP 15 days before ISEE shuts down?

ITRC: There's an increasing number of posts in the RSP ITRC Forum with people who have problems, and some people apparently internal to HP sometimes give out good answers (Andrew is one of them, good job!). At least it shows that many have awaken up and started to actually migrate.  

RSP in a VM: There are many angry posts about not being able to run RSP in a VM, and that's fully understandable. Someone found out how to change configuration files to fool RSP into installing into a VM, and posted a very detailed workaround, but the post got removed within three days. I prefer using physical hardware personally for the CMS, we purchased a dedicated server just for that. But come on: HP's argument that "RSP requires too much I/O to work reliably in a VM" is bullshit. I don't think VMware would like hearing such nonsense. Like we're talking about a real-time application here. Of course, SIM might generate a lot of IP traffic in large environments, but I wouldn't be surprised that there were so much bugs to purge out with the first releases, HP simply didn't want to have to support VMs as well.

CMS: After a weird problem I've decided to have my CMS reboot once per week. Besides that SIM is still as 5.2 SP2 and I haven't upgraded to 5.3 yet. When migration is 100% complete, then I'll update.

My own HP-UX migration: I've finished migrating the HP-UX 11.23 servers under my responsibility, everything works well. Gone into production early this week. I dispatched this to the field, the other sysadmins will take care of the servers in their own data centers. I scripted a lot of the requisites, such as creating a WBEM-specific unpriviledged user, uninstalling ISEE, pre-configuring SMH, opening IP Filter ports... the list goes on. I also uninstall and reinstall SysFaultMgmt each time, this re-registers the WBEM providers and has given good results.

ISEE: I read that ISEE will stay active until Oct 31st but that's for CS customers only. Non-CS customers, such as me, must be ready for June 1st. Some might see a conspiracy theory here so that less-important customers will crush all the bugs first and if I was a CS customer I'd be perfectly okay with that. :)  Humor aside, mission-critical shops take a lot of time to turn their boat around, and it is normal under these circumstances that they get more time. Not that my systems are not considered mission-critical... but CS was very expensive, and we decided against it. 

Two years with Integrity VMs

I've been running Integrity VMs for two years now. I started with HPVM 3.0, moved up to 3.5, then 4.0.

Here are the essentials of what has happened in that timeframe:

• There has been more and more demand for HP-UX VMs from my users, as they can be installed very quickly.
• The old "host consolidation" way of thinking is now dead and buried; instead of consolidating, let's just boot off a new VM! Hardware gets consolidated, but not operating systems. This requires management tools and procedures... as well as a lot more IP addresses.
• A few production, but not mission-critical, systems have been installed in HPVMs since I migrated to 4.0
• Performance with 3.0 was subpar, but with 3.5 came AVIO and this helped a lot. AVIO rocks. The performance with it is excellent.
• 4.0 introduced the new storage stack, with native multipathing and built-in APA
  
• I initially used a combination of the VxFS backend (slow) and LVM backend (painful to manage) but switched mostly to raw devices for increased performance and to benefit from my SAN features such as cloning and snapshots.
• I've had a few VM guest crashes. HP Support is good at troubleshooting the dumps quickly. Be sure to have a /var/adm/crash ready, or at least free space in /var, to be able to have dumps.
• Each release of HPVM seems to be rushed, as there is often an HPVM CORE patch available almost the same day as when the new revision was released to the general public. You have to search for it in the ITRC when you install the VM Host.
• We've had a small RHEL deployment and I asked an intern to install Red Hat Linux in an HPVM to evalutate it, it works, but we had to use an outdated version and it was deemed too exotic as a platform so I didn't pursue this project and had it deployed on Proliants and VMware VMs instead.

Here is what I would like to see in the future:

• NPIV support. VMware has had this since last year. This will make VMs truly transparent to the SAN administrator, and prevent potential mistakes on the VM host.
• A better VM Manager. Frankly, the bottom line is that most SMH-based tools truly suck and VM Manager goes into the lot. I've had so much problems with it that I rarely use it and became proficient with the CLI.
• Better integration with GlancePlus. The only way to have statistical data on VMs is to create an "Application" and view it from GlancePlus or Performance Manager. It works, but you have to think of configuring the Performance Agent each time you create or delete a VM. It would be nice for this to be done automatically. hpvmsar is a start, but how about a simple "esxtop" clone.
• Clustering of VMs with Online Guest Migration has to be easy to do. If it uses ServiceGuard, that is fine, but it has to be EASY. Virtual Center makes clustering VMs a two-minute job. I'd expect the same with HPVM.
• How about an "HPVM cluster in a big box", with a fully configured blade chassis full of clustered bl860c's... all that with a lean HP-UX host distribution that is completely flashed on SSDs, similar to ESXi, and for which we don't have to do anything under the hood. That would be really cool.

That's it for now.

Update to WEBES 5.5 : what's new?

The new features of WEBES 5.5 are documented here and the most important points that I can mention are support for Command View 9.x and the EVA 8400.

It gets auto-updated automatically when using the RSP Software Manager.

I noticed that this new version takes a lot of more time to discover and poll your systems. This seems to have reduced the CPU usage and memory footprint of DESTA_Service considerably, which is a good thing. However, after WEBES is upgraded from 5.4 to 5.5, it has to rediscover all your devices. This took over 4 hours at my site. Any event that happens during that time frame might get queued until it has finished discovering your systems.