Hacking Ignite-UX's expert recovery mode to scrub disks offline

Introduction

SAN arrays can be scrubbed independently (remember dilx?) but internal disks are more complicated to do. Many suggestions I've seen in the ITRC forums consist of logging into HP-UX, then wipe the disks using "dd /dev/zero" or similar tools. From my experience, this is risky. I used to do this over 10 years ago when decomissioning workstations, and the operating system eventually stopped working while the disks were scrubbing, leaving no proof that the they had indeed been wiped completely.

With a Proliant, no problem. Just boot up a Linux live CD such as System Rescue CD and it will come with a scrubber. Case closed. But with an Integrity server, it's more complicated as there aren't Linux live CDs available. Maybe with some elbow grease I could make one with CentOS but I don't know Linux enough to take on the challenge.


The Ignite-UX expert recovery mode

Ignite-UX comes with a rarely-used mode named "Expert Recovery" which puts on a RAM disk many tools you need to recover an unbootable system. Instead, we'll use the expert recovery shell to actually wipe out disks! While we could compile a statically-linked open source scrubbing tool such as diskscrub, to save time we'll bring in mediainit(1m) which, since the March 2009 release of 11.31, has a new scrubbing option. Since the man page does not describe precisely the algorithm used, I checked what mediainit actually writes, and from my findings, it follows the DoD 5520.22-M standard which consists of writing one character, the complement, then a random character. It should be enough for most people... but for classified stuff, nothing beats a drill press.


Steps

a/ Start by putting a working mediainit under /var/opt/ignite/scrub on your Ignite server. It has to come from a March 2009 11.31 release, as it's the earliest to support scrubbing. You'll also need to add the libpthread library since mediainit has a dependency on this library, and it's not included in the expert recovery environment.

Example:
ignite-server# mkdir /var/opt/ignite/scrub
ignite-server# cp /usr/bin/mediainit /var/opt/ignite/scrub
ignite-server# cp /usr/lib/hpux32/libpthread.so.1 /var/opt/ignite/scrub

b/ Ignite the server you want to wipe out. Igniting is beyond the scope of this howto, I personally use dbprofile and lanboot at the EFI to do this. If you are given choices between igniting 11.23 or 11.31, choose 11.31. Of course, if you're on a K class or some other older hardware that doesn't support 11.31, you're out of luck. Stop here, and try to compile diskscrub to add it to your 11.23 or 11.11 Ignite server.

Obtaining size of AUTO (226 bytes)
Downloading file AUTO (226 bytes)
1. target OS is B.11.23 IA
2. target OS is B.11.31 IA
3. Exit Boot Loader

Choose an operating system to install that your hardware supports: 2

c/ When you get to the Welcome to Ignite-UX screen, choose Run an Expert Recovery Shell. Configure your network and click OK. A RAM disk will be created, and some useful commands will be pulled from the Ignite Server. You'll be presented with a menu, where you must choose X - exit to shell.

            HP-UX NETWORK SYSTEM RECOVERY
                       MAIN MENU


   s.  Search for a file
   b.  Reboot
   l.  Load a file
   r.  Recover an unbootable HP-UX system
   x.  Exit to shell

This menu is for listing and loading the tools contained on the core media.
Once a tool is loaded, it may be run from the shell. Some tools require other
files to be present in order to successfully execute.

Select one of the above: x

Type 'menu' to return to the menu environment. Do not use 'exit'.

#

d/ mediainit and libpthread are missing from the environment, so they must be pulled from the Ignite server. To do this, we'll use tftp which is the protocol used to download software from the Ignite server.

First get mediainit and put it in /usr/bin:

# cd /usr/bin
# tftp ignite_server_ip_address
tftp> get /var/opt/ignite/scrub/mediainit
Received 88405 bytes in 0.0 seconds
tftp> quit
# chmod 755 /usr/bin/mediainit

Then get libpthread.so.1 and put it in /usr/lib/hpux32:

# cd /usr/lib/hpux32
# tftp ignite_server_ip_address
tftp> get /var/opt/ignite/scrub/libpthread.so.1
Received 1521497 bytes in 0.6 seconds
tftp> quit

e/ You're done! You now have downloaded a workable scrubber in the Expert Recovery Shell.

# /usr/bin/mediainit
usage: mediainit [-vrn] [-f fmt_optn] [-i interleave] [-p partition_size] pathname
usage: mediainit -S [-t scrub_count] [-c scrub_character] special_file

f/ The last step is identifying your disk devices under /dev/rdsk, and wipe them using the -S option:

# /usr/bin/mediainit -S /dev/rdsk/c0t0d0
WARNING: You have invoked the disk scrub option.
Using this option will completely destroy the data
on the specified disk. All the signals except SIGINT(ctrl-c)
will be disabled during disk scrub.
Are you SURE you want to proceed? (y/n) y

Disk scrub:PASS 1

Disk scrub:PASS 2

Disk scrub:PASS 3
...
mediainit: Disk scrubbing successful
#

With these default options, mediainit will write these hex characters in order to follow the DoD spec: First '0x30', then '0x66' , finally '0xc6'. 66 is the complement of 30, while c6 is the "random" character which is actually hard coded in mediainit. It's actually more interesting to use -S alone rather than with the -c (scrub character) and -t (number of times to scrub) options since these two options do not alternate between different characters and you must reinvoke mediainit manually to change them.

Good luck

Migrating LIVE from one datastore to another in an Integrity VM

Did you know that it is possible to easily migrate data using LVM's pvmove in Integrity Virtual Machines running HP-UX ? I tried it today and not only does pvmove works well, it's almost as fast as if I was on physical hardware. For those who are familiar with ESX's Storage VMotion, this is as close as you can get to achieving similar results. Using pvmove is not as slick as an svmotion since you need to do it directly on the guest. But it works, and that's good enough for me.

Why would you want to do this? Here are some examples:

• Migrating devices from scsi to avio_stor, without rebooting the VM (assuming it has the avio drivers, of course)
• Switching from a datastore to another. For example, moving from a flat file to a raw disk
• Moving data from a RAID-5 volume to RAID-1
• Migrating from one disk array to another

As an example, here is a sample procedure to move data from an LV datastore to a raw disk datastore, without having to copy data manually or turn off your VM.

Add your new raw device to your VM:

vmhost# hpvmmodify -P myvm -a disk:avio_stor::disk:/dev/rdisk/disk39
vmhost# hpvmstatus -P myvm

[Storage Interface Details]
Guest                                 Physical
Device  Adaptor    Bus Dev Ftn Tgt Lun Storage   Device
======= ========== === === === === === ========= =========================
disk    avio_stor    0   1   0   0   0 file      /ivm/myvm/disk1.vm
disk    avio_stor    0   1   0   3   0 lv        /dev/vg_myvm/rlv_vgdata
disk    avio_stor    0   1   0   4   0 disk      /dev/rdisk/disk39

In the VM itself, make an ioscan to discover the new device

myvm# ioscan
myvm# insf -eC disk   # insf required only on 11iv2
myvm# ioscan -kfnC disk

Class     I  H/W Path     Driver  S/W State   H/W Type     Description
=======================================================================
disk      0  0/0/1/0.0.0  sdisk   CLAIMED     DEVICE       HP      Virtual FileDisk
                    /dev/dsk/c0t0d0     /dev/dsk/c0t0d0s2   /dev/rdsk/c0t0d0    /dev/rdsk/c0t0d0s2
                    /dev/dsk/c0t0d0s1   /dev/dsk/c0t0d0s3   /dev/rdsk/c0t0d0s1  /dev/rdsk/c0t0d0s3
disk      3  0/0/1/0.3.0  sdisk   CLAIMED     DEVICE       HP      Virtual LvDisk
                    /dev/dsk/c0t3d0   /dev/rdsk/c0t3d0
disk      4  0/0/1/0.4.0  sdisk   CLAIMED     DEVICE       HP      Virtual Disk
                    /dev/dsk/c0t4d0   /dev/rdsk/c0t4d0

See that new "Virtual Disk" device? Just pvcreate it and add it to your the VG on which you need to migrate data. Here I'm using legacy devices since my VM runs 11iv2, but you can do this with agile devices if you wish.

myvm# pvcreate /dev/rdsk/c0t4d0
myvm# vgextend /dev/vgdata /dev/dsk/c0t4d0
myvm# vgdisplay vgdata

--- Physical volumes ---
PV Name                     /dev/dsk/c0t3d0
PV Status                   available
Total PE                    1249
Free PE                     0
Autoswitch                  On
Proactive Polling           On

PV Name                     /dev/dsk/c0t4d0
PV Status                   available
Total PE                    1279
Free PE                     1279
Autoswitch                  On
Proactive Polling           On

Then, use pvmove to move your physical extents from one datastore to another using pvmove, as you would on a physical server.

myvm# pvmove /dev/dsk/c0t3d0 /dev/dsk/c0t4d0
Transferring logical extents of logical volume "/dev/vgdata/lv_mydata"...

Here, LVM will be moving LIVE all your data from c0t3d0 (an LVM datastore) to c0t4do (a raw disk datastore). The process can take a while, since it's going slow voluntarily to prevent the process from interfering with normal operations.

When it's over, remove the empty PV from the VG

myvm# vgreduce vgdata /dev/dsk/c0t3d0
myvm# pvremove /dev/rdsk/c0t3d0
myvm# rmsf -H 0/0/1/0.3.0

Then, remove it from the VM host:

vmhost# hpvmmodify -P my_vm -d disk:avio_stor::lv:/dev/vg99/rlv_vgdata

And you're done. You've just migrated storage live in your VM.

Can the pvmove be done at the VM Host level?

Here we did the pvmove at the VM Guest level, i.e. inside the VM Guest. Can this be done on the VM Host itself? The answer is yes, but only if you're using LVM datastores. Just pvmove the PVs on the VM Host, and you can migrate data transparently.

I did useLVM datastores for a while, but they're a pain in the butt to manage, as you have two levels of volume groups to consider: one on the host, then one on the guest. There is also a lot of flexibility that is lost: you can't easily export and reimport the guest volume groups from virtual to physical servers, back and forth, when using anything except raw disks.

Yet there is one place where LVM datastores could be useful: to enable a Storage VMotion look-alike for VM guests that do not have any volume manager that can easily do this. I'm thinking here of VMs running Windows, for example.

Bye

R.I.P. ISEE

Today, ISEE closes for non-CS customers. I don't think I'll blog much about RSP much from now on, since I've completed my migration and it now works. 

I'm scheduled to meet the Product Manager for RSP in a few weeks at the HPTF, and I'll give him some of my comments as there are still a few things in it that I would like to see to make RSP better.

I posted most of my experiences in the last few months, but here is recap.  A 100% successful RSP installation boils down to this:

• If possible, start from scratch with a fresh CMS. SIM is easy to backup/restore so you won't loose any data. 
• Don't be tempted to install anything else on the CMS.
• Use RSP 5.20, as 5.10 has a dumbed-down Software Manager that doesn't tell you what it's doing.
• If you have EVAs, reinstall the SMS from scratch too, with SmartStart or the Proliant Support Pack, and ensure that it has only the required software to manage the EVA and link it to RSP
• Proliants running Windows and ESX are easy to configure, just use the SMH to send traps the the CMS.
• As for HP-UX... well, while the CIM-related tools are not terrific, SFM itself is worse. I've had lots of problems with it. You've been warned.

Good luck

I've been a long time member of the ITRC forums but I don't read the HP-UX forums all the time, since there is a shortage of good, interesting posts. What's the point of reading through countless questions that are clearly RTFM, or the "how to get certification????" type, or even worse, questions from members who've been there a while, asking lots of questions, but never assign any points to those who take some time to answer. These are users to which I usually do not care to reply.

Yet sometimes, I stumble upon threads that are interesting, and fueled up by comments of people who know what they're doing, and these are candy for me.

Want an example? Click here.

HPTF2009: Coming soon

I don't know how many sysadmins follow this blog, but I do have a few subscribers, so it must mean something.

I should attend the HPTF mid-June this year and as usual, I'll spend most of my time following up on HP-UX sessions. The difference this year, is that over time I've met some of the key people who are involved in the design of the OS and its utilities, and I intend to participate as much as I can in the customer advisory process. While I'll be the first to say that not everyone at HP is receptive to suggestions, some are, and these good fellas are involved with Connect and the ITRC forums. There will be some roundtable discussions at the "connections cafe", where I expect small groups of users to be matched with these guys. So it will be time to ask questions and comment on some issues related to their field. 

The ONE question I would like to ask, but cannot ask, is what's going on at HP with all the loss of morale that seems to transpire from all the chopping Mark Hurd has been doing at the expense of long-time, dedicated employees. The only person this could be asked to is Hurd himself (like he actually cares). I'm sure strict measures will be taken this year to prevent any deception from taking place at the HPTF should he actually appear -- Anyone noticed he's not mentioned as a keynote speaker anywhere for the event? I wouldn't be surprised if he did a George-Bush-surprise-visit-in-Irak kind of thing.

All the internal problems at HP are concerning me not only as a customer, but as someone who have personally known many HP employees for years, and the fact remains that while it seemed like a great place to work at 10 years ago, it's no longer the case. With demotivated employees, what new products can I expect to see in 5 years time? Maybe crap! 

New features for usermod and useradd in 11iv3

I discovered that HP-UX 11iv3 has a few new options bundled into useradd and usermod , options that I was anticipating for quite some time.

The first one is -p which lets you specify an encrypted password directly on the command-line, without having to modify /etc/shadow directly. That one will be very useful to me as I've scripted the creation of all our generic accounts, such as the WBEM account, and I have to lock and modify the passwd and shadow files before touching them... yuck.

Example:
# useradd -p 7Q4UTWb5U7dCA hpwbem

The second one is -F which, when used with usermod, lets you force changes even though the login is in use. For instance, one can finally use usermod with the root account, without having to modify the passwd or shadow files directly (which is dangerous with /etc/shadow since there is no vipw equivalent that lets you edit shadow files with proper locking).

# usermod -F -p 7Q4UTWb5U7dCA hpwbem

How a Ubuntu live CD can help testing braindead Windows drivers

I've been using my work laptop as my personal music player for years. SInce I'm in IT, it's with me everywhere I go, has tons of disk space, and I don't need to travel with an ipod. An ipod would be nice, but it needs to be charged, and I already have enough gadgets with a laptop, cell phone, and pager. I don't want something else.

Yesterday I received from my employer a brand new HP 6730b laptop. I started testing it, and it had a very serious issue: sound output was atrocious. And I'm not being picky here; it was just abyssmal. This has probably been unnoticed by our staff's QA process since built-in laptop speakers are rarely good (unless you have an alienware or ferarri), and not everyone uses their laptops to play music. The problem was apparent with earphones.

I tried another identical laptop model, and it sounded as bad.

I tried the same earphones on other laptops, and they sounded fine.

So the problem had to be the laptop. I tried searching the forums with "6730b headphones bad sound" (and all possble combinations) and didn't find anyone complaining with a similar issue. So what was the problem? Possibly a bad batch of DSPs? Maybe.

Of course, having to endure bad sound quality is not the kind of reason you can invoke to return your business laptop. The only choices were:

1. Suck it up, buttercup, and don't listen to music anymore.
2. Wait a few months until a new model is available from IT, and try real hard to break my laptop to get a new one.
3. Buy an external USB sound card (such as the Turtle Beach Audio Advantage Micro) and use it to bypass the lame built-in Soundmax HD Audio
4. (see below)

Neither of these choices were interesting. 

I tried choice #4: Boot up the laptop with a Ubuntu Live CD and check it out. And it sounded fine! Darn. So the culprit was Windoze after all.

I'll spare the end of the story. I could have brought it back to the IT staff, put in my favorite CD, and brag on how Ubuntu sounds better than Windows. But let's just say that fiddling with drivers (illegaly, I must say, since our PCs are locked out) fixed the issue. It now sounds fine. 

So what should you do if you want to investigate some weird hardware issue on your work laptop, for which you are limited in the scope of how far exactly you can go to fix it? Boot Ubuntu!