Olivier's hot tips to monitor HP-UX servers with SIM and RSP

1. Configure WBEM on your server
SIM and WEBES subscribe to WBEM events on your server in order to receive events. But you need to put root credentials in SIM's Global Protocol Settings for this to work. Whatever you do, don't add root's credentials anywhere. You should never have to hand out the root password to some slimy application unless you really know what you're doing. Create a dedicated WBEM user for this instead.

Add a user with "adduser", I named it hpwbem:
# useradd -u 505 -g users -s /bin/false -c "HP WBEM provider" -m -k /etc/skel hpwbem

Then use passwd to input a password of your choice.

Enable non priviledged users in the CIM:
# cimconfig -s enableSubscriptionsForNonprivilegedUsers=true -p
# cimconfig -s enableNamespaceAuthorization=true -p
# cimserver -s
# cimserver

Then add rights for hpwbem to the CIM:
# cimauth -a -u hpwbem -n root/cimv2 -R -W
# cimauth -a -u hpwbem -n root/PG_InterOp -R -W
# cimauth -a -u hpwbem -n root/PG_Internal -R -W
# cimauth -a -u hpwbem -n root/cimv2/npar -R -W
# cimauth -a -u hpwbem -n root/cimv2/vpar -R -W

Configure the hpwbem user, and its password, in SIM's Global Protocol Settings.

Now, have SIM subscribe to WBEM events for your server. It doesn't by default. On your CMS, type:

C:> mxwbemsub -a -n server_name

Once this is done, check on your server if you have SIM subscriptions by using evweb:
# evweb subscribe -L -b external

You should see three subscriptions named HPSIM_*.


2. Configure your system properties in SIM

Get into the System Properties of your server in SIM, then confirm that a serial and product number has been discovered. Sometimes the PN is missing for Integrity servers, so add it manually. Just to be sure, also recopy the SN and PN in the Customer-Entered serial number and product number fields in the Entitlement Information area. You'll be sorry if you don't do this. Next, set your Country code. If you don't do this, ISEE/RSP won't work. The other fields in the Entitlement Information area can normally be left blank.

Assign a site name, and at least a primary customer contact to your server. It's important, else I think no ticket will be generated by ISEE since there will be nobody to contact.


3. Configure RSP entitlement

Go in the ISEE client (Remote Support Configuration and Services under the Options menu) and confirm that your server is entitled. If it isn't, you can try clicking on the entitlement icon, and have it send a new entitlement request. As long as you're not entitled, RSP will not forward service calls to HP so it's critical that you get this fixed. Be sure you set the system properties correctly as mentioned above.


4. Configure WEBES

Get into WEBES (localhost:7906) and confirm that your server is in the Managed Entities list. Of course, there's no search feature, you'll probably have to check multiple pages in the Full List to find it. If your server appears, confirm that its system type is ManagedSystem - HPUX. If the server is of the wrong type, delete it, as it could stay that way for a while -- better be safe than sorry.

WEBES synchronizes its entity data with SIM, but it does this through telepathy or some other magic, I couldn't find out how it's done and if it can be forced (and nobody replied to me in the forums to help me...). Restarting desta doesn't do the trick.... the real trick is actually waiting, sometimes for a loooooong time, until your server appears as a managed entity. I suggest you wait until the next day.

Once your server is in WEBES, run evweb (see above) and confirm that there's a subscription named HPWEBES_*. You need to have one, else hardware events will not be caught by WEBES and forwarded to RSP...


5. Generate test events to confirm it actually work

Generate a test event with EMS:

# /etc/opt/resmon/lbin/send_test_event ia64_corehw

...then cross your fingers, hoping it will be reported. The following should happen:

a) the event will be shown in SIM, in the event tab of the server (this is what the SIM WEBM suscription is used for)
b) the event will be trapped by WEBES, and sent to ISEE (this is what the WEBES WBEM subscription is used for)
c) ISEE will send the event to HP, and you'll see in the server event log messages such as A service incident has been reported (this is what all the entitlement hassle is used for)

If you went down to step c, you're done. If it didn't work, go to step 1 and start again. I had to to this quite a few times. There's an old song in Quebec French named "refelemele". It basically means "doittomeagain". Chances are you'll be singing this along for a few days.

Goobye ISEE: Monitoring an EVA with RSP: day 3

I don't only have EVAs to monitor, but B-Series fibre switches as well. The RSP documentation has a few indications on how to configure Fabric OS to send SNMP traps to the CMS, but you're better off reading the OSEM setup guide which gives more details on the exact commands to use.

However, the switches don't appear magically in OSEM. Usually I think a device will appear once OSEM receives the first SNMP trap. But Fabric OS does not currently have any SNMP test trap that can be sent. Besides pulling off a FRU, which HP doesn't recommend, there's no solution for now. I'll try to pull off a SFP to see if it works.

As a sidenote, I also tried installing the Brocade SMI-S provider on the SMS since the switches are labeled as "unmanaged" by SIM but it doesn't seem to work. No matter how much I try running an Identify on the CMS, it doesn't discover the switches. Everything seems to be set up correctly, wbemdisco finds the devices, and I also modifid wbemportlist.xml to use port 60001. No results yet, the switches are still unmanaged (yes, I tried deleting one).

Goobye ISEE: Monitoring an EVA with RSP: day 2

Turns out that my WEBES problems not recognizing my SMS as a "CommandView Server" were a bug in WEBES 5.4. There is a document in the ITRC KB, that was published just yesterday, that describes the issue. Look for document #mmr_na-0229085.

The resolution is not really clear, however. Basically, you have to create a new Managed Protocol named "CommandView". In it, input your authentication information for CV. Then, delete your server from the managed entities. Stop and restart the desta service (net stop desta_service; net start desta_service). You'll then notice that the server will reappear in the managed systems list, still as a Proliant, but in its detais you'll see the ELMC is now using your CommandView Protocol.

Does it actually work? I don't know yet. Besides the "wccproxy test", I can't do much. There is no way to initiate a real test from the HSV controller besides pulling of an FRU part, and I'd rather not do this. I'll try to look if there's a test even somewhere in the service menu but I won't keep my fingers crossed.

Goobye ISEE: Monitoring an EVA with RSP: day 1

I have deferred upgrading the monitoring of my EVAs to RSP until Q1 2009 since I had a great deal of trouble with RSP last fall and was fed up.

HP Services proposed coming to help me (we have 6 EVAs and 6 SMSes), but I thought I'd try for myself for the first one to at least understand what they'll be doing, and be able to troubleshoot it once they're gone.

To increase my chances, I decided to start everything from scratch on the CMS and SMS. As far as these two servers are concernd, it doesn't get as "standard" as this:

• The SIM administrator and me installed SIM 5.2 on a freshly reinstalled Windows server, then we restored our database succesfully (see one of my previous post for my recommendations on this)
• I completely zapped my test SMS and reinstalled a vanilla Windows 2003 Enterprise, along with CV 6.0.2 and nothing more (we stick with 6.0.2 since it's the only version certified with Metrocluster).

Now does it work? Partly.

• SIM must have at least spoken to SMI-S, since EVAs appeared automagically in the system list. But there's not much information I can get from them.
• As the RSP "prerequisite" documentation that explains how to set everything up has no fucking example screenshot, who knows if the EVA entries in SIM are supposed to be in this state or not. Message to whoever's writing these guides: I'm sure you are allowed to put images there. Please do it!!!
• WEBES is still not able to communicate with CV since it sees the server as a generic "Proliant", and not a "CommandView Server". Is it because I'm running on a generic server instead of a real SMS? Maybe, but a generic server is officially supported. I don't know how to change this yet. More work needs to be done.

Since there are still problems and SIM 5.3 has just been released last Monday along with WEBES 5.4, I'll try to have SIM upgraded first and start from there. This will probably be the last straw. If it still doesn't work, I'm converting my SSSU monitoring script to a nagios plugin and I'll give it for free to everyone who's interested. If HP is happy to give me crappy software that doesn't work, then I'll let them handle the overhead paying a human to manage the service calls that I'll log manually. I just wasted too much time and energy on this.

The idiot's guide to (re-)installing SIM on Windows and making it actually work

My colleague and I have been busy in the last few days doing a complete re-install of SIM and RSP since we were running into problems with our server that would be tough to explain. To make a long story short, we decided that a fresh reinstall would fix things, and it looks like it did. Why are we running on Windows and not on HP-UX? Basically because 1) SIM was initially installed on Windows in our shop, 2) RSP only works on Windows and 3) My colleague is a Windows guy. :)

Here are my 10 suggestions if you want to do this. This might seem stupid for a Windows admin but I'm an HP-UX guy, remember.

1. Have a good backup
First of all, we made sure we had a good backup of the SIM database. HP has a whitepaper on the subject. But it says what to backup, but not necessarily how to back it up automatically. This was my first MS-SQL experience, and I ended up writing a custom script to back it up. I run it each day to dump the database, so that it can be backed up consistently.

2. Before reinstalling, confirm first that your data can be restored
Which I did by setting up a dummy VM running Windows, and restored data to a dummy SIM. It worked.

3. Use the Smart Start CD to Install Windows Server
I'm always sceptical of software that's self-labeled as "smart" and thought that we could just install a vanilla Windows server, then add all appropriate drivers and stuff... waste of time. Smart Start does all of this for you, and can install Windows from a CIFS-accessible .iso file.

4. Don't use a localized Windows and other software
Use a plain, honest-to-goodness U.S English version of Windows. If and when you run into problems, google will be a much better friend if you paste it error messages that are in english. If your company has a policy of installing software in a localized language, screw 'em.

5. Use the defaults to install *EVERYTHING*
Even if you don't like the defaults, at least they will work. We ran into a few bugs, especially with the database, and ended up thinking "if we were the QA guys at HP, how would we set up our server?" Chances are the answer to this is using the defaults! So don't try to tweak install optons, whether in SIM, RSP or MSSQL, unless you really know what you're doing. We didn't.

6. Don't run the software in your own account
Have it run with a generic account. If you use your personal account, SIM and MSSQL will work, but expect problems when your account gets deleted once you a) quit your job or b) get fired. Of course doing this is a good way to leave a time bomb at work in the case of b).

7. Update your server with Windows update between each software install
You'll probably end up going there 3-4 times

8. Run the SIM installer on the console
No need to use the iLO, you can type "mstsc /console" to do a terminal session. If you don't use the console, the RSP installer could fail miserably. Trust me.

9. Be patient when RSP is installing
It often asks you to wait "a few minutes" but experience here has shown me that it should rather be "a few hours" since it's downloading in the background a lot of software. Looks like the development team at HP tested this only on their gigabit network. In the real world, downloading hundreds of megabytes of bloated data through the internet can actually take quite some time.

10. Be prepared to reinstall everything, even Windows, if it doesn't work
There's an expression in French, un mal pour un bien, which means a bad thing for a good thing. We had problems with MSSQL which would have been impossible to fix cleanly, and decided that reinstalling Windows would be actually quicker than trying to make it work. It's not that bad, since by reinstalling Windows, yours truly actually took notes this time, and is sharing them with you!

Good luck