The hiden advantage of using Remote Support on HP-UX

Readers of my blog will know that I spent quite some time integrating HP-UX 11.23/11.31 with SIM and Remote Support on each and every of my servers, including even the older, neglected test/QA servers no one usually cares about.

I'm sure some must have thought I was crazy investing so much time on a feature that doesn't bring back much, because, they'll say, hardware doesn't break. It is partly true. Hardware doesn't break a lot with the exception of disks, fans and power supplies which can experience a higher failure rate than, say, anything else that's based on transistors. So, most efforts should be prioritized towards monitoring devices which have a lot of these, and this mostly applies to disk arrays.

There is, however, a hidden gem in using Remote Support pack with HP-UX, and it's the monitoring of system panics.

That's right, panics! I don't hear about the term as much as I used to in the old days, but the fact remains that they still happen, and can either be the result of a software bug or even an untrapped hardware problem. With HPVM guests, I've had my share of panics, too.

Remote Support comes to great help with panics. When a panic occurs, once rebooted the monitoring agents will notice it, WEBES will gladly flag it as important, and an event will be logged at the response center. If the panic happens overnight or when the sysadmin is not there (and it WILL happen - most of us are in the office only a small amount of time), hours will be saved in the process as someone will probably have already contacted the system contact about the issue.

There is not yet a feature to send to HP details on the crash dump when the event is opened, and it must be done manually. But I wouldn't be surprised this will come in the future. Wouldn't it be great, for example, if upon rebooting the server crashinfo could be ran automatically and send details to the engineer? One can only hope this will come in the future, to reduce even further the response time.

O.

Migrating from OSEM to WEBES 5.6

Here is a post on my experience migrating from OSEM to WEBES 5.6. It went well except for one minor problem with the MSA 2000.

First of all, you should know there are a few outstanding problems with WEBES 5.6, one of them being a security issue. I'm not sure I can disclose what they are as I didn't get them from official channels but I will say that a patch is expected sometime in December. If you don't need to run 5.6 right now in order to support specific hardware, you should stay with WEBES 5.5 and OSEM 1.4.8a in the mean time.

That being said, for those who wish to run WEBES 5.6, you can update it manually from Remote Support Software Manager. Also update to the latest version of Remote Support Eligible system List at the same time. The procedure is documented in the guide WEBES 5.6 and product coverage.

There is also some (redundant) information in the guides A.05.30 HP Insight Remote Support Advanced with WEBES 5.6 and OSEM to WEBES Migration Guide.

To migrate away from OSEM, if you have a standard installation with OSEM populated by info from SIM, it is as easy as simply uninstalling OSEM but you should read the above documents just to be sure. Once OSEM is uninstalled, stop WEBES (not documented, but I did it anyway) using "net stop desta_service" and "net start desta_service".

If you have hundreds of managed systems, it is better to wait at least an hour before testing if everything works well, as it takes a while for WEBES to stabilize and trap events once it is restarted. I also always confirm that e-mail notifications are enabled in order to have an alternate way of receiving notifications in case there is a problem up the food chain in SIM or ISEE.

Now you should test equipment that used to notify OSEM with SNMP traps to be sure they are being caught by WEBES and service events are opened at HP.

Here is what I tested successfully:

Proliant running Windows ... OK
Proliant running ESX ... OK
C3000 blade chassis ... OK
C7000 blade chassis ... OK
MSA2012i G1 disk array ... NO

The MSA 2000 G1 used to work with OSEM but no longer with WEBES. I've opened a ticket at the ITRC to have an official support statement. This is exacerbated by the fact that its events in SIM are reportedly sent as informational so those of you who "follow the red" could miss critical events.

O.

Olivier's hot tips to monitor HP-UX servers with SIM and RSP

1. Configure WBEM on your server
SIM and WEBES subscribe to WBEM events on your server in order to receive events. But you need to put root credentials in SIM's Global Protocol Settings for this to work. Whatever you do, don't add root's credentials anywhere. You should never have to hand out the root password to some slimy application unless you really know what you're doing. Create a dedicated WBEM user for this instead.

Add a user with "adduser", I named it hpwbem:
# useradd -u 505 -g users -s /bin/false -c "HP WBEM provider" -m -k /etc/skel hpwbem

Then use passwd to input a password of your choice.

Enable non priviledged users in the CIM:
# cimconfig -s enableSubscriptionsForNonprivilegedUsers=true -p
# cimconfig -s enableNamespaceAuthorization=true -p
# cimserver -s
# cimserver

Then add rights for hpwbem to the CIM:
# cimauth -a -u hpwbem -n root/cimv2 -R -W
# cimauth -a -u hpwbem -n root/PG_InterOp -R -W
# cimauth -a -u hpwbem -n root/PG_Internal -R -W
# cimauth -a -u hpwbem -n root/cimv2/npar -R -W
# cimauth -a -u hpwbem -n root/cimv2/vpar -R -W

Configure the hpwbem user, and its password, in SIM's Global Protocol Settings.

Now, have SIM subscribe to WBEM events for your server. It doesn't by default. On your CMS, type:

C:> mxwbemsub -a -n server_name

Once this is done, check on your server if you have SIM subscriptions by using evweb:
# evweb subscribe -L -b external

You should see three subscriptions named HPSIM_*.


2. Configure your system properties in SIM

Get into the System Properties of your server in SIM, then confirm that a serial and product number has been discovered. Sometimes the PN is missing for Integrity servers, so add it manually. Just to be sure, also recopy the SN and PN in the Customer-Entered serial number and product number fields in the Entitlement Information area. You'll be sorry if you don't do this. Next, set your Country code. If you don't do this, ISEE/RSP won't work. The other fields in the Entitlement Information area can normally be left blank.

Assign a site name, and at least a primary customer contact to your server. It's important, else I think no ticket will be generated by ISEE since there will be nobody to contact.


3. Configure RSP entitlement

Go in the ISEE client (Remote Support Configuration and Services under the Options menu) and confirm that your server is entitled. If it isn't, you can try clicking on the entitlement icon, and have it send a new entitlement request. As long as you're not entitled, RSP will not forward service calls to HP so it's critical that you get this fixed. Be sure you set the system properties correctly as mentioned above.


4. Configure WEBES

Get into WEBES (localhost:7906) and confirm that your server is in the Managed Entities list. Of course, there's no search feature, you'll probably have to check multiple pages in the Full List to find it. If your server appears, confirm that its system type is ManagedSystem - HPUX. If the server is of the wrong type, delete it, as it could stay that way for a while -- better be safe than sorry.

WEBES synchronizes its entity data with SIM, but it does this through telepathy or some other magic, I couldn't find out how it's done and if it can be forced (and nobody replied to me in the forums to help me...). Restarting desta doesn't do the trick.... the real trick is actually waiting, sometimes for a loooooong time, until your server appears as a managed entity. I suggest you wait until the next day.

Once your server is in WEBES, run evweb (see above) and confirm that there's a subscription named HPWEBES_*. You need to have one, else hardware events will not be caught by WEBES and forwarded to RSP...


5. Generate test events to confirm it actually work

Generate a test event with EMS:

# /etc/opt/resmon/lbin/send_test_event ia64_corehw

...then cross your fingers, hoping it will be reported. The following should happen:

a) the event will be shown in SIM, in the event tab of the server (this is what the SIM WEBM suscription is used for)
b) the event will be trapped by WEBES, and sent to ISEE (this is what the WEBES WBEM subscription is used for)
c) ISEE will send the event to HP, and you'll see in the server event log messages such as A service incident has been reported (this is what all the entitlement hassle is used for)

If you went down to step c, you're done. If it didn't work, go to step 1 and start again. I had to to this quite a few times. There's an old song in Quebec French named "refelemele". It basically means "doittomeagain". Chances are you'll be singing this along for a few days.

Goobye ISEE: Monitoring an EVA with RSP: day 3

I don't only have EVAs to monitor, but B-Series fibre switches as well. The RSP documentation has a few indications on how to configure Fabric OS to send SNMP traps to the CMS, but you're better off reading the OSEM setup guide which gives more details on the exact commands to use.

However, the switches don't appear magically in OSEM. Usually I think a device will appear once OSEM receives the first SNMP trap. But Fabric OS does not currently have any SNMP test trap that can be sent. Besides pulling off a FRU, which HP doesn't recommend, there's no solution for now. I'll try to pull off a SFP to see if it works.

As a sidenote, I also tried installing the Brocade SMI-S provider on the SMS since the switches are labeled as "unmanaged" by SIM but it doesn't seem to work. No matter how much I try running an Identify on the CMS, it doesn't discover the switches. Everything seems to be set up correctly, wbemdisco finds the devices, and I also modifid wbemportlist.xml to use port 60001. No results yet, the switches are still unmanaged (yes, I tried deleting one).

Goobye ISEE: Monitoring an EVA with RSP: day 2

Turns out that my WEBES problems not recognizing my SMS as a "CommandView Server" were a bug in WEBES 5.4. There is a document in the ITRC KB, that was published just yesterday, that describes the issue. Look for document #mmr_na-0229085.

The resolution is not really clear, however. Basically, you have to create a new Managed Protocol named "CommandView". In it, input your authentication information for CV. Then, delete your server from the managed entities. Stop and restart the desta service (net stop desta_service; net start desta_service). You'll then notice that the server will reappear in the managed systems list, still as a Proliant, but in its detais you'll see the ELMC is now using your CommandView Protocol.

Does it actually work? I don't know yet. Besides the "wccproxy test", I can't do much. There is no way to initiate a real test from the HSV controller besides pulling of an FRU part, and I'd rather not do this. I'll try to look if there's a test even somewhere in the service menu but I won't keep my fingers crossed.

Goobye ISEE: Monitoring an EVA with RSP: day 1

I have deferred upgrading the monitoring of my EVAs to RSP until Q1 2009 since I had a great deal of trouble with RSP last fall and was fed up.

HP Services proposed coming to help me (we have 6 EVAs and 6 SMSes), but I thought I'd try for myself for the first one to at least understand what they'll be doing, and be able to troubleshoot it once they're gone.

To increase my chances, I decided to start everything from scratch on the CMS and SMS. As far as these two servers are concernd, it doesn't get as "standard" as this:

• The SIM administrator and me installed SIM 5.2 on a freshly reinstalled Windows server, then we restored our database succesfully (see one of my previous post for my recommendations on this)
• I completely zapped my test SMS and reinstalled a vanilla Windows 2003 Enterprise, along with CV 6.0.2 and nothing more (we stick with 6.0.2 since it's the only version certified with Metrocluster).

Now does it work? Partly.

• SIM must have at least spoken to SMI-S, since EVAs appeared automagically in the system list. But there's not much information I can get from them.
• As the RSP "prerequisite" documentation that explains how to set everything up has no fucking example screenshot, who knows if the EVA entries in SIM are supposed to be in this state or not. Message to whoever's writing these guides: I'm sure you are allowed to put images there. Please do it!!!
• WEBES is still not able to communicate with CV since it sees the server as a generic "Proliant", and not a "CommandView Server". Is it because I'm running on a generic server instead of a real SMS? Maybe, but a generic server is officially supported. I don't know how to change this yet. More work needs to be done.

Since there are still problems and SIM 5.3 has just been released last Monday along with WEBES 5.4, I'll try to have SIM upgraded first and start from there. This will probably be the last straw. If it still doesn't work, I'm converting my SSSU monitoring script to a nagios plugin and I'll give it for free to everyone who's interested. If HP is happy to give me crappy software that doesn't work, then I'll let them handle the overhead paying a human to manage the service calls that I'll log manually. I just wasted too much time and energy on this.

RSP still sucks... but not big time anymore

The blog entry were I was saying that RSP sucks has created some attention, both in and out of the comments area. An update is in order. First of all, I won't censor this entry; it represents my initial feeling about RSP, a software bundle which made me waste lots of time, and whatever I think of it has not changed.

On the upside, following my rant on the ITRC forums (which was deleted quickly), some people at HP Canada noticed and they've put me in contact with colleagues in Colorado who were glad to listen my comments, and they promised to address some of the issues. Some of my concerns were: no support for VMs; no cookbook for HP-UX admins, lack of feedback from SWM, etc. I also had a quick talk with Brian Cox in Mannheim a few weeks later and he was aware of the problems HP-UX shops are facing with ISEE going away as some of them don't want to install Windows. Personally I don't care, but I would have rather run this on HP-UX if I could; I'm no Windows admin and feel more at home on Unix systems.

I've been running RSP as the only notification mechanism for a few Proliant(ESX) and Integrity(HP-UX) servers for over a month now, and it seems to work. All the events are sent to HP, and closed. I've also been able to have my C7000 blade chassis monitored too, although I couldn't find any documentation for this. I just set up the CMS as the trap destination, crossed my fingers, and test traps generate RSP events.

I evaluate that installing, debugging (and trying to understand) SIM and all the components that replace ISEE have taken me over 20 hours. That's a lot of work. So when a component will break in the future, I expect a phone call or e-mail from HP Support. If I don't get anything, I won't be in a good mood. I have many EVAs of different generations that will be migrated sometime in early 2009. They require more preventive maintenance, so this will be the real test.

In the mean time I'm asking all the support personnel to take a walk in their data center (we have 6) once in a while, looking for red lights. I thought these days were over, but RSP is a stack of multiple monitoring software solutions, and I haven't had proof yet that it can be trusted.

O.

Understanding all the RSP components

N.B. My updated diagram from December 2009 is here

This blog entry is updated regularly. Latest updates:

• November 4th 2008
• November 19th 2008
• December 10th 2008
• December 16th 2008
• Feburary 20th 2008

Having read (diagonally) over 1000 pages of documentation related to every component that RSP includes, here are my notes that might be of help. This is definitely not all accurate. When I find inconsistencies, I'll update this blog post.

The bottom line is that you no longer have a simple ISEE client running on your HP-UX host anymore. It's now much more complex than this.

There's a bunch of "new" tools that will become part of your life. In fact these are "old" tools that have been available for years. They're now tightly welded together, run on a central server (CMS) instead of locally on each monitored host, and for the most part do not need to be configured independently, but it's important to understand what each one does.

SysFaultMgmt (System Fault Management) - runs on the HP-UX server
It's the "new generation" of EMS, that speaks WBEM. Using WBEM, it can be integrated easily in SMH (System Management Homepage) and SIM (Systems Insight Manager). SysFaultMgmt used to work in parallel with traditionnal EMS monitors, but since HP-UX 11iv3 March 2008, it seems to switch off EMS and replaces it completely. EMS will be eventually EOL'd.

EVWeb - runs on the HP-UX server
A companion to SysFaultMgmt which is a GUI that lets you query and manage WBEM subscriptions. There's also an evweb CLI, which will let you extract events and see their contents (they look similar to EMS's event.log file). The CLI has a man page, it's not hard to use. Be careful: I've played with evweb from SMH, sometimes it crashed, and it resulted in some evweb CGI's spinning endlessly, taking 100% CPU. The CLI is probably more robust.

System Insight Manager agent - runs on Proliants running VMware ESX and probably Windows as well

This agent includes a good-old System Management Homepage, along with hardware diagnostics agents. If the agents detect that something goes wrong, they are configured to send an SNMP trap to the CMS.

OSEM - runs on the CMS
OSEM is an agent that analyzes SNMP events that are sent to it. It filters them, and translates them to a human-readable form which can be sent by e-mail and/or to ISEE. By filtering, I mean that will be find out if an SNMP trap send by a device is actually an important one, and decide if it's necessary to generate a service event for it.

OSEM supports mostly systems that reports their events using SNMP:

• Proliant servers running Linux, Windows or VMware ESX.
• Integrity Servers running Linux
• SAN switches
• MSA enclosures
• Bladesystem chassis (simply configure the OA to send SNMP traps to the CMS)

WEBES - runs on the CMS
WEBES is an analyzer that processes events in a similar fashion to OSEM that are sent to it from these primary sources:

• Event log on a Windows Server
• WBEM subscriptions
• Interactions with Command View to gather data for EVAs

From my understanding, it does not "translate" the WBEM events to a readable form as OSEM does, since the WBEM events already contain the information.

WEBES supports mostly:

• Integrity servers running HP-UX, through WBEM subscriptions
• EVAs by reading the event long on the Storage Management Server through ELMC, and by logging directly into Command View

Now there seems to be some places where WEBES and OSEM overlap each other, and I haven't understood yet to what extent these tools talk to each other. From the OSEM documentation, it seems that WEBES sends events to OSEM, and OSEM then manages the notification.

Why is there OSEM and WEBES? I'm not sure but it looks like OSEM has a Compaq history, while WEBES comes from Digital. ISEE in itself is HP. The tools have not been merged yet, are still actively developped and they will probably complement each other for a while.

ISEE - runs on the CMS
The new 5.x ISEE client is a new version of the 3.95 client, which is now integrated into SIM. Most of the configuration settings you used to put in the ISEE client are now configured there, from the Remote Support menu entry.

SIM - runs on the CMS
SIM is used to actually manage your servers, and WEBES and OSEM automatically sync their configuration with SIM. For instance, if you set yourself as the contact person for a server, both OSEM and/or WEBES configuration will be populated with what you put in SIM. So SIM is the only place where you actually need to do some manual configuration.

Basically, if you think that SIM takes care of handling events, you're wrong. It just _reports_ the events it receives directly and gathers from WEBES/OSEM. It also reports what ISEE does with the events. The exact way it gets the information from these agents is beyond me, I don't know how yet. SIM doesn't send any events to ISEE; RSP and OSEM do. SIM also receives SNMP traps and subscribes to WBEM events. But since they are not filtered, it will only log and "raw" events.

That's what I understand out of this for now. Hope that helps.