The e-mail came in late this afternoon: my HPTF 2009 abstract has been approved. If all goes well, I should be there this year to present Monitoring and profiling Virtual Connect performance with HP-UX or Windows.
This session will show how to use cacti to characterize and monitor the performance of your virtual connects, and iperf to actually test the potential bandwidth. These are FOSS multiplatform tools, and they work well. I'll also add some few caveats that I stumbled upon when we deployed our VC infrastructure last year.
See you there.
Friday, March 27, 2009
Friday, March 20, 2009
Configuring SysFaultMgmt and RSP on HP-UX
I've stumbled upon the following howto:
http://www.e-planit.net/doku_texte/hpsim-sfm-rsp.txt
I haven't gone through it entirely yet but it seems quite complete.
Good luck
http://www.e-planit.net/doku_texte/hpsim-sfm-rsp.txt
I haven't gone through it entirely yet but it seems quite complete.
Good luck
Saturday, March 14, 2009
The ridiculous price of HP Enterprise Software
A colleague of mine told me recently that "Enterprise" software that's usually despised by IT technical staff often finds its way in corporations using steaks and hookers. To this I would add good golf appartus. Unfortunately for these software sellers, I'm not fond of steak, I'm married, and I don't play golf... So my software purchases are based on technical merit. And I must say that increasingly, Open Source software wins in this arena. But that's another story.
Some Enterprise software are actually useful to IT techies, and even if they're costly, they can actually make you save in the long run. One example I can think of is OpenView Performance Manager. I've been using it for years and it helped us plan equipment purchases and consolidate on VMs. Its price could be considered expensive by some, but personally I find it reasonable when I see how useful it's been for our environment.
I might have been naive, but I expected a cost similar to OVPM when I asked about the price of a storage management software from HP which I won't name here. As a SAN admin, that software would have been very useful to me. It would have helped me provision and plan for years ahead our disk space trends.
But my inquiry came back with a price tag that was the absolute highest, by a large degree, that I've ever encounted up until now in my 10+ years in IT.
Man, what are these guys smoking?
Do they expect someone to actually pay this price, especially in these dire economic times? Give me a break. The cost is so insane that no matter what that software does, unless it actually prints money, there's no way a lone IT admin such as me could even think of recommending this to management. My chances would be better endorsing the purchase of a few Ferraris, at least they'd be fun to drive.
Joel Spolsky once wrote an essay detailing how to price software. If I remember correctly, it boiled do to this: you set the pricing strategy depending on the what your target customer is ready to pay.
Did you know that HP recently asked their employees to lower their salary ? No wonder. I don't know who can purchase software at these prices in 2009. The target customer for the management software I asked for is either someone who's IT budget is in tens (if not hundreds) of millions and don't give a damn about the price of their software, or they're the kind who has a soft spot for Angus beef, exotic dancers and nice golf clubs. I'm none of that. Looks like I'll just have to spend a few weeks gluing together some FOSS software, along with a few homemade scripts, and I'll have my own management solution for free.
Some Enterprise software are actually useful to IT techies, and even if they're costly, they can actually make you save in the long run. One example I can think of is OpenView Performance Manager. I've been using it for years and it helped us plan equipment purchases and consolidate on VMs. Its price could be considered expensive by some, but personally I find it reasonable when I see how useful it's been for our environment.
I might have been naive, but I expected a cost similar to OVPM when I asked about the price of a storage management software from HP which I won't name here. As a SAN admin, that software would have been very useful to me. It would have helped me provision and plan for years ahead our disk space trends.
But my inquiry came back with a price tag that was the absolute highest, by a large degree, that I've ever encounted up until now in my 10+ years in IT.
Man, what are these guys smoking?
Do they expect someone to actually pay this price, especially in these dire economic times? Give me a break. The cost is so insane that no matter what that software does, unless it actually prints money, there's no way a lone IT admin such as me could even think of recommending this to management. My chances would be better endorsing the purchase of a few Ferraris, at least they'd be fun to drive.
Joel Spolsky once wrote an essay detailing how to price software. If I remember correctly, it boiled do to this: you set the pricing strategy depending on the what your target customer is ready to pay.
Did you know that HP recently asked their employees to lower their salary ? No wonder. I don't know who can purchase software at these prices in 2009. The target customer for the management software I asked for is either someone who's IT budget is in tens (if not hundreds) of millions and don't give a damn about the price of their software, or they're the kind who has a soft spot for Angus beef, exotic dancers and nice golf clubs. I'm none of that. Looks like I'll just have to spend a few weeks gluing together some FOSS software, along with a few homemade scripts, and I'll have my own management solution for free.
Thursday, March 12, 2009
Time for a small overhaul
As there are some code snippets here, I decided to change the layout of my blog to make the articles wider and on a white background, it should be easier to read. As for the origin of Technocrat-UX... yes, it used to be a pun on technocrat.net which, unfortunately, shut down last december.
The purpose in life of vxfsd
Currently, the #1 Google keyword that has readers stumbling on this blog is "vxfsd" and they spend some time checking out the entry where I tell how to reduce vxfsd usage on 11iv2. It figures; vxfsd often shows up first in top(1) and there is not much documentation on what it does, let alone a man page. I think an explanation of what this this daemon is used for is in order.
Under Unix, an inode is a structure that contains file metadata (size, position on disk, owner, etc). You can find a .pdf with an example inode here. HP-UX puts these inode structures in a cache to accelerate I/O performance. That area is called the inode cache. It is a dynamic cache, meaning that it grows or shrinks its size RAM depending on the needs.
Where does vxfsd come in? It's a daemon whose job is to scan the inode cache, freeing inodes which have not been referenced since a while. The more inodes there are to scan and free, the more it spins and consumes CPU time.
When there is intensive read I/O on a server, such as during a backup, all files are read, so a great bunch of inodes get cached in the progress, with the cache size growing quickly. It is normal under these circumstances to see vxfsd spinning once that I/O pressure is over; it's doing its job, scanning the cache and freeing the inodes that haven't been recently needed. It should slow down once it's done.
As I described in my earlier post, the kernel can be tuned to use a static inode cache, and this will make vxfsd stop needing to scan the cache. It can actually be a good idea to do this on low-end systems (especially VMs) that have few or slow CPUs, as vxfsd can consume a fair percentage of CPU time on these types of systems. The trade off is that you could have lower I/O performance since the cache will no longer be able to grow to meet a sudden I/O demand. I normally don't care about vxfsd on systems that have multiple processors, and let it live with its default settings.
Take care
Under Unix, an inode is a structure that contains file metadata (size, position on disk, owner, etc). You can find a .pdf with an example inode here. HP-UX puts these inode structures in a cache to accelerate I/O performance. That area is called the inode cache. It is a dynamic cache, meaning that it grows or shrinks its size RAM depending on the needs.
Where does vxfsd come in? It's a daemon whose job is to scan the inode cache, freeing inodes which have not been referenced since a while. The more inodes there are to scan and free, the more it spins and consumes CPU time.
When there is intensive read I/O on a server, such as during a backup, all files are read, so a great bunch of inodes get cached in the progress, with the cache size growing quickly. It is normal under these circumstances to see vxfsd spinning once that I/O pressure is over; it's doing its job, scanning the cache and freeing the inodes that haven't been recently needed. It should slow down once it's done.
As I described in my earlier post, the kernel can be tuned to use a static inode cache, and this will make vxfsd stop needing to scan the cache. It can actually be a good idea to do this on low-end systems (especially VMs) that have few or slow CPUs, as vxfsd can consume a fair percentage of CPU time on these types of systems. The trade off is that you could have lower I/O performance since the cache will no longer be able to grow to meet a sudden I/O demand. I normally don't care about vxfsd on systems that have multiple processors, and let it live with its default settings.
Take care
Thursday, February 19, 2009
Olivier's hot tips to monitor HP-UX servers with SIM and RSP
1. Configure WBEM on your server
SIM and WEBES subscribe to WBEM events on your server in order to receive events. But you need to put root credentials in SIM's Global Protocol Settings for this to work. Whatever you do, don't add root's credentials anywhere. You should never have to hand out the root password to some slimy application unless you really know what you're doing. Create a dedicated WBEM user for this instead.
Add a user with "adduser", I named it hpwbem:
# useradd -u 505 -g users -s /bin/false -c "HP WBEM provider" -m -k /etc/skel hpwbem
Then use passwd to input a password of your choice.
Enable non priviledged users in the CIM:
# cimconfig -s enableSubscriptionsForNonprivilegedUsers=true -p
# cimconfig -s enableNamespaceAuthorization=true -p
# cimserver -s
# cimserver
Then add rights for hpwbem to the CIM:
# cimauth -a -u hpwbem -n root/cimv2 -R -W
# cimauth -a -u hpwbem -n root/PG_InterOp -R -W
# cimauth -a -u hpwbem -n root/PG_Internal -R -W
# cimauth -a -u hpwbem -n root/cimv2/npar -R -W
# cimauth -a -u hpwbem -n root/cimv2/vpar -R -W
Configure the hpwbem user, and its password, in SIM's Global Protocol Settings.
Now, have SIM subscribe to WBEM events for your server. It doesn't by default. On your CMS, type:
C:> mxwbemsub -a -n server_name
Once this is done, check on your server if you have SIM subscriptions by using evweb:
# evweb subscribe -L -b external
You should see three subscriptions named HPSIM_*.
2. Configure your system properties in SIM
Get into the System Properties of your server in SIM, then confirm that a serial and product number has been discovered. Sometimes the PN is missing for Integrity servers, so add it manually. Just to be sure, also recopy the SN and PN in the Customer-Entered serial number and product number fields in the Entitlement Information area. You'll be sorry if you don't do this. Next, set your Country code. If you don't do this, ISEE/RSP won't work. The other fields in the Entitlement Information area can normally be left blank.
Assign a site name, and at least a primary customer contact to your server. It's important, else I think no ticket will be generated by ISEE since there will be nobody to contact.
3. Configure RSP entitlement
Go in the ISEE client (Remote Support Configuration and Services under the Options menu) and confirm that your server is entitled. If it isn't, you can try clicking on the entitlement icon, and have it send a new entitlement request. As long as you're not entitled, RSP will not forward service calls to HP so it's critical that you get this fixed. Be sure you set the system properties correctly as mentioned above.
4. Configure WEBES
Get into WEBES (localhost:7906) and confirm that your server is in the Managed Entities list. Of course, there's no search feature, you'll probably have to check multiple pages in the Full List to find it. If your server appears, confirm that its system type is ManagedSystem - HPUX. If the server is of the wrong type, delete it, as it could stay that way for a while -- better be safe than sorry.
WEBES synchronizes its entity data with SIM, but it does this through telepathy or some other magic, I couldn't find out how it's done and if it can be forced (and nobody replied to me in the forums to help me...). Restarting desta doesn't do the trick.... the real trick is actually waiting, sometimes for a loooooong time, until your server appears as a managed entity. I suggest you wait until the next day.
Once your server is in WEBES, run evweb (see above) and confirm that there's a subscription named HPWEBES_*. You need to have one, else hardware events will not be caught by WEBES and forwarded to RSP...
5. Generate test events to confirm it actually work
Generate a test event with EMS:
# /etc/opt/resmon/lbin/send_test_event ia64_corehw
...then cross your fingers, hoping it will be reported. The following should happen:
a) the event will be shown in SIM, in the event tab of the server (this is what the SIM WEBM suscription is used for)
b) the event will be trapped by WEBES, and sent to ISEE (this is what the WEBES WBEM subscription is used for)
c) ISEE will send the event to HP, and you'll see in the server event log messages such as A service incident has been reported (this is what all the entitlement hassle is used for)
If you went down to step c, you're done. If it didn't work, go to step 1 and start again. I had to to this quite a few times. There's an old song in Quebec French named "refelemele". It basically means "doittomeagain". Chances are you'll be singing this along for a few days.
Monday, February 9, 2009
easyddr: a wrapper for scsimgr to help you manage your DDR
The DDR (device data repository) which I spoke about a few months ago is an interesting feature of the new HP-UX mass storage stack which lets you set attributes on-the-fly for storage devices matching a specific scope. Any new device which is presented to the system gets its custom attributes assigned dynamically.
While this is a nice feature, there are not thousands of use cases I can think of here. The stock DDR that comes with HP-UX seems to be populated with entries for tape devices, so my guess is that a lot of usefulness is related to tape drives (but I couldn't say, as our Data Protector infrastructure runs on Windows). But with disks, one case that is worth mentioning is that the SCSI queue depth can be fine-tuned for specific devices, enhancing performance.
While the DDR is useful, it's not really easy to use: it depends on fixed-length, whitespace-padded strings and it's hard to dispatch modifications to a bunch of servers at the same time. So I wrote a small script named easyddr.sh which eases its use somewhat.
Instructions:
1. Create a configuration file
The configuration file contains pairs matching wildcards and attributes. For example:
# Sample configuration file
/HSV:max_q_depth=32 esd_secs=60
/IR Volume:max_q_depth=8
This means that:
- All HSV (EVA) devices, no matter the controller version, will be set with a SCSI queue depth of 32 and an I/O timeout of 60 seconds.
- All IR Volumes (MPT) devices will be set with a SCSI max queue depth of 8
2. Run easyddr with your configuration file
Run easyddr:
# easyddr.sh /usr/local/etc/easyddr.cfg
It first reads the configuration file to see if any devices on your system matches your widcards. Then, if there is no DDR entry for a device, it adds it to the DDR without needing you to cut-and-paste it. The last step it takes is to set your attributes, then save them. The script always runs in preview mode by default. If everything seems okay and you want to apply the settings, re-run it with the "-apply" option:
# easyddr.sh /usr/local/etc/easyddr.cfg -apply
Caveat: there is no way to delete entries using easyddr once they are added. If you make a mistake, you'll need to run scsimgr ddr_del to remove them.
In my case, I install easyddr and the config file in my Golden Depot, and make it a baseline script that needs to be run when the server is initially installed. No more need to use the scsimgr command. Have fun!
Subscribe to:
Posts (Atom)