The ex-sysadmin

A homebrew ATSC multi-room PVR project

2011-11-04T08:07:00.001-07:00

I'm not involved in technical matters much at work right now and thus I've fallen back to updating my own home on my spare time. This is a six-month project that will consist of:

Canada has switched to ATSC on September 1st, so my main objective is to cut cable TV and modernize my 2 current standalone cable-company-supported-PVRs which have an interface that dates back to 2001.

In a nutshell, I'll do this in the following weeks:

No more lightning in the house: Installing and grounding an exterior OTA antenna
Thank you foxconn and the MediaPortal Team: Building and configuring a "budget" Windows 7 TV Server with MediaPortal
Rsync now, robocopy later: Dismantling my FreeNAS-based NAS to consolidate data on the TV Server.
PXE for the masses: Deploying 2-3 MediaPortal client nettops using PXE and OpenWRT

I've got some of the pieces in place. The Win7 server is running and I should be ready to test MediaPortal soon. Why Win7 and not Win2008? The reason is that my ATSC card (an AverTVHD Duet) doesn't have drivers for 2008, and I don't need a domain controller for my house anyway. For the nettops, I have one in hand already, and wish to try installing them using PXE (just for the kicks).

I'll try to post some pictures and details over the coming weeks. They should roughly follow the 4 steps above.

Olivier

Yes, I'm still alive.

2011-09-27T18:03:00.000-07:00

Not many posts lately huh?

My older blog, Technocrat-UX, consisted of a way for me to document quirks and techniques related to HP-UX, BladeSystems, and some other technologies. Technocrat-UX enjoyed some success as these were niche, but relevant, subjects.

That model doesn't fit well with The ex-sysadmin where I originally had the intention of documenting my new job as a systems architect. The main problem when designing IT architectures is that the ideas and diagrams that result of my efforts are not generic and reusable enough, thus not interesting. Furthermore, in a security perspective, a lot of work needs to be done to obfuscate the information - any information - before it is released. I can't, for instance, publish a networking topology just like that to the public.

Up until recently I did, however, have the intention of writing a paper and presentation documenting a reference architecture for IED event and measure collection following my 18 month experience with Cooper's products. But due to some restrictions, that has not been possible yet.

In the mean time I'm keeping the blog going with posts that I *think* could be interesting to sysadmins, architects and... ex-sysadmins.

O.

HP's Power Advisor

2011-08-30T06:04:00.000-07:00

This morning I had to use HP's Power Advisor to estimate the load of small servers I need to deploy (DL360G7s). I remember using an older tool some years ago but this new one is much better. It's available here:
http://h18004.www1.hp.com/products/solutions/power/advisor-online/HPPowerAdvisor.html

Dealing with MFT servers: a systems architect versus systems administrator love story

2011-08-26T11:41:00.000-07:00

In my past life as a system administrator, I once had to build from a ground up a secure MFT (managed file transfer) server. I've pulled it off by using the HP-UX infrastructure I was comfortable with, and built something from the ground up using OpenSSH. You wouldn't believe, however, how much tweaking had to be done to have the user accounts (which were stored in /etc/passwd and /etc/shadow) synchronized reliabily in a clustered system spanning two sites. It took me a few days to make sure everything was correct.

Now it is time to do it again at a new place. I have to design another highly-available MFT server that will be wedged between two DMZs, and besides supporting SFTP I want it to have an HTTPS-based, "drop box" feature for end-users do be able to upload files easily without needing an SFTP client. Oh, and by the way, I need it to be able to authenticate users with a Windows domain this time.

If I was still a sysadmin, I'd have to extend the first solution further by adding an Apache HTTPD server and some open-source file upload solution. Then, I'd have to find a solution *BOTH* for Apache and OpenSSH to authenticate users. OpenSSH would probably need to rely on PAM, and for Apache I don't have a clue. Yet no problema; I would just shrug and say I can do that, then spend a few days tying everything up. The end.

But, as a system architect, things don't work this way.

Why? Because I have to assume that there is no guarantee the sysadmin who will have to do the grunt work of building this up will be willing, or have enough experience, to install and configure a custom solution. And assuming he/she is willing to do it, I have to consider that each man-hour counts for serious dough within the frame of a project. With custom hacks like this, these can sum up to a lot of hours depending on whose desk the work falls on.

Therefore, I did what system architects do: I tried to pick a turnkey solution, and it will have to be shoved down the IT team's throat.

I always hated this when it happened before. Picture this: The architect goes on a golf course or whatever, and randomly picks a solution based on bullet points and checklist tables. Then the IT operations guy has to take whatever crappy, slow and expensive "enterprise" software the architect purchased at a pharaonic price, and make it work satisfactorily to fulfill a business need. More often than not, such lame software ends up in the garbage bin with the IT team developing its own in-house solution to patch things up.

As I've been on that side of the fence before, I try to do things differently to prevent this from happening. So when I technically can, I actually try out software before choosing it, when it's not too daunting for me to do so.

So back to the MFT. I went searching on the web and picked a market "enterprise" leader to try it out. Not only doesn't it support high availability easily, the software is clumsy and it took 30 minutes to run its installshield sequence on a Windows 2008 VM. Uninstalling took the same. And to have SFTP support, I actually had to pay a premium over the base price. Not good.

Few other vendors had solutions that seemed serious enough, though. By serious I mean that they have to offer technical support, and have some agility in dealing with enteprise customers. Then a colleague of mine found out a very elegant software named JSCAPE MFT Server. Installation is a snap and it's very easy to configure. I was up and running in a few minutes. And as a bonus, its feature set is actually useful and seems to have been designed based on user requests instead of some odd crystal ball. I've been trying it this morning and up to now, it works very well.

The MFT server itself works on Windows, Linux, some Unices and Mac OS X. Installing the RPM went without any problem on CentOS 6. It is managed by a Java-based GUI that I installed on Windows -- I wasn't fond of using a thick-client when compared to a web-based administration GUI, but their GUI is efficient and interface-rich without being clumsy. No bells and whistles, and it is fine that way.

The Java-based "Server Manager" does the job efficiently

Enabling the web-based transfer option was quick and easy to do. What helps is that the software comes with a manual that, without being too detailed, provides lots of screenshots and cookbook-like procedures to configure the server quickly. It took me maybe a minute or so to enable the web server, set up LDAP-based authentication, add a dummy user, and try it out.

The resulting web service might be bland but it does the job, once again no fireworks. This is very important as it will be deployed to users who might not always be too tech-savvy.

The web-based service might be simple, that's exactly what I want

As a system administrator, I would actually want to work with software like this because it's elegant. It does a few things, and it does it well. I like it when software feel natural, and everything works the first time without a glitch. As in every software, I'm sure there are some bugs somewhere, but it sure is a good start.

Is JSCAPE MFT Server, the iPod of MFT's? I'd say it's not far from it. Chances are that if my project is greenlit, I'll be the first in line to purchase it. Whoever wrote this, good work!

O.

Interesting thread on Slashdot

2011-08-12T11:56:00.001-07:00

Sysadmins and developers aren't the same, but they both share a strong technical background.

I suggest you read this thread. The first comment named "Stay Put" might be hilarious, it makes a lot of sense.

http://ask.slashdot.org/story/11/08/12/1433239/Ask-Slashdot-Am-I-Too-Old-To-Learn-New-Programming-Languages

The ex-sysadmin I am often asks himself if moving up was a good decision. It turns out it might be after all.

A presentation on IMS and PI might be coming

2011-08-08T18:18:00.000-07:00

After a hiatus in 2010 due to a career change, it's now time to start writing papers and building presentations again. I'll be submitting a paper for Cooper EAS's 2011 Smart Grid conference which will consist of my experience with a major Yukon IMS deployment I've been involved with as an IT architect. I'll also explain how we used the SMP gateway to link substations to the OSISoft PI data historian in order to collect critical data.

I'm not in academia so my work is in no way scientific. Furthermore, I'm part of a huge team which counts a fair share of people from IT operations, control engineering and electrical engineering, so my view is mostly IT-centric.

There are many air travel restrictions at the office so I hope I'll be able to make it. The worst case scenario would be driving from Montreal to Minneapolis for which, if it's any consolation, won't require a full body scan.

O.

Forcing laptop users to use only an Iron Key (and nothing else)

2011-07-04T18:20:00.000-07:00

I need to transfer files between two networks which need to be physically isolated for a few months until a beefed up and permanent security solution becomes possible.

The easiest way to do this on a budget consists of using USB keys to transfer files between two laptops, one which is connected on the intranet, the other on the secure network. If course, the "secure" laptop must be stripped to the bone and have an up-to-date Antivirus so it can trap known viruses that are currently in the wild. That won't prevent any new virus from coming in, but there is an urgent business need to transfer these files so there is not much that can be done in the short term.

I'm currently using IronKeys to ensure the integrity of the data, and also to prevent any data theft if a key is ever stolen. However, one must "encourage" end-users to use these keys, else they might end up using whatever key they lay their hands on to prevent having to enter a password.

On Windows XP, there is no way to do USB key filtering based on the key manufacturer. IronKey has a partner named DeviceLock that they suggest, but being a commercial product, it comes with a price. There are many other endpoint security tools that can be purchased to offer similar functionality, too. In my case, I was in a deadline and had missed the opportunity acquire software and charge it to the project, so it was preferable to use something free as a stopgap measure.

This afternoon, I've been making a few tests with USBSecure. It SEEMS to be free. There is no license, but all code is published so it can be tweaked if necessary. USBSecure is simple to configure: define users that use the computer, and whitelist the device IDs that are allowed on the system. I've been making a few tests for an hour or so, and it seems to work correctly. I might come back and give more details later.

Of course, transferring files between two adjacent PCs might look clumsy. But there are a lot of (justified) restrictions on secure control networks. What is sad is that that Stuxnet worked exactly this way, by propagating using USB keys. No matter how much we try to control their usage using endpoint security software, USB keys still remain a vector of infection for secure networks. Better long-term solutions must be done to ensure that any file transferred on a secure network is, indeed, clean. I'll be working on such solutions in 2011-2012.

O.

Why hosting enterprise-level videos on YouTube is not a good idea

2011-06-17T17:07:00.000-07:00

Last week, OSISoft sent their customers an e-mail pitching their new OSISoft Learning Channel on YouTube. Since they probably knew that many enterprise firewalls block YouTube, their communication pleaded that we should politely ask IT to authorize YouTube. They also put a reference to Buck Bard's blog post Don't be anti-social on social networking where he basically says that internal collaboration sites don't measure up to public ones like YouTube.

He's right on the social networking argument -- the corporate or "private club"-type social networking sites I've seen aren't so great when compared to the behemoths who've been able to get a foothold in the last five years. But isn't that what vCampus is, in essence? To benefit from its social networking features, one has to be a (paying) vCampus user. Maybe OSISoft could consider opening up parts of vCampus?

The problem which OSISoft acknowledged in their e-mail is that most social networking sites are blocked to many corporate users. Facebook is one thing, but I'll concentrate on YouTube in this post.

To illustrate this, let's compare YouTube to a television: if I came into the office of the CIO and asked for every cubicule in the office to have a cable TV set with every channel offered by the cable company, what would be his answer, you think? Even if all this came for free, I bet it would be "no". The reasoning being that nobody actually needs this, and it's a perfect way to lower productivity. Now if I came and asked for TV sets which only had access to all-news networks because the employees are financial analysts who actually need this to do their job, it might work.

The problem with YouTube is exactly that: it's a TV with millions of channels and there is no way to filter out content that is relevant to your workplace. Yet even though we have places where YouTube is barren due to questionable content, and rightly so, there are tech companies who keep on using it to publish their stuff. OSISoft are not the only ones, by the way -- ArcSight did the same thing two years ago.

What are the solutions?

The first one is not using YouTube at all to publish content. Which is too bad as YouTube is a really good platform to publish videos easily and cheaply. OSISoft probably doesn't want to invest thousands of dollars into a private streaming solution (and bandwidth), which is understandable, but they cut themselves out from some of their customers by choosing this path.

The second one would be for YouTube to make an "enterprise-level" version of their service, under a completely different name and domain, and charge a small fee to qualifying content publishers. Someone thought of this in 2007 and I have not seen a solution yet. The problem with this scenario is that over time, it will become another all-you-can-eat lineup. Victoria's Secret would probably end up calling themselves "enterprise-level", and I don't see where their videos would fit in a financial analyst's job.

So that's it. OSISoft's Learning Channel is on YouTube. Don't get me wrong -- I checked them out, and this initiative is very appreciated!! But since my employer doesn't let me watch YouTube, I'm stuck with watching these videos on my own time, at home or at the internet café.

Time to go grab a Latte.

O.

SFTP vs FTPS: tough choices

2011-04-27T16:12:00.000-07:00

Las week, I had to design in a hurry a secure file transfer mechanism between two DMZs on a zero budget which, in a nutshell, meant reusing the Windows servers that are already there, and not purchase any third party software.

I had to choose between using SFTP, a nice protocol, and FTPS, which I've been comparing to a bastard child for years.

I don't like FTPS mostly because it's a patch on FTP. For one, FTPS is harder to firewall than SFTP; it behaves exactly like the standard FTP with a control and data connection, the difference being that TLS is used to encrypt them. Like with standard FTP servers, the server must be configured with a fixed range of passive ports, and the firewall must let these ports through. Why? Because the firewall has no way of knowing what dynamic port has been assigned to a passive data connection... it can't sniff it out the control connection either, as it's encrypted!

Even though it's not exactly what I would call an elegant protocol, is FTPS actually easy to work with? The answer is yes: I was able to install IIS 7.5's FTP publishing service in 2008 R2 and have an FTPS server working within minutes. That is good enough. And in IT, good enough is, well, Good Enough.

So, here are my thoughts:

If your server will be hosted on any kind of Unix, choose SFTP. It has been built-in with OpenSSH for years. The drawback of OpenSSH is that it doesn't support virtual users, and this can make high availability tricky; you'll need to synchronize /etc/passwd entries, even if using AD authentication.

On the other hand, if you will host the service on Windows, you might be better off going with FTPS as it is included with IIS 7.5 and there is even high availability that is possible. To support SFTP on Windows, you either need to install unsupported open-source software (unacceptable in many secure, enterprise environments) or purchase a third-party product such as WS_FTP Server (which carries a premium if you need SFTP functionality).

As for CLI clients that support automation, no matter the platform you use, there are plenty to choose from. For SFTP, on Unix just use the sftp command and on Windows, try Putty's excellent psftp.exe. For FTPS, I suggest you try cURL which is multi-platform on Unix and Windows.

So, to conclude: SFTP if using a Unix server, FTPS if using a Windows server.

In my case, I'm going with FTPS.

O.

Gabriel Consulting Group survey on Oracle and HP-UX

2011-04-13T12:21:00.000-07:00

As many HP-UX admins still read this blog, I thought I'd post this. GCG is running a survey to have some insights on what you're thinking about Oracle's decision to stop developing products on Itanium, and Oracle in general:

http://survey.gabrielconsultinggroup.com/limesurvey/index.php?sid=73634&lang=en

I got this link from an article that Dan Olds posted on The Register.

I takes maybe 10-15 minutes to answer the survey and I think it is worth it, as the results will no doubt end up being published by HP somewhere down the road. Even though I'm sure they're independent, the questions and tone of the survey are not, er, I'd say, totally objective. I answered it not as an HP-UX admin (which I'm no longer), but as a systems architect for an enterprise that runs a mixture of HP-UX, AIX, Solaris, and Windows. So I tried to stay unbiased. You should do the same.

O.

PI DataLink Server and Excel Web App: A wedding cake dilemma

2011-04-06T11:24:00.001-07:00

photo by Michael Prudhomme

The project I'm attached to has in its list of technical requirements the installation of Excel Web App (EWA) along with PI DataLink Server (DLS). It is not clear what the customer intends on doing with it, but my guess is that it will be used to show PI data to end users using a web-based interface.

The DLS manual describes four user roles, two of which are directly related to Excel: a publisher and a reader. This pretty much resumes what it is designed for: some people, who are PI experts, develop and publish workbooks using a real Excel with PI DataLink, while common end users read them, using a browser. This apparently read-only nature of DataLink Server (which I need to confirm) is an important one, as from my understanding, it is positioned to be a simple web reporting platform.

I've recently had some time to experiment with these web features to try to predict what the developers will end up doing in the long run. I also had the hope of leaving the marketing pitch to marketers and finding what were the real advantages of going in that direction instead of sticking with a deployment based on the standalone Excel application.

I'm not an Excel whiz kid, and I'm even less a SharePoint expert. That being said, after a few mishaps, I've managed to make a proof of concept with DLS and EWA using the most dummy report I could build with my limited knowledge of PI:

The wedding cake dilemma

I'm glad to announce PI Datalink Server works as designed within the Excel Web App. However, when playing with it, I couldn't stop thinking about a three-layer wedding cake. Why? Because you see, pitting EWA against standalone Excel is like comparing that wedding cake to a slab of brownies. Both will easily feed dozens of people, but the wedding cake will take longer to assemble, be more expensive, and each layer will need to be supported by the one underneath (I also think the brownies will be tastier, but that is beyond the scope of this article).

I had no doubt that the combination of the three layers consisting of DataLink Server, Office Web Apps and SharePoint involved lots of other subsystems too. This presentation done by Microsoft last year confirmed my suspicions. IT Operations would have a hard time supporting all that if the dependency hell between all those subsystems ever hit the fan. Understandably, as a systems architect, I wasn't very comfortable in greenlighting the use of DataLink Server at first glance. Is it safe to assume that if an architecture is made like a wedding cake, it better offer something big in return or else it's not worth it?

I think that in that particular case, it will be worth it if your experts use PI DataLink a lot and they need to deploy ad hoc reports quickly to a controlled (i.e. not massive), read-only audience.

Using EWA and DLS for ad hoc reporting

The ugly sample report pictured above is what I would call an ad hoc report: It's a quickie, made in a hurry to fulfill an unexpected business need. These can be done in a matter of minutes and published as a web spreadsheet to be consumed by users who have no technical knowledge of PI. There is no need for these users to have Excel on their client, as everything runs in a stripped-down version of Excel straight in the browser. This could prove extremely useful when dealing with mobile devices in the future as I don't expect Excel and DataLink to be running on the iPad anytime soon.

Furthermore, since you don't have a bunch of standalone Excels running around in the wild, you don't have to:

Ensure all users have the correct Excel version;
Install PI Datalink on each of these Excels and maintain this installed base which can be substantial;
Deal with the security hassles of opening up network access to the PI infrastructure to every laptop in your WAN (you only need to open it to the server running DLS).

Interesting. One might expect a lot of reports to be created that way.

Preventing ad hoc report sprawling

Now comes a question: what do we do to prevent "ad hoc report sprawling"?

I think that ad hoc reports should be deployed to VIP users as prototypes, until the time comes to move to something better if they ever need to reach a wider audience. By "something better", I'm talking about a dedicated reporting system such as Crystal Reports for the kind of reports that pull data not only from PI, but also from AF and other sources. The kind of reports that are read daily by people who make business decisions based on their contents. The kind that end up on a printer, to be read to/by upper management.

These official reports should still be designed, deployed and stored on a dedicated platform. Why? Because:

EWA and and DLS have their limits; my understanding is that they can pull out data only from PI points, not AF (on the other hand, there are ways to combine web parts with DataLink Server, but I'm not good enough to try that out);
I also have a feeling that using EWA as a reporting solution might cause a performance impact both on your SharePoint and PI infrastructure as nothing will prevent John Doe from pressing CTRL-ALT-SHIFT-F9 (in caps, of course) all the time to be updated on the second. It's much slower on DLS than within the real Excel, so I think there is a performance hit. This impact needs to be evaluated, and thus why I talked about a controlled audience above.

Conclusion

The possibility of deploying ad hoc reports to read-only users who don't need to have Excel at all is the main advantage I've seen up to now to deploying an architecture based on PI DataLink Server, Excel Web App and SharePoint. However, as this might be a complex solution that your IT Operations will need to take care of in the long run, you need to be sure you really need it.

O.

Am I off the track on this? Have any comments? Please post below and I'll be glad to write an update to this article.

Oracle dumps Itanium. -1 for HP-UX, +1 for Integrity

2011-03-23T18:07:00.000-07:00

I'm no longer involved with HP-UX and Itanium but I've been an HP-UX geek for 10 years, and some of the readers of my previous blog (aptly named Technocrat-UX) were following me specifically for my comments on that market. I'm no analyst, but anyway my advice is free so here are my thoughts on this story which was unraveled today.

I'm not surprised about Larry's decision to stop development of Oracle on Itanium. It was just a matter of time before Oracle would try different stunts and measures - any measure - to try to save their SPARC platform and lock in customers. This one is a desperate measure indeed. I know a lot of SAP system administrators who won't be delighted to learn this. Some have been claiming for a while that Oracle is the new CA, and this couldn't be more true. If I was still an HP-UX admin, I'd be directly targeted by this decision. But I wouldn't say "Fuck HP". I'd say "Fuck Oracle", big time.

How does that look for HP-UX? Without an enterprise RDBMS, not good. Not good at all. But all is not over for Integrity. Rob Enderle picked up the story today and revealed some interesting information that was shown to him under an NDA:

Unfortunately for Oracle, HP just had a massive analyst event and in the server break-out had showcased under NDA the future for Itanium in new products. While I can’t share that future, it is NDA, and for those of us in the session there was no doubt that Itanium is going to continue. More importantly, the changes being made should make it vastly more cost effective than anything Oracle can announce on SPARC. You’ll understand what I mean in a few months, or if you have an HP relationship, ask HP what I’m talking about and you’ll have a big “ah hah” moment. But you won’t be able to share it any more than I can.

I've been thinking about his statement, trying to read between the lines. Here is my own speculation of what may be ahead. Note here that he's talking about the Itanium platform, there is no mention of HP-UX anywhere in his post. What can I make of this?

Here is what I know:

1. Enderle says that the new platform will be "vastly more cost effective than anything Oracle can announce on SPARC".

2. I bet that Microsoft are probably annoyed by Oracle (sorry, no time this evening to find an article to back this up).

3. I've learned from a trusted source (without signing a CDA) that the DL785 will retire and only the DL580 and DL980 series will be left. Which anyone can deduct from HP's web site, there is currently no G7 offering of the DL785.

...and here is what I predict:

1. Microsoft will be looking for an enterprise-level platform to harness MS-SQL, which has become over the years the "other" enterprise RDBMS.

2. All my current architecture projects are based on MS-SQL and I've learned a little about its licensing recently. This is software priced per CAL or per processor/socket (your choice), and each processor costs - well, costs a lot of money. Customers with thousands of users will want to get the most bang out of every processor they use. Does that sound like a return of Windows and MS-SQL to the Integrity platform? Hard to say if MS keeps a similar pricing with Tukwila's four cores, but it is possible. Microsoft could offer this as a vertically integrated solution pitched to customers who are currently relying on Oracle (or IBM's) solutions.

3. To cater to these mid-size Microsoft customers who aren't interested in blades (let alone Superdome 2s) for a reason or another, HP will release something like the rx5800 and rx9800 which will be based on the industry standard components of the big Proliants - namely, the 580 and 980. These are reliable and huge workhorses. By swapping only a few components, they'll save plenty and be able to offer these servers at a small premium over the x86 versions.

4. As for HP-UX? While the outcome isn't clear, I frankly don't expect Oracle to really stop releasing their RDBMS on HP-UX; I'm sure there are plenty of customers left and some bean counter at Oracle will realize the high risk of loosing them forever if a migration to SPARC is shoved down their throats. If they keep their stance, their loss. I haven't cared about Oracle for a while now.

These are my thoughts.

O.

Building a PI Lab for SharePoint 2010 and Excel Web App (Part 2)

2011-03-16T16:13:00.000-07:00

Didn't have as much time today, but here are my findings:

1. A standalone ProcessBook installation requires a huge dependency package. So if you intend on deploying this on hundreds of PCs, better think of it. It should be installed on a TS, or deported using Citrix. Of course, for large scale deployments, it is better to plan using Web Parts over ProcessBook...

2. PI Datalink 2010 is supported on Excel 2010 32-bit only. It's documented in the Release Notes but as usual I didn't RTFM, and had a 64-bit installation. The PI-DL installer doesn't tell you anything about this, and it results in the add-in not being installed in Excel as it should be.

3. If SharePoint was a blind date, I think I would stay polite, possibly pay the whole bill, then say goodbye with a kiss on the cheek... i.e. leaving all options open while sending a clear message at once. Web page editing is sluggish, which is unacceptable in 2011 for a web application and I don't care how slow the back-end is. And it is hungry as hell. Even a vanilla installation revealed a lot of clunkiness with some "oops" error messages and dead links. Bottom line is I don't like SharePoint as of now, but I'll have to get used to it.

4. The drawing on my first post isn't right. I'll need to fix it in an update.

O.

Building a PI Lab for SharePoint 2010 and Excel Web App

2011-03-15T12:38:00.000-07:00

I finally had some spare time today. No meetings, first time in a while. I had the chance to continue building my PI Lab and try to see how I can use SharePoint 2010 and Excel Web Services. My intentions are to harness the Web App version of Excel as much as I can, so that users can get PI data without needing Excel or PI DataLink at all. Developers will also be able to publish some pre-formatted reports on SharePoint.

I reinstalled almost everything from scratch to start afresh. It's not completed yet but here's what it should look like. Maybe it could inspire some of you. As a bonus, it shows the interactions between various OSISoft layers, which is not always clear to a neophyte such as I.

This is by no means what we'll have in production, everything has been installed with standard and typical (i.e. non-secure) settings and it is used to test and evaluate the interaction between Microsoft and OSISoft components only.

At the bottom, you have your honest-to-goodness vCampus PI System. It doesn't have any interfaces, but can provide some mock points such as good'ole CDT158 and BA:TEMP.1.

In the middle, a default dumb SharePoint 2010 server has PI-SDK feeding PI Data Services, and PI DataLink Server 2010. They, in turn, feed Web Parts and Office Web Apps (which, in fact, only has a usable Excel).

On the top, two terminal servers used to house the various clients. Why? Because The Man only installs and support IE6 and Excel 2003 in our PC environment. Nothing more. So I basically need to set up VMs which have the clients we need: ProcessBook, Excel 2010, and IE8 with the Adobe SVG player. One replicates the system which will host the clients used by our Joe Average User, which is web-only, and more worthy "Power Users", which can use Processbook if they like. Developers have their own environment with Excel 2010 from which they can publish directly on SharePoint. Hint: our production will go along the same way, using Citrix to deport the required applications. Screw The Man.

That's what is on my radar for now. I haven't finished making all these things work but when I'm done, I'll update and correct this post.

O.

My build of iperf is back

2011-03-09T16:11:00.000-08:00

***********************************************************
Update: You can get my build of iperf on Windows here:
http://www.mayoxide.com/iperf
***********************************************************

Wew! My post concerning my own build of iperf for Windows has proven to be the most popular of my blog up to now. And the file got downloaded so many times, it actually busted the download limit of mayoxide.com (my domain) 12 hours before me leaving on vacation overseas. I had to renew my hosting package in a hurry, as I was keeping it grandfathered to 2004 transfer quotas - 1 gigabyte, anyone? Now I have 10 times more and the download link is back (thanks to pjmco.ca, my hosting provider, pay them a visit; they're great).

Such a response shows me that there is a need for a more "official" page for the Windows build of iperf, which is a matter of 30 minutes of handwritten HTML. I'll be working on this soon, I promise. In the mean time, my earlier blog post is a nice placeholder.

O.

First IMS user group meeting

2011-02-23T16:22:00.000-08:00

Today, Cooper Power Systems held their first IMS Users Group teleconference. Building a user community was in their plans, and I am glad to see it slowly coming to a reality. I'll spare all the technical announcements because I'm not sure what can be made public. However, one of the things that stood out was a need to build up an online presence so that users can exchange ideas and solutions. Someone suggested using Google Groups but they already have a forum which they control, so my guess is that it will probably end up there.

Normally, a user group should be set up by users and remain independent. Linkedin is a good place to start one. However, as far as niche products go, IMS is as specific a product as it can get. That being said, Cooper EAS obviously doesn't have a pool of thousands of customers on which a user community can be easily built and communities need leaders. So it is better under the circumstances to keep the community leadership in their hands.

I believe that building a user group based on an unusual technology isn't necessarily a challenge... it's a feature that can be harnessed. Many specialized products have thriving user communities. For example, when I used to spend 100% of my time on HP business systems, the NonStop guys consisted of a fair crowd, tightly knit together with dedicated conferences and strong leadership, while HP-UX didn't benefit from such a community momentum (I was trying to change that slowly, but my career path steered me elsewhere). That hit me. Some people take pride in working with one-of-a-kind, high quality systems. Cooper encourages papers, so I expect some of their customers to participate in the best way they can in that manner.

Speaking of papers and conferences, Cooper EAS is planning a special IMS and SMP track at their next EAS conference. We'll see if more people can come than last year. And let's hope I'll be able to be one of them.

O.

P.S. Someone asked me if I had plans to continue my entries on IED integration in IT. I have a rough draft for part 2 ready, but didn't find the energy to finish it yet. I'll probably end up doing it eventually. I have to work from home on this on my own time, so bear with me.

Playing with PI Web Parts

2011-02-09T12:49:00.000-08:00

I've spent some free time at work setting up a basic SharePoint 2010 Server in order to test the functionality of OSISoft's PI Web Parts .

This is my first experience with SharePoint and PI Web Parts, and I'm not very impressed up to now.

Here are my thoughts:

SharePoint is terribly slow

Sharepoint is unbelievably slow. Granted, I have a small VM, but I'm using the default configuration, no bells and whistles. Usability suffers from extreme sluggishness, and I wouldn't want to design pages in SharePoint full-time as my objective in life is to keep a shred of sanity. When I use Web 2.0 apps, I expect Web 2.0 speed. Not cgi-bin-like responsiveness.

Web Parts rely on SVG (and, soon, MS Silverlight)

The "interesting" graphical Web Parts rely on SVG to generate graphics, like the PI Trend Web Part pictured above. Using SVG is not a problem per se, as it is a lightweight format which gives very usable results -- the trend graphics are live, and you can hover over parts of the graphic to get more info dynamically.

However, Internet Explorer is notorious for not supporting SVG natively. Furthermore, there is no alternative to configure these web parts to make static image files... so if you have a locked-down desktop with IE8 (or, even worse, IE6) and have no SVG viewer, you're fucked. The only solution consists of installing an old viewer from Adobe that has not been updated since 2005, and has been unsupported since 2009. Unacceptable in an enterprise environment. Calling the Man to install such a viewer on my laptop would result in the Man saying "no" and laughing his way back to the bank.

What further exacerbates me is OSISoft's commitment to migrate to Silverlight in the future. Deploying Silverlight will be another complex task in a locked-down enterprise environment. Of course, Microsoft already knows how to deal with this: I think they can't bundle Sliverlight with Windows 7 due to antitrust issues, but they will find a way to attach it into the next version of MS Office. So when the Man will decide it's time to upgrade the desktops from Office 1981 to Office 2030, we might get Silverlight as a bonus and be able to see some PI Web Parts. Woo!

In a world where many intranet sites are hardwired to IE6, and nobody wants to risk updating anyone to IE8 (let alone IE9), SVG and Silverlight are critical points that need to be taken care of.

O.

Fun with OSISoft's PI

2011-01-27T11:01:00.000-08:00

RRDtool has been a savior as a no-frills data historian, giving folks a free and high quality toolset to store and present time-series data. So when I first saw OSISoft's PI last year, my reaction was Oh no, not an "enterprise" version of RRDTool!

Many know that I'm very critical of expensive software that labels itself as "Enterprise", especially when it ends up not working as expected, assuming it works at all. When paying in the six digits for software, one expects it to at least be good, and provide value. But many times, it's a half-baked product, glued by imbeciles who are led by idiots, which is then sold by a dim-witted sales team who think their target customer base is a bunch of morons. That might be harsh, but it's the way I've felt for a long time with some software companies and their peddlers.

Needless to say, when I started architecting systems based on OSISoft's PI, I had low expectations.

It turns out I was wrong. PI (Plant Information), as its name implies, has roots deep into the Plant, not in the Enterprise. This is a significant difference that must have influenced its design a long way. I can't pinpoint exactly what makes it special, but at the base it's simply elegant and I feel perfectly comfortable when playing with their tools. Everything there looks like it has been put in it for a real purpose, not to show up well in feature comparison chart. A lack of buzzwords like "cloud computing" and "agile enterprise" makes me feel right at home.

There are some problems with OSISoft's suite of products, the top one in my list being the lack of good documentation. The documentation is either extremely high-level or very technical, with few images, there is no "in between" for someone like me who only needs to make a quick proof of concept then leave the implementation details to the IT team. Their vCampus subscription process is also hard to work it. And I'm getting increasingly frustrated with their support site which makes downloading each piece of software or document a tedious, three-step task.

But all these issues went magically away when I was able, in 15 minutes, to set up a mock operator screen for a mock reactor:

I like it when software lets you do things quickly, in a natural way. PI is exactly that. It comes built-in with a few data points, that can help someone quickly assemble a prototype to see its capabilities. I did this screen in PI Processbook. I can use it to show some of my colleagues , at a glance, what PI is all about; an image is worth a thousand words.

Sharepoint is currently being installed in another VM, and my next step will be to try to present data with PI Web Parts. I'd like to see if that Processbook screen can be converted as easily as OSISoft says it can. Geez, Sharepoint has just finished installing. Time to go see if I can pull it off in another 15 minutes.

O.

Iperf 2.0.5 for Windows

2010-12-13T17:58:00.000-08:00

***********************************************************
Update: You can get my build of iperf on Windows here:
http://www.mayoxide.com/iperf
***********************************************************

The original site that offered a Windows binary of Iperf 1.7.0 has been useful to me many times in the last few years but unfortunately, that page is no longer valid and I didn't find any recent 2.x version of Iperf compiled for Windows.

You can get my build of Iperf 2.0.5 for Windows here:
http://www.mayoxide.com/iperf/iperf-2.0.5-cygwin.zip

I've quickly tested it with Windows XP and Windows 7 and it seems to work okay. It was compiled under Cygwin, so don't be surprised if it comes with a (huge) .dll file.

O.

Integrating IEDs inside your IT infrastructure - Part 1

2010-12-01T17:32:00.000-08:00

Image: jscreationzs / FreeDigitalPhotos.net

Introduction

This should be a multi-part blog series that will introduce you to the control and data acquisition of substation-grade IEDs (Intelligent Electronic Devices) all the way to the data center. I'll write it as I have the time. If you have any comments or corrections, feel free to leave me a note.

Being a systems architect, not a control engineer, the emphasis of my writings will be on the IT side. I don't have any deep knowledge in the control field, having been exposed to these technologies only recently. When I tried looking for some information on the internet, there wasn't much to start with except Wikipedia entries that didn't fit together linearly. May this series help anyone who happens to follow my footsteps.

Part 1: It all starts at the IED

Wikipedia defines an IED quite well:

An Intelligent Electronic Device (IED) is a term used in the electric power industry to describe microprocessor-based controllers of power system equipment, such as circuit breakers, transformers, and capacitor banks.

Okay, so let me make my own definition, and it's all IT folks like me ought to know:

An IED is either a sensor that returns data, a control device installed in a substation, or "something that impacts the grid".

All these devices need to provide an interface to communicate back their data, and also some means to be configured by a control engineer or technician. A lot of this stuff have traditionally been relying on the serial RS-232 point-to-point interface (unless you're under 25, you've probably heard of RS-232 before; it is the standard 9-pin or 25-pin serial port on PCs on which you can hook up serial devices). Many IEDs also rely on RS-422 and RS-485 networks which have more features than the basic, low-speed RS-232: RS-422 is a "multi-drop" network, where one sender can be heard by up to 10 slave receivers, while RS-485 is a "multi-point" network that allows up to 32 arbitrary connections.

The upper layer protocol that IEDs use seem to be, a lot of times, Modbus or DNP3. Another interesting fact is that clock synchronization with these devices is often done using the IRIG-B protocol which has a lot more history than the usual (S)NTP protocol many network administrators are already familiar with. For one, IRIG-B can work on serial interfaces.

Many IEDs have recently started to rely on ethernet media and routable TCP/IP networks instead of point-to-point communications. TCP/IP can channel DNP3, Modbus and others, but the IEC 61850 protocol is also slowly becoming a leading standard. Using TCP/IP basically enables you to access the device from anywhere -- a nice feature, but a double-edged sword nonetheless: introducing a routable network in the substation, and hooking up IEDs to it, brings up many obvious security issues that weren't there before.

Remote control and data acquisition from these IEDs could be done using plain modems and dedicated POTS lines. But now that this telecommunication architecture is slowly moving towards routable, high speed IP networks, new ways to remotely manage the substation become available using commoditized IT technology. Many analysis and archiving possibilities spring up once this scattered data is centralized, which involves software that will be unheard of to many IT admins.

We see here that while it all starts at the IED, that IED needs a way to send back its data to the data center. So what once used to be the sole business of the "Control Guys" is also becoming one which also requires some assistance from the "IT Guys".

And yes, as I admitted firsthand at the top of this article, I'm an IT Guy.

O.

----

The next article will describe how to concentrate a bunch of IEDs together, and securely send their data to the data center.

Word clouds: a nice way to review your documents

2010-11-05T12:09:00.000-07:00

I've recently discovered on Kieren McCarthy's blog a picture of a word cloud. In essence, it's a cloud of words where each word occupies a space relative to its height.

The site Wordle lets you create such clouds for free, with a terrific design. I've pasted my whole architecture documents on this site to check out the results. Here is an example of what it results in:

In the above cloud, a few elements stand out: Acquisition (Data collection), PI, Données (data), PI-SMP, and points (tags). Anyone familiar with PI or Cooper's SMP will easily guess what that document is talking about.

Such clouds can be used to make a pro-looking cover page (or back page) for any official document; in my case, I just hang them in my cubicule. One can also use word clouds hem to size up a document, any document, in a quick glance: By looking at what words have the most weight, it is possible to have a quick idea of the document itself. And yes, it does look really nice.

O.

Thoughts on HP-UX, AIX, and Integrity

2010-10-22T17:51:00.001-07:00

Those who know me personally and have been following this blog know that I ditched my HP-UX admin career to become a systems architect in another division at my company. The reasons for this were mostly personal, although maybe 10% of my choice to move on was due to other reasons. Bottom line is that even though I miss the technical side, I'm glad I've made the switch to the "clouds" and became an architect.

I'm now mostly attached to stuff made by Cooper Power Systems and OSISoft, which is fine. Expect the content of this blog to switch to these two vendors over time. HP still has a place too, as I've spent so much time using their products. But here we're not a HP shop; we have some HP products of course, but also IBM, EMC, and other vendors on the floor.

Some HP reps came in this week to show us what was new this year. While I used to be very fond of HP since relying on a limited number of manufacturers was part of my one-sysadmin-for-all strategy, I must now step back and try to be as objective as possible.

That being said, there was the usual presentation about let-HP-shove-its-converged-infrastructure-down your-throat, then a half-hour presentation on Integrity systems (among others) during which I kept my mouth shut. I couldn't help myself thinking what's the future of this great platform. All talk about running Linux and Windows on the platform is now gone, which is a good thing, as we all know it can no longer be an option. Only HP-UX and OpenVMS are left. The BCS rep, knowing my division runs RHEL and AIX, told us to "please challenge us", meaning that we should evalutate HP-UX as a contender as much as possible.

I've worked with HP-UX for 10 years and love it. But, in my opinion, AIX is roughly equivalent. Although I haven't been administering AIX systems for 10+ years, I know enough that it is a mission-critical OS backed by a manufacturer who won't let my company down, same as HP-UX. If we were running Solaris, things would be very different. But for the moment, as an architect, I consider both HP-UX and AIX as equals: these are the last true "Enterprise" UNIX options available.

So, what's left for the Integrity platform? Not much. As AIX is the same as HP-UX at a glance, I can only think that Integrity is the same as POWER systems, give or take. So it's not a bad platform per se, but a niche one for sure.

It's too bad HP lost the big bet they've made on the Itanium. I still remember all the talk about Merced when I started my sysadmin career in the late 90s. Things didn't turn out as expected for sure, but even if HP had stayed on the PA-RISC bandwagon, they would be at the same spot they are right now. That doesn't mean HP-UX has no future - the OS still has a big place in my heart. I can only hope HP will eventually port the HP-UX kernel to x86, or make an HP-UX ecosystem and support infrastructure revolving around a Linux kernel. This is probably the best thing to do to this operating system to ensure its long term viability.

Have any comments? Please leave me feedback.

O.

Announced: An official Cooper EAS web forum

2010-10-20T08:14:00.000-07:00

I've been informed that Cooper EAS has to intention to build a community around their grid automation products such as IMS, a process to which I'll be glad to contribute when the time comes.

A first step into building a community is, in my opinion, to set up a web forum. To my surprise, EAS just announced one today. It is available here:

http://216.17.94.116/vbulletin/index.php

There is no DNS name yet, but I'm sure they'll fix this soon. Also note that access is limited to current customers only.

I think that a critical mass of their current customers is mostly interested in DR, which means that I don't expect to see many grid automation subjects in that forum to start. But it is a really, really good first step.

O.

Cooper Power Systems EAS, Stuxnet and control vs. IT

2010-10-05T18:04:00.000-07:00

I'm currently at the Cooper Power Systems EAS (Energy Automation Solutions) user conference in Minneapolis. I don't know much about DR, AMI, Smart Grids and such, but had to go there to at least learn the basics and be able to do a better architecture job.

I'm almost ashamed to admit that I'm an "IT guy". Seems that most who work in control don't like IT and I can't blame them. Many control systems are increasingly being linked to ethernet and IP-based networks, along with remote and consolidated interfaces, and this brings many challenges which only IT can address. Enhancing security of these systems is especially important, and many control users don't seem to view security as that important.

I've had an interesting chat with EAS's security guru about the Stuxnet worm. Many technical details have been leaking through Slashdot and elsewhere for a few weeks, thus I won't speculate on its possible origins or intents. But the bottom line is that Stuxnet does exist, and it is a staggering proof that even though its engineering is not within the reach of just anyone, SCADA systems are not immune to security threats.

Like we IT people have been disgusted by the security guys for years now, it's now the turn of control people to have to live with IT. Nice threesome. Looks like I'm stuck in the middle position. FML.

O.

REST: "understandable" web services

2010-09-28T12:33:00.000-07:00

Most of my experience as an ex-sysadmin with web services is with SOAP. And I couldn't speak about experience; the only thing I had to do back in these days was to setup Tomcat and Axis to host mysterious "SOAP" applications.

As soon as I started reading the manuals of Tomcat and Axis at each of these installations, I quickly ran away -- not only couldn't I understand anything, but these docs all started with the idea that whoever was reading them had a strong knowledge of Web Services and/or Java. Which, of course, I didn't.

At a high level, I understand what SOAP is. But since I've jumped in system architecture, what I've been hearing recently is this:

"We're modeling our web services architecture on REST".

Oh no, not a new buzzword. Wasn't SOAP complex enough? What the heck is REST?

Fear no more. It turns out REST has been out there for a while. A long, long while. On the web before many didn't know what the web was.

Anyone who knows the basics of HTTP will understand REST quickly. REST is not a heavy, XML-based protocol as in SOAP. It's a way of doing things the simple way, using straight HTTP to transfer information in - sorry if I interpret things little too much here - an ad hoc manner. No freaking XML and standards. Thus, your web browser is a REST client. If you want to go a little more deeper, another example would be Amazon's S3 which is a more complex service based on REST. In fact, if a service doesn't use SOAP, it probably uses REST.

REST is how anyone with a good-enough basic knowledge of HTTP, but doesn't give a shit about XML and layered protocols, will natively think web services should work. The way things should be. I'm hardwired to think this way, and SOAP always had me thinking that it was way overkill for a lot of uses.

I'm currently reading the following tutorial:
http://rest.elkstein.org/

Go read it, it is very well written.

Also take a look at the comment page here. Dr. Elkstein gives there a rough comparison between RPC, SOAP, REST.

O.

Building a low-power FreeNAS Server: Part 3

2010-09-17T07:55:00.000-07:00

My FreeNAS Server has been up for a week.

I assembled everything in my parts list and I'm glad to report that all components worked as predicted. I did not have any surprise.

Assembling a homebrew computer is a nice activity for kids. My 7-year old helped me plug all the headers and assemble the case. Counting all the explanations I had to give to him, we were over in a little over 30 minutes.

I then used m0n0wall's physdiskwrite to write an embedded amd64 image of FreeNAS directly on a leftover 1GB flash card I had from an HP tradeshow. Then I plugged it on my USB-header-to-USB-plug thingy directly inside the case. The two hard disks are not used to boot FreeNAS at all; their purpose is only to hold data.

I happen to no longer have any computer screens; I only have laptops at home. So I hooked up the PC to my flat screen TV to configure the BIOS and do the initial FreeNAS configuration.

There are a two things about using a mini-itx desktop board I don't like:

The Intel Desktop BIOS does not support console redirection (at least I didn't see any mention of this anywhere) so you need to configure it the old way with a screen and a keyboard;
The built-in video card, needed by the BIOS, cannot be deactivated. Its video memory is, naturally, shared with the system memory and the lowest memory footprint is 128Mb. That ended up taking 10% of the 1Gb of RAM in my server. Not a problem since I'll never need that gig, but a hassle anyway.

But for the price I paid that board, these were things I was ready to live with.

Now I've got a bootable system, and it works well. My next task is to ensure that it consumes the less power possible. I already configured hard disk spindowns but I'll try to see if I can do something with wake-on-lan. More to come later!

Adding a DynDNS client to OpenWRT's LuCI

2010-09-14T16:16:00.000-07:00

OpenWRT is terrific. The more I play with it, the more I like it. Even though I used to be a UNIX administrator (and a good-enough one, I think), I prefer using the LuCI interface as much as I can, keeping the CLI for repetitive tasks or debugging. LuCI is completely modular and lets you add packages depending on your specific needs. This is a good thing; it removes clutter from the interface and saves some precious space on your flash.

As an example, the following is a quick procedure that shows you how to add a DynDNS client functionality in LuCI.

Log into LuCI and switch to Administration mode.

Go into Overview -> LuCI Components.

In the Available Packages panel, install the luci-app-ddns package.

A Dynamic DNS menu entry will appear automagically.

Inside this menu, you can then add your DynDNS settings as usual. It might take a while before OpenWRT updates your status; be patient.

O.

The RNX-GX4, a well-priced Broadcom router

2010-09-03T20:31:00.000-07:00

Remember how I blogged about OpenWRT and Broadcom routers a few months ago? Turns out I've been looking for a new router and I found one which did what I wanted at a good price:
I introduce you to Rosewill's RNX-GX4. This is a rebranded Netcore NW618, a chinese router which is not available this side of the Pacific. What makes this router special is that it is tested with, and officially supports, DD-WRT at a reasonable price -- I got mine on sale, at almost a quarter of the price of a WRT54GL. It has the same quantity of RAM and flash as the WRT54GL, but a faster CPU. It's not a straight WRT54GL clone; there are some differences such as a serial flash chip which isn't as common as parallel flash. But the patches have been submitted to the various distributions, and up until now, it has been working well.

What further sets it apart from many other low-cost routers is how Rosewill openly brags about how it runs a BCM5354KFBG 240MHz CPU in the technical specs online, and even on the back of the retail box. When you see things like this, you know it is made for geeks. I also appreciate that the antennas can be removed. I don't intend to use the wireless radio on this device, so I'll put them out of the way.

I've started playing with it tonight. My intention is to replace my m0n0wall PC with a RNX-GX4 running OpenWRT. While I like m0n0wall very much, running it on a generic PC takes some real estate and electricity, and using a smaller devices would be a better fit. I've investigated purchasing an Alix board, to keep running M0n0wall, but there are many, many times the price of the RNX-GX4 so I decided against using one.

O.

Building a low-power FreeNAS Server: Part 2

2010-09-03T11:05:00.000-07:00

Part 2 of my series will be a shopping list of the materials I've picked to build by FreeNAS server.

Everything starts with the motherboard. I was looking for a low-power Mini-ITX board that had a built-in gigabit ethernet controller and I've decided to pick an Intel D510MO. This is a desktop board based on the Atom processor which is cheap and low power. There were third-party Atom boards that were maybe 10-15$ less than the Intel-branded one, but some reviewers complained on excessive heat and I didn't want to have heat problems. I was initially looking into the VIA C7 platform but it can't beat the Atom in terms of performance. I still don't know if the intel BIOS built on this board can work on a serial console. That would be surprising, but I will keep you posted.

I purchased two 1Tb Seagate 7200 RPM SATA disks -- nothing special here, except that 7200RPM was an imported factor for me. I want these disks to be as fast as possible when I'm copying large amounts of data.

For the case, I picked a cheap MicroATX one. Why MicroATX? Because Mini-ITX cases are expensive, and usually can't fit more than one hard disk. I selected a R102-P from Rosewill which happened to be 20$ on Newegg. That case is not only cheap, but it can hold 4 hard disks (which doesn't seem too common on MicroATX cases) and the front is very well ventilated with a lot of air holes right in front of the disks.

The case doesn't come with a power supply. And I didn't want to; since I was looking to be the most power-efficient possible, I picked a 80-Plus 250W power supply from Sparkle. 250W for an ATX form factor is also not that common, but I was really looking into getting what I need and not more -- an idling 500W PS consumes more power and one rated at 250W.

The last thing I bought is a gizmo made by Koutech - an adapter that converts a 10 pin USB header into a standard USB plug. Using this, I can put FreeNAS on a small USB key, and plug that key directly on the motherboard inside the case -- no dangling key outside the box. The motherboard doesn't have an IDE header, so I couldn't use a more common IDE to CF card converter. We'll see how this goes.

That's it for now. I'm currently in the process of having all this shipped to me and I will soon see how things work out.

Building a low-power FreeNAS Server: Part 1

2010-08-30T09:58:00.000-07:00

I've been looking into having a small file server in my home, to store my photographs and iTunes library. The most important aspects of that file server are, in order:

Low power requirements: It is on 24/7 and I want don't want to consume too much power
RAID-1: I want my data to be protected in case the hard disk crashes
Low cost
Good performance
Expandability: Nice features such as a bittorrent client are a plus, I want to be able to experiment with DLNA in the future, so the "hackability" factor is important.

There were two contenders here that met most requirements: DLink's DNS-323 and Synology's DS-210j (Buffalo has some too but they are hard to find online). They each had a drawback: the DNS-323 is reported by many to be subpar in terms of performance, while the DS-210j is expensive.

So I decided to build my own NAS instead. It will most probably be based on FreeNAS, assuming it is hackable enough to my taste. The overall price is below the DS-210j, and I expect performance to be up to my expectations. Low power being paramount, I had to hand pick all the components and my next posts will detail what I've chosen, and how I'll be building it. I'll try to put some nice pictures.

O.

wuauclt.exe and svchost.exe taking memory on XP

2010-08-19T18:12:00.000-07:00

Update 2010-08-21: The thread here indicates that Microsoft is investigating this as a priority 1 issue.

Normally I don't post PC and Windows-related stuff but there are not many recent posts on this August 2010 issue, so here it is.

I support my own PCs and those used by the extended family. For a few days a staggering issue has been happening. When configuring a Windows XP PC to use Microsoft Update (rather than the good old Windows update), svchost.exe and wuauclt.exe take so much memory that a low-memory PC (512Mb) will start swapping enough to freeze the whole system. I saw this happening on SP2 and SP3.

Yes, 512Mb is not a lot, but before calling me stupid, note that 512Mb used to be the standard configuration a few years ago for many low-end systems and it is officially supported by XP. That bug makes any low-memory PC unusable.

You should note that installing Microsoft Security Essentials switches any PC silently from WU to MU. That's how I introduced the problem and noticed it the first time on a laptop. I was about to reinstall it but found out that there have been some user reports here and here. Microsoft has not released a patch yet but I am sure many corporate users have stumbled on this problem and reported it officially.

The workaround in the mean time consists of connecting to Microsoft Update and choose to stop using Microsoft Update. The PC will revert to using Windows Update and everything will be working normally again.

O.

My top 2 articles on the internals of broadcom-based routers

2010-07-21T16:37:00.000-07:00

Embedded devices are energy efficient, and their limited memory and storage present some satisfying challenges. How have I missed this for all these years, I don't know, but it is time to play catch up. For a few weeks, I've been spending some of my free time reading on Openwrt and the WRT54GL and other devices of similar design.

Of course, flashing a custom firmware can be daunting enough for an average user, but an average geek like myself will want to know how exactly these devices work. There is a published book on the subject but upon reading the table of contents, I decided against buying it as I deemed it not technical enough.

So, I assumed there was information somewhere for people like me, info that didn't require me to read source code. And there is, but you have to search for it.

And what are the top two articles I found on the subject? Here they are.

1. First of all, there is a three year-old post entitled "Everything you need to know about Broadcom hardware" here in the Openwrt forums: https://forum.openwrt.org/viewtopic.php?id=11304

It is extremely informative on the boot process of these devices and it finally clarified for me how the CFE bootloader works. I didn't understand the SquashFS + JFFS2 combination details, thanks to this post now I do. You also should read about union mounts if you're not familiar with the topic (I wasn't, HP-UX does not support this!).

2. Then you should read about how VLANs and network interfaces work on the Broadcom platform -- this article is from the old OpenWRT wiki but well done, too. It is specifically for the Asus WL-500g but being a Broadcom design I can only suspect other routers are very similar, if not identical:

http://wiki.openwrt.org/oldwiki/openwrtdocs/networkinterfaces

This article explains why you have a bridge interface on your router. It especially shines in explaining how these routers isolate the interfaces cheaply using VLANs. In fact, there is only one twisted pair interface (eth0), the other one being the wireless antenna (eth2). I was under the impression that firewalls needed different and isolated interfaces, but the VLAN trick lets you do something similar on cheaper designs. And I guess it is good enough!

I might replace my m0n0wall PC soon in order to reclaim real estate in my utility room and save a few bucks on electricity. I just need a router, and disable the wireless radio. But I will NOT be purchasing a WRT54GL. Why? I'll tell you in a future post!

O.

Ah, Mom, I bricked my router!

2010-07-12T11:26:00.000-07:00

Remember I was running OpenWRT on a Bufallo WLI-TX4-G54HP ? After playing with OpenWrt a bit I spent some time setting up a Ubuntu-based build environment to be able to build my own custom firmware.

There's no particular reason for building a custom firmware since the package I want to ultimately run fits on the JFFS2 partition. I just wanted to get my hands dirty. But gasp... the documentation for OpenWRT is sparse and disseminated in four areas: an unfinished HTML manual, an old Wiki, a new Wiki being slowly migrated from the old one, and the OpenWRT forums. One really has to spend some time reading through all of them to understand how everything works, and I've covered maybe 10% of that. And that's okay. I don't pay a cent for OpenWRT and it's a distribution targeted to power-users... If I wasn't looking for a challenge, I'd be running Tomato instead.

Anyway. It turns out that I bricked my device yesterday after flashing that darn custom firmware. I didn't solder a serial or JTAG port, wasn't really looking into doing this, so I couldn't do much to troubleshoot. One thing that no longer worked, and this was supposed to be my planned way out, was the 2-3 seconds it pings on 192.168.1.1 when it's at the CFE bootloader so that I can TFTP a correct firmware. Since it no longer did this, I couldn't do that to unbrick it.

I was about to throw it in the garbage but found somewhere in the DD-WRT forums that some Buffalo routers listen to 192.168.11.1. I tried this address and it worked! Now as to why it decided to listen to 192.168.11.1 while the first time it was 192.168.1.1, I don't know. I must have pressed the reset button 50 times on this thing so who knows what it ended up doing.

Back to square one, I have a router that works, but no custom firmware.

O.

VMware player... what have I been missing?

2010-07-02T09:14:00.000-07:00

Up until now, after all these years, I never considered using VMware Player because as its name implies, it used to be be a player for virtual appliances. That's what it was initially designed to do, but it looks like it can do much more than I thought.

Since I used to be an ESX admin, I don't feel the need to learn to use something else and the only solution I knew to virtualize random stuff for free on my low-end laptop, if I stuck with VMware's offering, was to use VMware Server. I tried Server a few times over the years, but must admit that it isn't really interesting when compared to ESX. The web interface is clunky, and it does not integrate well on a "home" PC.

Little did I know that versions 3 and up of VMware Player now let you create virtual machines at your liking, without needing to use pre-made appliances. I've tried it and it works well, and it is much easier to use than Server's web interface.

I've been "converted".

O.

Looking for my next technical venture

2010-06-24T19:32:00.000-07:00

My regular readers know that I used to be an HP-UX admin until a few months ago, and I threw the towel on my sysadmin career.

I'm left with an interesting-enough IT-related job at work, but I can't blog about it easily. The work I'm doing cannot be easily contributed back in a generic way. And the technologies I'm learning are mostly specific industrial stuff and I don't think anyone would be interested in reading about this. Case in point: Is there anybody here, expect perhaps me, interested in IRIG-B ? Yeah, I thought so.

So I'm slowly investigating where I should spend maybe 2 hours a week learning the ins and outs of a new technical thing, and possibly start contributing to the community in whatever means I can, once I feel good enough with it (which might end up taking years). Since I work with embedded devices at work, I've been getting interested in the embedded market and it looks like my next venture might be with OpenWrt but as I don't have many practical uses for it at home, I'm still not sure.

I just found out that a smaller-than-netbook device named běn NanoNote went into production recently, and it runs OpenWrt. What is special with the NanoNote is that both the software and the hardware schematics are completely open designs. At 99$, it's as cheap as it can get. It's almost the same price as a dumb HP digital picture frame I purchased a few days ago, yet I can do whatever I want with it.

Whatever I want with it... I've been spending half an hour staring at the NanoNote specs and pictures, thinking about what I could use it for. Remember that it's very limited in memory and flash, and you can't fatten it up with thousands of bloated applications. In our iPhone/Android era, I simply don't know what people would think of this. But I'll admit it has one advantage against smartphones up its sleeves: it is cheap, and can be mass-produced unencumbered by patents and legal restrictions (at least, I think so).

One can only hope that someone smarter than me will invent a proper use for this device.

O.

What running backfire on a WLI-TX4-G54HP really looks like

2010-06-21T17:24:00.001-07:00

The 10.03 release of OpenWrt is named "Backfire", which is a cocktail made of the following ingredients according to the login banner:

I happen to have these three drinks at home, so I fixed myself one to celebrate yesterday's experience with this device. Here's what it looks like, with a WLI-TX4-G54HP:

I thought it would have been a three-layer drink, but it gets mixed up quickly in tha glass. I never tried a Backfire before, and it does taste good. Both in the glass, and in the router.

Running OpenWrt on a WLI-TX4-G54HP

2010-06-20T18:12:00.001-07:00

I have an on hand a Buffalo WLI-TX4-G54HP. This is a wireless-to-ethernet bridge. What that bridge does is acually the reverse of an access point: it lets you plug any device that doesn't support wireless, such as an old Xbox, and connect it to a wireless network. I actually used it with my locked-down corporate laptop which had its wireless fuction "deactivated for security reasons". :-)

I was thinking of purchasing a WRT-54GL or an Alix board. The WRT54GL, being a hobbyist device, is pricey for what you get (even on eBay) and I was hesitant. Since I had that Buffalo bridge doing nothing, I thought that I might as well hack it with an alternate firmware and see what I can do with it.

The WLI-TX4-G54HP is not specifically documented as runnable with a third party firmware, my take is that not many of these bridges have been sold so nobody reported it. Yet I found some specs hinting me that it was running on a Broadcom 5352 which is the same as the chip used in the WRT54GL. It also has the same amount of RAM and flash, which is a good thing. So sure enough, there were some Buffalo routers based on the 5352 that were officially supported by OpenWrt, but no mention of the WLI-TX4-G54HP. I decided to take a chance and flash it anyway. And it worked:

The only method of flashing OpenWRT on this device is to use the TFTP method. There are no signed firmware available that you can install from the router's webpage. It worked on my first try using tftp.

That's it for now. I'll do some more tests as time permits, and will see how I can submit that device in OpenWrt's compatibility lists.

Why OpenWrt? And why not Tomato or DD-WRT? Because from what I've seen until now, OpenWrt seems to be the most "open" solution available. All source code is available and GNU licensed. Furthermore, it has a lot of command-line interfaces and is targeted to experienced Linux admins.

N.B. If you're running Windows 7, you'll notice there is no longer a tftp or telnet client included as with Windows XP. Look here for a quick fix to include them:

http://www.leateds.com/2009/telnet-for-windows-vista-windows-7/

O.

Thoughts on NTP and a possible homebrew project

2010-06-15T19:12:00.000-07:00

After two months without working with UNIX I'm already missing it, and I'm looking for a project to have fun at home and remain technical -- all I do at work is using Word and Excel, and it's starting to make me crazy.

After accumulating them for over 20 years, I recently gave or threw away most of my computer parts (which included two vintage 5 1/4 drives, what a shame). The only PCs remaining are various laptops which are used by my family. I also have a rock solid m0n0wall appliance hidden in my utility room, and I don't want to zap it right away because all my network depends on it.

Those who know me well are aware that I've had a personal fascination for years with NTP. Being a licenced ham operator, building a public stratum-1 server synchronized to CHU using Linux's CHU driver would have been a kickass project to undertake 10 years years ago, with the satisfaction later down the road of contributing to the NTP pool when it became mainstream. However, owning a fixed IP address is costly, not counting bandwidth, and limiting the usefulness of a time server to my own internal network wouldn't give me much. And what's the purpose of using CHU or WWVB when you can sync using a GPS anyway; the only situation I can think of is if you can't have a clear path to a satellite, or need a cheap solution to extract the time from a reliable source. That's the premise those nice radio clocks that set their time automatically are built on.

My second fascination is with embedded devices. They don't consume much power, they're small and fun to work with. The cheapest way to own one to play with would logically be to buy a WRT54GL and flash it with a third party power-user firmware such as OpenWRT. However, the WRT54GL is based on old technology (2002), and thus fairly expensive for what you get. To add basics such as a serial port, you need to crack open the case and solder wires. Fixed storage is limited to 4Mb, that's not a lot of space in 2010 numbers. Bottom line, using the WRT54GL for a homebrew project can get expensive and cumbersome quickly, and paying big bucks for an underpowered device doesn't excite me much. Thus, I'm almost ready to purchase an ALIX 2d13. It's around double the price of a WRT54GL when you count shipping, a case and a CF card. But but it packs a LOT more power and expandability and this embedded device should be able to offer enough power to last 10 years.

One thing I was thinking about is to combine both by adding a soundcard (miniPCI or USB) to the ALIX board, plugging in a shortwave radio, and building a homemade CHU-compatible NTP time server. And why not try WWVB as a "part two". It could have been used in areas where satellite is not accessible. But I quickly found out I wouldn't have been the first one to think of this -- Meinberg already has one. Their only drawback is that it's fixed to one station (they're German, so of course they offer one synced to DCF77). But once your choice is made, you can't change it.

So back to square one. I won't invest tons of money to recreate what has already been done. And this is where I'm at, as of today, with my research on embedded x86 devices.

I think I'll end up building a generic server on the ALIX. Some tasks such as PXE installs don't seem to be well documented on this platform, and I think that testing and documenting a PXE environment could be of benefit to whoever has a bunch of these to flash. We'll see.

Take care

O.

The Information Paradox

2010-05-31T17:57:00.000-07:00

Almost every IT shop has a methodology. Where I'm currently working, they're using Macroscope which, from what I see in the IT market in Montreal, has been in good use around many places here for two decades (I even studied one of its ancestors in College in 1994)

Fujitsu is now in charge of that methodology, and they offer for free an electronic version of a book named The Information Paradox which, I thought, could have helped me understand better that methodology.

I was dead wrong. I tried reading the first two chapters and couldn't finish a paragraph without phasing out and thinking about, oh, various subjects such as home improvement, Star Trek, Bangladesh or Yonge Street. Yep, that's right, a technical guy like me just cannot read this book and remain sane. It's all talk about IT Portfolio, governance, and some other nonsense which doesn't ring a bell. But who actually reads this book? Lots of people, it seems, and I figure they're all working for what I used to call the IT Gestapo. Now that I jumped the barrier to architecture, they're supposed to be my friends now. Yet this friendship is only on paper; I think I'll never be able to share a beer with such people who would talk about IT portfolio the same way I talk about, say, the latest FreeBSD release.

So it looks like with a B.Sc., I'm not qualified to read that damn book. And it's fair game: people with B.A.'s could probably never understand why APUE is one of my favorite books.

QLogic disabled BIOS in a blade can cause problems

2010-05-25T17:58:00.001-07:00

Today I was called for help concerning a weird problem with blades that "didn't work" on the Virtual Connect FC. I'm no longer a sysadmin and shouldn't know anything but since I've worked with blades for three years, I must have become some kind of hot property at my new workplace. :)

In this case, the WWNs of the blades never showed up in Brocade's fabric manager, even though the configuration of the VC domain seemed correct with all profiles set correctly. I double checked everything -- most of the configuration in the VC-FC was correct, except a few missing things but nothing spectacular.

These blades had no OS installed yet, so the QLogic HBA driver couldn't be brought up to initialize the HBAs... thus the fabrics could never detect them. That should be expected to be "normal", but not so: how are you supposed to boot on SAN if you can't even install the OS in the first place?

Turns out that the QLogic BIOS was disabled on all the blades. Calling-up the BIOS configuration with CTRL-Q, and enabling it for all SAN-connected blades, fixed the problem.

O.

So, what's going on?

2010-04-30T10:13:00.000-07:00

So what's up with me? Now that UNIX and SAN admin are behind me, what am I doing as a system architect? There's obviously no further HP-UX administration going on, which means the readership of this blog will drop dramatically. I do have a link for you admins, though: look at this fascinating discussion in the ITRC that has been going on for a while about its future.

Deep technical content for The ex-syadmin blog can only be written if I get my hands dirty, which I'm not sure I'll be doing much for a while.

I probably can't disclose exactly what I'm doing, but I'll limit myself to saying that my current project consists of laying the grounds of data acquisition systems that use industrial-grade devices and software manufactured by Cooper Power Systems. Some of the win32 software parts are encapsulated using XenApp, which reminds me a lot of X-Windows. :) Everything needs to be bound to NERC security standards (one of the many standards we need to comply with) and it is my understanding that many legacy systems are in the process of being "NERC'ed", which means I'll have lots of interesting work to come.

It would be hard to blog about all the implementation details while still remaining generic enough to separate myself from my workplace; thus I won't speak too much, at least for the moment. Once I have a better understanding of the technologies I'm working with, I might come back with technical posts and recipes similar to what I did for Technocrat-UX.

O.

Details on the future EVAs

2010-04-27T06:50:00.001-07:00

I couldn't say much since I was bound by a CDA, but I can link to The Register team who has posted some details along with what should be the new name for the future EVAs:
http://www.theregister.co.uk/2010/04/27/hp_eva_p6000/

Concerning release dates, sorry, I have to keep my mouth shut.

The new Integrity line has also been announced yesterday. It uses a modular, or should I say stackable, blade design. I'm no longer involved with them but if I was still an HP-UX admin, you bet I'd be excited. Note the new Superdome 2; this is a major redesign which will most probably mean the end of the cell-based servers. I didn't have the time to check if and what OLAR features will still be available to customers using the bl890c, as it was a selling point for the rx7640 and rx8640.

Yet this might be too late. RHEL no longer supports Integrity, nor will Windows soon. For customers looking for 32-core systems or more, this will still make the huge Proliants DL7xxx (and upcoming DL9xxx) interesting alternatives to the smaller blades.

O.

A visit at the "ESS CEC"

2010-04-22T16:18:00.000-07:00

Earlier this week, I went for one day at HP's Enterprise Storage & Servers Customer Experience Center, which is located on HP's campus in Houston. You'll probably deduce that I was there to get some info on current and future storage offerings from HP, and I did. But I signed a CDA and can't disclose what exactly I learned there... however some new products have just begun shipping, such as LTO5 drives. As for sdates for other future announcements, well...of course I can't give dates and specifics. But the HPTF usually serves as a platform to announce new products, doesn't it?

One treat that topped the day pretty well was a visit of the Factory Express shop floor where all servers and blades are assembled, and optionally pre-racked. I also saw some PODs being assembled. Once again, I can't say much without getting my ass kicked but whew! They have a nice operation there, I was impressed. Of course, I couldn't take any picture either, but a fellow who probably wasn't under a CDA leaked some pictures via Twitter a few weeks ago during a storage-related event and you can try searching to see if they're still available.

Speaking of the HPTF, I won't be there this year.

O.

The ITech Summit, Compellent, and cheap storage

2010-04-13T15:55:00.000-07:00

The Infrastructure Technology Summit will be held in Montreal and Toronto in the next two weeks, and Calgary and Vancouver are coming in September. This event seems to be the descendant of the older SAN/NAS summit which was held yearly. HP won't be there... which is too bad because I remember seeing Chet Jacobs present right here in Montreal at the SAN/NAS summit maybe 7 years ago. And those who know StorageWorks know Chet; his presentation style is unforgettable.

HP aside, one SAN manufacturer I've been very interested in since last year is Compellent . As an ex-SAN administrator, I was having a lot of problems with unreasonable disk space demands that were reaching into chunks of terabytes, and Compellent's copy-on-write thin provisionning sure seemed like an easy to implement solution that didn't require putting a gazillion agents on servers. The 4/6/8x400 EVAs are, to say the least, average when compared to what Compellent makes. Let's hope the next-generation EVAs will be better. However, not everyone is jumping into Compellent's bandwagon as I personally feel it still has some mileage to do to prove itself as a reliable brand name in the enterprise where long-term commitments and support are very important. A recent article at the Register tends to show what effect having no household name in that world is having on Compellent.

Speaking about terabytes, users think storage is cheap because they can purchase a Hitachi 1Tb USB drive at Best Buy for, what, 100$ now?? Unbelievable. They can even choose to get a Seagate 500Gb hard drive filled with popular movies for the same price! Asking for terabytes is therefore no big deal, right? Well, boys and girls, Enterprise Storage ain't cheap. And there are good, valid reasons for this. I'll make a blog post about that subject soon.

O.

Some last HP-UX tips before I go

2010-04-07T10:28:00.000-07:00

I'm officially finishing my job as an HP-UX/SAN/VMware/Blade admin tomorrow, and I'm moving up the food chain next Monday to my new job. I'm supposed to come back to train my replacement but don't know when it will happen.

Here are some tidbits I found in the last two weeks:

swfixrealm(1m) - Appeared in HP-UX 11.31. It lets you correct the default realm in SD ACL files in one shot. Very useful if you create a depot on a system that has its hostname changed, such as one that was ignited with a golden image.

The new 11.31 Bastille has a parameter named AccountSecurity.unowned_files that wasn't there with 11.23. It is enabled by default and will silently chown files that aren't owned by anyone (i.e. belonging to a uid which is undefined in /etc/passwd) to the bin user. Same for groups. Be careful with this on a server that serves a bunch of home directories, or an NFS server. It might be normal that some files aren't owned by anyone.

Newer versions of Ignite dropped the "Wizard" installation screens which, while braindead, was the one I was suggesting support personnel use to install HP-UX using the Golden DVDs I made. They're in remote offices, so no remote igniting is possible, thus why I burn them custom DVDs. Loosing that interface means they'll have to use the Advanced TUI and it is less user-friendly.

O.

Windows on Itanium: no more.

2010-04-05T16:46:00.000-07:00

Corrected April 6th: I forgot NonStop

I just saw on Slashdot that Microsoft is dropping the ball on Windows on the Itanium platform. While I'm not surprised, that pretty much gives more weight to my prediction that the future of the Integrity platform, at least as we currently know it, is uncertain.

This sure isn't good news for BCS. Of course, everyone knows that most Integrity customers are using HP-UX, OpenVMS and NonStop so that shouldn't mean HP will loose that much revenue following the canning of Windows on ia64. But HP will loose potential customers for sure, and can say goodbye within a few years to ones are currently running SQL Server on Superdomes. I've heard through the grapevine that many Superdomes are actually running Windows, and less installed domes means higher engineering costs pushed down to whoever will remain to purchase high-end and midrange Integrity systems.

That's really too bad. Being a really-soon-to-be-ex-HP-UX admin I can only feel sad when I see such news. But let's be clear: HP still has three strong operating systems to run in Integrity, so that doesn't mean the end of it all.

O.

The sysadmin dilemma

2010-03-31T17:11:00.000-07:00

I just stumbled upon Matt Simmons' "Culture of Quitting" post where, besides talking about the fascinating concept of Up or Out, he puts in a nutshell his motivations for being a systems administrator:

No, I (and probably you), have intrinsic motivation. I don’t expect direct rewards (or even outward appreciation, typically) from doing my job. The reward is that my infrastructure works the way it should. Sure, I have certain long term goals, but I can’t accomplish them if I don’t accomplish my short term goals first.

That's where I got my motivation, too, for quite some time. I don't know any sysadmin who wouldn't be proud of putting in place a high quality, resilient infrastructure. But the problem with this, though, is that not only can it get expensive, it never breaks. And when nothing goes sour once in a while, it's hard to get noticed (and further motivated) by all levels of management unless you're lucky enough to work for people who are perfectly aware of what you're doing.

So what should be done to get a tap on the shoulder? Be a below-average sysadmin? In other words, don't produce results too quickly, don't try to optimize everything right away so that performance issues are apparent, constantly say no to user demands... so you can come back later as a hero by finding solutions to "complex problems" to save the day? That might not make any sense, yet I'm slowly starting to think it does: Under some circumstances, the only way to actually show you're doing something productive is to spend a great deal of your time addressing issues which are visible to management.

This is the base of what I've learned to mockingly call "the sysadmin dilemma": If you're doing good work, you won't get noticed too much, and will risk either staying where you are for years with no chance of being gratified, or worse, you'll end up having to justify your job. On the other side of the dilemma, do bad work which costs big bucks to your employer, and you'll be shown the door quickly.

So I think the best path to take if you want to avoid a sysadmin dilemma is to put your target on being average. Be necessary, but don't be too good. But it is obvious to me that being an "average" kind of person will probably not be a fitting motivation for types like me and Mr. Simmons.

Up or Out.

That sucks, but life in the enterprise does, doesn't it.

O.

See a candidate coding live during a remote interview

2010-03-29T08:20:00.001-07:00

Following a discussion on Slashdot, I stumbled upon a site named See[Mike]Code where an online temporary "interview" room can be set up (for free) and you can use it to evaluate coders in real-time.

Simple and clever. The site brings up a unique URL for the interviewer and another one for the candidate. Everything the interviewee types is replicated, realtime, to the interviewer. That's an easy way to to evaluate the technical merits of someone without the trouble of bringing him/her in for a formal interview.

O.

The redundant IT team

2010-03-24T14:40:00.000-07:00

Following my resignation as the senior sysadmin at my division, I've had many of my colleagues come into my office, saying (amicably) "Darn! You've just put us in deep shit!". That's a matter of perspective; nothing should start breaking apart the moment I leave, and many servers can live without any maintenance for a while. No, I'm really not leaving them in distress, as for many years, I've been thinking the following:

Everyone is replaceable.

...a quote which probably makes me fit for management, because that's exactly what managers think. And they aren't wrong, not at all. They are right. No key IT staff, be it a sysadmin, developer or tech support person, should hold all the information and knowledge to keep any part of your business running. This is especially true with small IT teams of less than 50 resources where there is no implicit redundancy that covers everyone.

Yet, that's the position that many IT managers will put their staff into. When you're counting beans, the only thing that matters is keeping a predefined quality of service at the lowest cost. While hardware manufacturers will be hard-pressed to justify various expenses due to all the redundancies they pack into their stuff, remember that people need to be redundant too! And when the shit hits the fan, no amount of preparation will be of any help if you can't count on support staff who is experienced with your systems. Which will introduce my second quote:

If you can afford redundant hardware, you should be able to have redundant people too.

By "redundant" I don't mean to double up your resources 2 to 1. But you really, really have to reduce the chances of someone being the sole owner of a key role without an assigned teammate.

These rules should then be followed to put into place a redundant IT staff:

1. Make sure no one in your staff has a unique set of skills and knowledge. Someone must be able to replace any missing resource, to some level, quickly. This is for valid for everyone in your IT organization, as nobody answering the phone on your tech support hotline (or giving wrong information) could end up being as bad as somebody else inadvertently initiating an IPL on the mainframe.

2. Require a standard and consistent set of documentation for hardware and software solutions before putting anything into production. Docs that are formatted with a standard set of sections will be easy to skim through by anyone who needs it. The UNIX man pages are a stellar example of consistency. If you don't have the time to mess with recognized processes and standards, then simply don't; a word processor template is all you need to get your guys going.

3. Organize regular lunch'n'learn sessions where someone presents a technology subject to its peers. Not a deep dive, but just an overview so they know about it. Insist on quality presentation documents as they can become a great reference later down the road when training new recruits. And don't be a cheap bastard: if you want to motivate people to come and listen, you better pay the lunch.

4. Treat your key resources well, so that they aren't tempted to go elsewhere. By "key" resources, I mean anyone who has deep knowledge of proprietary systems, and for which there is a shortage of qualified workforce on the street. For that matter, enforce rule #2 with them. Even if you can count on someone else to fill in when they say their goodbyes, you'll still need to hire someone further down the road, so it's best to make sure they don't leave.

5. Don't hire smart asses and douche bags. And I'm serious about this. The smart ass is easy to spot in an interview; that guy will claim that he knows everything, sometimes amalgamating nonsensical buzzwords, so just grill him with very technical questions prepared in advance by your staff and he'll fail miserably. The second kind is harder to spot. In my career, I've crossed a few computer science folks who were very talented but also impossible to work with; these aren't team players, they don't trust anyone, and keep everything for themselves. They must be avoided at all cost as they have the ability to sink your operation. Besides a personality test, which they might be able to trick, there's not much you can do to find out except asking for references. So do you get me here? You need to find a balance between social and technical skills depending on the type of job you're offering; low social skills don't fit well with tech support but might be acceptable for a senior developer. Hell, that point is getting so long, I think I'll make it a blog post all by itself one day.

6. Hire people that are ready to hold many hats. Let the truth be told: some people are happy to be single task-oriented and will make sure it stays that way. These do not fit well in a small IT team, as you can't motivate them to learn about new technology, especially if it's under one of their colleague's responsibility. They're NOT autodidacts and the first they'll always do is ask to be "trained" for just about anything. If you have someone who can use a jig saw, but refuses to even try using a reciprocating saw without going a few years to the School of Reciprocating Saw Professionals, then you have a problem. Of course, when working with a unionized crew, the rules are very different and I don't think that blog post will be of any help to you.

7. And last, nobody likes change management and avoids it like the plague unless they're forced to implement it to follow some crapass compliance rule. But change management, if done well with a minimal mount of red tape, can be very beneficial to your team. Any time someone changes something, it will be documented, which makes fixing any mistake easier if that person is not reachable. Up until now, most "enterprise" change management I've seen seem to be a bunch of expensive, incomprehensible software stacks. I've yet to find a simple and easy-to-use web-based system so I can only encourage you to spend a few days making your own.

This set of rules is by no means scientific. They're the ones I would do my best to apply, was I in a management position for an IT team. Fortunately for me, I'm not. Managing an IT team presents its own set of challenges: with limited money you have to keep your employees happy, the users happy, while at the same time ensuring that your enterprise's survival isn't in peril by ensuring that a reasonable risk management is done. Running a redundant team is one of the best way to lower that risk.

The page has turned. Let's get the new blog going!

2010-03-23T17:28:00.001-07:00

Let's get the new blog going! Why not right here, right now! I might have decided to stop being a hardcase sysadmin, but that doesn't mean I have to stop blogging. Technocrat-UX is dead, long live Technocrat-UX! And let's welcome The ex-sysadmin!

Is there a life after the systems administrator? You tell me! In my blog, you'll follow an ex-sysadmin's endeavour into a world of business process intrigues, vague specs and internal politics. Experience has showed me that sysadmins don't always defeat the bad guys and get the girl at the end of the story. The question is, do system architects? This blog will try to find out the truth!

Expect pragmatic answers to the most elaborate questions. Simple solutions to complex problems. And of course, my own vision of market trending and analysis that you'll love to hate. Who says systems architecture has to be bleak?

But seriously, I first need some time to know what I'm doing. That should take a few weeks, if not months. So I'll document the process I'm going through to leave the sysadmin behind, and prepare the carpet for the architect.

O.

No more Technocrat-UX

2010-03-23T07:35:00.000-07:00

Dear readers,

Instead of letting this blog die, I thought it would be better to make one last post to say goodbye, at least for the time being.

After almost 10 years as a full-time system administrator, I decided to change my career path to become a systems architect. Which, at its bottom line, means that I'll stop writing shell scripts to write plans and documents instead. I will no longer be dedicated to HP technology, and HP-UX in particular. The decision has been hard to make, as I love my job as a sysadmin. Over time, I've participated to many HP-related events and met very interesting people at HP. I will miss this, a lot.

Over time, I've been saddened to see that my workplace doesn't put a lot of, er, "financial" value to its technical staff, no matter how much effort I've put into developing the best technical career I could. The only way to keep going up the salary ladder while remaining technical would have been to quit and try my chances with consulting, and I wasn't interested in doing this for personal reasons. So, I decided to stay with my public sector employer, relinquishing my purely technical job, and time will tell if I will like it.

I'm keeping this blog open for the time being, since the content should be relevant for a few years. If I ever start producing content related to my new experiences, I'll keep the same URL (omasse.blogspot.com) and switch the name from Technocrat-UX to something else.

Thanks to everyone who've been reading this blog since 2008. You've sent me many comments and suggestions that kept me going. Writing for Technocrat-UX has been a very good experience and I sure hope that the content has been useful to you.

Take care.

Olivier S. Massé

HP-UX 11iv3 U6 will include an optional parallel rc sequencer

2010-03-11T12:32:00.001-08:00

One of the questions users ask me most is How come HP-UX takes such a long time to boot and shutdown?

I always reply that the startup rc sequencer works serially, thus any subsystem that takes a long time to either start (CIM) or stop (OVPA) will have a negative impact as everyone else will be waiting in queue. And yes, I also add that Linux's been doing it in parallel for years.

In the old days, HP-UX used to stand above many other BSD-derived Unix flavors with its really nice startup checklist. Yet, as startup times are getting longer and longer, a new parallel sequencer was needed and HP announced one a few days ago through a whitepaper. I expect startup and shutdown times to decrease at least twofold in the long term with this.

Details are in this whitepaper here:
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02036939/c02036939.pdf

The RCEnhancement bundle is available in the software depot and it sure looks promising. At first glance, I'm not sure I like the way it's being implemented with the "rcutil" command when compared to some Linux offerings I've seen which use config files. On the upside, administrators used to the current SYSV sequence will feel comfortable right away using this one.

I know nothing of what's going on in the labs but my take is that the parallel sequencer is a backport of what's being developped for 11iv4. If that is the case, chances are strong that HP will rewrite many of their startup scripts under 11iv4 to use the new sequencer, thus promoting "now boots 200% faster!" as a marketing incentive to encourage users to migrate to v4 when the time comes.

O.

Fun dealing with 'at'

2010-02-25T18:52:00.000-08:00

I'm stuck with a design decision taken before my time by the development team which consists of using the native OS service as the task scheduler for our custom SCADA application. The decision was purely logical; they were migrating away from a previous timesharing OS that had, from what I heard, enterprise-class batch and scheduling services and it was taken for granted that "Unix" (HP-UX to be precise) would be able to handle task scheduling well.

The result is that to save a programmer a few days, the decision was taken to delegate task scheduling to the Operating System and be done with it: when the software needs to run something at a later date, it spawns the scheduler and leaves the responsibility to the OS to run the job.

Where that design decision hurts is that nobody realized that on Unix, as far as scheduling goes, you're pretty much limited to the stock cron or at (both being the same software, by the way) and these can be a real pain in the but to manage on modern systems.

I have nothing against at in itself. There's nothing wrong with it. As a bare bones task scheduler, it does the job and has been doing it for what, maybe 40 years now. Many system administrators have learned to depend on at to schedule nightly jobs. But it shows signs of its age, and has nothing that should appeal to a developer in need of a task scheduler: It doesn't do dependencies; running at -l doesn't show much; its logging features are, to be honest, close to nonexistent; jobs are saved with a file name representing an epoch offset which, while clever, isn't really a nice way of presenting data.

As a sysadmin, I ran into a lot a trouble over the years when trying to support a bunch of application-level at jobs. Here are some examples:

At saves all its tasks under /var/spool/cron/atjobs. That's nice, but what do you do with clustered applications that are packaged with ServiceGuard? There is no easy way to migrate the jobs across nodes when a failover occurs. I had to write a special daemon that monitors the atjobs directory just to handle that.
Support personnel were used on their previous OS to hold, release, and reschedule jobs on the fly. At doesn't support that. When you want to reschedule a job with at, you need to extract what that job runs, delete it, then reschedule it yourself. That's not nice. I had to write a complete wrapper around at just to do that.
You don't know what a task consists of, except of what user is running it, and what epoch-offset name it has. That's not very useful when you have an application that scheduled 50 different jobs over a week. I had to change my wrapper to be able to show a few lines of the contents of each job.
When cold-reinstalling a server, you have to be sure you saved the jobs somewhere as the users will expect you to recover them. Sure, nobody forgets the crontab, but that darn atjobs directory needs to be saved, too.

I'm so fed up with this that I'm thinking of writing my own distributed task scheduler, that would address most of the issues above, while still keeping a standard at front-end that would not mess up any application depending on its format. What do you think?

N.B. Yes, I took a look at vixie-cron a few years ago but didn't think it would be worth trying to make it work on HP-UX as I didn't gain much using its atfront-end over the one shipped with HP-UX. If anyone thinks otherwise, drop me node.

Steps to take in SIM/RSP when upgrading HP-UX Servers

2010-02-15T08:04:00.000-08:00

When cold-reinstalling an HP-Ux server from 11.23 to 11.31, steps need to be taken to be sure that it is correctly linked to SIM and Remote Support.

Here are the steps I take without needing to delete the server in SIM, this way I keep all its past events. These are the quickest I've found over the last year:

1. Go in SIM. Find the server and open its system properties. Uncheck "Prevent Discovery [...] from changing these system properties"

2. Run an "Identify Systems" on the server. Once this is done, it should now show 11.31 as the OS version.

3. SIM won't subscribe to WBEM events when doing an identify, only a discovery. So you need to manually subscribe to WBEM events on the CMS (mxwbemsub -a -n hostname).

4. WEBES will not resubscribe its WBEM events either. To force it, you need to log into WEBES (http://cmsaddress:7906), click the "Configure Managed Entities" icon, find your server, check it, and delete it (that's right, delete it). Then, restart WEBES by doing "net stop desta_service" and "net start desta_service" on the CMS. Within a few minutes it will resubscribe automagically to the HP-UX server.

5. You can confirm you have SIM and WEBES subscriptions on your server by running "evweb subscribe -b external -L"

Good luck

O.

Thumbs up to ServiceGuard Manager

2010-02-10T07:11:00.000-08:00

Being a CLI kind of guy I've never been really attracted to ServiceGuard Manager, especially the first web-based versions. However, since it started having a map view again , I find myself increasingly proposing it to support personnel who find it more intuitive than using the CLI. Training time is decreased at least three fold by using SG Manager.

Today, I decided to try to build a small package from scratch using the GUI instead of making the config files manually and was delighted by its ease of use. I won't publish too much screenshots as those I took contain confidential data and it would take me a while to obfuscate them. But here are two teasers:

The general look is polished, and very intuitive. The interface is responsive. Online help is readily available, with a question mark icon and sometimes with pop-on bubbles. This makes creating packages an easy task which is done in minutes without needing to go through the ServiceGuard manual.

Behind the scenes, SG Manager takes care of migrating the configuration files on all nodes itself. You don't need to copy them manually. Furthermore, they're very easy to read. Here is an example of a config file generated by SG Manager:


#  module name and version
operation_sequence          $SGCONF/scripts/sg/package_ip.sh
operation_sequence          $SGCONF/scripts/sg/service.sh
package_description         Quorum Server
module_name                 sg/basic
module_version              1
module_name                 sg/package_ip
module_version              1
module_name                 sg/priority
module_version              1
module_name                 sg/monitor_subnet
module_version              1
module_name                 sg/failover
module_version              1
module_name                 sg/service
module_version              1
package_type                FAILOVER
NODE_NAME                   mtlrelux00
NODE_NAME                   mtlprdux00
auto_run                    yes
node_fail_fast_enabled      no
run_script_timeout          no_timeout
halt_script_timeout         no_timeout
successor_halt_timeout      no_timeout
script_log_file             $SGRUN/log/$SG_PACKAGE.log
log_level                   0
PRIORITY                    NO_PRIORITY

failover_policy             CONFIGURED_NODE
failback_policy             MANUAL

# Package monitored subnets...
monitored_subnet 1.2.3.0
local_lan_failover_allowed  yes

# Package subnets and relocatable IP addresses ...
ip_subnet 1.2.3.0
ip_address 1.2.3.10

# Package services...
service_name                            qs
service_cmd                             /usr/lbin/qs >>/var/adm/qs/qs.log 2>&1
service_restart                         3
service_fail_fast_enabled               no
service_halt_timeout                    0

Instead of a sea of comments, there are only a few well-placed onces, which make re-editing and fine-tuning configuration files an easy task.

Nice piece of work! I think I've been converted to ServiceGuard Manager.

O.

Remote Support Advanced 5.40 has been released

2010-02-09T11:38:00.000-08:00

Version 5.40 has been released last week. I was waiting for 5.40 to show up in RSSWM and thought it would update itself automatically. It hasn't done so yet, and it is not clear if RSSWM will eventually take care of updating to 5.40. The Release Notes indicate that for current customers to upgrade, a package must be downloaded from the HP Software Depot, so my take is that it won't be pushed by RSSWM.

This time I do not intend to forcibly update. Version 5.30 has been running fine for me for a while now, I found it to be mature and stable. It might be a better idea for current mission-critical customers to wait until they update to HP SIM 6.0 to do RSP at the same time (unless RSSWM pushes it without warning). That is probably what I will end up doing. However, I don't know any experienced SIM admins who risk upgrading to a new SIM release before a service pack is released a few months later. So I'm actually NOT planning to update to SIM 6.0 / RSP 5.40 before next summer.

Here is a list of the main new features. The most significant one, from what I've seen users asking for in the ITRC forums, is the official support for CMS's running in virtual machines.

Added virtualization support for the Central Management Server
Support for HP Systems Insight Manager 6.0
Improved scalability of the Central Management Server
New Basic Configuration collections for MSA2000 storage and OpenVMS on Integrity servers
Introduction of Unified Communications monitoring
Windows 2008 operating system support for the HP Remote Support Network Component
Web-Based Enterprise Services (WEBES) v5.6 and WEBES v5.6 Update 2 are the most current supported analysis engines

WEBES 5.6U2 is required to monitor most recent HP hardware. Current users who do not wish to update to RSP 5.30 right away can install WEBES 5.6U2 from RSSWM and delay updating to 5.40 until later.

O.

Using olrad to remotely flag PCI slots

2010-02-08T11:38:00.000-08:00

Many rack-mountable Integrity servers from the rx3600 and up support OLAR, which is an acronym of "online addition and replacement" that applies in many cases to PCI cards. Cell-based servers also support OLAR of complete cells. The System Management Homepage offers some OLAR-related commands but over time I've learned to use the CLI-based olrad command which I trust more than the GUI.

The olrad command can be used not only to replace cards, but also to flash a LED under specific PCI slots. This is very useful when you send an operator on site to plug wires; using olar, you can flag the exact slot where you want a cable to be plugged, and save time.

Here is a quick procedure to see how to do this:

1. Run ioscan to show the hardware path of your device

Example:


# ioscan -kfnC lan
Class     I  H/W Path        Driver S/W State   H/W Type     Description
=========================================================================
lan       0  0/0/1/1/0/6/0   igelan   CLAIMED     INTERFACE    HP A9784-60002 PCI/PCI-X 1000Base-T FC/GigE Combo Adapter
lan       1  1/0/1/1/0/6/0   iether   CLAIMED     INTERFACE    HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan       2  1/0/1/1/0/6/1   iether   CLAIMED     INTERFACE    HP AB290-60001 PCI/PCI-X 1000Base-T 2-port U320 SCSI/2-port 1000B-T Combo Adapter
lan       3  1/0/12/1/0/6/0  igelan   CLAIMED     INTERFACE    HP A9784-60002 PCI/PCI-X 1000Base-T FC/GigE Combo Adapter

2. Run "olrad -q" to obtain a table matching hardware paths with slot numbers.

Example:


# olrad -q
                                                        Driver(s)
                                                        Capable
Slot     Path            Bus    Max  Spd  Pwr  Occu Susp OLAR  OLD  Max   Mode
                        Num    Spd                                 Mode
0-0-0-1  0/0/8/1         140    133  133  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-2  0/0/10/1        169    133  133  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-3  0/0/12/1        198    266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-4  0/0/14/1        227    266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-5  0/0/6/1         112    266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-6  0/0/4/1         84     266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-7  0/0/2/1         56     133  133  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-0-8  0/0/1/1         28     133  133  On   Yes  No   Yes   Yes  PCI-X PCI-X
0-0-1-1  1/0/8/1         396    133  133  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-1-2  1/0/10/1        425    133  133  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-1-3  1/0/12/1        454    266  133  On   Yes  No   Yes   Yes  PCI-X PCI-X
0-0-1-4  1/0/14/1        483    266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-1-5  1/0/6/1         368    266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-1-6  1/0/4/1         340    266  266  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-1-7  1/0/2/1         312    133  133  Off  No   N/A  N/A   N/A  PCI-X PCI-X
0-0-1-8  1/0/1/1         284    133  133  On   Yes  No   Yes   Yes  PCI-X PCI-X

3. Run "olrad -I ATTN slot_number" to flash the LED under the desired slot.

Example:


# olrad -I ATTN 0-0-0-8

4. When you're done, turn off the LED on your slot using "olrad -I OFF slot_number"

Example:


# olrad -I OFF 0-0-0-8

Cold-Updating small ServiceGuard clusters -- FAST!

2010-01-27T17:55:00.000-08:00

Here's my guerilla procedure to cold-update small ServiceGuard clusters without doing an official rolling upgrade.

I'm currently migrating many small two-node ServiceGuard clusters which are scattered in different sites from SG 11.18 / HP-UX 11.23 to SG 11.19 / HP-UX 11.31. I decided to upgrade not only the OS, but the clustering software too for the simple reason that I didn't want to stick with 11.18 and have to update SG later down the road... With 11.19, I should be good for a few years.

The "rolling upgrade" procedure documented in the Admin Guide doesn't work in such a scenario as last time I checked, it only supports running an update-ux on the nodes one after another. I don't do update-ux, I prefer cold-reinstalling my systems with my heavily customized Golden Image. And since I wanted to take advantage of the downtime to move to 11.19, I fell in the "unsupported" arena.

Here's how I'm pulling it off with a procedure that takes a mere 60 seconds more downtime than a straight failover:

1. Update the failover node
1a) reconfigure the packages to be runnable only on the main node
1b) reconfigure the cluster to remove the failover node (you'll end up with a one node cluster)
1c) dump the golden image on the failover node
1d) install and configure the requirements for SG 11.19 on the failover node (it takes maybe 10 minutes if you've documented the process correctly, I know it for fact)
1e) set up a configuration file for a brand new one-node cluster on the failover node. If using lock disks, you can either use new lock disks and start it right away, or prepare config files which you're sure will work and start the cluster at step 2b.
1f) bring in the package configuration files and volume groups on the failover node, and configure these packages to be runnable only on the failover node. Run a cmcheckconf on them but do NOT run cmapplyconf yet because they're still used on the other cluster!

2. Move the packages to the failover node
2a) stop the packages on the cluster running on the main node
2b) remove the cluster bit on the VGs (vgchange -c) to prevent SG from identifying the disks as part of a cluster
2c) cmapplyconf the packages on the failover node (you might need to run vgchange -c again)
2d) start the packages
Total downtime: maybe a few minutes more than a standard failover but not much. With a well-prepared scenario with pastable commands, it takes me less than 60 seconds to do 2b and 2c.

3. Upgrade the main node
3a) dump the golden image on the main node
3b) install SG on the main node
3c) have that node join the cluster running on the failover node
3d) configure the packages to be runnable on both nodes

4. Bring back the packages to the main node
Simply move back the packages as you would in a normal cluster. Downtime will be the same as during a standard failover.

O.

The Software Conspiracy, downloadable edition

2010-01-23T16:29:00.001-08:00

I was going over my bookshelves and stumbled on a book I read some 10 years ago, named "The Software Conspiracy". The title pretty much says it all. Back then, I had just received by B.Sc. and was starting a career as a Unix admin; some thought I was crazy of not following the same path as others of that era which became part of an army of MCSEs. The fact is, I've always been very critical of the software industry, and back in these days of Microsoft in particular. UNIX just plain worked.

I still disklike a lot of "enterprise" software titles. However, I think software is actually better, for two reasons. First, it updates more easily, and unattendingly. Secondly, the move towards appliances, be it a smartphone or a DVD player, increased the quality of software overall as when an appliance doesn't work, you send it back.

I just found out the book was released for free in .pdf in 2005. It's a fascinating read.

HP-UX's future is in question. Does it lie in Linux? Or not?

2010-01-22T10:38:00.000-08:00

Juanma's blog made a follow-up to my december post where I expressed concerns on HP-UX and Integrity's future.

I've thought more about it over weeks and here is what I think of it.

First we have Brian Cox's (in)famous blog post where he openly states that Linux should be considered in the long run by enterprise customers. I'm not against what he said, au contraire. He had the courage to confirm what many were thinking. But what does this mean? Is HP abandonning HP-UX? Not necessarily, but I wouldn't be surprised if this announcement means between the lines that HP has plans for Linux in the future.

We first have to look at the market HP is pitching HP-UX to: enterprise customers. One could argue that as long as HP-UX keeps a resonable TCO and is well regarded by all the Gartners of this world, it should continue faring well in the enterprise. But the fact remains that even if the average enterprise architecture team usually chooses a platform based on who wins an RFP or what industry analysts have to say, the pressure to move to Linux will just intensify over time.

Gartner aside, HP-UX does have great qualities that are fit for the enterprise environment. For one thing, it doesn't change much, which can actually make it cheaper to run than operating systems that are constantly being revised. And for customers who are looking for a vertically integrated infrastructure where almost everything comes from one an only vendor, HP-UX, and AIX, for that matter, are a no brainer.

Yet, developing, supporting, and maintaining your own operating system is a costly venture which, unless you're a strong leader, doesn't have a good payback. What pays back is the support and services you'll sell.

The bread and butter for HP doesn't lie in HP-UX itself; it lies in their enterprise vertical platform which, in my opinion, consists not only of the hardware and the operating system, but all the services you can offer under the same umbrella. Could an HP-supported Linux be the answer to the slow decline of HP-UX? I sure think so. HP would bring to Linux a very strong support and services infrastructure, that would please enterprise customers. They also already have a bunch of manageability tools that are invaluable to the enterprise, many already available on Linux.

Furthermore, HP's all about converged infrastructure now. That's what they've been nailing their customers about and believe me, they want them to get the message. We all know that Oracle is becoming a menacing converged player: they have the software, the hardware, and the operating system. HP will probably not allow itself to become runner-up.

One has to wonder, though, what kind of deal they struck with Microsoft with their Frontline partnership. I'm sure Ballmer asked Hurd not to venture too much into the Linux business when signing the deal. This is speculation, of course, but a possibility no less. And if it is true, then we won't see HP release their own Linux for quite some time, and HP-UX will stay there as long as it can bring them money.

O.

Integrity iLO2 Advanced Pack trial license

2010-01-21T16:28:00.000-08:00

HP offers a free 30 day trial of iLO2 Advanced Pack here:
http://h71028.www7.hp.com/enterprise/cache/279991-0-0-225-121.html

Scroll down to "Multi-OS" and you'll see licence keys that can be installed within seconds to enable the Advanced Pack features. The most useful one, in my opinion, is Virtual Media which lets you mount a remote .iso image and present it to the server as if it was a real DVD.

These keys could be extremely valuable if you ever need Virtual Media in a hurry on a remote server which doesn't have a licensed advanced pack. I can think of many situations where this could happen. I won't copy the keys here, so I suggest you save them in a text file and hold on to them in case the web page disappears.

Olivier.

ServiceGuard Manager once again has a map view!

2010-01-19T11:50:00.000-08:00

Remember the days when ServiceGuard Manager's java interface was so cool it was prominently shown in HP's video where they blow up servers? We actually used this high-level GUI to demo ServiceGuard to upper Management, as it was the most effective way of explaining to non-technical people what a ServiceGuard cluster consisted of.

You can imagine my deception when, a few years ago, HP decided to ditch that client-based GUI in favor of a nasty, ugly web-based implementation where you lost all the ease-of-use that was a hallmark of ServiceGuard Manager.

Well, I just found out that the map view has finally been restored in the web version. ServiceGuard 11.19 patch PHSS_40152 was released a few months ago, and it introduces release 2.00.10 of ServiceGuard Manager which, finally, restores the much-needed map view. You get something like this:

This is much better and more intuitive than the older tabular view, and furthermore you're once again able to drag packages from one node to the other.

Usually, I would praise HP for doing something like this. But considering that they actually removed the map view and took years to bring it back, the only thing I can say is that it was about time.

O.

So, what is HP SIM good for?

2010-01-16T17:56:00.001-08:00

Over time, HP Systems Insight Manager has grown to become an important part of our infrastructure. The path to success has not been easy; many man-hours have been invested by both me and our previous CMS administrator into making everything work reliably. The best advice I can give you is to install it from scratch on dedicated hardware.

So what do we use SIM for? Mainly for three things:

System inventory
Contract and warranty management
Monitoring of hardware failures

We use SIM to monitor and manage a few hundred devices scattered in 7 data centers. It is used with almost every piece of HP hardware we have, from Proliant to Integrity, from MSAs to EVAs. Integrating SIM with Remote Support has been bumpy, to say the least, but once I made everything work, it has become a well-appreciated service. Support personnel like the fact that service calls are opened automatically at HP.

SIM and Remote Support helped us slowly move to a Preventive Maintenance way of thinking, instead of a Corrective Maintenance approach. We're now able to monitor the health of just about any piece of hardware, and have power supplies and disks replaced within a few hours when problems happen. I can't quantify how much our risk of downtime or data loss has decreased by using Remote Support, but it is certainly an appreciable amount.

On the downside, SIM has many other functions that we don't need, but they come bundled in anyway and cannot be hidden. Its interface is not very intuitive, with drop-down menus that tend to integrate many different concepts. The SIM UI has many performance and design problems which makes it a hard to learn and understand. Maybe 6.0 will be better.

It seems that HP is pushing SIM as the interface to to just about anything to manage servers -- from what I've seen in the recent years, they talk about it much more than OpenView and boast it as a component of the Converged Infrastructure ploy. However, Insight is not a brand I've grown to trust over time. SIM does the job, but it is clunky, slow, and I find its documentation to be rebarbative.

O.

New EVAs are coming in 2010

2010-01-11T10:19:00.000-08:00

I didn't sign an NDA but since some of the information I've learned come from people who probably did, I cannot disclose precisely what's coming. But I can cite public sources that I found on the internet.

The Register found this out last fall and wrote a good description of what's coming.

A rumor I've seen on the internet is that Dot Hill might have something to do with the underlying technology. Which might no be so far-fetched after all. But I will believe it when I see it - something tells me this is a LeftHand design.

O.

Is a crossroad coming? You tell me.

2010-01-05T16:40:00.000-08:00

Last new year's eve, my folks came over for dinner with my family. There were plenty for everyone but I unfortunately ran out of champagne glasses. Being the HP-UX groupie I am, I simply took out a promo HP-UX 25th glass I had in the cupboard and used it for the first time.

It sure looks nice, doesn't it?

But as we are starting 2010, it is also time for me to think about what HP-UX's future will become, and thus what my future career will consist of.

I'm a blogger; I'm no industry analyst. I might be making wild guesses, but the post where I raised concerns on HP's long time commitment toward HP-UX and the Integrity platform sure didn't close 2009 well for me. I've invested lots of time over the years on the HP 9000, then on Integrity, and all this started paying off only after years of dedication.

I don't have any sources inside Hewlett-Packard but on the surface, as a average customer, I've seen many things which have changed within the company in the last 5 years. But the debate must remain strictly technical: Is HP-UX still able to to the job, at a resonable TCO? The answer for me is still yes. For how long? Probably a while, but I'm still nervous anyway.

Over the years, HP-UX has never let me down, both as an enterprise-class operating system and as a career path. But nevertheless, there are times when we reach a crossroad, and in these times we must either continue on the same path, hoping for another crossroad further down the road, or simply switch our direction right away.

I think I'll be standing in a crossroad soon.

How to breed life in an old laptop

2009-12-22T17:37:00.001-08:00

My older Acer TravelMate laptop was getting so slow, I decided a few weeks ago to change it for a a flashy HP with Windows 7.

I still use it from time to time. Like right now. But I was still faced with a clunker that took at least 7 minutes to boot up and be usable, with the hard disk spinning endlessly. 7 minutes! I tried all the Remove-Windows-Rot advice I found, and uninstalled every piece of software I didn't use, then ran all sorts of Crap-Cleaners and Page-File-Optimizers and Disk-Defragmenters. Nothing worked.

I finally decided to:
1. Upgrade my memory to the maximum I could
2. When 1 didn't proove to be satisfying, I reinstalled Windows.

1. Memory Upgrade
I wanted to update from 512Mb to 2Gb, which is the maximum the laptop can hold. I looked on eBay but, what do you know, 1Gb SODIMMs are more expensive than the new stuff (especially when you add the shipping costs). Newegg.ca had cheaper memory modules than eBay, which happened to be brand new, completely legit and under warranty, so I went with them. I ended up buying two modules from G.Skill. I didn't know this company, but apparently they're well known in the gamer and overclocker scene. They actually manufacture their own ICs. I installed them, ran Memtest86+ and everything went well.

2. Reinstall from Scratch
Even with 2Gb or RAM, the rot was still there, with the hard disk being abused like what Jason Scott would call a "drunk cheerleader dropped in the exercise yard of a prison". I went down from 7 minutes to 4 minutes, and that still was unacceptable. I was fed up, so I simply booted off the Recovery CDs (yes, I DO have them!), and went through the pain of visiting Windows Update 8 times, rebooting each time. Total time: 4 hours, but it was worth it. I didn't install NOTHING else on the laptop except the vanilla Windows XP. To prevent crapping my registry, I decided that all add-ons will come from PortableApps.com. Bloatware like Adobe Reader and Quicktime, move away please - you're not welcome on my vanilla PC.

I now have a Laptop which, while not good enough to handle Youtube HD channels, is workable enough to boot in less than two minutes and offer a completely sane Windows XPerience.

If anyone has suggestions, send them in. I would like to know what you do on your side to prevent Windows Rot.

O.

Understanding Insight Remote Support

2009-12-22T11:41:00.000-08:00

Here is an update to my "Understanding Insight Remote Support" (formerly "Understanding RSP Components") flow diagram. This one removes OSEM from the picture, which means that there is one less tool to worry about, and adds generic SNMP devices to the mix:

Note that this diagram only shows devices I'm familiar with. If you have any comments, I'll be glad to modify it.

O.

WEBES 5.6 update 1 is out

2009-12-22T11:11:00.000-08:00

WEBES 5.6 update 1 has been released recently. It is required to support new hardware, namely the behemoth AMD-based Proliant DL785. Users interested in learning more can take a look at this document which explains what 5.6 and 5.6U1 are all about -- from what I understand, users who have not voluntarily upgraded to 5.6 yet are still at 5.5 and will be updated once RSP 5.40 comes out in early 2010.

For current 5.6 users, the minor update was silently pushed through the RSSWM last week-end and at my site, the upgrade went flawlessly.

I noticed that a bug that was plaguing me since 5.5 has finally disappeared, when you click in the managed systems list on an Integrity/HP-UX Server, the interface now gets back with the info panel instead of timing out.

O.

The outcome doesn't look that good for both HP-UX and Integrity

2009-12-18T09:02:00.001-08:00

First of all, we have Brian Cox's blog recent post comparing what he thinks of HP-UX and Linux. Read it here:
http://www.communities.hp.com/online/blogs/musings-on-mcc/archive/2009/12/16/linux-vs-unix-shouldn-t-they-be-equals.aspx

Here is a quote:
Similarly, if you asked me to choose between HP-UX and Linux for a customer’s most demanding workload, I would typically recommend HP-UX. However, if my customers’ time horizon is five years from now, then I would seriously consider Linux (by the way, you could replace OpenVMS for HP-UX and Windows for Linux in the above comparison and I would give you a similar answer).

I've met Brian personally last year, and he's a level-headed guy. Preferences for a platform versus another aside, what he says here makes sense on both a business and technical perspective.

Then, around the same time, rumors pop up indicating that Red Hat will be canceling their Itanium port:
http://www.theregister.co.uk/2009/12/18/redhat_rhel6_itanium_dead/

What will the future be for HP-UX and Integrity? Red Hat apparently abandoning ia64, with Novell being unsure if they'll continue, are especially bad news for BCS. That leaves us with one less operating system for the Integrity line, and it turns out it's one that Cox suggested potential mission critical customers should investigate if planning for 5 years down the road. The outcome for the excellent Integrity line doesn't look that good.

As far as I'm concerned, as a current HP-UX / Integrity customer, it's business as usual for now and will be for a few years to come. We're starting to renew our systems next year and this won't change our plans. But I think it is time to seriously plan my long-term strategy for post 2015.

O.

Performing a chmod on a symbolic link

2009-12-09T11:47:00.000-08:00

On HP-UX, symbolic links cannot have their permissions changed. When doing a chmod on a symbolic link, the chmod operation is performed on the file it references.

A little background information is in order. When a symbolic link is created, it sets its permissions depending on the current umask. So if you have a umask set to 027, it will create a link like this:
# umask 027
# ln -s /stand/vmunix /tmp/link1
# ls -al /tmp/link1
lrwxr-x--- 1 root sys 13 Dec 9 14:50 /tmp/link1 -> /stand/vmunix

While a very restrictive umask such as 777 will do this:
# umask 777
# ln -s /stand/vmunix /tmp/link1
# ls -la /tmp/link1
l--------- 1 root sys 13 Dec 9 14:50 /tmp/link1 -> /stand/vmunix

So what do you do if someone created a bunch of symbolic links with a umask of 000, and you have scattered symlinks that look like they're world-writable files?

The technical answer would be to ignore them. As most file operations except the link()-related apply to the file referenced by the symbolic link itself, I do not think this is a security problem. But of course, when you're being scrutinized by a security auditor, explanations like this one often don't have any merit. It's less hassle to just satisfy whatever the auditor wants, and correct these symbolic links.

The problem arises when you notice that chmod doesn't work on a symbolic link. And this isn't specific to HP-UX; Linux doesn't allow this either, but I found that FreeBSD has a "-h" option to chmod that addresses the issue. How can you fix that?

The only solution I found by looking into the ITRC forums is to delete the symlink, and re-create it with an appropriate umask. This can be done really quickly but the process won't be atomic so I can't garantee this will be completely unnoticed by your applications.

Here is a short script I've written named lchmod which will ease the operation:


#!/bin/sh
if [ "${1}" = "" -o "${2}" = "" ]
then
       echo "Usage: lchmod  "
       return 1
fi
umask=${1}
symlink=${2}

if [ ! -h ${symlink} ]
then
       echo "Symlink '${symlink}' does not exist"
       return 1
fi
destination=$(/bin/ls -l ${symlink} | sed 's/.*-> //g')
umask ${umask}
rm ${symlink}
ln -s ${destination} ${symlink}

Say you've got this link:
# ls -la /tmp/link3
lrwxrwxrwx 1 root sys 13 Dec 9 14:59 /tmp/link3 -> /stand/vmunix

Simply run lchmod like this and the link will be recreated with a umask set to 027:
# lchmod 027 /tmp/link3
# ls -al /tmp/link3
lrwxr-x--- 1 root sys 13 Dec 9 15:00 /tmp/link3 -> /stand/vmunix*

O.

Using USB dongles with ESX-based virtual machines

2009-12-09T09:18:00.000-08:00

The post where I mentioned how I made a pool of external fax modems work with ESX guests using a Digi PortServer has proven to be one of the most popular of my blog.

Recently, I've been faced with a similar challenge: Is it possible to virtualize Windows servers which host software that requires a copy protection USB dongle? The answer is yes!

Since I was a happy camper with the PortServers, I once again checked what Digi had to offer and found their AnywhereUSB line of network-enabled USB hubs. Simply put, these devices work like this: The hub has a LAN port, and you can use it to access USB devices through your LAN. You simply need to add a special driver to your Windows server that will "fake" a local USB port, while in fact it redirects the traffic to the remote hub. This works flawlessly with physical servers and most importantly VMs, and you can VMotion them around at will.

Digi has written a concise whitepaper that describes how to connect the Anywhere USB to VMware ESX guests here: http://www.digi.com/pdf/wp_ESXServer_AnywhereUSB.pdf. The setup is done within a matter of minutes.

The 2-port version, according to Digi's online store has a list price of 287$USD while the 5-port version is 349$USD.

But whatever you do, don't do the same mistake I did and buy a 5-port hub, thinking that each independent port can be shared among multiple servers in the same manner like Digi's serial servers can - I found out that the Anywhere USB can be connected to only one server at the time. The whitepaper above claims that you can have "multiple USB hubs per virtual machine", but don't confuse this with "multiple virtual machines per USB hub". I don't think the 5-port version is very useful for many cases unless you need to plug a lot of devices on the same VM.

Also, remember that the hub only has a 100Mb/s connection and will downgrade USB 2.0 devices to work at USB 1.1 speed. This is fine for many cases such as with a dongle, but any use requiring a high-performance data rate will be better served by using a physical server.

The fact that you can't share the hub with multiple VMs is a serious design limitation that will require you to deploy a lot of these devices if you ever need to virtualize dozens of servers that use dongles. The 287$ cost for each VM has to be considered in this context, but compared to having to install and manage a physical server, this is as cheap as it can get.

O.

Update Dec 10th 2009: I found another product that is a lot cheaper than Digi's. While it would do the job in a SOHO environment, it's built by a vendor that I wouldn't trust for enterprise systems. At 287$, better buy yourself peace of mind, and especially long-term support.

When something doesn't work...

2009-12-06T16:40:00.000-08:00

Here is my definition of something that "doesn't work": It is a product you try before you buy (or, if you're unlucky, you buy outright) and you're not able to use it.

When I buy computer peripherals for my personal use at home, usually the cheaper, the better. And even though they're engineered overseas and poorly translated, they almost always work.

Then why in hell are there many instances where I've seen enterprise software and hardware that does not work when I try it? This is plain nonsense.

Recently I've been trying an OTP solution and, geez, it looks like the manufacturer did *everything* to discourage me from buying their product. Yet I feel compelled to do it as a favour to my team, because it looks like Corporate IT chose this for the VPN access and I don't want my personnel to end up with two different tokens from two different vendors.

I won't go as far as saying the product itself sucks. But its marketing sure as hell does. I won't give out details or name that vendor right now. It's better to give myself some time to vent. But I'm slowly getting pissed and, guess what, I don't like wasting my time.

O.

Moving physical extents within a PV

2009-12-01T10:45:00.001-08:00

A new pvmove feature appeared in 11.31 which lets you move physical extents within a PV. This can be very useful to move PEs to make space for a LV which has a contiguous allocation policy such as the swap LV, /stand or /.

To use this, simply specify a start range and end range, and tell pvmove to move a range of extents within the same PV.

Example:

root@bonyeune[~]# pvmove /dev/disk/disk21_p2:00736-01248 /dev/disk/disk21_p2
Transferring logical extents of logical volume "/dev/vg00/lvol3"...
Transferring logical extents of logical volume "/dev/vg00/lvol4"...
Transferring logical extents of logical volume "/dev/vg00/lvol5"...
Transferring logical extents of logical volume "/dev/vg00/lvol6"...
Physical volume "/dev/disk/disk21_p2" has been successfully moved.
Volume Group configuration for /dev/vg00 has been saved in /etc/lvmconf/vg00.conf

In the previous example, I moved a range of 512 PEs from #736 to #1248 further inside the same PV. This freed up PEs between 736 to 1248.

Is it possible to move around a LV with a contiguous policy using this technique? To my surprise, I tried it, and it the answer seems to yes. But there are some limitations. I'll need to set up a VM to be able to experiment further. There are other interesting commands such as "pvmove -n" which lets you move a whole LV without needing to specify PEs like above. I'll make a better post once I've had to time to try it out.

O.

Virtual Connect for Dummies

2009-11-20T10:26:00.000-08:00

Grab it while you can

I just went over it and although it is a true honest-to-goodness "For Dummies" book, this is by no means a complete book but rather a 75 page marketing stint by HP.

There isn't much deep technical info in the book but it can be very handy if you need to introduce Virtual Connect to someone who is new to the technology. The informal and clear writing style of the For Dummies books will be familiar to many, and this will encourage people to read it. However, there are few figures and logical diagrams, which means that for serious training you'll be better off reading the official documentation. For someone already familiar with networking and blades, the product briefs for Virtual Connect are very clear in describing what it does.

By the way, after almost 2 years of running Virtual Connect modules, I'm still very excited by them. They saved me a bunch of work and hassles dealing with our networking team.

O.

The hiden advantage of using Remote Support on HP-UX

2009-11-20T07:14:00.000-08:00

Readers of my blog will know that I spent quite some time integrating HP-UX 11.23/11.31 with SIM and Remote Support on each and every of my servers, including even the older, neglected test/QA servers no one usually cares about.

I'm sure some must have thought I was crazy investing so much time on a feature that doesn't bring back much, because, they'll say, hardware doesn't break. It is partly true. Hardware doesn't break a lot with the exception of disks, fans and power supplies which can experience a higher failure rate than, say, anything else that's based on transistors. So, most efforts should be prioritized towards monitoring devices which have a lot of these, and this mostly applies to disk arrays.

There is, however, a hidden gem in using Remote Support pack with HP-UX, and it's the monitoring of system panics.

That's right, panics! I don't hear about the term as much as I used to in the old days, but the fact remains that they still happen, and can either be the result of a software bug or even an untrapped hardware problem. With HPVM guests, I've had my share of panics, too.

Remote Support comes to great help with panics. When a panic occurs, once rebooted the monitoring agents will notice it, WEBES will gladly flag it as important, and an event will be logged at the response center. If the panic happens overnight or when the sysadmin is not there (and it WILL happen - most of us are in the office only a small amount of time), hours will be saved in the process as someone will probably have already contacted the system contact about the issue.

There is not yet a feature to send to HP details on the crash dump when the event is opened, and it must be done manually. But I wouldn't be surprised this will come in the future. Wouldn't it be great, for example, if upon rebooting the server crashinfo could be ran automatically and send details to the engineer? One can only hope this will come in the future, to reduce even further the response time.

O.

HP-UX Community Links

2009-11-17T10:03:00.000-08:00

Okay... over time I've accumulated a few HP-UX resources, here is what I've stumbled upon up until now. I'll keep this post updated, so send me your comments.

Of course, I couldn't start the list without mentioning the ITRC forums which replaced the trusted HP-UX sysadmin list in functionality. Unfortunately, the interface and features have not evolved much in, what, 10 years, which limits its outreach to a database of questions and answers. Some regulars over there do put a community spirit in the HP-UX forum, but over the years I've seen a decrease in the quality of many of the questions and I don't participate as much as I used to. Furthermore, any post that is too critical of HP or its products can get deleted by a moderator, which overcomes an important aspect of having a community-driven independent site.

The real "community" effort should logically comme from Connect and they have an Enterprise Unix SIG (special interest group) and HP-UX group. However, even though they are linked from the ITRC forums, these groups are not very active. Their web interface has a lot of usability problems, which might not encourage anyone wanting to start a discussion there.

I've been surprised to find an HP-UX group on Linkedin, though, it has a group named "HP-UX users" which counts 600 people, and seems to be quite active. Facebook has an "HP-UX" group too but almost nobody posts anything there. I think Linkedin is much better suited to this usage.

Now to blogs. I've found a few people who run HP-UX-related blogs, likewise to mine. Some have RSS feeds you can subscribe to.
Victor Balogh
Steven Protter (a top ITRC member)
Daniel Parkes

I also subscribe to the blog "Musings on Mission Critical Computing", which isn't very technical inbut shows an insight into where top people at HP could be steering HP-UX.

On Twitter, I found a lot of HP feeds available. One of them is named HP_UX_Docs who posts liks to recent documentations and community resources. Well worth subscribing to.

O.

Planning racks and cabling using Visio

2009-11-13T10:11:00.000-08:00

For years, HP has been offering on visiocafe.com Visio stencils that help plan not only the racking and dispotition of your servers, but their cabling too. Everything, from blade servers, to SANs, to power distribution units is available. The drawings can be made quickly in a matter of minutes and every component uses the same scale. I can compare the stencils to a bunch of Lego blocks.

By preparing myself in advance using these stencils, I've been able to greatly reduce the the time it takes me to perform physical installations and evaluate precisely how many cables are needed before going on site. This is especially useful when dealing with unmanned data centers where things need to be done right on a unique visit. If you're not familiar with Visio, it is easy to learn, and I can assure you that you'll save time in the long run.

I also use the stencils when drawing maps of my environment, especially network-centric ones. By using the "real" device images instead of white boxes, I end up with maps that are readily understandable by IT staff and they look great!

O.

Quick start with the HP-UX CIFS Client

2009-11-06T07:46:00.000-08:00

I've been slowly migrating web servers from IIS over the years to HP-UX Web Server Suite to benefit from increased PHP performance, and a need has come up to access data stored on a Windows share.

The HP-UX CIFS Client is an OEMed version of Objective Development GmbH's Sharity which is a software that lets you mount Windows shares on a variety of platforms. I don't know the specifics but it used to be a userland tool, and HP has extended it to to make it a kernel module. The engineer in charge of the CIFS client actually posts once in a while in the ITRC forum.

The CIFS client is already built in the default 11.31 installation, so there is no need to install it. It is very easy to use. It uses a clever hack to manage who owns files in the CIFS mount: your users log on independently to the CIFS server, and will be able to see the whole filesystem as their own.

In a nutshell, here is how to mount a CIFS share. Your HP-UX box doesn't have to be in a domain.

1. Obtain from your Windows admin a login/password to an account authorized to use the share

2. Activate the CIFS client
# vi /etc/rc.config.d/cifsclient
RUN_CIFSCLIENT=1
# /sbin/init.d/cifsclient start

3. Create a mountpoint and mount it
# mkdir -p /cifs/myshare
# mount -F cifs myserver:/myshare /cifs/myshare

At this step, the filesystem is mounted but it is not possible to access it. You first need to have a UNIX user log in to the CIFS Server using cifslogin, then access to the share will be possible. You can do this as root but it is better to use an unprivileged standard user.

4. Log in to the CIFS server as an unprivileged user
# su - user
user$ cifslogin -U winuser -P winpassword myserver

5. Save the user credentials in the CIFS datbase so they can be reused automatically next time:
user$ cifsdb myserver

6. Return to root and save the share the CIFS database so it can mount automatically next time:
# cifsdb /cifs/clsweb_donnees

This should enable the share to remain mounted across reboots. It is also supposed to work if the Windows server reboots but I have not tested it yet. Those of you who would prefer to use the automounter to mount shares dynamically can also do so, which can be useful if you have a bunch of home directories to take care of.

O.

Using Gartner's Magic Quadrants

2009-10-30T11:38:00.000-07:00

When evaluating security products, one way to go is to check out what Gartner has to say. I found out last week that they produce yearly what they call Magic Quadrants on a variety of products, and this can help you choose which ones you're going to look into.

Here is an example of such a Quadrant, using made-up products:

Of course, being inside, or outside, of the Quadrant doesn't mean ANYTHING to me. I'm not evaluating here the veracity of the Magic Quadrant, just its purpose. When choosing software, I prefer "going with my heart" when I can.

But there are cases where using the Quadrant can be of help. For example, if I happen to like Gorbatcheck and it turns out that it's favorably placed in the Quadrant, that's another thing up my sleeve that I can pitch to management. It's also a good ticket to my own peace of mind as being backed by Gartner gives a sense of immunity if the product turns out to be below expectations.

There is also the case where Gorbatcheck might be based on an open-source product, let's say Gorbrafree. Of course, Gorbafree won't be in the Quadrant, but the Quadrant can be used to give more credibility to Gorbafree over something less stellar such as GZK. If you're on a tight budget, that's a way to introduce Gorbafree until you're ready to move on to Gorbacheck.

How do you obtain these quadrants? Try Wikipedia!

HP-UX 11i: Mission-critical UNIX

2009-10-29T14:41:00.001-07:00

HP was announcing that webcast for weeks, and I decided to check it out. I had seen the one from last year and I didn't find this one much different in terms of presentation or content. The detailed results of the report are here.

First, if I recall correctly the study is done on a sample of around 250 people, give or take, I don't remember the exact number. That's not, in my opinion, a high number. But they insist on the fact that the answers come from data center IT tech staff, not CIOs, which at least comforts me as a systems administrator.

According to the study, UNIX(r) is still alive and well and still a strategic OS, but mostly in the enterprise. I insist on UNIX with capitals and the registered trademark, as they didn't include Linux in that category. They only evaluated AIX, Solaris, and HP-UX which are the "big three" Unixes left. I can resume the presentation to this: HP is better than the rest, sprinkled with numbers from Gabriel Consulting Group and a few slides I've seen countless times previously such as the OE "Christmas present" diagram and the HP-UX roadmap. Cox also spent almost 7 minutes on Green IT, speaking about how HP is "green" and reduces carboard boxes. Interesting, but maybe a bit out of subject. The presentation is high-level, and targeted mostly at people evaluating a migration to HP-UX more than existing customers. No wonder it's on cio.com!

Dan made a smalll mistake during the presentation. He mentioned that Virtualization was a strong point for the HP platform, and he is right. But he then followed on how the tendency would be to virtualize different operating systems on the integrity platform, all the way from Linux, to Windows, to OpenVMS, HP-UX, and then he hesitated and brought up "Tandem", which Brian corrected as "Nonstop OS". Well guys, I don't think they are plans to run Nonstop on HPVM. If there are, I stand corrected!

I couldn't comment if HP is the "best". I haven't worked on multiple platforms in 6 years. But whenI did work with HP-UX, Solaris and AIX, I preferred HP-UX due to its better management feature then. I can only expect it is still even better now.

O.

To me, it is now true. The web is (almost) the (home) platform

2009-10-27T11:03:00.001-07:00

I was holding up from purchasing a new laptop for over 6 months cause everyone told me that Vista sucked. I finally did it last Friday with a brand new one running Windows 7. An HP, of course. All that for a price that was half of what one could pay for an honest espresso machine! And the quarter of an equivalent Macbook (which is why I'm not a Mac user, they're just too expensive).

So what do I think of Windows 7?

Not much, actually. It doesn't seem to be on a suck-o-meter at first glance, which is a good thing. But I simply don't care. I'm not excited. I'm completely indifferent.

Why? Because I've realized that while my previous XP installation had accumulated some useful software over time, I'm not sure this will happen with my new laptop.

Thus why, last week-end while staring at my shiny new laptop, I found myself thinking :

>> Where do I want to go today?

That's a very interesting question.

Windows 7 is sure slicker than Windows XP, but the latter had raised the bar already in 2001 by adding a well-deserved feature that had been missing from Microsoft's consumer Windows line since its inception: it didn't crash.

While I used Outlook Express before, I didn't bother installing Windows Live Mail this time and decided to just use my webmail. I didn't bother shelling out money for Office, as I don't use it much and Google apps works just fine.

My brand new laptop is therefore just an appliance to run a browser. Besides maybe retouching my digital photographs, I simply have not much use for all the processing power, and slick Windows 7 features, of that laptop.

Is the web soon to become the platform for my home usage? You bet.

O.

Old motif habits die hard

2009-10-23T09:49:00.000-07:00

Just saw an e-mail today from our development group. They're planning on making a graphical dashboard for our proprietary application which runs on HP-UX. Everything on the backend has been CLI or text-based up until now, with the client running on Windows with MFC. But for that dashboard reserved to application administrators, developing a Motif application came out.

Yes, you read me right, Motif. Whew! Haven't seen development with Motif in a while.

Of course they didn't mean by this they would start develop in Motif right this morning (at least, I hope so). Maybe GTK+ would be more fit, and a good browser-based app would be even better.

My point is that Motif used to be associated with the commercial UNIX flavors for so long that saying "I'll make a Motif app on UNIX" is still a catch-all phrase like saying "I'll transfer files with FTP". Yes, these technologies have been there for a while and still work, but are clunky and outdated... and don't take me wrong, I used to root for Motif! That was in the mid 1990s. The Motif toolkit was, for its time, quite customizable using X resources. The no-frills window manager mwm offered a refreshing, KISS interface that actually worked on a workstation without crashing like Windows95 used to do twice a day.

All that was 15 years ago. Today, everything runs in a browser.

Migrating from OSEM to WEBES 5.6

2009-10-22T09:09:00.000-07:00

Here is a post on my experience migrating from OSEM to WEBES 5.6. It went well except for one minor problem with the MSA 2000.

First of all, you should know there are a few outstanding problems with WEBES 5.6, one of them being a security issue. I'm not sure I can disclose what they are as I didn't get them from official channels but I will say that a patch is expected sometime in December. If you don't need to run 5.6 right now in order to support specific hardware, you should stay with WEBES 5.5 and OSEM 1.4.8a in the mean time.

That being said, for those who wish to run WEBES 5.6, you can update it manually from Remote Support Software Manager. Also update to the latest version of Remote Support Eligible system List at the same time. The procedure is documented in the guide WEBES 5.6 and product coverage.

There is also some (redundant) information in the guides A.05.30 HP Insight Remote Support Advanced with WEBES 5.6 and OSEM to WEBES Migration Guide.

To migrate away from OSEM, if you have a standard installation with OSEM populated by info from SIM, it is as easy as simply uninstalling OSEM but you should read the above documents just to be sure. Once OSEM is uninstalled, stop WEBES (not documented, but I did it anyway) using "net stop desta_service" and "net start desta_service".

If you have hundreds of managed systems, it is better to wait at least an hour before testing if everything works well, as it takes a while for WEBES to stabilize and trap events once it is restarted. I also always confirm that e-mail notifications are enabled in order to have an alternate way of receiving notifications in case there is a problem up the food chain in SIM or ISEE.

Now you should test equipment that used to notify OSEM with SNMP traps to be sure they are being caught by WEBES and service events are opened at HP.

Here is what I tested successfully:

Proliant running Windows ... OK
Proliant running ESX ... OK
C3000 blade chassis ... OK
C7000 blade chassis ... OK
MSA2012i G1 disk array ... NO

The MSA 2000 G1 used to work with OSEM but no longer with WEBES. I've opened a ticket at the ITRC to have an official support statement. This is exacerbated by the fact that its events in SIM are reportedly sent as informational so those of you who "follow the red" could miss critical events.

O.

The grep of all games

2009-10-18T17:57:00.000-07:00

I normally don't talk about games in this blog, but I think a special mention should be made to World of Goo, a game I bought on Wiiware a few months ago and recently finished. It's actually the first game I've played seriously in years. Why? Because like a tool as ubiquitous as grep, it has a simple concept and few rules. It is easy to get the hang of it quickly. It is launched in a matter of seconds. And you can stop where you are, and come back later. Simply put, it's a masterpiece in terms of design, that even my two young boys mastered in a matter of minutes.

Give the game a shot. You will not be disappointed.

World of Goo Trailer 3
par 2dboy

Log management is done. Now, on to change control. And more pie charts!

2009-10-15T12:13:00.000-07:00

First, let me introduce you to the interesting subject of pie charts in the land of security compliance software:

I've had enough of all these software products whose selling point is that they're able to make pie charts. Like in our economic times, someone's job would be to sit in front of a screen all day staring at Pie Charts, making Pie Charts, and reading Reports With Pie Charts (if you're one of these people then sorry - I just can't understand how you can cope with this job). As a systems administrator, I want something that is, in order: 1. easy to use and deploy 2. responsive and 3. fits compliance requirements. If these include pie charts, then let it be, but that shouldn't be the only feature to look for. Seems that when your business consists of charging big bucks for software, looking for a bigger piece of the... er, pie, being no-frills is not a good sales argument.

Now I feel better. Let's move on.

I've posted extensively on various log management solutions I've been looking into in the last few weeks. Oh yes, and let's not forget my rant on the lack of info available on the website of some vendors that could have helped me get an idea of what their product does without needing to bug their sales team. Turns out there has been a total of 5 contenders (and not all of them had great websites, by the way). My business case is almost over, and while I won't disclose what I intend to recommend between ArcSight, Q1labs, a Balabit/Splunk hybrid solution and RSA, let's just say assisting to demos and speaking to a fair number of sales reps got me exhausted.

I could almost joke that had I decided upfront to use rsyslog and program a few perl scripts to extract the required compliance-related data manually, I might had been able to pull it off quicker for free. And if the auditor came in wanting pie charts, I could have been able to plot them in Lotus 1-2-3 like I used to do in high school and print them on a sheet of sprocket-fed paper.

Now it is time to turn that wheel again.

Yes, that's right, I have to do the process all over since I'm now looking for a Change Control solution that supports most of my devices. Change Control = knowing what, and possibly how, predefined critical files have changed on my servers from a certain reference point. None of the vendors above have one available I could piggyback on except Splunk's fschange which is not end-to-end enough and doesn't support HP-UX anyway.

I've looked into what's available and the names Tripwire and Solidcore pop up. I've used the academic Tripwire in the past, it did the job, but I need something that is based on a central server and supports multiple platforms, Windows being one of them. Maybe OSSEC? Perhaps, it is already running here successfully under the radar... but in my enterprise world, FOSS, especially when its intention is to reach compliance, is a hard sell even if it costs close to nothing.

Any suggestions on what I should look into?

O.

HPTF 2010: Yes. In Sin City. Again.

2009-10-13T14:15:00.000-07:00

The HPTF website has not been updated yet, but the HPTF facebook group posted an official announcement that it will happen again, still at Mandalay Bay. I was hoping for another venue, somewhere in the bay area would have been nice, but ah, well. I'm not the one who decides.

Thoughts on the Sidekick fiasco

2009-10-12T11:07:00.000-07:00

... What a fiasco. I can only feel sorry for the sysadmin team in charge of the data at Danger, is this due to their incompetence or simply pressure to deliver? We'll probably never know. But some heads will be rolling for sure, Microsoft will be associated with this mess for years to come, and this might be the end of their foray with the smartphone. This is another one of these data loss event that will go down in history. It's also a strong point against the Cloud, for anyone thinking about outsourcing their data.

It seems to have all happened during a "SAN upgrade". When you update anything on your SAN, you better have a DR site ready, and stop your replication before doing the upgrade. And that doesn't give you the luxury of not backing up your data correctly, which these guys at Danger didn't seem to be doing.

I won't start bashing any particular vendor in this blog, if you're interested in finding out who the rumors point to, I'll let you do your own search. No, it is not HP, but it's not far either. I can't give details, but it's not the first time I hear about a "routine" firmware update on a storage array that goes south. Yes, it is true that SANs are supposed to be upgradable online. But the more and more I think of it, the more I'm comparing the firmware update of a disk array to upgrading the thrusters of a jet while it's in the air. Yes, a jumbo jet can fly while one of the motors is stopped, but would you put your life at stake flying during an upgrade unless you really needed to? I wouldn't.

O.

Monitoring an MSA 2000 G1 with SIM and Remote Support

2009-10-08T08:11:00.000-07:00

I tried it, and it works. Here is a quick checklist:

On the MSA:

Go into Manage -> Event Notification -> SNMP Configuration
Configure your read/write community and the IP address of the CMS.

On the CMS:

Discover or identify the MSA if you have not done so already. If you have two controllers, you only need to discover one controller management IP address, SIM does not correlate together both controllers.
In the system properties, the product number will not be identified correctly. The product number burned in the MSA seems to be a valid HP part number, but the product number I had under contract differed and was a 6 letter number, so I copied it from my contract directly. Add the correct product number in the customer field under Contract and Warranty Information, as well as in Product Number field on the top, just in case. Check the two Prevent the discovery... checkboxes to prevent your mocked product number to be overwritten in the future.
Just to be sure, I also added the Care Pack directly in the system properties instead of relying on it being detected from the HP back-end, due to the problems I've had with the product number.
Re-check the entitlement in Remote Support, it should be green.

Go back on the MSA, and send an SNMP test trap. The event should be logged in SIM and a service event will be opened. N.B. I only tested this with OSEM as of now, as I have not yet had the time to migrate the SNMP monitoring to WEBES.

O.

Log management for the system administrator

2009-10-07T17:54:00.000-07:00

I've had an increased number of readers who have been following this blog since my first posts detailing my log management hurdles, so here is an update on what's been going on.

I've limited myself to talking to a small number vendors, for various reasons I won't explain here. But I'll tell you what I think you should ask yourself when considering purchasing a log management solution:

Do you want an appliance, or software that runs on your own infrastructure?
Do you want your log data to be translated to a high-level format, keep your raw logs, or do both?
Do you plan on deploying this yourself or do you need an onsite consultant?
Do you favor a solution that is easy to use or one that is feature rich?
Do you have the human resources to maintain the solution once it's installed?
And, of course, what is your budget?

Getting answers to these questions is, well, complicated. Buying software is like purchasing a suit: you have the choice of doing it online, at a rock bottom price, with no help whatsoever and without trying it on. You can also go downtown to stroll down a few department stores, where you can get a feel of what's available, look at the price tags freely, and possibly get some minor adjustments done. Or you can go to a full-service luxury store, where someone will help you pick the perfect suit. Whatever you do is up to you, but I think you get my point.

If you're the department-store type of person, you can assemble some of the components by yourself. While getting your hands dirty will give you more control on the solution and possibly save some money, you need to be sure you'll be compliant with your auditor's requirements once you're done.

Instead of an appliance, getting the specs and a quote for an enterprise-grade x86 server running Linux or Windows isn't rocket science. Enough said.

To centralize your logging, if you're already familiar with syslog-ng, Balabit's Premium Edition of Syslog-ng has few secrets, they have a well-written whitepaper on the suject, and you can even get an instant quote online. If you're on a zero budget, rsyslogd a free alternative but I think syslog-ng might sound better to possible auditors, as they've been hearing about it for years.

As for the log drilling itself, which I decided in my documents to call deferred log analysis, I still don't know what can make the job as I have not finished that part of my architecture yet. I've seen both free and commercial solutions, and up until now Splunk seems to be a strong contender in this area. But I still need to figure out exactly what our tech people will be drilling for, and what the auditors will be looking for in terms of high-level, bells-and-whistles reports, before making my own decision.

The last part is the real-time log analysis, for which some IT security people tell me that it is "not automatable". I have doubts on this statement. While enterprise-wide solutions require dedicated staff, our needs are at a departmental level; I therefore think it is possible to pull it off with limited human resources. We'll see.

Connecting a MSA2012i through a Virtual Connect with ESX

2009-10-04T16:38:00.000-07:00

A year ago, I ordered the required building blocks to install a small ESX cluster in a remote office: a C3000, a few blades, and a MSA2012i. It was my first iSCSI implementation. It took a while to get it racked because my team was busy elsewhere, but now that it's done, I had to experiment a bit to make it work correctly.

The MSA is not an HP design. It's made by a Carlsbad, CA company named Dot Hill. The documentation and web interface are not up to HP's usual standards. (the interface has been upgraded with the MSA 2000 G2, but I have a G1). Furthermore, there is not much information explaining how the controller failover works, and this is important to set it up correctly. There is a very good document here in the ITRC KB that you must read before deploying these devices.

Go read it, right now, and come back to this post when you're done.

Here's how I integrated this through a Virtual Connect. That's not how you should do it, that's how I did it; if there are better solutions, please drop me a comment as I would be glad to hear about what alternatives are possible. If you google around, you'll see that some people have made similar setups to this one.

Single Controller Setup

Above is how you should hook everything up with a single controller MSA. The reason for using two different subnets is to isolate them as if you were on two SAN fabrics. If you use the same subnet, ESX will gladly team both pNICs under the same vswitch, and since one pNIC is active at a time, you won't be able to see both paths at the same time. There might be workarounds but I suggest you save yourself some trouble and use two separate subnets. Be sure to create a vmkernel interface on each one of these subnets, as well as a two service consoles too.

How does failover work? Well, esxcfg-mpath will report two paths for each iSCSI device. So you are free to shut down or update the firmware of one of your Virtual Connect's with no downtime. I tried it, and it works as if you were on a fibre channel SAN.

Dual Controller Setup

With two controllers, you are required to add two switches because of the way the controller failover is designed. It did not find a proper way to hook up both controllers to the Virtual Connect - it insists on teaming the two controllers, and shutting down Controller A doesn't turn off its link so the VC doesn't failover to network Controller B.

In the iSCSI initiator, configure only 192.168.10.10 and 192.168.11.10 - don't bother with the IP addresses of controller B. Although the MSA2012i is not supposed to be active-passive - I've had trouble configuring paths on both controllers at the same time. If you're experiencing long delays booting ESX or scanning your iSCSI HBAs, be sure to reference only to the IPs of the master controller in your iSCSI initiator setup.

If controller A fails or is shut down, controller B will takeover the IP addresses of A automatically and you'll be able to resume I/O. ESX will not even switch from one path to another, as the path is bound to the IP address -- 192.168.10.10 should be out of reach for 30 seconds and come back magically.

As I said, this might not be the best solution, but it worked for me. If I ever revise mine, I'll update this post. Good luck.

O.

Integrity fibre channel card firmware quick dive

2009-09-30T18:36:00.000-07:00

I've had a fixation on the firmware of these cards for a while. Why? Because while I never update firmware on LAN cards, I still have many 2Gb fibre cards in my environment and they are based on designs probably made sometime around 2005-2006. Their firmware took a while to become mature and support specifics on the Integrity platform such as vPars, so there have been a few firmware releases in their lifetime.

There are two official and documented ways to update the firmware on QLogic-based fibre cards shipped with Integrity servers running HP-UX. The first one, which is also the easiest, consists of putting in the server or through Virtual Media a recent IA Offline Diagnostics CD (which comes with up to date drivers for many cards) and run fcd_update at the EFI Shell. The other one, which I personally prefer, requires you to copy the firmware files on the EFI partition, reboot to the EFI shell, and run fcd_update from there.

However, few administrators actually need to update this firmware as each release of the HP-UX fcd driver comes with the required RISC firmware and it updates the card automatically if required. The only situation where one might need to update the firmware manually is when booting on SAN as it might require updating the EFI driver. More on this below.

That's where being able to do it online can come handy if you want to save time. Why? Because flashing the firmware offline requires, from my experience, around 10 minutes per port and that can become very cumbersome if you're dealing with a rx8640 with multiple vPars or a Superdome. That online process is not well documented but I found this document here which explains how to do it using fcmsutil. It is actually easy: Simply run fcmsutil once to update the RISC firmware, and run it again for the EFI firmware.

So what is the difference between that RISC and EFI firmware? The same document linked above, although not well written, provides some definite answers. The RISC firmware is the storage processor on the card, which, not surprisingly, is based on a RISC chip; it it used to implement the fibre channel protocol. On the other hand, the EFI "firmware" is in fact an EFI driver embedded in a 2nd flash ROM on the card which is loaded by the EFI when booting the server. vPars themselves also go a bit deeper and I won't go too much in details here but they require an additional layer named fPars, or firmware partitions (Alan Hymes from HP has good slides on this), and that EFI firmware must support them if you're running vPars.

I hope this clears things up for you. Good luck with your firmware update endeavours!

O.

Moving a C3000 and C7000

2009-09-28T17:02:00.000-07:00

Today, I had to move two blade chassis, a C7000 and C3000, to two different locations downtown. We no longer had the original packaging and this being sensible equipment, my fellow sysadmins and I didn't want to hire movers and risk having some parts broken. HP does offer an official moving service, and they will cover anything that breaks once at your destination, but it can be costly. As two blade chassis can fit quite well in a minivan, they can be moved around easily as long as you're cautious.

You know you're dealing with true geeks when you see a bunch of guys shoveling on a hand truck a naked C7000, tied to it with old orange fiber optic cable because they didn't find anything else. That image was so cool, I should have taken a picture. But man, these suckers are heavy. Even with all the blades, power supplies, fans and interconnects removed, you'll still need to be two to hold them up. And whatever yo do, don't drop'em, especially if you have off-the-shelf Hush Puppies right underneath.

Looks like things are still like what they used to be!

2009-09-26T15:42:00.000-07:00

Last Wednesday, I tried to renew my subscription to a consumer protection magazine I've been reading for the last 11 years. I don't know how many subscribers it has, but it must not be beyond 100 or 200K so it's fair to expect their web services to be limited. Yet, they offered the possibility of renewing over the internet, so sure, I decided to save some carbon dioxide and use their web interface instead of snail mail.

Wrong idea. By following their subscription process, I ended up in the profile of another customer and saw his personal info. I didn't do any effort to do get there, and by that I really mean NONE. It just popped up in my browser. Looks like our sessions got mixed up. Man, even something using the infamous formmail would have given me a better sense of security! Looks like things are still like what they used to be.

There was no credit card info, but enough data to try doing a fraudulent phone call since I not only knew the guy's birth year, but also his address, phone number, and the pinnacle of it all: that he was subscribed to a highly respected magazine, along with the expiration date of his subscription... Social engineering anybody? Sure, many people put all this on display on facebook, but I'm note sure that customer would have liked me calling him up.

While a mom-and-pop operation could be a little more excusable, I'm surprised considering the nature of that publication that such a thing could happen. I left them an e-mail with a screenshot and sure hope they'll fix this soon. We're not in 1995; we're in 2009, and a bug like this shouldn't have gone unnoticed. And no, two business days later, I didn't get any reply to my mail whatsoever.

Needless to say I decided to delete all information in my profile... and send everything through the mail.

O.

Comparing log management products

2009-09-25T17:29:00.000-07:00

In the last few weeks, I've been looking into SIEMs and log management products. Yes, you know it already, I've blogged extensively on how I was upset that I had to go through a sales channel to get a bit of info, but promised I would give out details on what I preferred between ArcSight and Splunk.

It turns out that doing a public comparison of these products won't be easy as ArcSight gives out technical info only under NDA. While I can probably announce loudly that "their appliances log stuff", I can probably say no more. So technical details will remain sealed to my business documents. Sorry. One thing I can say, however, it that their range of products seem to be the Cadillac of log management, and everything I could possibly think of needing to better score at our next audit will be in it.

Concerning Splunk, I inquired about ESS using the "contact sales" button as I didn't find much details on that application. They left me a VM some 4 business days after my initial request for info although I said in it I preferred e-mail, and that didn't rub me the right way (I hate voicemail but that subject is more fitting for a future blog post). No follow-up e-mail. I'll try to call them back when I'll be near a phone when it's California time, and with all these governance'n'compliance-related meetings I'm assisting to these days, it might turn out to be never.

Q1labs read my blog, knew I was looking for log management products, and gave themselves the trouble to track me down and find me at my workplace. I normally would have turned them away, but they showed some good will by having someone call me up in french, and their products being designed in Fredericton N.B., I just had to give them a chance. I saw what they make and it's similar in spirit to what ArcSight does, and their selling point is that their technology is simpler and quicker to deploy than ArcSight's. It sure looks interesting.

I'll see what political pressures I'll face internally but compared to some other cost centers in our company, for us IT is an expense, not a revenue. What will determine whoever wins might come down to be strictly business... as long as the tool does the job and has the feature set we're looking for, the financial aspect might end up having the most weight.

I'm all new to pleasing this IT Governance gestapo that came out of nowhere to bully our small, under-the-radar-IT dream team. But from what I understand until now, I first need to submit a "business opportunity" document to them to justify my funding, giving ball park figures and a few vendors, THEN I can make another "business case" document to explain which one I've chosen. Such a process takes time, and when I cannot give any clear timeframe, it's no wonder that these sales people get their hopes down.

Want to know why I prefer Open Source software? Because since it costs nothing, I've been able to pull it off for years without having to go through this shit. Now I'm knee-deep in it.

O.

HPTF 2010:if it happens, what would you like to see?

2009-09-23T18:29:00.000-07:00

HPTF 2010 has not been confirmed yet. But should it happen for a fifth year, I sure hope to be able to make it again as I enjoy presenting to my peers very much. As abstracts must usually be submitted in January, I started thinking about what I would like to talk about in 2010.

I'll keep it to a technical presentation on what I know most and like the most, and that is - what a surprise - HP-UX.

The year 2008 was spent on increasing the availability and resilience of all mission critical systems under my responsibility. In 2009, my research and efforts have been increasingly towards manageability and security. The security aspect is totally not under my control, and I should rather talk about compliance rather than security. The two might be complementary but they're totally different. And I don't find that subject interesting.

So I think my 2010 paper will be in the manageability area. This being said, my current ideas for subjects are:

Integrating HP-UX systems in a Nagios Core monitoring environment
Easy and secure monitoring of HP-UX servers with SIM and Remote Support
How I manage my HP-UX environment without getting paged

You're welcome to cast your vote on what you would like the most.

Are enterprise software details accessible to the average joe?

2009-09-15T19:31:00.001-07:00

The post where I bashed an enterprise security software vendor because it wasn't possible to obtain technical information on their products without leaving personal information, and going through the sales channel, got me a lot of e-mails. Well, I wasn't exactly right. I discovered that other vendors in the SIEM industry follow similar standards and don't provide much information, except a feature list, without requiring visitors to register first. Even one product which is spun off from an open source project seems to do the same ! And no, I won't tell their names explicitly this time as I don't want this post to end up on Twitter and get blown out of proportion again. This blog is named Technocrat-UX, not Cranky-UX.

Having used lots of infrastructure security software over the years, where I never had any trouble getting an idea of what these product did exactly, all in a discipline where disclosure is paramount, I was surprised by the way SIEM products are presented. They're in their right to do it that way, but to me, a website is like a store, and if it makes me feel like I've just crossed the door of a very special car dealer instead of my corner Toyota dealership, my interest wanes quickly. Maybe it's just me. After all, I'm a Unix guy.

Perhaps companies that sell products based on business requirements, rather than technical requirements, have a modus operandi I'm not familiar with? Maybe they're, justifiably so, only targeting people with a business education instead of a scientific one? This is possible. So let's check. I've assembled a list of six "Enterprise" software products, and spent 15 minutes checking their websites to see if they have information relevant for a systems administrator. I've voluntarily excluded Open Source software since that wouldn't have been very fair. I also excluded HP software, as I've accumulated 10 years experience of searching through their web maze.

This is way, way, far from thorough. But here are my quick results.

Databases:

Oracle 11g: Has lots of information freely available, and documentation is free to access.
IBM DB2: Same as Oracle. Even better arranged than Oracle, with technical documentation easy to access.

Enterprise Content Management:

Opentext Document Management: I need to register just to see a spec sheet. Yuck.
EMC Documentum: I was curious about EMC, since they also make kickass hardware and own VMware, but for Documentum I also need to register to see info. DoubleYuck.

ITIL-related service request systems:

CA Service Desk: I wasn't expecting much from CA but I was pleasantly surprised. They have lots of info, and access to manuals is free. I'll see CA differently from now on.
BMC Remedy Service Desk: Information is passable, and manuals are not available.

WEBES 5.6 just got released

2009-09-14T17:49:00.000-07:00

WEBES 5.6 has just been released. There are no release notes on HP's web site but from what I've been able to gather, the two major changes are that it now uses PostgreSQL as its embedded database instead of relying on SQL Server, and it seems to replace OSEM outright.

I don't know if I will have time to try it out this week but I'll follow-up as soon as I can. For instance, I'll test if the old OSEM monitored devices I have will work out of the box (namely, Proliants and B-Series fibre switches). I'm also curious to see if the old SQL Server database will be deleted during the upgrade. I had some interface timeouts when checking HP-UX managed systems with 5.5 and I can't wait to see if they are resolved.

O.

Updating a server to 11iv3 while keeping it in SIM

2009-09-01T12:13:00.000-07:00

Here is how to update a server to 11iv3 and keep everything working in SIM and RemoteSupport.

Update or re-install the server following your own procedure.
If necessary, configure all the requisites on the updated server so that it can be integrated correctly with SIM (there are too much to detail here, but this post can help)
Once the update is done, log into SIM and show the System Properties page of your server. Confirm that the two checkboxes "Prevent the discovery from changing these system properties" are unchecked.
Launch a discovery on your system. In 5.3, the process has changed: you need to create a discovery job and specify directly the target server in it.
Subscribe to WBEM events from your server from the Options->Events menu

If using RemoteSupport:

Redo an entitlement check to be sure that it your server is still entitled correctly.
This part is important, you need to restart WEBES (stop director, start director) or else I don't know if and when it will resubscribe to events. I waited 24 hours and it didn't subscribe, so screw it, I restarted it (I know that sucks, but I didn't find out how to force a resubscription besides restarting WEBES). Restarting the director results in WEBES subscribing to your server eventually, this might take a while depending on how many managed nodes you have.
Confirm with "evweb subscribe -b external -L" that there are SIM and WEBES subscriptions and run "sfmconfig -t -a" to test the delivery of events to SIM and the RemoteSupport back-end.

Good luck

Using OFM to update firmware on rx7640/rx8640 series

2009-08-31T17:30:00.000-07:00

In the past, updating firmware on cell-based servers was a daunting task, requiring an FTP server usually piggybacked directly on the MP, and lots of manual commands to flash each part independently. Not anymore. HP wanted to charge me to come in and flash a bunch of servers so this gave me the opportunity to do it myself. I flashed among these a two-cell rx7640 using OFM, and it's now dead easy: simply download a .iso file, burn it on a CD, and boot on it. It uses OFM which has been available for low-end Integrity servers for a while. We're still far from the Proliant Firmware Maintenance CD, but nevertheless it's still much better than nothing! One detail: the MP has to be configured to "allow upload of firmware updates from the OS", which is enabled in CM>SO. You still need to cut off AC power though at some point, so an onsite update is still mandatory.

And for those who still have rx7620s, there's no OFM version of the latest firmware and you still have to do it the long way. While it's not as trivial, it is at least well documented.

Apache.org hacked. What the hell were they thinking?

2009-08-28T08:48:00.000-07:00

As many will know already, apache.org has been hacked yesterday. While events like these are rare, and sometimes look like science fiction, the path taken to exploit their servers was a relatively easy one that, if I understand it correctly, shows gross negligence from their part.

Here is my analysis of what the apache team posted today:

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org. The account was accessed using SSH key authentication from this host.

Having your SSH keys stolen is a possibility. With automated tasks, keys are not protected by a passphrase, so anyone who gains access to them can easily use them for their own purpose. The first line of defense is to protect file access to your private keys as much as possible, and use a dedicated user to own it. How were these keys stolen? That could possibly be inside job, and you can probably bet that it didn't require root privileges to grab it. If it did require root, then that should narrow down the culprit unlesss the provider got highjacked, too.

Or perhaps that "3rd party hosting provider" didn't bother protecting the key at all, leaving it world-readable, and didn't chroot its inbound data transfer accounts, so anyone who has FTP access to the server to upload his own stuff could have stumbled upon it by snooping on the server. For that part, I don't know.

But the next part is particularly interesting:

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts. These files were then rsynced to our production webservers by automated processes. At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

Now get this. From what I can see here, there a few problems here:

Whoever owns the SSH key can upload stuff on minotaur.apache.org. That's not really a problem per se. But there is probably no filtering done on the IP address to limit inbound connections to the provider's netblock, so he can come from possibly anywhere. That might be a usability requirement; in that case, you can bet they'll probably think about requiring port knocking from now on to at least mitigate the possibility of this happening again (and I insist on the verb mitigating, as knocking is more obscurity than real security). And for this to be "safe", the hosting provider will have to keep the knocking sequence as safe as the key.

Whether the account is kept under a tight leash on on minotaur, such as using a chroot jail or whatever else, doesn't make a difference! Why? Because data uploaded to this account is rsynced automatically from that account to the servers running www.apache.org, unverified. So you can possibly upload any nasty code you would like to compromise anyone reading a page on www.apache.org using an exploitable browser.

Hell, who knows, maybe these hackers have been injecting compromised pages for a few days as what seems to have tipped the apache admin off are rogue processes on their servers. They were launched remotely quite easily, as data can be rsync'ed straight in cgi-bin/ ! Now how good is that?

While not thinking about details like that inside a corporation is standard practice, and tolerable in many cases as there is an implicit trust within the organization, as soon as you have a server with a gateway exposed publicly on the net you need to take precautions to isolate it from your production. In this case, it's clear to me that the Apache group didn't think this completely through. Being the authors of a secure and great web server, being hacked like this will probably go down in as one of the shameful events in the Apache group's history.