Wednesday, March 31, 2010

The sysadmin dilemma

I just stumbled upon Matt Simmons' "Culture of Quitting" post where, besides talking about the fascinating concept of Up or Out, he puts in a nutshell his motivations for being a systems administrator:

No, I (and probably you), have intrinsic motivation. I don’t expect direct rewards (or even outward appreciation, typically) from doing my job. The reward is that my infrastructure works the way it should. Sure, I have certain long term goals, but I can’t accomplish them if I don’t accomplish my short term goals first.

That's where I got my motivation, too, for quite some time. I don't know any sysadmin who wouldn't be proud of putting in place a high quality, resilient infrastructure. But the problem with this, though, is that not only can it get expensive, it never breaks. And when nothing goes sour once in a while, it's hard to get noticed (and further motivated) by all levels of management unless you're lucky enough to work for people who are perfectly aware of what you're doing.

So what should be done to get a tap on the shoulder? Be a below-average sysadmin? In other words, don't produce results too quickly, don't try to optimize everything right away so that performance issues are apparent, constantly say no to user demands... so you can come back later as a hero by finding solutions to "complex problems" to save the day? That might not make any sense, yet I'm slowly starting to think it does: Under some circumstances, the only way to actually show you're doing something productive is to spend a great deal of your time addressing issues which are visible to management.

This is the base of what I've learned to mockingly call "the sysadmin dilemma": If you're doing good work, you won't get noticed too much, and will risk either staying where you are for years with no chance of being gratified, or worse, you'll end up having to justify your job. On the other side of the dilemma, do bad work which costs big bucks to your employer, and you'll be shown the door quickly.

So I think the best path to take if you want to avoid a sysadmin dilemma is to put your target on being average. Be necessary, but don't be too good. But it is obvious to me that being an "average" kind of person will probably not be a fitting motivation for types like me and Mr. Simmons.

Up or Out.

That sucks, but life in the enterprise does, doesn't it.


Monday, March 29, 2010

See a candidate coding live during a remote interview

Following a discussion on Slashdot, I stumbled upon a site named See[Mike]Code where an online temporary "interview" room can be set up (for free) and you can use it to evaluate coders in real-time.

Simple and clever. The site brings up a unique URL for the interviewer and another one for the candidate. Everything the interviewee types is replicated, realtime, to the interviewer. That's an easy way to to evaluate the technical merits of someone without the trouble of bringing him/her in for a formal interview.


Wednesday, March 24, 2010

The redundant IT team

Following my resignation as the senior sysadmin at my division, I've had many of my colleagues come into my office, saying (amicably) "Darn! You've just put us in deep shit!". That's a matter of perspective; nothing should start breaking apart the moment I leave, and many servers can live without any maintenance for a while. No, I'm really not leaving them in distress, as for many years, I've been thinking the following:

Everyone is replaceable.

...a quote which probably makes me fit for management, because that's exactly what managers think. And they aren't wrong, not at all. They are right. No key IT staff, be it a sysadmin, developer or tech support person, should hold all the information and knowledge to keep any part of your business running. This is especially true with small IT teams of less than 50 resources where there is no implicit redundancy that covers everyone.

Yet, that's the position that many IT managers will put their staff into. When you're counting beans, the only thing that matters is keeping a predefined quality of service at the lowest cost. While hardware manufacturers will be hard-pressed to justify various expenses due to all the redundancies they pack into their stuff, remember that people need to be redundant too! And when the shit hits the fan, no amount of preparation will be of any help if you can't count on support staff who is experienced with your systems. Which will introduce my second quote:

If you can afford redundant hardware, you should be able to have redundant people too.

By "redundant" I don't mean to double up your resources 2 to 1. But you really, really have to reduce the chances of someone being the sole owner of a key role without an assigned teammate.

These rules should then be followed to put into place a redundant IT staff:

1. Make sure no one in your staff has a unique set of skills and knowledge. Someone must be able to replace any missing resource, to some level, quickly. This is for valid for everyone in your IT organization, as nobody answering the phone on your tech support hotline (or giving wrong information) could end up being as bad as somebody else inadvertently initiating an IPL on the mainframe.

2. Require a standard and consistent set of documentation for hardware and software solutions before putting anything into production. Docs that are formatted with a standard set of sections will be easy to skim through by anyone who needs it. The UNIX man pages are a stellar example of consistency. If you don't have the time to mess with recognized processes and standards, then simply don't; a word processor template is all you need to get your guys going.

3. Organize regular lunch'n'learn sessions where someone presents a technology subject to its peers. Not a deep dive, but just an overview so they know about it. Insist on quality presentation documents as they can become a great reference later down the road when training new recruits. And don't be a cheap bastard: if you want to motivate people to come and listen, you better pay the lunch.

4. Treat your key resources well, so that they aren't tempted to go elsewhere. By "key" resources, I mean anyone who has deep knowledge of proprietary systems, and for which there is a shortage of qualified workforce on the street. For that matter, enforce rule #2 with them. Even if you can count on someone else to fill in when they say their goodbyes, you'll still need to hire someone further down the road, so it's best to make sure they don't leave.

5. Don't hire smart asses and douche bags. And I'm serious about this. The smart ass is easy to spot in an interview; that guy will claim that he knows everything, sometimes amalgamating nonsensical buzzwords, so just grill him with very technical questions prepared in advance by your staff and he'll fail miserably. The second kind is harder to spot. In my career, I've crossed a few computer science folks who were very talented but also impossible to work with; these aren't team players, they don't trust anyone, and keep everything for themselves. They must be avoided at all cost as they have the ability to sink your operation. Besides a personality test, which they might be able to trick, there's not much you can do to find out except asking for references. So do you get me here? You need to find a balance between social and technical skills depending on the type of job you're offering; low social skills don't fit well with tech support but might be acceptable for a senior developer. Hell, that point is getting so long, I think I'll make it a blog post all by itself one day.

6. Hire people that are ready to hold many hats. Let the truth be told: some people are happy to be single task-oriented and will make sure it stays that way. These do not fit well in a small IT team, as you can't motivate them to learn about new technology, especially if it's under one of their colleague's responsibility. They're NOT autodidacts and the first they'll always do is ask to be "trained" for just about anything. If you have someone who can use a jig saw, but refuses to even try using a reciprocating saw without going a few years to the School of Reciprocating Saw Professionals, then you have a problem. Of course, when working with a unionized crew, the rules are very different and I don't think that blog post will be of any help to you.

7. And last, nobody likes change management and avoids it like the plague unless they're forced to implement it to follow some crapass compliance rule. But change management, if done well with a minimal mount of red tape, can be very beneficial to your team. Any time someone changes something, it will be documented, which makes fixing any mistake easier if that person is not reachable. Up until now, most "enterprise" change management I've seen seem to be a bunch of expensive, incomprehensible software stacks. I've yet to find a simple and easy-to-use web-based system so I can only encourage you to spend a few days making your own.

This set of rules is by no means scientific. They're the ones I would do my best to apply, was I in a management position for an IT team. Fortunately for me, I'm not. Managing an IT team presents its own set of challenges: with limited money you have to keep your employees happy, the users happy, while at the same time ensuring that your enterprise's survival isn't in peril by ensuring that a reasonable risk management is done. Running a redundant team is one of the best way to lower that risk.


Tuesday, March 23, 2010

The page has turned. Let's get the new blog going!

Let's get the new blog going! Why not right here, right now! I might have decided to stop being a hardcase sysadmin, but that doesn't mean I have to stop blogging. Technocrat-UX is dead, long live Technocrat-UX! And let's welcome The ex-sysadmin!

Is there a life after the systems administrator? You tell me! In my blog, you'll follow an ex-sysadmin's endeavour into a world of business process intrigues, vague specs and internal politics. Experience has showed me that sysadmins don't always defeat the bad guys and get the girl at the end of the story. The question is, do system architects? This blog will try to find out the truth!

Expect pragmatic answers to the most elaborate questions. Simple solutions to complex problems. And of course, my own vision of market trending and analysis that you'll love to hate. Who says systems architecture has to be bleak?

But seriously, I first need some time to know what I'm doing. That should take a few weeks, if not months. So I'll document the process I'm going through to leave the sysadmin behind, and prepare the carpet for the architect.


No more Technocrat-UX

Dear readers,

Instead of letting this blog die, I thought it would be better to make one last post to say goodbye, at least for the time being.

After almost 10 years as a full-time system administrator, I decided to change my career path to become a systems architect. Which, at its bottom line, means that I'll stop writing shell scripts to write plans and documents instead. I will no longer be dedicated to HP technology, and HP-UX in particular. The decision has been hard to make, as I love my job as a sysadmin. Over time, I've participated to many HP-related events and met very interesting people at HP. I will miss this, a lot.

Over time, I've been saddened to see that my workplace doesn't put a lot of, er, "financial" value to its technical staff, no matter how much effort I've put into developing the best technical career I could. The only way to keep going up the salary ladder while remaining technical would have been to quit and try my chances with consulting, and I wasn't interested in doing this for personal reasons. So, I decided to stay with my public sector employer, relinquishing my purely technical job, and time will tell if I will like it.

I'm keeping this blog open for the time being, since the content should be relevant for a few years. If I ever start producing content related to my new experiences, I'll keep the same URL ( and switch the name from Technocrat-UX to something else.

Thanks to everyone who've been reading this blog since 2008. You've sent me many comments and suggestions that kept me going. Writing for Technocrat-UX has been a very good experience and I sure hope that the content has been useful to you.

Take care.

Olivier S. Massé

Thursday, March 11, 2010

HP-UX 11iv3 U6 will include an optional parallel rc sequencer

One of the questions users ask me most is How come HP-UX takes such a long time to boot and shutdown?

I always reply that the startup rc sequencer works serially, thus any subsystem that takes a long time to either start (CIM) or stop (OVPA) will have a negative impact as everyone else will be waiting in queue. And yes, I also add that Linux's been doing it in parallel for years.

In the old days, HP-UX used to stand above many other BSD-derived Unix flavors with its really nice startup checklist. Yet, as startup times are getting longer and longer, a new parallel sequencer was needed and HP announced one a few days ago through a whitepaper. I expect startup and shutdown times to decrease at least twofold in the long term with this.

Details are in this whitepaper here:

The RCEnhancement bundle is available in the software depot and it sure looks promising. At first glance, I'm not sure I like the way it's being implemented with the "rcutil" command when compared to some Linux offerings I've seen which use config files. On the upside, administrators used to the current SYSV sequence will feel comfortable right away using this one.

I know nothing of what's going on in the labs but my take is that the parallel sequencer is a backport of what's being developped for 11iv4. If that is the case, chances are strong that HP will rewrite many of their startup scripts under 11iv4 to use the new sequencer, thus promoting "now boots 200% faster!" as a marketing incentive to encourage users to migrate to v4 when the time comes.