Documentation and Asking Questions

(Rant warning. Well, if I never write anything contentious, you might all drop off to sleep without me noticing …)

Over the past few years I’ve had many conversations with different DBAs (at least four teams worth!), System Administrators, Managers and even her indoors (poor woman!). It’s a subject that’s very dear to my heart as several of the unfortunate recipients of my arguments would tell you 😉 Frankly, people are sick of me rabbiting on about documentation and processes, so this blog has been bubbling around in my little brain for a long time. (i.e. Can I just emphasise that this blog describes just about every new site I turn up at and not just the latest?! That statement is completely true and I’d be upset if this was taken as a description of how things are at Pythian. In fact, they’re far better than average when it comes to documentation and particularly processes but, like all places, they’re not perfect. Not least, because the place is stuffed full of DBAs 😉)

Other recent inspirations include Jeff Hunter’s blog on Cookbooks, which I agree with wholeheartedly, and one of the quotations on the front of Jonathan Lewis’ blog.

“Sun Tzu: When trouble is solved before it forms, who calls that clever?“

I don’t think good documentation was the motivation for that, but it does to get to the heart of my distrust of the ‘Hero’ DBA.

Let’s start with advice repeated many times on the Internet in guidelines for newsgroup postings, blogs, articles, flame-mail (even the funny ones, between friends). You should look for an answer for yourself first before firing off questions to large communities, looking for easy answers. Not to do so is lazy and leads to constant repetition of the same questions. You’re also less likely to learn new things in the search for answers to your specific questions. The gist of this argument is that people will normally be happier to answer your questions if you’ve tried to find out the answer for yourself first.

I have some sympathy with this point of view, particularly when there is plentiful vendor documentation freely available. In an office populated mainly by people who operate in today’s electronic domain, it isn’t surprising that there’s a certain kudos attached to those that can work things out for themselves without constantly asking questions.

Let me explain why I think this approach is utterly wrong for the workplace and that everything that’s been done before should be passed on to the next person as clearly as possible.

What a waste of time. When I’m working, I want specific, accurate answers as soon as possible. Whilst I might learn some interesting new facts whilst searching for the salient ones, I might also waste a lot of time chasing up blind alleys. If someone else knows the answer and it’ll take them 30 seconds to tell me it, I struggle to see the value in taking 10 minutes to work it out for myself, beyond some sort of boost to my tech ego. This is a job we’re doing, not an interesting little hobby (and I say that as someone who seems to enjoy it more than most). Multiply that 10 minutes for each new employee and maybe it’s better just to write it down and tell people where it is. It’s one of the reasons Pythian keep a record of everything the DBAs do – so that you can look at previous work.

Look, I know about Oracle, but I don’t know about each individual site’s processes, procedures, hostnames, passwords, which bloke to talk to get an ID card, which lady can create me a secure account on server y, what that specific error means and so on. Of course you know the information – you’ve worked in the place for 2 years! Maybe you had to work it out for yourself and it took you a month or two to get everything sorted out, in dribs and drabs. Don’t you think it might be a tad wasteful for each person to go through the same process? Maybe some people think – ‘well I had to learn it the hard way, why shouldn’t he?’ – but I just don’t get that point of view.

What a waste of skill. I don’t mean me but, let’s face it, most people who work as DBAs are pretty highly paid and I’d hope that we’re not getting paid to flail around, trying to work out what we’re doing. i.e. There are better ways of spending our time, including – shock, horror – documenting things so that only one person has to go through the cycle of learning and then share it with everyone else. We have enough on our plate keeping up with new features, new problems that no-one’s ever seen before and new applications coming through the door. Let’s maximise our time on what’s new and interesting, not re-work.

Documentation imposes self-discipline. If you can’t document it, there’s probably something wrong with the way you’re doing it. The minute someone starts saying to me, ‘Ah, well, that shouldn’t normally happen, so don’t worry about it, or go and speak to John when it does and he’ll help’, I start to worry about the process itself. Maybe by documenting it, it’ll flush out the flaws in the process. Is there really a process so complicated that only one person can do it? Better still, when someone else attempts a task that ‘only John’ can do, it’s the perfect test of the documentation. The chances are it will fail – it always does – but that’s when you get the chance to improve things.

Individual information and power. Speaking of ‘only John’, none of us should be indispensable, although it’s natural for most people to enjoy the feeling. When I see a situation where there’s only one person who can perform a task, I question the management. It’s not John’s fault that he’s the only guy who can perform the task. In fact, good on him for having the skills (although he might start getting sick of the workload and pressure, too!). However, it’s bad for business – what happens when he gets run over by a bus? (I suppose he’ll at least go with the knowledge that he was the only true expert on process Y, but I doubt that would give much comfort.)

Information and skills are incomplete. If everyone learns things through some initial basic knowledge and then on-the-job learning through trial and error then everyone ends up with different skills which are almost inevitably incomplete. The best example I can think of here is Unix shell programming (of all varieties). I bet experienced DBAs will recognise this straight away. My shell programming skills are limited. Why? Because I learnt them by looking at other people’s scripts, hacking around, having a go myself and learning through trial and error. Now that might have been rewarding for me, but is that really the best way to learn an important skill for my job and can I really be sure that I’m not one small misunderstanding away from screwing things up really badly? When I do, will it be an interesting exercise to fix it, or just plain amateurism? Wouldn’t it be better to learn the thing properly than to impress people with the myriad subjects I know a bit about and can ‘get by’ on?

Let’s say that I do manage to work almost everything out for myself? What happens when I find out that actually there’s a few snippets I missed here and there? Does everyone have to make the same mistakes based on the same little bits that they were missing? Doesn’t that make you as miserable as it makes me and, for what, so we can admire our ability to think on our feet?

Caring about our colleagues. I know that might sound a bit touchy-feely but, ultimately, the aspect of this that upsets me most is that it’s so damn selfish. I love making sure that other people know what they’re doing because I know what a horrible feeling it is to not know what you’re doing. I wouldn’t wish it on anyone. It’s selfish in the wider sense, too. Are we all to remain geeks, sitting in our corners, stuffed full of pride at the things we know that others don’t, or are we in it together and want to become better as a team? I realise writing good documentation and answering people’s apparently dumb questions is time-consuming and can be a bit frustrating sometimes, but don’t we all grow better together as a result?

Phew, I think I need a lie down!