For the latest in a (hopefully short) series of Burns screw-ups, I just need to cast my mind back to last Saturday. I’d finished my final on-call weekend at Pythian the weekend before<\/i> but, in a bid to give my team lead a night’s sleep and take advantage of our different time-zones, I grabbed the pager when I woke up stupidly early on Saturday morning – about 5am, I think.<\/p>\n
Pythian have a pretty robust system for handling pages which, for each team, includes the on-call DBA for that team; followed by the on-call DBA for another team that is our backup; followed by our manager; followed by everyone in the company. Each stage lasts 15 minutes and we get new pages every 5.<\/p>\n
So, anyway … while I was working on some personal stuff, the on-call DBA for our backup team came on-line, fixing a problem. We had an IM chat when he was done and I suggested I pick up his pager too, so he could get some sleep. (If you didn’t notice the first stupid mistake at this point, you’re not concentrating.<\/i>) He was very grateful, but he’s done us plenty of favours in the past.<\/p>\n
I carried on fixing a couple of minor problems, working on a few personal things and when I was finished, I decided to lie down on the settee to watch some TV, knowing I only had about an hour more on the pager, then I’d hand it back to the guys in Ottawa. (Surely by now you can spot the second big mistake?)<\/i><\/p>\n
A couple of moments later (because that’s how it seemed) my mobile phone woke me up with a text message alert – a page. Mmmm, except, when I went to look at my monitor, this was what was sitting in my Inbox.<\/p>\n
!Fully! Escalated Page for Client X<\/p>\n
Well, the text wasn’t really that big in Thunderbird, but it seemed<\/i> that big! Oh, and there were 45 minutes worth of other pages, building up to the big moment! I think my first two words when I spoke to my (now-awake) team lead on IM were – ‘Arrrgghhh!’ and ‘Sorry!’. Why was he awake? Well, it wasn’t because he’d been paged – I’d<\/i> been paged. It wasn’t because the backup DBA had called him to wake him up – I<\/i> was the backup DBA (twit!). No, it was because one of the other on-call DBAs, on being woken up, had called the guy who was supposed to be on-call – my team lead.<\/p>\n
That !Fully! Escalated<\/i> means everyone in the company 🙁 You can only imagine the apology email …<\/p>\n
Actually, people were OK about it and it wasn’t a serious problem we were being paged for, but lessons learned, <\/p>\n
Looking on the bright side? Well, <\/p>\n