Recovery Design

I sometimes think that a blog seems incomplete unless the author whines about how busy they’ve been recently. It would be easy to miscontrue that as a plea for sympathy. Maybe it is sometimes but another strong possibility is that it arises from mild feelings of guilt about posting frequency. There’s no reason to feel guilty, but some of us feel unreasonably guilty from time to time 🙁 It’s the parents to blame, I reckon 😉

Either way, I’ve been extremely busy over the past month, but at least work has provided potential blog material if not the time to blog it. That’s more than enough whining and on to a subject that’s always interested me – Recovery Design.

I thought long and hard over that description and there were other candidates I might have chosen – Disaster Recovery Planning and Designing Highly Available Systems are just a couple, or readers could fill in their own favourite. I’ll explain the requirements a little.

We’re implementing a new system consisting of perhaps 20-30 servers including five databases. In fact the Oracle components appear at first sight to be a small part
of the architecture but the simple fact that two of the databases can’t
suffer an outage of more than a few minutes is an indication that
they’re a small but probably critical component.

• A two server cluster containing two databases with a maximum down-time of 5 minutes
• A two server cluster hosting two databases with a maximum down-time of 30 minutes
• A two server cluster hosting a single database with a maximum down-time of 6 hours.

In all cases, there are application components on the servers too.

I’m not going to go into much detail about the purpose of the system, to preserve client confidentiality, although I’d love to because it’s fascinating. Suffice to say it’s a system that thousands of customers will connect to directly rather than being an internal business system.

My current client takes a pretty enlightened approach to designing systems like this. Although all of the software is being provided by third parties, those companies, the company’s internal technical experts and business representatives work on the initial high level design together before the expenditure is signed off and the implementation can begin. For example, it’s very difficult to establish the maximum down-time for a component without involving technical and business people. (Even then, it wasn’t easy!) I became involved as the Oracle guy and my first task was to review the third-party’s technical design.

Next up, I’ll look at that initial design and the tough questions I asked. For now, I’m off to put together my latest version of the technical design. It might be Sunday and I’m not supposed to be working, but deadlines are tight and I enjoy the work anyway 😉