For the next few parts of this mini-series, I’m going to start looking at the initial high level design for the system and how well it meets the requirements. <\/p>\n
First of all I’m going to talk about the most critical database from the business perspective. I’ll call it the User Session database. It’s the most critical because, if it’s down, customers will lose their connections to what should be a 24x7x365 service. There are multiple app servers managing connections so if one of the app servers fails, the service is capable of restarting their sessions on another app server and retrieving their session information from the database. All very cool and reassuring – all that pretty redundancy. Erm, but if the database is unavailable, all of the sessions will eventually time-out, won’t they? Won’t all of the app servers prove worthless then? The database, therefore, is a Single Point Of Failure (remember that).<\/p>\n
The two most important bits of information I need at this stage are the proposed design (from the last blog<\/a>, although I noticed I’d used the lax ‘database instances’ terminology so I’ve corrected it here and there)<\/p>\n and the true up-time requirements. This depends on how the application manages it’s connections and you often need to dig pretty deep with the vendor to arrive at a final answer. Our best answer at the moment is ‘less than 5 minutes’.<\/p>\n So what’s wrong with this picture?<\/p>\n Well, during the first meeting, I suggested that if maximum up-time is the focus, RAC might not be the best answer. I appreciate that might come as a shock to some people (it certainly did to the meeting), but consider this.<\/p>\n If a given server has an expected uptime of 99.9%, then what’s the expected uptime of two of those servers in a two-node RAC cluster? Is it 100% (unbreakable)? 99.9%? 99.8%? Do we even know? Is it perhaps more<\/i> likely to fail than a single node? Are we factoring in the possible failures of the underlying cluster filesystem or RAC or basic human error on a more complex configuration? My point is this, why oh why (rant approaching) do some people buy the message that RAC must mean 100% up-time? What will it protect you from? Node or Instance failure. How often do modern servers fail when balanced against how often something goes wrong because a) the software failed or b) a human being screwed up? <\/p>\n However, I’d say that there’s an even bigger hole in this design. One that a careless elephant could drop through when out for an afternoon stroll. Remember I mentioned single points of failure? Well, we have two instances for this database on the RAC cluster, but, stone the crows – only one database! What happens if something goes wrong with that? How is RAC going to help us then?<\/p>\n The answer I was given was – well nothing’s going to go wrong with that because it’s on high-end RAID-ed storage. Give me a break! Even if I forget <\/p>\n Even if I forget such things, what happens when some idiot (perhaps this idiot) mis-types something and screws up the database? What do you think happens more often these days, user error or hardware (system) failure?<\/p>\n Let me be crystal clear about this. There is only one database. Have you never had anything go wrong with the database itself? So what are you going to do when that<\/i> happens. How are you going to patch the thing?<\/p>\n To understand this stuff, you just need to have worked with OPS (Oracle Parallel Server) or RAC RAC is nothing new. Clusters are nothing new. There are limits to what they will protect you from.<\/p>\n Next time, I’ll address the single database problem.<\/p>\n","protected":false},"excerpt":{"rendered":"\n
\n
\nor clusters in the past, but if you haven’t, here’s an excellent article<\/a> to
\nreinforce what I’m trying to say.\n<\/p>\n