I’ve left one of my favourite topics
\nfor the penultimate episode of this mini-series. The more sites you work
\nat and the more performance problems you work on, the more you begin to
\nlearn the one essential truth of modern system architecture. <\/p>\n
It’s never the SAN. <\/em> My cuddly toy mate, Flatcat, could analyse It *was* the SAN and I don’t enjoy The last part will discuss the problems P.S. Despite the tone of this post (which has been kicking around my brain cells for a while) I’m looking forward to Christmas more than a middle-aged man should. So, I’m not feeling grumpy and you’ll forgive me if I log off now and pick up any comments later. Then again, I’m not expecting too many comments for a few days ….<\/p>\n","protected":false},"excerpt":{"rendered":"
Here’s a true story. Rest assured that
\nI’ve watched the same story develop several times so there’s no possibility
\nyou can identify an individual client. In fact, the thing you’re most likely
\nto recognise is a similar experience of your own 😉 <\/p>\n\n
\nbatch schedule on a significant system. The problem is that the execution
\ntimes from night to night are unacceptably variable, for similar data volumes
\nand identical code. The batch window is already tight, so this leads to
\nthe system being unavailable during critical hours. The business is upset,
\nparticularly as they are fined if they don’t deliver certain transaction
\nmessages within set deadlines. <\/li>\n
\neveryone assumes there’s a problem with the database. Despite the usual
\ngrumbles at this point, I have some sympathy with that perspective because
\nit’s a key component of the overall system that’s both critical and quite opaque
\noutside the DBA team. Besides, we have the best instrumentation so are
\nlikely to be able to help. <\/li>\n
\n(this is a complex batch with hundreds of jobs in multiple job streams
\n– a little difficult to trace) and notice that the average single block
\nread time varies from night to night and there is a close correlation between
\nthe nights when i\/o performance is poor and the batch jobs take longer
\nto run. The vast majority of wait events are related to i\/o. At this stage,
\nwe ask the O\/S or Storage guys to take a look at our numbers and investigate
\ni\/o performance. They do so and come back with “no,
\neverything looks fine to us”. Which is a little tricky for me to deal with,
\ngiven that you’ve demonstrated single block read times over 30ms! Still,
\nwhat can you do? You ask for expert help and the experts tell you everything
\nis ok. <\/li>\n
\nof the batch jobs to show the individual wait times. These look even worse
\n– some are as high as 90m\/s! We start to look at filesystem configuration options, reducing the i\/o workload and several other bright ideas as we desperately cast around for a solution. <\/li>\n
\nto make it last. At some point, however, the business has had enough of
\nthis nonsense and calls in the storage vendor. <\/li>\n
\nmet every employee of every storage vendor and so I’m sure there are bright
\nguys there (maybe I’ve been unlucky) but, invariably, there is ‘no problem’ with the storage. Again.
\nThere never is. More to the point, they join in the general finger-pointing
\nin the direction of the DBAs, start asking for initialisation parameter
\nvalues (guaranteed to drive me daft, is that one, because it’s just a knee-jerk
\nreaction) and explain to the business that the back-end infrastructure
\nis working well. Of course, the last thing the vendor is going to do is
\nto explain that the system that they helped the business to specify (typically
\non cost per Megabyte, rather than bandwidth) is under-specified. Oh, and
\nif you’ve spent three million quid on something nice, big and shiny, the
\nlast thing you want to hear about are its limitations. <\/li>\n
\nmight start postulating theories which ‘prove’ it might not
\nbe the SAN and I’ve had a few extended conversations on this subject with sys admins (Hi, Mike 😉), but here’s the clincher. A true clincher. <\/li>\n
\nis due for deployment. When the database is moved to the new infrastructure,
\nsingle block read times are reduced to single figures and are consistent.
\nThe performance problems are completely solved. “Ah”, say the
\nStorage guys, “that’s just because not all of the databases havet been
\nmoved yet. When they are, you should expect to see similar performance
\nlevels as before.” <\/li>\n
\nmoved and the new improved performance levels are maintained. <\/li>\n<\/ul>\n
\nthis situation and see it for what it was (and he’s not exactly a computer
\nexpert). <\/p>\n
\nwasting 9 months of my life debating it with people who I’m looking to
\nfor expertise, not hand-waving. If this was a one-off story, I might not
\nmind, but I sometimes feel like I’m going round in circles. <\/p>\n
\ncaused by The Reality Gap.<\/p>\n