“I might attend a couple of presentations later, but I doubt it.“
In the end I decided I’d had enough of my own presentation and I really wanted to see James Morle and Eric Grancher talk about storage so I decided to just pop back to the conference quickly and keep a low profile, other than a quick visit to the OUG Scotland table at the Community Focus Pubs.
Yeah, a low profile. Right.
I arrived just into James Morle’s introduction and had difficulty finding a seat I could slip into but managed something a few rows from the front. What a mistake! Not attending James’ presentation, it was informative and funny as usual and having ‘Driveheads Revisited’ as your title isn’t a bad start. His presentation was about SSDs, the fact that 64-bit Operating Systems are pretty much ubiquitous now and contained some useful, actual *numbers* as usual. On the 64-bit side, he questioned why people don’t have much bigger database block buffer caches now that they can address more memory. In my recent experience, it’s because people are still being tight with memory if they’re on propietary systems, but he’s right, why not take some pressure off the SAN by buffering more data in the cheaper server memory, rather than the expensive SAN cache?
In terms of actual numbers, he suggested that redo log writes are so critical to Oracle performance that they should be going through a battery-backed writeback cache and so you should see response times of the order of a couple of milliseconds and be concerned about anything higher. Interesting thing to check later.
Then it all started to go wrong. In his latest attempt to illustrate the folly of RAID-F(whatever) for write performance, he conducted a (cough) fun game as a demonstration. He was looking for 4 volunteers. Well, I’m not much of a volunteer for that kind of thing, so I kept my head down while three others volunteered for the easy jobs then, when no-one was coming forward for the fourth role, which he described as needing someone fit and despite me keeping my head down, SMS-ing from my mobile, he walked to the front of the stage and said ‘Doug’. Damn. What followed was ten minutes of me running up the stairs in Hall 5, which is cinema like and steep. Yep, let’s pick the smoker as the obvious choice. I made it through it, but only just!
Good presentation, but I’ll be wearing a bag over my head next time I go to hear him speak. Some people probably thinks that’s a good idea in general 😉
He advertised the next presentation I wanted to see, which was Eric Grancher’s on “Oracle and storage IOs, explanations, experience at CERN and SSD tests”. As mentioned in the title, Eric works at CERN and is a friend and fellow Oak Table Network member who organised a visit to the LHC for me and Swanny at work. (A word for Swanny – Eric was asking after you.) I really enjoyed this one because working at CERN means that Eric gets to try out lots of cool technology so as well as talking about SSD performance this was the first of two presentations where someone was able to talk about their experience testing Exadata performance. In this case, he focussed on the offloading of file creation work to the storage and the results sounded mighty impressive. Good stuff and I picked up something from his presentation that I mentioned in mine the next day – the close correlation he’d found between ASH samples and trace files in the tests he’d been running. i.e. ASH can be a much closer representation of reality than you might expect for something consisting of one second samples.
Oh, I also remember he talked about how reported high I/O times might be because of overloaded CPUs rather than the storage. Which might not be instantly obvious, but I’ve seen it before too.
Then it was the community focus pubs, so I popped round to the Scottish table, which was a roaring success as usual, then headed back to my hotel to carry on working on Wednesday’s presentation.
Bigger buffer cache to spare SAN cache? WTF? Won’t that choke the cpus?
That’s why I generally don’t blog too much about the actual content of presentations. Presentations already being a reduced version of system complexity and then I go and reduce it further!
Having said that, there’s no doubt that you could just move a bottleneck from one place to another. I’m not sure that I have seen many heavily CPU-choked systems recently, relative to the number of SAN or SAN-cache choked systems. Of course your mileage may vary but I’d hope readers of this blog would take a quick comment like that and treat it with some thought of their own before taking it too seriously.
I still think it’s a good idea, though, but maybe I just don’t understand enough about architecture and in case I was misrepresenting what James said, I’ll let him know.
No, it will not choke the CPUs, at least not until significantly more work is achieved first.
Think about the process of Oracle reading a block in from disk: It has to go via a buffer in the cache anyway, once it is read from disk. So the CPU only ever gets to see a memory incarnation of the block that previously only resided on disk. If the block is already cached, it is only the physical read that is eliminated – which cannot be viewed as a bad thing.
After saying all that, it is possible that a particular system may have the following profile:
– many fast/good queries that are mostly serviced from cache
– a few slow queries that are mostly serviced from disk
In this case, the relative speed of the disk may be bottlenecking just the large/slow queries, leaving plenty of CPU for the fast queries.
However, I’m not a believer in using system bottlenecks as an implicit workload prioritisation mechanism.