{"id":1470,"date":"2009-02-08T12:00:00","date_gmt":"2009-02-08T12:00:00","guid":{"rendered":"http:\/\/orcldoug.com\/blog\/?p=1470"},"modified":"2009-02-08T12:00:00","modified_gmt":"2009-02-08T12:00:00","slug":"time-matters-throughput-vs-response-time","status":"publish","type":"post","link":"http:\/\/orcldoug.com\/blog\/2009\/02\/08\/time-matters-throughput-vs-response-time\/","title":{"rendered":"Time Matters: Throughput vs. Response Time"},"content":{"rendered":"

Niall Litchfield<\/a> made an interesting comment in an email thread that prompted this post. <\/p>\n

“… I can see that once you start to define workload, or transactions, in business terms (I need to get all these things done, what works best overall?) then workload response time*<\/sup> does make sense, both as a metric, and more pertinently as a tuning target. I think, and have done for a while, that for much of the last 10 years the tuning dialog has been around query or session optimization with either an explicit or implicit assumption that optimization of particular queries will result in workload optimisation from a business perspective. In many cases this is true, I suspect though that for many cases it isn’t. Response time will still be the core metric, but perhaps some of the tuning methods will vary. It may be as simple as redefining stage 1 of the tuning process from <\/p>\n

‘<\/em>define the task of interest to the business’
to
‘<\/em>define the workload of interest to the business'”<\/em><\/p><\/blockquote>\n

I agree with Niall, particularly with regard to the current emphasis on optimising individual sessions or queries, rather than the overall workload.<\/p>\n

I’ll state this another way. Maybe …<\/p>\n

1) the fastest individual session response time<\/em> <\/p>\n

… isn’t always the most important optimisation goal if you’re trying to achieve …<\/p>\n

2) the highest workload throughput<\/em>? <\/p>\n

(That’s why I have a few slides early on in the course I wrote for
\nOracle that define a few ways of looking at system performance,
\nincluding response time, throughput and scalability.) <\/p>\n

I’ll try to illustrate these two different tuning goals using Dominic Giles’ Swingbench<\/a> utility. I executed the Sales Order Entry (SOE) benchmark with different numbers of concurrent users for a period of 3 minutes. The Oracle instance, hardware configuration (dual-core laptop with the usual lousy hard drive) and application code were identical for all of the tests. The only significant variable was the number of concurrent sessions. Here are the results (which Swingbench helpfully displays in the Output tab after each test run).<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Concurrent Sessions<\/strong><\/td>\nAvg. Response Time (ms)<\/strong><\/td>\nTransactions Completed<\/strong><\/td>\n<\/tr>\n
1<\/td>\n79<\/td>\n4,203<\/td>\n<\/tr>\n
2<\/td>\n108<\/td>\n6,772<\/td>\n<\/tr>\n
4<\/td>\n133<\/td>\n10,481<\/td>\n<\/tr>\n
8<\/td>\n198<\/td>\n13,346<\/td>\n<\/tr>\n
12<\/td>\n244<\/td>\n13,639<\/td>\n<\/tr>\n
16<\/td>\n310<\/td>\n14,798<\/td>\n<\/tr>\n
20<\/td>\n337<\/td>\n14,749<\/td>\n<\/tr>\n
24<\/td>\n369<\/td>\n14,176<\/td>\n<\/tr>\n
28<\/td>\n428<\/td>\n15,181<\/td>\n<\/tr>\n
32<\/td>\n563<\/td>\n13,278<\/td>\n<\/tr>\n
36<\/td>\n533<\/td>\n14,151<\/td>\n<\/tr>\n
40<\/td>\n587<\/td>\n13,302<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Without question, the average response time is best when only one user is running the test, and gets worse steadily as the number of active sessions grows. However, having only one active session also delivers the worst number of completed transactions during the test. If we want to process as many transactions as possible in those 3 minutes, then it looks like we should have around 28 concurrent execution streams.<\/p>\n

Do we care that the average transaction completion time has increased to 587 milliseconds? Well, if the transaction is part of a user interaction, then the answer is undoubtedly yes. I’m sure your users like the fastest possible response and if we have 28 active sessions, then an individual user is going to experience performance 5 times slower than if they were the only user. Oh, and that’s just the average response time – the maximum will be much worse!<\/p>\n

If, however, this is all part of an overnight batch process and the tuning goal is to get through all of the workload as quickly as possible, then the number of transactions completed is our focus and who cares if each transaction takes a little longer? In this case, it’s better to load the system more heavily and increase<\/em> the response time of individual transactions. Scheduling the workload such that the transaction response time increases might seem counter-intuitive. For example, if I were to trace one of the sessions at random during each of the test runs and I analysed the trace files, wouldn’t ‘performance’ look much ‘better’ in the first test than the last, because I’m only focussing on one session?<\/p>\n

Is<\/em> performance better in the first test than the last? It all depends on what you mean by performance.<\/p>\n

Now, let me address what I might appear to be implying here, that tracing and tuning individual sessions is flawed. In fact, all of these tests would have yielded even more impressive results if I had tuned the SOE application at the session level *first* so that each transaction uses less resources and completes more quickly. That’s a good thing. I do understand that analysis and tuning at the session scope is vital but, if each individual session’s work has been optimised already, there are other aspects we need to consider and looking at a single session trace will not give me all of the information I need. (Gosh, I’d better be careful because I’m getting close to suggesting that Statspack and AWR might be useful tools for some situations after all!)<\/p>\n

However, when looking at this useful improvement in throughput on my laptop by increasing the load during a basic test, be very careful.<\/p>\n

– Whilst this particular ‘batch process’ processed more work in the same period of time with more concurrent sessions, it also slowed response time for *everything else* on my laptop (server). That’s why it’s not sensible to mix batch and interactive workloads.<\/p>\n

– As the number of sessions increases past a certain point, the number of completed transactions goes *down* as the system starts to suffer contention so we get further away from our goal whilst using more resources. Achieving the correct balance is the key.<\/p>\n

– Looking at the results again, is the best balance to run 28 concurrent sessions when we can deliver almost the same throughput from 16? Well, I’d plump for 16 for this job on this configuration every time, if it meets the system requirements. Squeezing every last ounce of performance from a system is likely to lead to a serious performance problem one day because we’re reducing the breathing room to cope with unplanned events.<\/p>\n

All I’m really saying in this post is that performance analysis, response time tuning, workload management (call it what you will) can’t and shouldn’t be limited to looking at the performance of an individual session in isolation.<\/p>\n

* Is it just me, or does ‘Workload Response Time’ sound weird? It’s the same idea as session response time – how quickly do we get from the beginning to the end – but it seems different to me. That might be my lack of understanding, reading, or rigour, but I find the idea of a system, workload or multiple users having an aggregated response time a strange idea to get my head around. I understand start – do lots of work – stop. But is the time that that takes a response time or throughput? Have I just got too used to Response Time being directly related to a user sitting, waiting for a response?<\/p>\n","protected":false},"excerpt":{"rendered":"

Niall Litchfield made an interesting comment in an email thread that prompted this post. “… I can see that once you start to define workload, or transactions, in business terms (I need to get all these things done, what works best overall?) then workload response time* does make sense, both as a metric, and more… Continue reading Time Matters: Throughput vs. Response Time<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1470","post","type-post","status-publish","format-standard","hentry","category-uncategorized","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":1429,"url":"http:\/\/orcldoug.com\/blog\/2008\/08\/20\/time-matters-db-time\/","url_meta":{"origin":1470,"position":0},"title":"Time Matters – DB Time","date":"August 20, 2008","format":false,"excerpt":"[In retrospect, the title of that first blog post might have suited the subject, but doesn't translate too well for subsequent related blog posts. That was a lack of planning or foresight on my part. These blog posts are tumbling out of my head in a fairly incoherent way. Maybe\u2026","rel":"","context":"With 13 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1496,"url":"http:\/\/orcldoug.com\/blog\/2009\/05\/07\/adaptive-thresholds-in-10g-part-1-metric-baselines\/","url_meta":{"origin":1470,"position":1},"title":"Adaptive Thresholds in 10g – Part 1 (Metric Baselines)","date":"May 7, 2009","format":false,"excerpt":"Some features in this post require a Diagnostics Pack license.[I really didn't want to get into another multi-part blog post, but this has grown longer than I hoped, so I'll split it up ...]One of the more interesting components of the 10gR2 instrumentation improvements is the use of Metric Baselines\u2026","rel":"","context":"With 7 comments","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/18.133.199.212\/wp-content\/uploads\/recovered\/oem1.JPG?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1497,"url":"http:\/\/orcldoug.com\/blog\/2009\/05\/07\/adaptive-thresholds-in-10g-part-2-time-grouping\/","url_meta":{"origin":1470,"position":2},"title":"Adaptive Thresholds in 10g – Part 2 (Time Grouping)","date":"May 7, 2009","format":false,"excerpt":"Some features in this post require a Diagnostics Pack license.Metric Baselines were designed to be easy to implement. There are only two options :-1) Pick how much recent\u00a0activity you want to use and let Oracle recompute the statistics based on the most recent metric values over time. (Moving Window) 2)\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"\/serendipity\/uploads\/640x377xoem3.jpg.pagespeed.ic.P-CkJpYqsK.jpg","width":350,"height":200},"classes":[]},{"id":1582,"url":"http:\/\/orcldoug.com\/blog\/2010\/03\/10\/hotsos-2010-day-3-an-excellent-one-part-1\/","url_meta":{"origin":1470,"position":3},"title":"Hotsos 2010 – Day 3 – An excellent one (part 1)","date":"March 10, 2010","format":false,"excerpt":"Well, that was a nice start to the day! Someone came up to me just before Wolfgang Breitling's presentation to point out that something from my presentation had helped him fix a problem at work last night. It was the OEM Raw Data drill-down that shows you the underlying ASH\u2026","rel":"","context":"With 2 comments","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1050,"url":"http:\/\/orcldoug.com\/blog\/2006\/08\/10\/historical-perspective-part-2\/","url_meta":{"origin":1470,"position":4},"title":"Historical Perspective (Part 2)","date":"August 10, 2006","format":false,"excerpt":"Let's look at items 2 and 3 from the list in my previous blog1) You know you have a performance problem and can re-create it by running a specific part of the application, be it a user interaction or batch job.2) You have an intermittent but recurring performance problem which\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1498,"url":"http:\/\/orcldoug.com\/blog\/2009\/05\/10\/adaptive-thresholds-in-10g-part-3-setting-thresholds\/","url_meta":{"origin":1470,"position":5},"title":"Adaptive Thresholds in 10g – Part 3 (Setting Thresholds)","date":"May 10, 2009","format":false,"excerpt":"Some features in this post require a Diagnostics Pack license.One of my favourite aspects of blogging is that if I were to write a proper article or conference paper, I'd be likely to fix any problems with it as I go (well, the ones I notice!). With blogging, it's more\u2026","rel":"","context":"With 7 comments","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/18.133.199.212\/wp-content\/uploads\/recovered\/oem7.jpg?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/posts\/1470","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/comments?post=1470"}],"version-history":[{"count":0,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/posts\/1470\/revisions"}],"wp:attachment":[{"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/media?parent=1470"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/categories?post=1470"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/orcldoug.com\/blog\/wp-json\/wp\/v2\/tags?post=1470"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}