On Wed, 20 Nov 2013, Tim Connors <tconnors@rather.puzzling.org> wrote:
On your most overloaded (cpu/fork rate/context switches - ignoring memory network, disk and swap etc), what is the maximum number of context switches per second per core (ie divide sar -w output by 16 if you have a 16 core box) you measure?
Of all the systems I can measure in that regard (ones with sar installed and running) the highest is an average of 3428.79 for a 10 minute period. That's a workstation running KDE and Chromium which probably had some cron jobs running at the time (but nothing particularly intensive, maybe an iView download). That system in question has 4 cores. Of my servers the ones that have any significant load are running Xen. That means that they have several copies of sar for different DomUs (which is difficult to collate at best) and also means that there may be context switches between DomUs that don't show up in sar output and which I don't know how to measure (if you know then let me know and I'll get the output). I really doubt that 850 contest switches per second is any sort of hardware limit.
Does anyone know what the maximum number of context switches per core you can expect on xeon level hardware?
I'm trying to claim we get overloaded when we reach a little less than 10,000 cswch/s per second, but we've lost all the historical data.
Indeed, is there going to be a maximum for a given piece of hardware (eg, maximum amount of interrupts that can be generated per second; time spent in the interrupt handler that all has to be handled by only one CPU hence explaining why CPU system usage never looks alarming (divide by 8 on some servers, by 16 on others); big kernel lock somewhere in the context switch code)?
When we have these overloads, nothing else we measure seems to be approaching any limit. The servers have plenty of CPU left, and there's no real difficulty logging into them. Anything else I should be looking at? Fork rate is tiny (1 or 2 per second). Network bandwidth is fine. Not sure that I've noticed network packet limitations (4k packets per second per host when it failed last time, generating 16000 interrupts/second total per host).
What is going wrong in the "overload"? Why not just write a context switch benchmark? It should be simple to have a 50+ pairs of processes and for each pair have them send a byte to a pipe and then wait to receive a byte from another pipe. http://manpages.ubuntu.com/manpages/hardy/lat_ctx.8.html From a quick Google search it seems that my above idea has already been implemented. http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html https://github.com/tsuna/contextswitch The above looks interesting too. A google search for the words context, switch, and benchmark will find you other things as well. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/