20 Nov
2013
20 Nov
'13
12:47 a.m.
Hi all,
On your most overloaded (cpu/fork rate/context switches - ignoring memory
network, disk and swap etc), what is the maximum number of context
switches per second per core (ie divide sar -w output by 16 if you have a
16 core box) you measure?
Does anyone know what the maximum number of context switches per core you
can expect on xeon level hardware?
I'm trying to claim we get overloaded when we reach a little less than
10,000 cswch/s per second, but we've lost all the historical data.
--
Tim Connors
20 Nov
20 Nov
1:06 a.m.
On Wed, 20 Nov 2013, Tim Connors wrote:
Hi all,
On your most overloaded (cpu/fork rate/context switches - ignoring memory
network, disk and swap etc), what is the maximum number of context
switches per second per core (ie divide sar -w output by 16 if you have a
16 core box) you measure?
Does anyone know what the maximum number of context switches per core you
can expect on xeon level hardware?
I'm trying to claim we get overloaded when we reach a little less than
10,000 cswch/s per second, but we've lost all the historical data.
Indeed, is there going to be a maximum for a given piece of hardware (eg,
maximum amount of interrupts that can be generated per second; time spent
in the interrupt handler that all has to be handled by only one CPU
hence explaining why CPU system usage never looks alarming (divide by 8
on some servers, by 16 on others); big kernel lock somewhere in the
context switch code)?
When we have these overloads, nothing else we measure seems to be
approaching any limit. The servers have plenty of CPU left, and there's
no real difficulty logging into them. Anything else I should be looking
at? Fork rate is tiny (1 or 2 per second). Network bandwidth is fine.
Not sure that I've noticed network packet limitations (4k packets per
second per host when it failed last time, generating 16000
interrupts/second total per host).
--
Tim Connors
1:28 a.m.
On Wed, 20 Nov 2013, Tim Connors <tconnors(a)rather.puzzling.org> wrote:
What is going wrong in the "overload"?
Why not just write a context switch benchmark? It should be simple to have a
50+ pairs of processes and for each pair have them send a byte to a pipe and
then wait to receive a byte from another pipe.
http://manpages.ubuntu.com/manpages/hardy/lat_ctx.8.html
From a quick Google search it seems that my above idea has already been
implemented.
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
https://github.com/tsuna/contextswitch
The above looks interesting too. A google search for the words context,
switch, and benchmark will find you other things as well.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
> On your most overloaded (cpu/fork rate/context
switches - ignoring memory
> network, disk and swap etc), what is the maximum number of context
> switches per second per core (ie divide sar -w output by 16 if you have a
> 16 core box) you measure?
Of all the systems I can measure in that regard (ones with sar installed and
running) the highest is an average of 3428.79 for a 10 minute period. That's
a workstation running KDE and Chromium which probably had some cron jobs
running at the time (but nothing particularly intensive, maybe an iView
download). That system in question has 4 cores.
Of my servers the ones that have any significant load are running Xen. That
means that they have several copies of sar for different DomUs (which is
difficult to collate at best) and also means that there may be context
switches between DomUs that don't show up in sar output and which I don't know
how to measure (if you know then let me know and I'll get the output).
I really doubt that 850 contest switches per second is any sort of hardware
limit.
Does anyone
know what the maximum number of context switches per core you
can expect on xeon level hardware?
I'm trying to claim we get overloaded when we reach a little less than
10,000 cswch/s per second, but we've lost all the historical data.
Indeed, is there going to be a maximum for a given piece of hardware (eg,
maximum amount of interrupts that can be generated per second; time spent
in the interrupt handler that all has to be handled by only one CPU
hence explaining why CPU system usage never looks alarming (divide by 8
on some servers, by 16 on others); big kernel lock somewhere in the
context switch code)?
When we have these overloads, nothing else we measure seems to be
approaching any limit. The servers have plenty of CPU left, and there's
no real difficulty logging into them. Anything else I should be looking
at? Fork rate is tiny (1 or 2 per second). Network bandwidth is fine.
Not sure that I've noticed network packet limitations (4k packets per
second per host when it failed last time, generating 16000
interrupts/second total per host).
4:16 a.m.
On Wed, 20 Nov 2013, Russell Coker wrote:
On Wed, 20 Nov 2013, Tim Connors
<tconnors(a)rather.puzzling.org> wrote:
What is going wrong in the "overload"?
Something hits a tipping point, the number of apache worker slots
(3000-6000 depending on hardware specs) rapidly fills up, then apache
stops accepting new connections and www.bom.gov.au goes dark (since this
happens on all machines in the load balanced cluster simultaneously).
woops!
Does
anyone know what the maximum number of context switches per core you
can expect on xeon level hardware?
I'm trying to claim we get overloaded when we reach a little less than
10,000 cswch/s per second, but we've lost all the historical data.
Indeed, is there going to be a maximum for a given piece of hardware (eg,
maximum amount of interrupts that can be generated per second; time spent
in the interrupt handler that all has to be handled by only one CPU
hence explaining why CPU system usage never looks alarming (divide by 8
on some servers, by 16 on others); big kernel lock somewhere in the
context switch code)?
When we have these overloads, nothing else we measure seems to be
approaching any limit. The servers have plenty of CPU left, and there's
no real difficulty logging into them. Anything else I should be looking
at? Fork rate is tiny (1 or 2 per second). Network bandwidth is fine.
Not sure that I've noticed network packet limitations (4k packets per
second per host when it failed last time, generating 16000
interrupts/second total per host). Why not just write a context switch benchmark? It
should be simple to have a
50+ pairs of processes and for each pair have them send a byte to a pipe and
then wait to receive a byte from another pipe.
http://manpages.ubuntu.com/manpages/hardy/lat_ctx.8.html
From a quick Google search it seems that my above idea has already been
implemented.
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
https://github.com/tsuna/contextswitch
The above looks interesting too. A google search for the words context,
switch, and benchmark will find you other things as well.
Believe me I searched. All the snot in my head seems to be clogging up my
synapses today unfortunately.
But the blog entry looks good. I imagine that the 140,000
cswitches/second on 16 core machines running httpd+php interpreter is
pretty much a fundamental limit on E5410 level hardware, given that apache
is heavy weight enough that it's going to be more towards the 50,000µs end
of the spectrum presented in that blog.
Now I just have to convince the powers that be that php is a stupid thing
to rely on when you don't have to, and it's obviously that recent change
that broke the system that formerly coped with many times the amount of
traffic that it now croaks on.
Now I've got some benchmarks to run. I mean, fight some fires.
--
Tim Connors
21 Nov
21 Nov
1:38 a.m.
On 20 November 2013 15:16, Tim Connors <tconnors(a)rather.puzzling.org> wrote:
Something hits a tipping point, the number of apache worker slots
(3000-6000 depending on hardware specs) rapidly fills up, then apache
stops accepting new connections and www.bom.gov.au goes dark (since this
happens on all machines in the load balanced cluster simultaneously).
Ah, you are probably already well aware of this, so stop me if so..
In my experience, there's definitely an upper bound to the number of
web-serving worker threads you can run on a machine, beyond which you
start to see a drop in the aggregate performance rather than gain.
Three to six thousand slots sounds like a lot for one machine, to me.*
I wondered why so many? Are you not running a reverse-proxy
accelerator in front of Apache? (eg. Varnish or some configurations of
nginx)
If you were just serving static content directly, I'd go with
something lighter-weight than Apache; and if you're serving dynamic
content (ie. the php you mention) then I'd definitely not do so
without a good reverse-proxy in front of it, and a much-reduced number
of apache threads.
Sorry this doesn't really help with the context-switching question,
but maybe helps with the overall issue.
-Toby
* But I'm a bit out of date; current spec hardware is quite a bit more
powerful than it was last time I was seriously working with
high-thread-count code.
When we have these overloads, nothing else we measure
seems to be
approaching any limit. The servers have plenty of CPU left, and there's
no real difficulty logging into them. Anything else I should be looking
at? Fork rate is tiny (1 or 2 per second). Network bandwidth is fine.
Not sure that I've noticed network packet limitations (4k packets per
second per host when it failed last time, generating 16000
interrupts/second total per host).
What is going wrong in the "overload"?
27 Nov
27 Nov
2:06 a.m.
On Thu, 21 Nov 2013, Toby Corkindale wrote:
On 20 November 2013 15:16, Tim Connors
<tconnors(a)rather.puzzling.org> wrote:
Something hits a tipping point, the number of apache worker slots
(3000-6000 depending on hardware specs) rapidly fills up, then apache
stops accepting new connections and www.bom.gov.au goes dark (since this
happens on all machines in the load balanced cluster simultaneously).
Ah, you are probably already well aware of this, so stop me if so..
In my experience, there's definitely an upper bound to the number of
web-serving worker threads you can run on a machine, beyond which you
start to see a drop in the aggregate performance rather than gain.
That I believe. But the reason I bring up context switch limitations is
that we've still got 80% cpu free when we hit whatever limit we're
hitting. iowait is bugger all; system (ie, what I would have thought
would be accurately accounting for any context switch overhead, but maybe
it misses a few cpu cycles between user space relinquishing control and
the kernel swapping its pointers in and starting to account for resources
against itself?) is small; so what else can it be (notwork is fine,
interrupts appear fine, but again I don't know whether theres an invisible
limitation there; NFS server and disk array are trivially loaded)?
Here's a sar -u output from last Saturday when we filled 5000 slots on
some of the cluster nodes and started refusing connections (scary thought,
because then the load balancer drops out the first offending node, and so
the others have to take up the slack, and all drop out too):
00:00:01 CPU %user %nice %system %iowait %steal %idle
06:20:01 all 8.25 0.00 9.44 0.25 0.00 82.05
sar -w:
00:00:01 cswch/s
06:10:01 80444.64
06:20:01 108955.60
06:30:01 55500.00
(I've seen 150000 on these 16 core machines before it really started
struggling. The older 8 core nodes in the cluster achieve pretty much
half that)
When we have these overloads, nothing else we measure
seems to be
approaching any limit. The servers have plenty of CPU left, and there's
no real difficulty logging into them. Anything else I should be looking
at? Fork rate is tiny (1 or 2 per second). Network bandwidth is fine.
Not sure that I've noticed network packet limitations (4k packets per
second per host when it failed last time, generating 16000
interrupts/second total per host).
What is going wrong in the "overload"? Three to six thousand slots sounds like a lot for one
machine, to me.*
I wondered why so many? Are you not running a reverse-proxy
accelerator in front of Apache? (eg. Varnish or some configurations of
nginx)
We're government. Let's throw resources at it (until we run out of money)
rather than think about it carefully (actually, it's served us pretty well
up til now. But someone made Wise Choices last year, and then <rant
elided>).
If you were just serving static content directly,
I'd go with
something lighter-weight than Apache; and if you're serving dynamic
content (ie. the php you mention) then I'd definitely not do so
without a good reverse-proxy in front of it, and a much-reduced number
of apache threads.
The php wasn't a big thing until a few months ago. It's obvious that it's
causing the problems, but chucking a cache in front of each node will be
impossible now that we're not allowed to buy any equipment or replacements
for so-old-they're-out-of-warranty machines (annoyingly, the offending
.php page could just as easily be a static page, but we outsourced that to
an "industry expert"). The httpd.conf configuration is complex enough
that it'll never be replaced with another httpd server particularly now
that the only two people who knew enough about it in the web group have
retired.
* But I'm a bit out of date; current spec hardware
is quite a bit more
powerful than it was last time I was seriously working with
high-thread-count code.
Heh. I spent all day in a hot aisle of one of our data centres tracing
unlabelled cables a few weeks ago. Some of these 64 core blades are
seriously powerful boxes. Fortunately it wasn't a day like today.
--
Tim Connors
5:59 a.m.
On 27 November 2013 13:06, Tim Connors <tconnors(a)rather.puzzling.org> wrote:
[snip]
If you were just serving static content before, then Apache
(w/sendfile) is fairly efficient and can handle a lot of simultaneous
connections.
I really wouldn't do it myself for a busy site, and it's susceptible
to a few problems, but it's something you can mostly get away with.
As soon as you throw dynamically-generated stuff in (ie. CGI of any
sort) all that changes.
You see, Apache can use a kernel-extension (sendfile) to attach an
open filehandle (to static content) directly to a socket, and then the
kernel just handles the rest, so it uses very little memory or CPU
time or context switching.
But dynamic content involves spawning an addition process, doing a
whole lot of memory allocation, lots of file i/o and system calls, and
worse: that heavy execution environment has to stick around as long as
it takes to send the results off to a client, gradually feeding in a
few kilobytes at a time. If your client is on a slow connection
(mobile, dial-up, busy adsl, DoS attack) then you're holding up a lot
of resources for a long time.
Again, you are probably aware of this -- but I'm trying to illustrate
just *how much* heavier the PHP processes are compared to serving
static content. I'm not particularly surprised that servers which
could handle things fine w/static content are falling over now.
I do advise limiting the maximum number of threads per machine, to
some amount lower than where you see the problems occurring.
Set up a benchmarking rig against one machine (that's taken out of the
load-balancing pool) and find out where the optimum amount is -- I'm
pretty sure you'll find it's lower than at the many-thousand-processes
mark. It's better to allow connections to pile up in the "pending"
queue, but be able to process them quickly, than to accept them all
and then serve them all very slowly or not at all.
Secondly, if you're really stuck with Apache, and can't put decent
reverse proxy accelerators in front of them, then try switching over
the event-based worker?
http://httpd.apache.org/docs/current/mod/event.html
Toby
Three to six
thousand slots sounds like a lot for one machine, to me.*
I wondered why so many? Are you not running a reverse-proxy
accelerator in front of Apache? (eg. Varnish or some configurations of
nginx)
We're government. Let's throw resources at it (until we run out of money)
rather than think about it carefully (actually, it's served us pretty well
up til now. But someone made Wise Choices last year, and then <rant
elided>).
If you were just serving static content directly,
I'd go with
something lighter-weight than Apache; and if you're serving dynamic
content (ie. the php you mention) then I'd definitely not do so
without a good reverse-proxy in front of it, and a much-reduced number
of apache threads.
The php wasn't a big thing until a few months ago. It's obvious that it's
causing the problems, but chucking a cache in front of each node will be
impossible now that we're not allowed to buy any equipment or replacements
for so-old-they're-out-of-warranty machines (annoyingly, the offending
.php page could just as easily be a static page, but we outsourced that to
an "industry expert"). The httpd.conf configuration is complex enough
that it'll never be replaced with another httpd server particularly now
that the only two people who knew enough about it in the web group have
retired.
29 Nov
29 Nov
6:08 a.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
On Wed, 27 Nov 2013, Toby Corkindale wrote:
I do advise limiting the maximum number of threads per
machine, to
some amount lower than where you see the problems occurring.
Set up a benchmarking rig against one machine (that's taken out of the
load-balancing pool) and find out where the optimum amount is -- I'm
pretty sure you'll find it's lower than at the many-thousand-processes
mark. It's better to allow connections to pile up in the "pending"
queue, but be able to process them quickly, than to accept them all
and then serve them all very slowly or not at all.
ListenBacklog:
https://sites.google.com/site/beingroot/articles/apache/socket-backlog-tuni…
512 by default (according to that page, but http://<server>/server-info
doesn't list the current value), but the kernel in its current config only
allows 128 per port. Aha, we'll change that next week (I actually thought
that when it got to MaxClients, ie, number of slots filled up, it didn't
accept any new connections at all. At least, in practice, we find that
once it hits maxclients, the servers start dropping connections very soon
after (and this propagates through the load balancers). That would be
explained by the kernel limit being to only allow an extra 10% above the
current number of max slots. Easy to fix.
Secondly, if you're really stuck with Apache, and
can't put decent
reverse proxy accelerators in front of them, then try switching over
the event-based worker?
http://httpd.apache.org/docs/current/mod/event.html
That would be good, but rhel5&6 are still on apache 2.2, and event is
marked as experimental there :(
http://httpd.apache.org/docs/2.2/mod/event.html
We'll be stuck on rhel5 in production for years to come up at the current
rate.
I wonder about worker vs prefork? linux processes are lightweight, so I
don't imagine threading is going to be much better. We only fork one
process per second typically, and I don't think there'll be many
differences in context switch overhead between the two. Worker apparently
"sucks for php", but I don't know whether that's for mod_php or cgi or
whatever.
I like the sound of mod_pagespeed:
https://www.digitalocean.com/community/articles/how-to-get-started-with-mod…
but the risk of rewriting stuff on the fly won't be accepted for most of
our website.
Hey, we just rediscovered a longstanding problem in that the most common
static image (5 million hits in an hour, image hasn't changed in a year)
on the site was in a directory that was marked as non cacheable! Whee!
--
Tim Connors
Midrange Systems | ITB | Bureau of Meteorology
Phone: (03) 9669 4208 | E-mail: T.Connors(a)bom.gov.au
6:30 a.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
On 29 November 2013 17:08, Tim Connors <tconnors(a)rather.puzzling.org> wrote:
Ugh :(
On Wed, 27 Nov 2013, Toby Corkindale wrote:
[snip]
Secondly, if
you're really stuck with Apache, and can't put decent
reverse proxy accelerators in front of them, then try switching over
the event-based worker?
http://httpd.apache.org/docs/current/mod/event.html
That would be good, but rhel5&6 are still on apache 2.2, and event is
marked as experimental there :(
http://httpd.apache.org/docs/2.2/mod/event.html
We'll be stuck on rhel5 in production for years to come up at the current
rate. I wonder about worker vs prefork? linux processes are
lightweight, so I
don't imagine threading is going to be much better. We only fork one
process per second typically, and I don't think there'll be many
differences in context switch overhead between the two. Worker apparently
"sucks for php", but I don't know whether that's for mod_php or cgi or
whatever.
That's OK though, because you're following the highly recommended
practice of not using the same Apache instance for dynamic (ie. php)
content as for static content, right? So you can switch the static
handling over to mpm worker, and leave PHP on the prefork setup.
T
9:53 p.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
On Fri, 29 Nov 2013 05:08:58 PM Tim Connors wrote:
I wonder about worker vs prefork? linux processes are
lightweight, so I
don't imagine threading is going to be much better. We only fork one
process per second typically, and I don't think there'll be many
differences in context switch overhead between the two. Worker apparently
"sucks for php", but I don't know whether that's for mod_php or cgi or
whatever.
Well I switched from prefork to worker on my personal Debian VM so that it
could cope with lots of PHP, I kept running out of RAM otherwise.
Worked really well and my problems went away.
Now of course it's not loaded anything like the BoM servers, but this is the
first I've heard that it "sucks for php".
cheers,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP
10:01 p.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
You may also want to investigate Apache in worker mode with php-fpm running via fCGI. I
switched a VM to that combo recently and the memory consumption by Apache dropped
significantly. Also, you can then disable lots of apache modules to make it even leaner.
Also, consider new relic monitoring of your servers. That allowed us to spot all sorts of
PHP and MySQL related bottlenecks.
Sent from my mobile device.
On 30 Nov 2013, at 8:53 am, Chris Samuel
<chris(a)csamuel.org> wrote:
On Fri, 29 Nov 2013 05:08:58 PM Tim Connors
wrote:
I wonder about worker vs prefork? linux processes are lightweight, so I
don't imagine threading is going to be much better. We only fork one
process per second typically, and I don't think there'll be many
differences in context switch overhead between the two. Worker apparently
"sucks for php", but I don't know whether that's for mod_php or cgi or
whatever.
Well I switched from prefork to worker on my personal Debian VM so that it
could cope with lots of PHP, I kept running out of RAM otherwise.
Worked really well and my problems went away.
Now of course it's not loaded anything like the BoM servers, but this is the
first I've heard that it "sucks for php".
cheers,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP
_______________________________________________
luv-main mailing list
luv-main(a)luv.asn.au
http://lists.luv.asn.au/listinfo/luv-main
2 Dec
2 Dec
11:46 p.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
On 30 November 2013 08:53, Chris Samuel <chris(a)csamuel.org> wrote:
On Fri, 29 Nov 2013 05:08:58 PM Tim Connors wrote:
It's even in the PHP manual:
http://www.php.net/manual/en/faq.installation.php#faq.installation.apache2
I wonder about worker vs prefork? linux
processes are lightweight, so I
don't imagine threading is going to be much better. We only fork one
process per second typically, and I don't think there'll be many
differences in context switch overhead between the two. Worker apparently
"sucks for php", but I don't know whether that's for mod_php or cgi or
whatever.
Well I switched from prefork to worker on my personal Debian VM so that it
could cope with lots of PHP, I kept running out of RAM otherwise.
Worked really well and my problems went away.
Now of course it's not loaded anything like the BoM servers, but this is the
first I've heard that it "sucks for php".
11:55 p.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
On 03/12/13 10:46, Toby Corkindale wrote:
It's even in the PHP manual:
I suspect that's about using mod_php, not PHP per se, it says:
# If you want to use a threaded MPM, look at a FastCGI
# configuration where PHP is running in its own memory space.
which is pretty much what I do (mod_fcgid) and it works well.
cheers,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
3 Dec
3 Dec
9:44 a.m.
New subject: Context switches per second [SEC=UNCLASSIFIED]
On 3 December 2013 10:55, Chris Samuel <chris(a)csamuel.org> wrote:
On 03/12/13 10:46, Toby Corkindale wrote:
I'd agree with that. Using fastcgi is similar to using a reverse
proxy. Rather than apache running the dynamic content generation, it's
just dispatching queries to back-end app servers and then forwarding
the results back to the client. (Which is different to trying to
efficiently manage many php interpreters within a multi-threaded
apache process)
Given what we've heard about Tim's environment, I suspect the machines
will have taken the simplistic option, and be running PHP within the
Apache process, rather than as separate processes connected to Apache
via a socket.
T
It's even in the PHP manual:
I suspect that's about using mod_php, not PHP per se, it says:
# If you want to use a threaded MPM, look at a FastCGI
# configuration where PHP is running in its own memory space.
which is pretty much what I do (mod_fcgid) and it works well.
3397
days inactive
3410
days old
13 comments
5 participants
participants (5)
-
Avi Miller -
Chris Samuel -
Russell Coker -
Tim Connors -
Toby Corkindale