[ILUG] Hanging apache

[Colm MacCarthaigh]

On Thu, Jan 25, 2007 at 01:04:57PM +0000, Niall O Broin wrote:
> On 25 Jan 2007, at 12:19, Colm MacCarthaigh wrote:
> 
> >This almost always turns out to be a dynamic scripting module, usually
> >mod_php and a blocking operation in some code chewing up the process.
> 
> This is very likely to be the answer, as the site is based on some  
> awfully cruddy PHP. But I'm a little puzzled as to why this hangs the  
> server completely. It only happens sometimes, so should it not then  
> only block one child? Or perhaps what happens is that this is a  
> gradual process, which hangs child after child until the entire  
> allowed number of children has been spawned and is hanging.

That's exactly it :-) The worst kind of errors are DNS, because there
are a few failures which are just eternally blocking.  connect() to db's
and so on will usually timeout, but a getaddrinfo() call can really sit
there for a long long time. This happens enough times, and you'll run out
of spare servers.

> >The best thing to do is to attach gdb to the process and to do a
> >backtrace and see what functions it's blocking in.
> 
> I suspect this won't help very much - it's going to show me there's a  
> problem in mod_php, I'll be bound. But I'll give it a whirl - it may  
> well be that it'll narrow it down a little more than that.

php built-in functions map to sane symbol names, it's usually not too
hard, though it's very hard to figure out any user-provided functions as
that's all interpretted.

> >In apache 2.x, mod_forensic_log is a good way of isolating the
> >requests that actually cause the problems.
> 
> OK - so I'll have more tools at my disposal when we move it to apache  
> 2 - that's good to know.

Turns out it's in >= 1.3.30 too;

http://httpd.apache.org/docs/1.3/mod/mod_log_forensic.html

Shows how much attention I pay ;-)

-- 
Colm MacCárthaigh                        Public Key: colm+pgp at stdlib.net