[ILUG] anyone have any experience of java thread problems on linux ?

[Matthew French]

Bryan Hunt mentioned:
> Thing about this thing is that every page request will fire up a stateless
> session bean which will then connect to a pool of entity beans. As they're
all
> running in the one server this is creating a lot of threads. Apparently
things
> start to go funny at approx 500 connections.

Off list I also managed to get the following information:
> JVM: IBM JDK 1.3
> App server: JBoss 3.0RC1
> Web server: Jetty 4.0

Also, the most crucial piece of information, the error message:
> [java] 17:04:10.371 EVENT  LOW ON THREADS: SocketListener at 0.0.0.0:8085
> [java] 17:04:42.042 WARN!! OUT OF THREADS: SocketListener at 0.0.0.0:8085
> [java] Can't open /proc/4730/stat with unexpected errno 24 - unable to \
> determine native thread state

Firstly, JBoss does not limit the number of threads it can be running at a
time, but as far as I can make out it is also not responsible for creating
them - this is handled by either the RMI invoker (not being used in this
case) or the embedded web server. Therefore number of threads = number of
HTTP connections (+ some small number of management threads).

Thus the first stop should be the web server: jetty. Jetty has a
configuration parameter "MaxThreads" that is set to 255 by default. In the
stock JBoss installation, this is hidden in a file called
{jboss}/server/deploy/jetty-plugin.sar. This is a JAR file that must be
unpacked to access the jboss-service.xml file which contains this parameter,
so I suspect you would not have modified it.

To confirm, a quick look at the source file:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/jetty/Jetty/src/org/mortbay/h
ttp/SocketListener.java

shows that it is Jetty, and not JBoss or Java that is generating this error
message. In other words the web server is running out of threads.

However, a further reading of the code shows that Jetty generates the error
messages, and stops using HTTP keep-alives when it hits the thread limit,
but it will still accept new connections and create new threads.

What is curious is the "errno 24" which shows this might be unrelated. A
quick glance at linux/include/asm/errno.h shows that error 24 indicates that
the system has run out of file descriptors. "proc/4730/stat" would be a
status file for PID 4730 and presumably the JVM is attempting to open it to
check the thread state - but cannot because it has run out of file
descriptors. This would mean that the number of threads is not the problem,
but you said you have tried to change ulimit with no affect?

I have a sneaking suspicion that jetty is not closing file handles properly
when it gets saturated. This is not unusual because the garbage collector
will not have a chance to close open files.

Thus I can recommend several different things to try:
* Reduce the number of Jetty threads.
* Increase the number of Jetty threads.
* Try Tomcat instead of Jetty (pretty easy in JBoss - ask me if you want to
know how).
* Use Apache to restrict the number of active threads, which it does much
more effectively than either Tomcat or Jetty.

Your goal should be to reduce, not increase, the number of threads. Threads
take up memory, as well as processor space. A good web server, such as
Apache, will have a limited number of threads to deal with requests, and
will stick new requests into a queue. Even if the threads are idle, waiting
for the database to handle a request, it will be more effecient for the
database to finish a transaction before procedeeding on to the next.

Now, to bring this somewhat back on topic, I must emphasise that this in no
way indicates that Linux is a better or worse OS. This problem may well be a
result of Linux being _too_ efficient and is in fact palming the load onto
off on to the EJB server too quickly. Windows might "scale" better because
it takes longer to handle requests, or because it has a different queuing
mechanism for new connections.

Generally, when doing performance testing it is very important to have a
solid methodology. One must be sure to compare like with like - is it the
same hardware? Is it the same database? Is the database running on the same
server?

It is also important to test the end result, and not the back end. For
example, you might find that Linux is doing 300 transaction per minute when
it breaks, but an equivalent Windows setup cannot get past 120tpm. And one
should *not* try and test by initiating 1,000 connections at once. For J2EE
testing it is also important to test sustained throughput - because the JIT
takes some time to work its magic, the first few iterations should be
ignored. The test "client" should also be a separate machine.

Finally, performance testing can be a complex and thankless task - but if
you want to believe the results it is important to do it right. My
experience has been that Linux outperforms Windows, but that may not be true
for all applications. Either way, my justification for using Linux would not
be performance, but manageability and reliability. Unfortunately it is not
looking too reliable at the moment, which is why I am interested to get to
the bottom of this problem.

HTH,

- Matthew (who should really be doing other less interesting work)



_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com