Hi Bill --
I looked at your jstacks, and the glassfish threads aren't blocked;
they're just all doing I/O. So from the appserver perspective, that all
appears to be working normally.
Given the difference you report below in the number of files used, I'd
guess that your system is becoming I/O bound. I'm not sure what the
org.rrd4j.core.RrdDb package is, but I'd guess it's probably doing lots
of synchronous writes to the filesystem, and once you reach a threshold,
everything gets into increasing contention and increasing time. Have you
run iostat -xn during the test, and does it show that I/O might be a
problem?
Where is it writing the files? Can you try putting the files written
into /tmp and see if that solves the issue? Then we'd know that the I/O
is the problem (and then to fix it, you could look into faster disks,
ZFS, or a disk array with a disk cache).
-Scott
On Thu, 2008-01-31 at 12:15, William Fretts-Saxton wrote:
> Hi Scott,
>
> I've combined your idea of using jstack, with Harsha's previous
> recommendation on lowering the number of files written to. I did 3
> runs, using 20 files, 42 files, and 83 files written to, to benchmark,
> and came up with some information.
>
> Firstly, I've attached the "jstack" output for each of these runs
> (jstack.zip). If you could take a look @ them, I'd appreciate it.
> Files with .0 suffix is before the run began, .1 is while the number of
> clients goes to about 100, and .2 and .3 are while the client load is
> peaking. Please let me know if you find anything.
>
> As far as web service call times, I found the following:
>
> Average call to web service:
>
> 20 files: .2 seconds
> 42 files: 3.1 seconds
> 83 files: 61 seconds
>
> As you can see, the increase is astronomical.
>
> Please let me know if any more information is helpful.
>
> -Bill
>
> Scott Oaks wrote:
> > Assuming that you have tuned the thread pools correctly (and it sounds
> > like you've looked at that), then something is causing a bottleneck. It
> > sounds almost like there's some sort of deadlock between your client(s)
> > and server.
> >
> > One easy trick to find this is to make sure that you're using JDK 6
> > (6_04 is best), and then to use the JDK 6 jstack command to get the
> > stack of the appserver. [You can use various tools, including asadmin,
> > to get stack dumps, but jstack will be fastest.] If threads are all
> > blocking on a resource or something, you can usually tell from the
> > stack dump: all of the GrizzlyWorkerThreads will be doing something
> > other than calling waitForIoTask (or a similarly named method, depending
> > on which precise version you're running).
> >
> > You can send me a stack dump if you have questions about it.
> >
> > -Scott
> >
> > On Wed, 2008-01-30 at 18:57, William Saxton wrote:
> >> Hi all,
> >>
> >> Since the EJB issues I was having earlier appear to be due to a known
> >> bug, I've switched over to using a Web Service to handle my application
> >> server needs. The usage is the same, I have about 700 clients accessing
> >> an application server, every 5 minutes, so around 3-4 clients will be
> >> accessing it every second.
> >>
> >> It takes less than 1 second for a client to send the web service the
> >> XMLified data, the application server to save the data to about 80
> >> different files, and for it to return an "ok". When I turn on all of
> >> the clients, though, it takes anywhere from 10 seconds to over 5 minutes
> >> (which I timeout) for the request to be completed.
> >>
> >> I've tried these JVM tweaks:
> >> http://weblogs.java.net/blog/sdo/archive/2007/12/a_glassfish_tun.html
> >> as well as setting the HTTP acceptor threads to 16 and
> >> request-processing threads to 16. These don't seem to have done
> >> anything. There is very little CPU usage (~90% idle) and little local
> >> disk I/O (where the data is saved) so I don't know where the bottleneck
> >> is. I'm wondering whether the application server is simply queuing up
> >> too many of the client requests when it could be handling more.
> >>
> >> Anyone have an ideas? Since I'm internal to Sun, I can always give
> >> someone access to the app server if they wanted to do a quick glance.
> >>
> >> Thanks.
> >>
> >> -Bill
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> >> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> > For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>