users@glassfish.java.net

Re: Glassfish scaling/contention problems in http queue and jdbc pool

From: <glassfish_at_javadesktop.org>
Date: Mon, 17 May 2010 04:36:39 PDT

> Yes, it's standard synchronized thread pool
> implementation, GFv3.1
> will have more optimized implementation, which will
> be less
> synchronized, it could slightly improve perf.
> numbers. But I don't
> think this thread pool is the narrow place in your
> testing.

When doing sampling over longer periods of time, this problem shows up very regularly, like once every 5-10 seconds.

New pool sounds interesting. Do you know if it is possible to somehow backport it and do some tests with the new pool? Or maybe its simply easier to try GF3v1?

> I've checked the dump.
> You have a lot of worker threads. which may cause
> redundant CPU
> thrashing. Just out of curiosity, can you create
> similar test, which
> doesn't access DB, but let's say runs some
> calculation logic instead,
> and check if you'll be able to load CPUs. This way we
> will be able to
> recognize if DB access is the narrow place or not.
>

Yes, i tried running a crypto on a random string for each request to a HTTP servlet and return it to the client, like so:

[code]
    public synchronized String encrypt() {
        String plaintext = "" + new Random().nextDouble();
        MessageDigest md = null;
        try {
            md = MessageDigest.getInstance("SHA");
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException(e.getMessage());
        }
        try {
            md.update(plaintext.getBytes("UTF-8"));
        } catch (UnsupportedEncodingException e) {
            throw new RuntimeException(e.getMessage());
        }
        byte raw[] = md.digest();
        String hash = (new BASE64Encoder()).encode(raw);
        return hash;
    }
[/code]

Still we cannot get the CPU loaded to more than around ~65%.

During this particular test, i also watched "sun.rt._sync_ContendedLockAttempts" by doing "jstat -J-Djstat.showUnsupported=true -snap <pid> | grep _sync_"..

Seems we have over 5000 lock conflicts every second...
[Message sent by forum member 'ekrisjo']

http://forums.java.net/jive/thread.jspa?messageID=470137