Hi Brent,
Took a quick look - I couldn't build the project as your gradle build
file has a script plugin on a server I can't reach:
apply from: '
http://dev.iecokc.com:8081/nexus/service/local/repositories/iec/content/gradle/common/1.0/common-1.0.gradle'
You're code is certainly helpful to look at - but I can help better if
I can locally reproduce/profile, etc.
I'm glad you mentioned that the query in fillProduct is not the
problem -- probably would have been my first consideration given the
union and subqueries =)
A couple question since I can't build --
- when you say that the return statement goes into limbo,I assume you
verified that with timehacks, etc?
- Is your 50k return size...is that serialized binary size being
passed remotely, or is that measured another way?
- can you send the 50kB dummy data generator or files? Is that the
test data provided? Those are 5kB each (XML), not 50kB....
- did you also change your JVM version? What versions are you
testing on with GF3.1 and GF 4.1?
- Have you done any additional tuning such as request partitioning
for your remote EJBs, or are they still using the default settings? As
a side note, you may want to look into this regardless if you're not
already doing this...since you're using so many remote EJBs and you
indicated you're having slowdowns across applications, you'll be able
to prioritize. [1]
- what is your heap memory min/max set to?
I want to investigate serialization/deserialization since you've
eliminated jdbc, but you indicated you have pumped through 50kB of
data without the DB and had no problems...can you confirm you tested
50kB (I only ask because the file sizes in test). How is system
memory looking when you're EJB is "in limbo"...any large amounts of
paging going on? What about your JVM profiling...how is the GC acting
during this time?
Anything you can do to simplify this test code a bit (ie if you've
eliminated the JDBC from the problem, can you remove it and load code
from classpath for testing?) would be a great help.
[1]
https://glassfish.java.net/docs/4.0/performance-tuning-guide.pdf
On Mon, Oct 27, 2014 at 3:04 PM, Brent Vaughn <Vaughn_at_iecokc.com> wrote:
> Sorry it has taken me so long to get back on this. Here is an application
> that is currently running very slowly on GF 4.1
>
> https://github.com/jamie-lb/GF4ProblemProject
>
>
>
> Here is what our investigation found.
>
>
>
> We inject the PrmsIecProductDao EJB into a CDI bean and run the
> getProduct(String) method on the ProductDao interface. The CDI bean looks
> something like this…
>
>
>
> @EJB(mappedName = "ejb/BillOfMaterials/iec")
>
> private ProductDao prodDao;
>
>
>
> public void someMethod(){
>
> Product product = productDao.getProduct(“Some product id”);
>
> }
>
>
>
>
>
> We have found that a Product object with the return size of 50 KB takes 40
> seconds to return in GF 4.1. It took less than 2 seconds in GF 3.1 for a
> Product object of similar size. Once we took out the database call in GF
> 4.1 and filled the object with 50KB of dummy data mimicking the same
> structure as a database filled Product object, the return time in GF 4.1 was
> less than two seconds. The problem isn’t the database query. The database
> query runs in less than 1 seconds. The return statement in the
> PrmsProductDao.getProduct(String) method just goes into limbo. It will take
> 40 seconds to return back to the caller.
>
>
>
> Thanks for looking into this. Is there anything else I can provide to help
> in this?
>
>
>
>
>
> From: Steven Siebert [mailto:smsiebe_at_gmail.com]
> Sent: Thursday, October 23, 2014 9:53 AM
>
>
> To: users_at_glassfish.java.net
> Subject: [gf-users] Re: Glassfish 4 remote ejb very slow
>
>
>
>>3) what does your EJB do (what resources does it access, etc)?
>> a. It uses @EJB to inject local and remote ejbs
>
> Are these remote EJBs on the same GF server?
>
>
>>Where do I send the test app to?
>
> You can upload the test app to wherever is most convenient for
> you...github.com is a common place, for example.
>
>>I will try to get it later, but right now I am getting torn apart at work
>> over this. I will try to get it later today or tomorrow.
>
> OK. Your other answers rule out some considerations, a reduced app would
> help us diagnose the problem with you.
>
> S
>
>
>
>
>
> On Thu, Oct 23, 2014 at 10:40 AM, Brent Vaughn <Vaughn_at_iecokc.com> wrote:
>
> We have 80 applications on the same GF 4.1 instance. Probably half of
> those use remote ejbs.
>
>
>
> 1) what kind of load are handling when you have this slowdown...or is
> it always?
>
> a. We have 80 applications on the same GF 4.1 instance. Probably
> around 30 of those use remote ejbs. It slows down across all the
> applications that use remote ejbs
>
> 2) what are your ejb pool settings?
>
> a. The defaults from when you install. Initial and Minimum Pool Size
> is 0. Maximum Pool Size is 32. Pool Resize Quantity is 8. Pool idle
> timeout is 600 seconds
>
> 3) what does your EJB do (what resources does it access, etc)?
>
> a. It uses @EJB to inject local and remote ejbs
>
> 4) anything in your log?
>
> a. Nothing that I saw
>
> 5) is this app on the same physical machine, using the same hardware,
> or did you also change the host?
>
> a. All the apps are running on the same physical machine
>
> 6) any other external changes?
>
> a. Not that I know of
>
>
>
> Where do I send the test app to? I will try to get it later, but right now
> I am getting torn apart at work over this. I will try to get it later today
> or tomorrow.
>
>
>
> From: Steven Siebert [mailto:smsiebe_at_gmail.com]
> Sent: Thursday, October 23, 2014 6:51 AM
> To: users_at_glassfish.java.net
> Subject: [gf-users] Re: Glassfish 4 remote ejb very slow
>
>
>
> We're gonna need some more information about this...
> - what kind of load are handling when you have this slowdown...or is it
> always?
>
> - what are your ejb pool settings?
>
> - what does your EJB do (what resources does it access, etc)?
>
> - anything in your log?
>
> - is this app on the same physical machine, using the same hardware, or did
> you also change the host?
>
> - any other external changes?
>
>
>
> The easiest way to get an answer to your question, and give you a leg up in
> submitting a bug if there is one, is if you can reduce the problem within a
> small test application? If so, can you share that test application so we
> can test and check it out ourselves?
>
> S
>
>
>
> On Wed, Oct 22, 2014 at 4:29 PM, Brent Vaughn <Vaughn_at_iecokc.com> wrote:
>
> We upgraded today to Glassfish 4.1. We have a problem with our remote EJBs
> returning very slowly. What used to return in 10 seconds on Glassfish 3.1
> now returns in 2 minutes. The “slowness” appears to be in the return from
> the Remote EJB to the client that called it. Any idea why this is
> occurring?
>
>
>
>