* The gradle build file has been updated. It should work now.
* There is now a DummyPrmsProductDao class loads dummy data.
* I can confirm with time hacks that it is the return statement going into limbo that is causing the slow down.
* We measured the size of the objects using this method in a JSF test project
public static int serialize(Object obj) throws IOException {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(obj);
return baos.toByteArray().length;
}
* The tests aren't related to this problem directly. They are just part of the project.
* With Glassfish 4.1 we tried Java 8 build 20 on my machine and Java 8 build 25 on another developers machine
With Glassfish 3.1 we tried Java 7, but I don't know the build number.
* No, I have not done any additional tuning to GF 4.1. Just whatever comes out of the box.
* Our heap jvm args: -Xmx16G on our dev server and -Xmx8G on my local GF 4.1
* Yes, I can confirm we tested 50kb of data on both data sets. It works great without the jdbc interaction. That is what is puzzling to me
* I profiled my local GF 4.1 instance with Yourkit java profiler. The snapshot has been added to the GitHub project. Something I noticed in the Events category in the snapshot at #18489 is where it goes into "limbo". A substantial amount of team is spent between #18488 and #18489. The EJBTestBean.serialize method is the next line of code in our JSF test project after it returns from GF 4.1 EJB inner workings. The times will be exaggerated because I had every option turned on in the profiler.
-----Original Message-----
From: Steven Siebert [mailto:smsiebe_at_gmail.com]
Sent: Monday, October 27, 2014 9:17 PM
To: users_at_glassfish.java.net
Subject: [gf-users] Re: Glassfish 4 remote ejb very slow
Hi Brent,
Took a quick look - I couldn't build the project as your gradle build file has a script plugin on a server I can't reach:
apply from: '
http://dev.iecokc.com:8081/nexus/service/local/repositories/iec/content/gradle/common/1.0/common-1.0.gradle'
You're code is certainly helpful to look at - but I can help better if I can locally reproduce/profile, etc.
I'm glad you mentioned that the query in fillProduct is not the problem -- probably would have been my first consideration given the union and subqueries =)
A couple question since I can't build --
- when you say that the return statement goes into limbo,I assume you verified that with timehacks, etc?
- Is your 50k return size...is that serialized binary size being passed remotely, or is that measured another way?
- can you send the 50kB dummy data generator or files? Is that the test data provided? Those are 5kB each (XML), not 50kB....
- did you also change your JVM version? What versions are you testing on with GF3.1 and GF 4.1?
- Have you done any additional tuning such as request partitioning for your remote EJBs, or are they still using the default settings? As a side note, you may want to look into this regardless if you're not already doing this...since you're using so many remote EJBs and you indicated you're having slowdowns across applications, you'll be able to prioritize. [1]
- what is your heap memory min/max set to?
I want to investigate serialization/deserialization since you've eliminated jdbc, but you indicated you have pumped through 50kB of data without the DB and had no problems...can you confirm you tested 50kB (I only ask because the file sizes in test). How is system memory looking when you're EJB is "in limbo"...any large amounts of paging going on? What about your JVM profiling...how is the GC acting during this time?
Anything you can do to simplify this test code a bit (ie if you've eliminated the JDBC from the problem, can you remove it and load code from classpath for testing?) would be a great help.
[1]
https://glassfish.java.net/docs/4.0/performance-tuning-guide.pdf
On Mon, Oct 27, 2014 at 3:04 PM, Brent Vaughn <Vaughn_at_iecokc.com> wrote:
> Sorry it has taken me so long to get back on this. Here is an
> application that is currently running very slowly on GF 4.1
>
> https://github.com/jamie-lb/GF4ProblemProject
>
>
>
> Here is what our investigation found.
>
>
>
> We inject the PrmsIecProductDao EJB into a CDI bean and run the
> getProduct(String) method on the ProductDao interface. The CDI bean
> looks something like this…
>
>
>
> @EJB(mappedName = "ejb/BillOfMaterials/iec")
>
> private ProductDao prodDao;
>
>
>
> public void someMethod(){
>
> Product product = productDao.getProduct(“Some product id”);
>
> }
>
>
>
>
>
> We have found that a Product object with the return size of 50 KB
> takes 40 seconds to return in GF 4.1. It took less than 2 seconds in
> GF 3.1 for a Product object of similar size. Once we took out the
> database call in GF
> 4.1 and filled the object with 50KB of dummy data mimicking the same
> structure as a database filled Product object, the return time in GF
> 4.1 was less than two seconds. The problem isn’t the database query.
> The database query runs in less than 1 seconds. The return statement
> in the
> PrmsProductDao.getProduct(String) method just goes into limbo. It
> will take
> 40 seconds to return back to the caller.
>
>
>
> Thanks for looking into this. Is there anything else I can provide to
> help in this?
>
>
>
>
>
> From: Steven Siebert [mailto:smsiebe_at_gmail.com]
> Sent: Thursday, October 23, 2014 9:53 AM
>
>
> To: users_at_glassfish.java.net
> Subject: [gf-users] Re: Glassfish 4 remote ejb very slow
>
>
>
>>3) what does your EJB do (what resources does it access, etc)?
>> a. It uses @EJB to inject local and remote ejbs
>
> Are these remote EJBs on the same GF server?
>
>
>>Where do I send the test app to?
>
> You can upload the test app to wherever is most convenient for
> you...github.com is a common place, for example.
>
>>I will try to get it later, but right now I am getting torn apart at
>>work over this. I will try to get it later today or tomorrow.
>
> OK. Your other answers rule out some considerations, a reduced app
> would help us diagnose the problem with you.
>
> S
>
>
>
>
>
> On Thu, Oct 23, 2014 at 10:40 AM, Brent Vaughn <Vaughn_at_iecokc.com> wrote:
>
> We have 80 applications on the same GF 4.1 instance. Probably half of
> those use remote ejbs.
>
>
>
> 1) what kind of load are handling when you have this slowdown...or is
> it always?
>
> a. We have 80 applications on the same GF 4.1 instance. Probably
> around 30 of those use remote ejbs. It slows down across all the
> applications that use remote ejbs
>
> 2) what are your ejb pool settings?
>
> a. The defaults from when you install. Initial and Minimum Pool Size
> is 0. Maximum Pool Size is 32. Pool Resize Quantity is 8. Pool idle
> timeout is 600 seconds
>
> 3) what does your EJB do (what resources does it access, etc)?
>
> a. It uses @EJB to inject local and remote ejbs
>
> 4) anything in your log?
>
> a. Nothing that I saw
>
> 5) is this app on the same physical machine, using the same hardware,
> or did you also change the host?
>
> a. All the apps are running on the same physical machine
>
> 6) any other external changes?
>
> a. Not that I know of
>
>
>
> Where do I send the test app to? I will try to get it later, but
> right now I am getting torn apart at work over this. I will try to
> get it later today or tomorrow.
>
>
>
> From: Steven Siebert [mailto:smsiebe_at_gmail.com]
> Sent: Thursday, October 23, 2014 6:51 AM
> To: users_at_glassfish.java.net
> Subject: [gf-users] Re: Glassfish 4 remote ejb very slow
>
>
>
> We're gonna need some more information about this...
> - what kind of load are handling when you have this slowdown...or is
> it always?
>
> - what are your ejb pool settings?
>
> - what does your EJB do (what resources does it access, etc)?
>
> - anything in your log?
>
> - is this app on the same physical machine, using the same hardware,
> or did you also change the host?
>
> - any other external changes?
>
>
>
> The easiest way to get an answer to your question, and give you a leg
> up in submitting a bug if there is one, is if you can reduce the
> problem within a small test application? If so, can you share that
> test application so we can test and check it out ourselves?
>
> S
>
>
>
> On Wed, Oct 22, 2014 at 4:29 PM, Brent Vaughn <Vaughn_at_iecokc.com> wrote:
>
> We upgraded today to Glassfish 4.1. We have a problem with our remote
> EJBs returning very slowly. What used to return in 10 seconds on
> Glassfish 3.1 now returns in 2 minutes. The “slowness” appears to be
> in the return from the Remote EJB to the client that called it. Any
> idea why this is occurring?
>
>
>
>