Looks like there is a known bug in MYSQL driver that does not properly
timeout connections. What happens is the MySQL server side of the
connection times out and closes, but the MySQL client doesn't detect
this and just sits there..
Checkout
http://mongrel.rubyforge.org/wiki/FAQ#Q:Mongrelstopsworkingifitsleftaloneforalongtime.
for the workaround.
-vivek.
Vivek Pandey wrote:
> Mark,
>
> Looks like you need someway to renew database connection. Not sure if
> its an issue in the jdbc MySQL adapter or the jdbc-mysql gem. Since
> firewall is causing idle connection termination so what you are
> looking for is for someway something to reconnect. Nick Sieger is the
> author of the gem. Maybe he can suggest how to renew a new
> connection. Try posting your findings to user_at_jruby.codehaus.org,
> maybe someone ran in to such issue before and might have some
> suggestions.
>
> -vivek.
>
> Mark Westling wrote:
>> Hi Vivek,
>>
>> I noticed something interesting in the production.log file. I'd said
>> before that the RoR application froze after a lengthy idle time, and
>> it seems this isn't correct. What's happening is that the first
>> request after idle takes a very long time to execute. Here's the log:
>>
>> Nov 24, 2009 8:12:47 AM com.sun.grizzly.config.GrizzlyServiceListener
>> WARNING: Interrupting idle Thread: http80-WorkerThread(3)
>> Nov 24, 2009 8:20:07 AM com.sun.grizzly.config.GrizzlyServiceListener
>> INFO:
>> Processing HomeController#index (for at 2009-11-24 08:20:07) [GET]
>> Redirected to http://202.140.96.154/user_sessions/new
>> Filter chain halted as [:require_user] rendered_or_redirected.
>> Completed in 11ms (DB: 1) | 302 Found [http://202.140.96.154/]
>>
>> Nov 24, 2009 8:23:12 AM com.sun.grizzly.config.GrizzlyServiceListener
>> INFO:
>> Processing HomeController#index (for at 2009-11-24 08:07:46) [GET]
>> Rendering template within layouts/application
>> Rendering home/index
>> Completed in 925705ms (View: 11, DB: 925676) | 200 OK
>> [http://202.140.96.154/]
>>
>> Note the DB time of 925s. The customer has advised me that all
>> database queries go through a firewall that terminates idle
>> connections but I didn't think anything of this, figuring that I'd
>> either see a DB connection error or there would be a reconnect that
>> would automatically fix it.
>>
>> I did a thread dump during this 925s wait period and found this thread:
>>
>> "http80-WorkerThread(3)" daemon prio=10 tid=0xb31cf400 nid=0x5d25
>> runnable [0xb2ff1000]
>> java.lang.Thread.State: RUNNABLE
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at
>> com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)
>>
>> at
>> com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)
>>
>> at
>> com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)
>>
>> - locked <0x760a3548> (a
>> com.mysql.jdbc.util.ReadAheadInputStream)
>> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1910)
>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2304)
>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803)
>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1573)
>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1665)
>> at com.mysql.jdbc.Connection.execSQL(Connection.java:3170)
>> - locked <0x7609fd38> (a java.lang.Object)
>> at com.mysql.jdbc.Connection.execSQL(Connection.java:3099)
>> at com.mysql.jdbc.Statement.executeQuery(Statement.java:1169)
>> - locked <0x7609fd38> (a java.lang.Object)
>> at
>> jdbc_adapter.RubyJdbcConnection$6.call(RubyJdbcConnection.java:254)
>> at
>> jdbc_adapter.RubyJdbcConnection.withConnectionAndRetry(RubyJdbcConnection.java:1072)
>>
>> at
>> jdbc_adapter.RubyJdbcConnection.executeQuery(RubyJdbcConnection.java:247)
>>
>> at
>> jdbc_adapter.RubyJdbcConnection.execute_query(RubyJdbcConnection.java:234)
>>
>> at
>> jdbc_adapter.RubyJdbcConnection$i_method_multi$RUBYINVOKER$execute_query.call(jdbc_adapter/RubyJdbcConnection$i_method_multi$RUBYINVOKER$execute_query.gen)
>>
>> at
>> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:310)
>>
>> at
>> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
>>
>> at
>> ruby.jit.ruby.usr.local.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.active_record.connection_adapters.jdbc_adapter._execute9030225_18734324.__file__(jdbc_adapter.rb:569)
>>
>> at
>> ruby.jit.ruby.usr.local.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.active_record.connection_adapters.jdbc_adapter._execute9030225_18734324.__file__(jdbc_adapter.rb)
>>
>> at
>> org.jruby.ast.executable.AbstractScript.__file__(AbstractScript.java:51)
>> at
>> org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:187)
>>
>> at
>> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:330)
>>
>> at
>> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:189)
>>
>> at
>> ruby.jit.ruby.usr.local.jruby_minus_1_dot_4_dot_0.lib.ruby.gems.$1_dot_8.gems.activerecord_minus_jdbc_minus_adapter_minus_0_dot_9_dot_2.lib.active_record.connection_adapters.jdbc_adapter.execute5048333_18734324.block_0$RUBY$__block__(jdbc_adapter.rb:560)
>>
>> [...]
>>
>> Here are my MySQL-related gems:
>>
>> activerecord-jdbc-adapter (0.9.2)
>> activerecord-jdbcmysql-adapter (0.9.2)
>> jdbc-mysql (5.0.4)
>>
>> And here's a snippet from database.yml:
>>
>> production:
>> adapter: <%= jdbc %>mysql
>> encoding: utf8
>> database: myapp_production
>> host: 172.0.0.223
>> username: myacct
>> password: mypass
>> reconnect: true
>> pool: 100
>>
>> Any suggestions?
>>
>> Thanks,
>> Mark
>>
>>
>> On Mon, Nov 16, 2009 at 2:51 PM, Vivek Pandey <Vivek.Pandey_at_sun.com
>> <mailto:Vivek.Pandey_at_sun.com>> wrote:
>>
>> Mark Westling wrote:
>>
>> Hi Vivek,
>>
>> I've opened an issue on RubyForge as you suggested. gfrake
>> didn't help; in fact, I can reproduce the errors using a newly
>> created rails app. So, I suspect it's some kind of
>> configuration problem. (I did check and I'm using rubygem
>> 1.3.5).
>>
>> I just posted on your bug. I see you are using jdk 6 u18 EA. The
>> exception is form xml parser which is part of jdk 6. Can you try
>> with jdk6 u17.
>>
>> Going back to the original problem (connection freezing after
>> a long idle), should I paste a thread dump (from 0.9.5,
>> obviously) here on the list or open another issue?
>>
>> You can paste it here.
>>
>> -vivek.
>>
>> Thanks!
>> Mark
>>
>>
>>
>> On Thu, Nov 12, 2009 at 2:13 PM, Vivek Pandey
>> <Vivek.Pandey_at_sun.com <mailto:Vivek.Pandey_at_sun.com>
>> <mailto:Vivek.Pandey_at_sun.com <mailto:Vivek.Pandey_at_sun.com>>>
>> wrote:
>>
>> I double checked the tests and see everything working just
>> fine.
>>
>> I am not doubting this as an issue at your end but will
>> mention it
>> anyway, since 0.9.6.dev has (dev) suffix, which gem treats as
>> pre-release version. There are issues using the prerelease
>> versions with rubygem < 1.3.5. So I would recommend doing
>>
>> gem update --system
>>
>> This will update your rubygem to 1.3.5 and 0.9.6.dev after
>> installed will be identified as the latest version.
>>
>> You can open an issue ticket at
>> http://rubyforge.org/projects/glassfishgem/. Also provide
>> steps on
>> how I can reproduce it.
>>
>> -vivek.
>>
>>
>> Vivek Pandey wrote:
>>
>> Hmm. Our tests are running fine with this gem. I guess
>> I know
>> what might be happening.
>>
>> Can you try
>>
>> $ gfrake clean
>>
>> and then run your application. This new glassfish gem is
>> supposed to cleanup previous tmp/.glassfish left from
>> previous
>> gem run maybe there is some bug and thats what is
>> causing this
>> failure.
>>
>> I will spend some time tomorrow to investigate whats
>> going on
>> with daemon mode. Some resource injection seem to be
>> failing.
>>
>> -vivek.
>>
>> Mark Westling wrote:
>>
>> Thanks Vivek, I installed the build#120 version of
>> the gem
>> (0.9.6.dev) but now I'm running into other problems.
>>
>> When I set the glassfish.yml configuration to run in
>> daemon mode (enable: true), I get the following in
>> the
>> production.log file:
>>
>> Nov 12, 2009 4:05:53 AM
>> com.sun.enterprise.v3.server.SystemTasks
>> setSystemPropertiesFromEnv
>> SEVERE: cannot determine host name, will use
>> localhost
>> exclusively
>> java.net.UnknownHostException:
>> mobileadapp02.myclient-domain.com
>> <http://mobileadapp02.myclient-domain.com>
>> <http://mobileadapp02.myclient-domain.com>
>> <http://mobileadapp02.myclient-domain.com>:
>> mobileadapp02.
>> myclient-domain.com <http://myclient-domain.com>
>> <http://myclient-domain.com>
>> <http://myclient-domain.com>
>> at
>>
>> java.net.InetAddress.getLocalHost(InetAddress.java:1353)
>> at
>>
>> com.sun.enterprise.util.net.NetUtils.getCanonicalHostName(NetUtils.java:95)
>>
>>
>> at
>>
>> com.sun.enterprise.v3.server.SystemTasks.setSystemPropertiesFromEnv(SystemTasks.java:152)
>>
>>
>> [...]
>> at org.jruby.Main.run(Main.java:117)
>> at org.jruby.Main.main(Main.java:97)
>> Nov 12, 2009 4:24:35 AM
>> com.sun.enterprise.v3.server.AppServerStartup run
>> INFO: GlassFish v3 startup time : Embedded(200ms)
>> startup
>> services(161ms) total(361ms)
>> Nov 12, 2009 4:24:35 AM AppServerStartup run
>> INFO: [Thread[GlassFish Kernel Main Thread,5,main]]
>> started
>> Nov 12, 2009 4:24:35 AM
>> org.hibernate.validator.util.Version <clinit>
>> INFO: Hibernate Validator null
>> Nov 12, 2009 4:24:35 AM
>>
>> com.sun.enterprise.v3.services.impl.GrizzlyProxy$2$1 onReady
>> INFO: Grizzly Framework 1.9.18-e started in: 81ms
>> listening on port 80
>> Nov 12, 2009 4:24:35 AM
>> com.sun.enterprise.v3.server.ApplicationLifecycle
>> startContainers
>> SEVERE: Cannot start container jruby
>> java.lang.NullPointerException
>> ser.java:104)
>> at
>>
>> org.glassfish.scripting.jruby.JRubyContainer.postConstruct(JRubyContainer.java:97)
>>
>>
>> at
>>
>> com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:174)
>> at
>>
>> com.sun.hk2.component.ConstructorWomb$1.run(ConstructorWomb.java:87)
>> at
>> java.security.AccessController.doPrivileged(Native
>> Method)
>> at
>>
>> com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:84)
>>
>>
>> [...]
>> at org.jruby.Main.run(Main.java:272)
>> at org.jruby.Main.run(Main.java:117)
>> at org.jruby.Main.main(Main.java:97)
>> Nov 12, 2009 4:24:35 AM
>> org.glassfish.api.ActionReport failure
>> SEVERE: Aborting, Failed to start container
>> org.glassfish.scripting.jruby.JRubyContainer
>> Nov 12, 2009 4:24:35 AM
>> org.glassfish.api.ActionReport failure
>> SEVERE: Exception while deploying the app
>> java.lang.Exception: Aborting, Failed to start
>> container
>> org.glassfish.scripting.jruby.JRubyContainer
>> at
>>
>> com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:448)
>>
>>
>> at
>>
>> com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:246)
>>
>>
>> at
>>
>> org.glassfish.kernel.embedded.EmbeddedDeployerImpl.deploy(EmbeddedDeployerImpl.java:206)
>>
>>
>> at
>>
>> org.glassfish.kernel.embedded.EmbeddedDeployerImpl.deploy(EmbeddedDeployerImpl.java:146)
>>
>>
>> at
>>
>> org.glassfish.scripting.gem.GlassFishMain.startGlassFishEmbedded(GlassFishMain.java:145)
>>
>>
>> [...]
>>
>>
>> If I change it to non-daemon mode (enable: false),
>> I get
>> the following on the console:
>>
>> [root_at_mobileadapp02 sched2]# jruby -S glassfish
>> Parsing config file:
>> /usr/local/sigma/sched2/config/glassfish.yml
>> Starting GlassFish server at: 0.0.0.0:80
>> <http://0.0.0.0:80>
>> <http://0.0.0.0:80> <http://0.0.0.0:80> in production
>>
>> environment...
>> Writing log messages to:
>> /usr/local/sigma/sched2/log/production.log.
>> Press Ctrl+C to stop.
>> java.lang.UnsupportedOperationException: Cannot
>> create
>> XMLStreamReader or XMLEventReader from a
>> javax.xml.transform.stream.StreamSource
>> at
>>
>> com.sun.xml.internal.stream.XMLInputFactoryImpl.jaxpSourcetoXMLInputSource(XMLInputFactoryImpl.java:285)
>>
>>
>> at
>>
>> com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader(XMLInputFactoryImpl.java:145)
>>
>>
>> at
>>
>> org.jvnet.hk2.config.ConfigParser.parse(ConfigParser.java:102)
>> at
>>
>> org.glassfish.kernel.config.DefaultConfigParser$2.run(DefaultConfigParser.java:93)
>>
>>
>> [...]
>>
>>
>> Any suggestions?
>>
>> Thanks,
>> Mark
>>
>>
>> On Wed, Nov 11, 2009 at 9:46 AM, Vivek Pandey
>> <Vivek.Pandey_at_sun.com <mailto:Vivek.Pandey_at_sun.com>
>> <mailto:Vivek.Pandey_at_sun.com <mailto:Vivek.Pandey_at_sun.com>>
>> <mailto:Vivek.Pandey_at_sun.com
>> <mailto:Vivek.Pandey_at_sun.com>
>> <mailto:Vivek.Pandey_at_sun.com
>> <mailto:Vivek.Pandey_at_sun.com>>>> wrote:
>>
>> Mark,
>>
>> Just looking at the warning message I can say what
>> really might be
>> going on. Perhaps someone from grizzly team can
>> comment
>> on this
>> particular issue.
>>
>> In the meantime can you give the latest
>> glassfish gem (not
>> released yet) a try? It will be released soon
>> after
>> more testing
>> but it has latest stable Grizzly and glassfish v3
>> kernel bits. I
>> think its worth a try. Get build#120 from
>>
>> http://hudson.glassfish.org/job/glassfish-gem-build/.
>>
>> If the latest gem does not fix your problem,
>> can you
>> send the
>> thread dump? kill -QUIT pid or ctrl + \
>>
>> -vivek.
>>
>>
>>
>> Mark Westling wrote:
>>
>> I'm running my Rails project using the
>> Glassfish v3
>> gem and if
>> there's no activity for an hour or two
>> hours, the
>> next request
>> hangs and logs the following error:
>>
>> Nov 11, 2009 3:15:44 AM
>> com.sun.grizzly.config.GrizzlyServiceListener
>> WARNING: Interrupting idle Thread:
>> http80-WorkerThread(0)
>>
>> I'm using the default parameters in the
>> glassfish.yml
>> configuration file. This occurs in both
>> daemon and
>> non-daemon
>> mode. The only unusual thing I might be
>> doing is
>> creating (and
>> closing) several processing threads during
>> some
>> requests to
>> take advantage of multiple cores for lengthy
>> computations, but
>> the problem seems to occur even if I don't
>> run any
>> thread-creation processes after a restart. In
>> other words, I
>> can restart glassfish, wait a few hours
>> with no
>> requests, and
>> see the same error.
>>
>> Configuration:
>> jruby 1.4.0 (ruby 1.8.7 patchlevel 174)
>> (2009-11-02
>> 69fbfa3)
>> (Java HotSpot(TM) Client VM 1.6.0_18-ea)
>> [i386-java]
>> rails 2.3.3
>> glassfish gem 0.9.5
>> database adapter: jdbcmysql
>> Linux version 2.6.18-164.el5PAE
>> (mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>>
>>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>>>
>>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>>
>>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com
>> <mailto:mockbuild_at_x86-002.build.bos.redhat.com>>>>) (gcc
>> version
>> 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP
>>
>> Any suggestions?
>>
>> --Mark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>>
>>
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>>>
>> For additional commands, e-mail:
>> webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>
>> <mailto:webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>>
>> <mailto:webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>
>> <mailto:webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>>
>> For additional commands, e-mail:
>> webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>
>> <mailto:webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>>
>> For additional commands, e-mail:
>> webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>
>> <mailto:webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: webtier-unsubscribe_at_glassfish.dev.java.net
>> <mailto:webtier-unsubscribe_at_glassfish.dev.java.net>
>> For additional commands, e-mail:
>> webtier-help_at_glassfish.dev.java.net
>> <mailto:webtier-help_at_glassfish.dev.java.net>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: webtier-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: webtier-help_at_glassfish.dev.java.net
>