Hi, I'll answer your questions below, but I'll provide more observations
somewhere else.
On Wed, Jul 25, 2012 at 3:04 PM, Oleksiy Stashok <oleksiy.stashok_at_oracle.com
> wrote:
> for
> $ ab -n 3000 -c 300
> Netstat showed (the worsest result):
> # netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
> 1 CLOSING
> 371 ESTABLISHED
> 53 FIN_WAIT1
> 19 FIN_WAIT2
> 7 LISTEN
> 33 SYN_RECV
> 1023 TIME_WAIT
>
> Can you pls. the netstat stats after the same GF load?
>
>
new test for the same load, netstat below:
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
9 CLOSE_WAIT
324 ESTABLISHED
1 FIN_WAIT2
1 LAST_ACK
10 LISTEN
23 SYN_RECV
390 TIME_WAIT
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
6 CLOSE_WAIT
329 ESTABLISHED
1 FIN_WAIT1
1 FIN_WAIT2
1 LAST_ACK
10 LISTEN
17 SYN_RECV
470 TIME_WAIT
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
7 CLOSE_WAIT
327 ESTABLISHED
1 FIN_WAIT1
1 FIN_WAIT2
1 LAST_ACK
10 LISTEN
17 SYN_RECV
471 TIME_WAIT
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
7 CLOSE_WAIT
327 ESTABLISHED
1 FIN_WAIT2
1 LAST_ACK
10 LISTEN
19 SYN_RECV
472 TIME_WAIT
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
9 CLOSE_WAIT
324 ESTABLISHED
1 FIN_WAIT2
1 LAST_ACK
10 LISTEN
23 SYN_RECV
390 TIME_WAIT
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
9 CLOSE_WAIT
324 ESTABLISHED
1 FIN_WAIT2
1 LAST_ACK
10 LISTEN
25 SYN_RECV
390 TIME_WAIT
root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
13 CLOSE_WAIT
316 ESTABLISHED
1 FIN_WAIT2
10 LISTEN
32 SYN_RECV
357 TIME_WAIT
server load were load average: 0.14, 0.32, 0.17
much actually lower than tomcat's
# uptime
17:57:37 up 4:51, 1 user, load average: 0.62, 0.36, 0.20
root_at_lvps176-28-13-94:/home/glassfish/glassfish/domains/domain1/logs# uptime
17:59:08 up 4:52, 1 user, load average: 0.82, 0.53, 0.27
but could take much less requests, seen plentry in log file:
[#|2012-07-25T15:50:41.218+0000|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=77;_ThreadName=Thread-2;|SelectorFactory.
Can not create a selector
java.io.IOException: Too many open files
> The adventage I got with Tomcat so far is :
>
> - it didn't hang out yet
> - it clearly hits Virtuozzo limits
>
> Can you pls. elaborate what kind of limits it hits (reported by uptime?).
>
>
No uptime limit it hit,
tomcat hit tcpsndbuf limit by virtuozzo,
more details about this limit at
http://wiki.openvz.org/UBC_secondary_parameters
Can you pls. attach GF domain.xml?
>
>
I played with various domain and the similar problem were noticed, the
first change only included the change to http-pool-thread to higher number
(i.e. 200).
The current domain.xml is as follows:
<domain log-root="${com.sun.aas.instanceRoot}/logs"
application-root="${com.sun.aas.instanceRoot}/applications" version="5">
<system-applications>
<application context-root=""
location="${com.sun.aas.installRootURI}/lib/install/applications/__admingui"
name="__admingui" directory-deployed="true" object-type="system-admin">
<module name="__admingui">
<engine sniffer="web"></engine>
<engine sniffer="security"></engine>
</module>
</application>
</system-applications>
<applications>
<application context-root="/OpenRewrite"
location="${com.sun.aas.instanceRootURI}/applications/OpenRewrite/"
name="OpenRewrite" object-type="user">
<property name="appLocation"
value="${com.sun.aas.instanceRootURI}/applications/__internal/OpenRewrite/OpenRewrite.war"></property>
<property name="preserveAppScopedResources" value="false"></property>
<property name="defaultAppName" value="OpenRewrite"></property>
<module name="OpenRewrite">
<engine sniffer="security"></engine>
<engine sniffer="web"></engine>
</module>
</application>
<application context-root="/NonWwwNumbeo"
location="${com.sun.aas.instanceRootURI}/applications/NonWwwNumbeo/"
name="NonWwwNumbeo" object-type="user">
<property name="appLocation"
value="${com.sun.aas.instanceRootURI}/applications/__internal/NonWwwNumbeo/NonWwwNumbeo.war"></property>
<property name="preserveAppScopedResources" value="false"></property>
<property name="defaultAppName" value="NonWwwNumbeo"></property>
<module name="NonWwwNumbeo">
<engine sniffer="security"></engine>
<engine sniffer="web"></engine>
</module>
</application>
<application context-root="/NonWwwOpenRewrite"
location="${com.sun.aas.instanceRootURI}/applications/NonWwwOpenRewrite/"
name="NonWwwOpenRewrite" object-type="user">
<property name="appLocation"
value="${com.sun.aas.instanceRootURI}/applications/__internal/NonWwwOpenRewrite/NonWwwOpenRewrite.war"></property>
<property name="preserveAppScopedResources" value="false"></property>
<property name="defaultAppName" value="NonWwwOpenRewrite"></property>
<module name="NonWwwOpenRewrite">
<engine sniffer="security"></engine>
<engine sniffer="web"></engine>
</module>
</application>
<application context-root="/NonWwwDecisionCrowd"
location="${com.sun.aas.instanceRootURI}/applications/NonWwwDecisionCrowd/"
name="NonWwwDecisionCrowd" object-type="user">
<property name="appLocation"
value="${com.sun.aas.instanceRootURI}/applications/__internal/NonWwwDecisionCrowd/NonWwwDecisionCrowd.war"></property>
<property name="preserveAppScopedResources" value="false"></property>
<property name="defaultAppName"
value="NonWwwDecisionCrowd"></property>
<property name="java-web-start-enabled" value="false"></property>
<module name="NonWwwDecisionCrowd">
<engine sniffer="security"></engine>
<engine sniffer="web"></engine>
</module>
</application>
<application context-root="/LivingCost"
location="${com.sun.aas.instanceRootURI}/applications/LivingCost/"
name="LivingCost" object-type="user">
<property name="appLocation"
value="${com.sun.aas.instanceRootURI}/applications/__internal/LivingCost/LivingCost.war"></property>
<property name="defaultAppName" value="LivingCost"></property>
<module name="LivingCost">
<engine sniffer="security"></engine>
<engine sniffer="web"></engine>
</module>
</application>
<application context-root="/DecisionCrowd"
location="${com.sun.aas.instanceRootURI}/applications/DecisionCrowd/"
name="DecisionCrowd" object-type="user">
<property name="appLocation"
value="${com.sun.aas.instanceRootURI}/applications/__internal/DecisionCrowd/DecisionCrowd.war"></property>
<property name="defaultAppName" value="DecisionCrowd"></property>
<module name="DecisionCrowd">
<engine sniffer="security"></engine>
<engine sniffer="web"></engine>
</module>
</application>
</applications>
<resources>
<jdbc-resource pool-name="__TimerPool" jndi-name="jdbc/__TimerPool"
object-type="system-admin"></jdbc-resource>
<jdbc-resource pool-name="DerbyPool"
jndi-name="jdbc/__default"></jdbc-resource>
<jdbc-connection-pool
datasource-classname="org.apache.derby.jdbc.EmbeddedXADataSource"
res-type="javax.sql.XADataSource" name="__TimerPool">
<property name="databaseName"
value="${com.sun.aas.instanceRoot}/lib/databases/ejbtimer"></property>
<property name="connectionAttributes" value=";create=true"></property>
</jdbc-connection-pool>
<jdbc-connection-pool is-isolation-level-guaranteed="false"
datasource-classname="org.apache.derby.jdbc.ClientDataSource"
res-type="javax.sql.DataSource" name="DerbyPool">
<property name="PortNumber" value="1527"></property>
<property name="Password" value="APP"></property>
<property name="User" value="APP"></property>
<property name="serverName" value="localhost"></property>
<property name="DatabaseName" value="sun-appserv-samples"></property>
<property name="connectionAttributes" value=";create=true"></property>
</jdbc-connection-pool>
</resources>
<servers>
<server name="server" config-ref="server-config">
<application-ref ref="__admingui"
virtual-servers="__asadmin"></application-ref>
<application-ref ref="OpenRewrite" virtual-servers="
www.openrewrite.com"></application-ref>
<application-ref ref="NonWwwNumbeo" virtual-servers="numbeo.com
"></application-ref>
<application-ref ref="NonWwwOpenRewrite" virtual-servers="
openrewrite.com"></application-ref>
<application-ref ref="NonWwwDecisionCrowd" virtual-servers="
decisioncrowd.com"></application-ref>
<application-ref ref="LivingCost" virtual-servers="www.numbeo.com
"></application-ref>
<application-ref ref="DecisionCrowd" virtual-servers="
www.decisioncrowd.com"></application-ref>
<resource-ref ref="jdbc/__TimerPool"></resource-ref>
<resource-ref ref="jdbc/__default"></resource-ref>
</server>
</servers>
<nodes>
<node node-host="localhost" name="localhost-domain1" type="CONFIG"
install-dir="${com.sun.aas.productRoot}"></node>
</nodes>
<configs>
<config name="server-config">
<http-service>
<access-log></access-log>
<virtual-server id="server"
network-listeners="http-listener-1,http-listener-2"></virtual-server>
<virtual-server id="__asadmin"
network-listeners="admin-listener"></virtual-server>
<virtual-server id="numbeo.com" hosts="numbeo.com"
default-web-module="NonWwwNumbeo" sso-cookie-http-only="false"
network-listeners="http-listener-1"></virtual-server>
<virtual-server id="www.numbeo.com" hosts="www.numbeo.com,
ww2.numbeo.com" default-web-module="LivingCost"
sso-cookie-http-only="false"
network-listeners="http-listener-1"></virtual-server>
<virtual-server id="openrewrite.com" hosts="openrewrite.com"
default-web-module="NonWwwOpenRewrite" sso-cookie-http-only="false"
network-listeners="http-listener-1"></virtual-server>
<virtual-server id="www.openrewrite.com" hosts="www.openrewrite.com"
default-web-module="OpenRewrite" sso-cookie-http-only="false"
network-listeners="http-listener-1"></virtual-server>
<virtual-server id="decisioncrowd.com" hosts="decisioncrowd.com"
default-web-module="NonWwwDecisionCrowd" sso-cookie-http-only="false"
network-listeners="http-listener-1"></virtual-server>
<virtual-server id="www.decisioncrowd.com" hosts="
www.decisioncrowd.com" default-web-module="DecisionCrowd"
sso-cookie-http-only="false"
network-listeners="http-listener-1"></virtual-server>
</http-service>
<iiop-service>
<orb use-thread-pool-ids="thread-pool-1"></orb>
<iiop-listener port="3700" id="orb-listener-1" address="0.0.0.0"
lazy-init="true"></iiop-listener>
<iiop-listener port="3820" id="SSL" address="0.0.0.0"
security-enabled="true">
<ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as"></ssl>
</iiop-listener>
<iiop-listener port="3920" id="SSL_MUTUALAUTH" address="0.0.0.0"
security-enabled="true">
<ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as" client-auth-enabled="true"></ssl>
</iiop-listener>
</iiop-service>
<admin-service system-jmx-connector-name="system"
type="das-and-server">
<jmx-connector port="8686" address="0.0.0.0"
security-enabled="false" auth-realm-name="admin-realm"
name="system"></jmx-connector>
<property name="adminConsoleContextRoot" value="/admin"></property>
<property name="adminConsoleDownloadLocation"
value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
<property name="ipsRoot"
value="${com.sun.aas.installRoot}/.."></property>
<das-config></das-config>
</admin-service>
<connector-service></connector-service>
<web-container>
<session-config>
<session-manager>
<manager-properties></manager-properties>
<store-properties></store-properties>
</session-manager>
<session-properties></session-properties>
</session-config>
</web-container>
<ejb-container
session-store="${com.sun.aas.instanceRoot}/session-store">
<ejb-timer-service></ejb-timer-service>
</ejb-container>
<mdb-container></mdb-container>
<jms-service default-jms-host="default_JMS_host">
<jms-host host="localhost" name="default_JMS_host"></jms-host>
</jms-service>
<security-service>
<auth-realm
classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
name="admin-realm">
<property name="file"
value="${com.sun.aas.instanceRoot}/config/admin-keyfile"></property>
<property name="jaas-context" value="fileRealm"></property>
</auth-realm>
<auth-realm
classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
name="file">
<property name="file"
value="${com.sun.aas.instanceRoot}/config/keyfile"></property>
<property name="jaas-context" value="fileRealm"></property>
</auth-realm>
<auth-realm
classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm"
name="certificate"></auth-realm>
<jacc-provider
policy-provider="com.sun.enterprise.security.provider.PolicyWrapper"
name="default"
policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
<property name="repository"
value="${com.sun.aas.instanceRoot}/generated/policy"></property>
</jacc-provider>
<jacc-provider
policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider"
name="simple"
policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory"></jacc-provider>
<audit-module classname="com.sun.enterprise.security.Audit"
name="default">
<property name="auditOn" value="false"></property>
</audit-module>
<message-security-config auth-layer="SOAP">
<provider-config provider-type="client"
provider-id="XWS_ClientProvider"
class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="dynamic.username.password"
value="false"></property>
<property name="debug" value="false"></property>
</provider-config>
<provider-config provider-type="client"
provider-id="ClientProvider"
class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="dynamic.username.password"
value="false"></property>
<property name="debug" value="false"></property>
<property name="security.config"
value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
</provider-config>
<provider-config provider-type="server"
provider-id="XWS_ServerProvider"
class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="debug" value="false"></property>
</provider-config>
<provider-config provider-type="server"
provider-id="ServerProvider"
class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="debug" value="false"></property>
<property name="security.config"
value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
</provider-config>
</message-security-config>
<message-security-config auth-layer="HttpServlet">
<provider-config provider-type="server"
provider-id="GFConsoleAuthModule"
class-name="org.glassfish.admingui.common.security.AdminConsoleAuthModule">
<request-policy auth-source="sender"></request-policy>
<response-policy></response-policy>
<property name="restAuthURL" value="
https://localhost:
${ADMIN_LISTENER_PORT}/management/sessions"></property>
<property name="loginPage" value="/login.jsf"></property>
<property name="loginErrorPage"
value="/loginError.jsf"></property>
</provider-config>
</message-security-config>
<property name="default-digest-algorithm"
value="SHA-256"></property>
</security-service>
<transaction-service
tx-log-dir="${com.sun.aas.instanceRoot}/logs"></transaction-service>
<java-config debug-options="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9009"
system-classpath="" classpath-suffix="">
<jvm-options>-XX:MaxPermSize=192m</jvm-options>
<jvm-options>-XX:PermSize=64m</jvm-options>
<jvm-options>-Djava.awt.headless=true</jvm-options>
<jvm-options>-Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder</jvm-options>
<jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
<jvm-options>-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed</jvm-options>
<jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
<jvm-options>-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf</jvm-options>
<jvm-options>-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as</jvm-options>
<jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options>
<jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options>
<jvm-options>-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext</jvm-options>
<jvm-options>-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver</jvm-options>
<jvm-options>-DANTLR_USE_DIRECT_CLASS_LOADING=true</jvm-options>
<jvm-options>-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory</jvm-options>
<jvm-options>-Dosgi.shell.telnet.port=6666</jvm-options>
<jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
<jvm-options>-Dosgi.shell.telnet.ip=127.0.0.1</jvm-options>
<jvm-options>-Dgosh.args=--nointeractive</jvm-options>
<jvm-options>-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/</jvm-options>
<jvm-options>-Dfelix.fileinstall.poll=5000</jvm-options>
<jvm-options>-Dfelix.fileinstall.log.level=2</jvm-options>
<jvm-options>-Dfelix.fileinstall.bundles.new.start=true</jvm-options>
<jvm-options>-Dfelix.fileinstall.bundles.startTransient=true</jvm-options>
<jvm-options>-Dfelix.fileinstall.disableConfigSave=false</jvm-options>
<jvm-options>-XX:NewRatio=2</jvm-options>
<jvm-options>-server</jvm-options>
<jvm-options>-Xmx2048m</jvm-options>
<jvm-options>-Xms1024m</jvm-options>
<jvm-options>-Dproduct.name=</jvm-options>
</java-config>
<network-config>
<protocols>
<protocol name="http-listener-1">
<http xpowered-by="false" timeout-seconds="3"
default-virtual-server="server" max-connections="30">
<file-cache></file-cache>
</http>
</protocol>
<protocol security-enabled="true" name="http-listener-2">
<http xpowered-by="false" default-virtual-server="server"
max-connections="250">
<file-cache></file-cache>
</http>
<ssl
classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as"></ssl>
</protocol>
<protocol security-enabled="true" name="admin-listener">
<http xpowered-by="false" default-virtual-server="__asadmin"
max-connections="250" encoded-slash-enabled="true">
<file-cache></file-cache>
</http>
</protocol>
<protocol security-enabled="true" name="sec-admin-listener">
<http default-virtual-server="__asadmin"
encoded-slash-enabled="true">
<file-cache></file-cache>
</http>
<ssl client-auth="want"
classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as"></ssl>
</protocol>
<protocol name="admin-http-redirect">
<http-redirect secure="true"></http-redirect>
</protocol>
<protocol name="pu-protocol">
<port-unification>
<protocol-finder protocol="sec-admin-listener"
name="http-finder"
classname="com.sun.grizzly.config.HttpProtocolFinder"></protocol-finder>
<protocol-finder protocol="admin-http-redirect"
name="admin-http-redirect"
classname="com.sun.grizzly.config.HttpProtocolFinder"></protocol-finder>
</port-unification>
</protocol>
</protocols>
<network-listeners>
<network-listener port="8080" protocol="http-listener-1"
transport="tcp" name="http-listener-1"
thread-pool="http-thread-pool"></network-listener>
<network-listener port="8181" protocol="http-listener-2"
transport="tcp" name="http-listener-2"
thread-pool="http-thread-pool"></network-listener>
<network-listener port="4848" protocol="pu-protocol"
transport="tcp" name="admin-listener"
thread-pool="admin-thread-pool"></network-listener>
</network-listeners>
<transports>
<transport max-connections-count="800" name="tcp"
idle-key-timeout-seconds="20" acceptor-threads="2"></transport>
</transports>
</network-config>
<thread-pools>
<thread-pool name="admin-thread-pool" max-thread-pool-size="50"
max-queue-size="256"></thread-pool>
<thread-pool max-thread-pool-size="200" name="http-thread-pool"
max-queue-size="500" idle-thread-timeout-seconds="120"></thread-pool>
<thread-pool name="thread-pool-1"
max-thread-pool-size="200"></thread-pool>
</thread-pools>
<monitoring-service>
<module-monitoring-levels http-service="HIGH"
connector-connection-pool="HIGH" connector-service="HIGH" jvm="HIGH"
thread-pool="HIGH" web-container="HIGH"></module-monitoring-levels>
</monitoring-service>
<group-management-service>
<failure-detection></failure-detection>
</group-management-service>
</config>
<config name="default-config">
<http-service>
<access-log></access-log>
<virtual-server id="server" network-listeners="http-listener-1,
http-listener-2">
<property name="default-web-xml"
value="${com.sun.aas.instanceRoot}/config/default-web.xml"></property>
</virtual-server>
<virtual-server id="__asadmin"
network-listeners="admin-listener"></virtual-server>
</http-service>
<iiop-service>
<orb use-thread-pool-ids="thread-pool-1"></orb>
<iiop-listener port="${IIOP_LISTENER_PORT}" id="orb-listener-1"
address="0.0.0.0"></iiop-listener>
<iiop-listener port="${IIOP_SSL_LISTENER_PORT}" id="SSL"
address="0.0.0.0" security-enabled="true">
<ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as"></ssl>
</iiop-listener>
<iiop-listener port="${IIOP_SSL_MUTUALAUTH_PORT}"
id="SSL_MUTUALAUTH" address="0.0.0.0" security-enabled="true">
<ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as" client-auth-enabled="true"></ssl>
</iiop-listener>
</iiop-service>
<admin-service system-jmx-connector-name="system">
<jmx-connector port="${JMX_SYSTEM_CONNECTOR_PORT}"
address="0.0.0.0" security-enabled="false" auth-realm-name="admin-realm"
name="system"></jmx-connector>
<property name="adminConsoleDownloadLocation"
value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
<das-config></das-config>
</admin-service>
<web-container>
<session-config>
<session-manager>
<manager-properties></manager-properties>
<store-properties></store-properties>
</session-manager>
<session-properties></session-properties>
</session-config>
</web-container>
<ejb-container
session-store="${com.sun.aas.instanceRoot}/session-store">
<ejb-timer-service></ejb-timer-service>
</ejb-container>
<mdb-container></mdb-container>
<jms-service addresslist-behavior="priority"
default-jms-host="default_JMS_host">
<jms-host port="${JMS_PROVIDER_PORT}" host="localhost"
name="default_JMS_host"></jms-host>
</jms-service>
<log-service log-rotation-limit-in-bytes="2000000"
file="${com.sun.aas.instanceRoot}/logs/server.log">
<module-log-levels></module-log-levels>
</log-service>
<security-service>
<auth-realm
classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
name="admin-realm">
<property name="file"
value="${com.sun.aas.instanceRoot}/config/admin-keyfile"></property>
<property name="jaas-context" value="fileRealm"></property>
</auth-realm>
<auth-realm
classname="com.sun.enterprise.security.auth.realm.file.FileRealm"
name="file">
<property name="file"
value="${com.sun.aas.instanceRoot}/config/keyfile"></property>
<property name="jaas-context" value="fileRealm"></property>
</auth-realm>
<auth-realm
classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm"
name="certificate"></auth-realm>
<jacc-provider
policy-provider="com.sun.enterprise.security.provider.PolicyWrapper"
name="default"
policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
<property name="repository"
value="${com.sun.aas.instanceRoot}/generated/policy"></property>
</jacc-provider>
<jacc-provider
policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider"
name="simple"
policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory"></jacc-provider>
<audit-module classname="com.sun.enterprise.security.Audit"
name="default">
<property name="auditOn" value="false"></property>
</audit-module>
<message-security-config auth-layer="SOAP">
<provider-config provider-type="client"
provider-id="XWS_ClientProvider"
class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="dynamic.username.password"
value="false"></property>
<property name="debug" value="false"></property>
</provider-config>
<provider-config provider-type="client"
provider-id="ClientProvider"
class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="dynamic.username.password"
value="false"></property>
<property name="debug" value="false"></property>
<property name="security.config"
value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
</provider-config>
<provider-config provider-type="server"
provider-id="XWS_ServerProvider"
class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="debug" value="false"></property>
</provider-config>
<provider-config provider-type="server"
provider-id="ServerProvider"
class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
<request-policy auth-source="content"></request-policy>
<response-policy auth-source="content"></response-policy>
<property name="encryption.key.alias" value="s1as"></property>
<property name="signature.key.alias" value="s1as"></property>
<property name="debug" value="false"></property>
<property name="security.config"
value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
</provider-config>
</message-security-config>
</security-service>
<transaction-service tx-log-dir="${com.sun.aas.instanceRoot}/logs"
automatic-recovery="true"></transaction-service>
<diagnostic-service></diagnostic-service>
<java-config debug-options="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=${JAVA_DEBUGGER_PORT}"
system-classpath="" classpath-suffix="">
<jvm-options>-XX:MaxPermSize=192m</jvm-options>
<jvm-options>-XX:PermSize=64m</jvm-options>
<jvm-options>-server</jvm-options>
<jvm-options>-Djava.awt.headless=true</jvm-options>
<jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
<jvm-options>-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed</jvm-options>
<jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
<jvm-options>-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf</jvm-options>
<jvm-options>-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as</jvm-options>
<jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options>
<jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options>
<jvm-options>-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext</jvm-options>
<jvm-options>-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver</jvm-options>
<jvm-options>-DANTLR_USE_DIRECT_CLASS_LOADING=true</jvm-options>
<jvm-options>-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory</jvm-options>
<jvm-options>-XX:NewRatio=2</jvm-options>
<jvm-options>-Xmx512m</jvm-options>
<jvm-options>-Dosgi.shell.telnet.port=${OSGI_SHELL_TELNET_PORT}</jvm-options>
<jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
<jvm-options>-Dosgi.shell.telnet.ip=127.0.0.1</jvm-options>
<jvm-options>-Dgosh.args=--noshutdown -c noop=true</jvm-options>
<jvm-options>-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/</jvm-options>
<jvm-options>-Dfelix.fileinstall.poll=5000</jvm-options>
<jvm-options>-Dfelix.fileinstall.log.level=3</jvm-options>
<jvm-options>-Dfelix.fileinstall.bundles.new.start=true</jvm-options>
<jvm-options>-Dfelix.fileinstall.bundles.startTransient=true</jvm-options>
<jvm-options>-Dfelix.fileinstall.disableConfigSave=false</jvm-options>
</java-config>
<availability-service>
<web-container-availability></web-container-availability>
<ejb-container-availability
sfsb-store-pool-name="jdbc/hastore"></ejb-container-availability>
<jms-availability></jms-availability>
</availability-service>
<network-config>
<protocols>
<protocol name="http-listener-1">
<http default-virtual-server="server">
<file-cache></file-cache>
</http>
</protocol>
<protocol security-enabled="true" name="http-listener-2">
<http default-virtual-server="server">
<file-cache></file-cache>
</http>
<ssl
classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="s1as"></ssl>
</protocol>
<protocol name="admin-listener">
<http default-virtual-server="__asadmin" max-connections="250">
<file-cache></file-cache>
</http>
</protocol>
<protocol security-enabled="true" name="sec-admin-listener">
<http default-virtual-server="__asadmin"
encoded-slash-enabled="true">
<file-cache></file-cache>
</http>
<ssl client-auth="want"
classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl"
cert-nickname="glassfish-instance"></ssl>
</protocol>
<protocol name="admin-http-redirect">
<http-redirect secure="true"></http-redirect>
</protocol>
<protocol name="pu-protocol">
<port-unification>
<protocol-finder protocol="sec-admin-listener"
name="http-finder"
classname="com.sun.grizzly.config.HttpProtocolFinder"></protocol-finder>
<protocol-finder protocol="admin-http-redirect"
name="admin-http-redirect"
classname="com.sun.grizzly.config.HttpProtocolFinder"></protocol-finder>
</port-unification>
</protocol>
</protocols>
<network-listeners>
<network-listener port="${HTTP_LISTENER_PORT}"
protocol="http-listener-1" transport="tcp" name="http-listener-1"
thread-pool="http-thread-pool"></network-listener>
<network-listener port="${HTTP_SSL_LISTENER_PORT}"
protocol="http-listener-2" transport="tcp" name="http-listener-2"
thread-pool="http-thread-pool"></network-listener>
<network-listener port="${ASADMIN_LISTENER_PORT}"
protocol="pu-protocol" transport="tcp" name="admin-listener"
thread-pool="http-thread-pool"></network-listener>
</network-listeners>
<transports>
<transport name="tcp"></transport>
</transports>
</network-config>
<thread-pools>
<thread-pool name="http-thread-pool"></thread-pool>
<thread-pool max-thread-pool-size="200"
name="thread-pool-1"></thread-pool>
<thread-pool name="admin-thread-pool" max-thread-pool-size="50"
max-queue-size="256"></thread-pool>
</thread-pools>
<group-management-service>
<failure-detection></failure-detection>
</group-management-service>
<management-rules></management-rules>
<system-property name="ASADMIN_LISTENER_PORT"
value="24848"></system-property>
<system-property name="HTTP_LISTENER_PORT"
value="28080"></system-property>
<system-property name="HTTP_SSL_LISTENER_PORT"
value="28181"></system-property>
<system-property name="JMS_PROVIDER_PORT"
value="27676"></system-property>
<system-property name="IIOP_LISTENER_PORT"
value="23700"></system-property>
<system-property name="IIOP_SSL_LISTENER_PORT"
value="23820"></system-property>
<system-property name="IIOP_SSL_MUTUALAUTH_PORT"
value="23920"></system-property>
<system-property name="JMX_SYSTEM_CONNECTOR_PORT"
value="28686"></system-property>
<system-property name="OSGI_SHELL_TELNET_PORT"
value="26666"></system-property>
<system-property name="JAVA_DEBUGGER_PORT"
value="29009"></system-property>
<monitoring-service>
<module-monitoring-levels></module-monitoring-levels>
</monitoring-service>
<connector-service></connector-service>
</config>
</configs>
<property name="administrative.domain.name" value="domain1"></property>
<secure-admin enabled="true"
special-admin-indicator="3047aff3-3214-4ac9-aa5e-a5dad78b2eea">
<secure-admin-principal dn="CN=localhost,OU=GlassFish,O=Oracle
Corporation,L=Santa Clara,ST=California,C=US"></secure-admin-principal>
<secure-admin-principal dn="CN=localhost-instance,OU=GlassFish,O=Oracle
Corporation,L=Santa Clara,ST=California,C=US"></secure-admin-principal>
</secure-admin>
<load-balancers></load-balancers>
<lb-configs></lb-configs>
<clusters></clusters>
</domain>
Thanks.
>
> WBR,
> Alexey.
>
>
> *
> *
> *
> *
> *
> *
> On Tue, Jul 24, 2012 at 6:33 PM, Oleksiy Stashok <
> oleksiy.stashok_at_oracle.com> wrote:
>
>> Hi Mladen,
>>
>>
>> increased the number of connections in time_wait state.
>> 110 CLOSE_WAIT
>> 251 ESTABLISHED
>> 2 FIN_WAIT2
>> 10 LISTEN
>> 1 SYN_RECV
>> 404 TIME_WAIT
>>
>>
>> Connection queue of 500 were full to accomodate further requests. I had
>> to lower it so webserver doesn't run out of available sockets (1024, common
>> value with Virtuozzo).
>>
>> I doubt connection queue size will help you prevent more sockets to be
>> accepted. IMO you can keep the default value.
>> You can write simple app. like:
>>
>> Socket[] sockets = new Socket[1024];
>> int i = 0;
>> try {
>> for (; i < 1024; i++) {
>> sockets[i] = new Socket(yourServerHost, yourServerPort);
>> }
>> } catch (Exception e) {
>> System.out.println("Failed on connection #" + i);
>> }
>>
>> to double-check if connection queue size makes any different.
>>
>>
>>
>> If I understand correctly TIME_WAIT is when server waits for response
>> from client it received the data and it can last up to 240ms. This time
>> shall not be lowered in Linux, it can have big consequences.
>>
>> Well, if you're so limited in resources, may be you can try to decrease
>> this timeout.
>> Anyway, agree w/ your conclusion below :))
>>
>> Thanks.
>>
>> WBR,
>> Alexey.
>>
>>
>>
>> Now after turning the server down and up, it starts to increase again:
>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
>> 35 ESTABLISHED
>> 20 FIN_WAIT1
>> 4 FIN_WAIT2
>> 74 LAST_ACK
>> 10 LISTEN
>> 1 SYN_RECV
>> 119 TIME_WAIT
>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
>> 43 ESTABLISHED
>> 1 FIN_WAIT1
>> 23 FIN_WAIT2
>> 10 LISTEN
>> 2 SYN_RECV
>> 228 TIME_WAIT
>>
>> OK, I decided to try to crash it from my localhost:
>>
>> I run from my localhost: # ab -n 10000 -c 1000
>> http://www.numbeo.com/cost-of-living/
>> to try to crash my own webserver, this is how it looks like:
>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
>> 7 CLOSE_WAIT
>> 337 ESTABLISHED
>> 25 FIN_WAIT1
>> 12 FIN_WAIT2
>> 10 LISTEN
>> 74 SYN_RECV
>> 610 TIME_WAIT
>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
>> 343 ESTABLISHED
>> 28 FIN_WAIT1
>> 16 FIN_WAIT2
>> 10 LISTEN
>> 119 SYN_RECV
>> 1051 TIME_WAIT
>>
>> ab finished with
>> Completed 1000 requests
>> Completed 2000 requests
>> apr_socket_recv: Connection reset by peer (104)
>> Total of 2520 requests completed
>>
>> and now sockets seems to come back to normal value:
>> # netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
>> 3 CLOSE_WAIT
>> 119 ESTABLISHED
>> 2 FIN_WAIT1
>> 21 FIN_WAIT2
>> 10 LISTEN
>> 34 SYN_RECV
>> 179 TIME_WAIT
>>
>> During the period I can see it run out of sockets in the log:
>> [#|2012-07-24T10:08:40.855+0000|WARNING|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=11;_ThreadName=Thread-2;|GRIZZLY0006:
>> Exception accepting channel
>> java.io.IOException: Too many open files
>>
>>
>> and server load is not bad at all:
>> # uptime
>> 12:12:29 up 15:47, 1 user, load average: 0.24, 0.84, 0.55
>>
>>
>> *Conclusion*: it looks to me that although webserver process the
>> request for 3ms in average, the socket remains open much longer to be sure
>> the response has been received by the client. So at the moment of heavy
>> load testing, 120 request were processing but 1600 sockets were kept for
>> TIME_WAIT and ESTABLISHED phase of tcp.
>> This problem I might face it not be web server related issue but rather
>> lack of available sockets under virtualization for TIME_WAIT. Perhaps for
>> me time has come to migrate to dedicated server and to put available
>> sockets to 65K.
>>
>>
>>
>>
>> On Tue, Jul 24, 2012 at 11:58 AM, Mladen Adamovic <
>> mladen.adamovic_at_gmail.com> wrote:
>>
>>> Hi guys, currently with 1 max alive request per connection, the
>>> webserver started to refuse connections, connection queue with max 500 is
>>> full.
>>>
>>> server.log looks clean and this is summary of the netstat:
>>>
>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>> -c
>>> 96 CLOSE_WAIT
>>> 143 ESTABLISHED
>>> 1 FIN_WAIT1
>>> 1 FIN_WAIT2
>>> 10 LISTEN
>>> 29 TIME_WAIT
>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>> -c
>>> 96 CLOSE_WAIT
>>> 138 ESTABLISHED
>>> 6 FIN_WAIT2
>>> 10 LISTEN
>>> 20 TIME_WAIT
>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>> -c
>>> 99 CLOSE_WAIT
>>> 134 ESTABLISHED
>>> 1 FIN_WAIT2
>>> 10 LISTEN
>>> 28 TIME_WAIT
>>> root_at_lvps176-28-13-94:~# uptime
>>> 11:52:46 up 15:27, 1 user, load average: 0.30, 0.06, 0.02
>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>> -c
>>> 111 CLOSE_WAIT
>>> 223 ESTABLISHED
>>> 1 FIN_WAIT2
>>> 10 LISTEN
>>> 2 SYN_RECV
>>> 17 TIME_WAIT
>>>
>>>
>>> and the number of connections seems to increase:
>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>> -c
>>> 110 CLOSE_WAIT
>>> 251 ESTABLISHED
>>> 2 FIN_WAIT2
>>> 10 LISTEN
>>> 1 SYN_RECV
>>> 404 TIME_WAIT
>>>
>>>
>>> Connection queue seems to be full although processing time is 3ms:
>>>
>>>
>>> onitor (34 Statistics)
>>> File Cache Statistics : http-listener-1
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> ContentMissesCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of
>>> misses on cached file content
>>> OpenCacheEntriesCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of
>>> current open cache entries
>>> MaxMappedMemorySize 0 byte(s) Jul 24, 2012 9:54:48 AM Jul 24, 2012
>>> 9:54:48 AM -- Maximum memory map size used for caching
>>> HitsCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of cache lookup
>>> hits
>>> InfoMissesCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of misses
>>> on cached file info
>>> ContentHitsCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of hits
>>> on cached file content
>>> HeapSize 0 byte(s) Jul 24, 2012 9:54:48 AM Jul 24, 2012 9:54:48 AM -- Current
>>> cache size in bytes
>>> MaxHeapSize 0 byte(s) Jul 24, 2012 9:54:48 AM Jul 24, 2012 9:54:48 AM -- Maximum
>>> heap space used for cache
>>> InfoHitsCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of hits on
>>> cached file info
>>> MappedMemorySize 0 byte(s) Jul 24, 2012 9:54:48 AM Jul 24, 2012 9:54:48
>>> AM -- Size of mapped memory used for caching
>>> MissesCount 0 count Jul 23, 2012 9:57:57 PM -- -- Number of cache
>>> lookup misses
>>> Keep Alive Statistics : http-listener-1
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> CountTimeouts 48452 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:47:58
>>> AM -- Number of keep-alive connections that timed out
>>> MaxRequests 1 count Jul 23, 2012 9:57:57 PM Jul 23, 2012 9:57:58 PM -- Maximum
>>> number of requests allowed on a single keep-alive connection
>>> CountHits 217839 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:47:59 AM
>>> -- Number of requests received by connections in keep-alive mode
>>> SecondsTimeouts 5 seconds Jul 23, 2012 9:57:57 PM Jul 23, 2012 9:57:58
>>> PM -- Keep-alive timeout value in seconds
>>> CountConnections 188 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:47:59
>>> AM -- Number of connections in keep-alive mode
>>> CountFlushes 96184 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:47:52 AM
>>> -- Number of keep-alive connections that were closed
>>> CountRefusals 73011 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:47:43
>>> AM -- Number of keep-alive connections that were rejected
>>> Connection Queue Statistics : http-listener-1
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> CountQueued15MinutesAverage 3365 count Jul 24, 2012 9:54:48 AM Jul 24,
>>> 2012 9:54:48 AM -- Average number of connections queued in the last 15
>>> minutes
>>> CountQueued5MinutesAverage 203 count Jul 24, 2012 9:54:48 AM Jul 24,
>>> 2012 9:54:48 AM -- Average number of connections queued in the last 5
>>> minutes
>>> CountQueued1MinuteAverage 0 count Jul 24, 2012 9:54:48 AM Jul 24, 2012
>>> 9:54:48 AM -- Average number of connections queued in the last 1 minute
>>> TicksTotalQueued 0 count Jul 23, 2012 9:57:57 PM Jul 23, 2012 9:57:57 PM
>>> -- (Unsupported) Total number of ticks that connections have spent in
>>> the queue
>>> PeakQueued 500 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:51:45 AM -- Largest
>>> number of connections that were in the queue simultaneously
>>> CountOpenConnections 202 count Jul 24, 2012 9:54:48 AM Jul 24, 2012
>>> 9:54:48 AM -- The number of open/active connections
>>> CountTotalConnections 163494 count Jul 23, 2012 9:57:57 PM Jul 24, 2012
>>> 9:54:48 AM -- Total number of connections that have been accepted
>>> CountTotalQueued 248658 count Jul 23, 2012 9:57:57 PM Jul 24, 2012
>>> 9:51:45 AM -- Total number of connections that have been queued
>>> CountQueued 500 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:51:45 AM -- Number
>>> of connections currently in the queue
>>> CountOverflows 888 count Jul 23, 2012 9:57:57 PM Jul 24, 2012 9:54:48 AM
>>> -- Number of times the queue has been too full to accommodate a
>>> connection
>>> MaxQueued 500 count Jul 23, 2012 9:57:57 PM Jul 23, 2012 9:57:58 PM -- Maximum
>>> size of the connection queue
>>> Thread Pool Statistics : http-listener-1
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> CoreThreads 5 count Jul 23, 2012 9:57:57 PM Jul 23, 2012 9:57:58 PM -- Core
>>> number of threads in the thread pool
>>> CurrentThreadsBusy 200 count Jul 23, 2012 9:57:57 PM Jul 24, 2012
>>> 9:47:59 AM -- Provides the number of request processing threads
>>> currently in use in the listener thread pool serving requests
>>> TotalExecutedTasksCount 247958 count Jul 23, 2012 9:57:57 PM Jul 24,
>>> 2012 9:47:52 AM -- Provides the total number of tasks, which were
>>> executed by the thread pool
>>> MaxThreads 200 count Jul 23, 2012 9:57:57 PM Jul 23, 2012 9:57:58 PM -- Maximum
>>> number of threads allowed in the thread pool
>>> CurrentThreadCount 200 count Jul 23, 2012 9:57:57 PM Jul 24, 2012
>>> 9:47:59 AM -- Provides the number of request processing threads
>>> currently in the listener thread pool
>>>
>>>
>>>
>>> Monitor (18 Statistics)
>>> Web Container: Servlet Statistics : LivingCost
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> ActiveServletsLoaded 12count Jul 23, 2012 9:58:08 PM Jul 24, 2012
>>> 8:50:52 AM High Water Mark: 12 count
>>> Low Water Mark: 0 count
>>> Number of Servlets loaded
>>> TotalServletsLoaded 12 count Jul 23, 2012 9:58:08 PM Jul 24, 2012
>>> 8:50:52 AM -- Total number of Servlets ever loaded
>>> ServletProcessingTimes 867166 millisecond Jul 23, 2012 9:58:08 PM Jul
>>> 24, 2012 9:47:58 AM -- Cumulative Servlet processing times
>>> Web Container: Session Statistics : LivingCost
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> PersistedSessionsTotal 0 count Jul 23, 2012 9:58:08 PM -- -- Total
>>> number of sessions ever persisted
>>> RejectedSessionsTotal 0 count Jul 23, 2012 9:58:08 PM -- -- Total
>>> number of sessions ever rejected
>>> SessionsTotal 51635 count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:47:58
>>> AM -- Total number of sessions ever created
>>> ActiveSessions 2074count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:55:31 AM High
>>> Water Mark: 4468 count
>>> Low Water Mark: 0 count
>>> Number of active sessions
>>> ExpiredSessionsTotal 49561 count Jul 23, 2012 9:58:08 PM Jul 24, 2012
>>> 9:55:31 AM -- Total number of sessions ever expired
>>> ActivatedSessionsTotal 0 count Jul 23, 2012 9:58:08 PM -- -- Total
>>> number of sessions ever activated
>>> PassivatedSessionsTotal 0 count Jul 23, 2012 9:58:08 PM -- -- Total
>>> number of sessions ever passivated
>>> Web Container: Request Statistics : LivingCost
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> RequestCount 222664 count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:56:20
>>> AM -- Cumulative number of requests processed so far
>>> MaxTime 3896 millisecond Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:56:20 AM
>>> -- Longest response time for a request; not a cumulative value, but the
>>> largest response time from among the response times
>>> ProcessingTime 2 millisecond Jul 23, 2012 9:58:08 PM Jul 24, 2012
>>> 9:56:20 AM -- Average request processing time
>>> ErrorCount 958 count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:47:50 AM -- Cumulative
>>> value of the error count, with error count representing the number of cases
>>> where the response code was greater than or equal to 400
>>> Web Container: JSP Statistics : LivingCost
>>> Name
>>> Value
>>> Start Time
>>> Last Sample Time
>>> Details
>>> Description
>>> JspReloadedCount 166 count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:01:43
>>> AM -- Total number of JSP pages that were reloaded
>>> TotalJspCount 166 count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:01:43 AM
>>> -- Total number of JSP pages ever loaded
>>> JspCount 166count Jul 23, 2012 9:58:08 PM Jul 24, 2012 9:01:43 AM High
>>> Water Mark: 166 count
>>> Low Water Mark: 0 count
>>> Number of active JSP pages
>>> JspErrorCount 0 count Jul 23, 2012 9:58:08 PM -- -- Total number of
>>> errors triggered by JSP page invocations
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jul 24, 2012 at 12:08 AM, Mladen Adamovic <
>>> mladen.adamovic_at_gmail.com> wrote:
>>>
>>>>
>>>>> When you disable keep-alive connections, does it change anything
>>>>> w.r.t. a number of CLOSE_WAIT connections?
>>>>>
>>>>>
>>>> Well, I cannot always reproduce the problem, the last time it ran for
>>>> 12 hours before crashing. Note that this is production server so it doesn't
>>>> make sense for me to run "experiments against the real user base".
>>>>
>>>> Currently the webserver is running with modified configuration, which
>>>> includes keep-alive max requests to 20, connection time out to 15 seconds
>>>> and this is how number of connections did look like recently:
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 105 ESTABLISHED
>>>> 1 FIN_WAIT1
>>>> 42 FIN_WAIT2
>>>> 2 LAST_ACK
>>>> 10 LISTEN
>>>> 4 SYN_RECV
>>>> 112 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 102 ESTABLISHED
>>>> 1 FIN_WAIT1
>>>> 33 FIN_WAIT2
>>>> 2 LAST_ACK
>>>> 10 LISTEN
>>>> 4 SYN_RECV
>>>> 123 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 88 ESTABLISHED
>>>> 1 FIN_WAIT1
>>>> 20 FIN_WAIT2
>>>> 2 LAST_ACK
>>>> 10 LISTEN
>>>> 8 SYN_RECV
>>>> 137 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -a | grep "CLOSE_WAIT" | wc -l
>>>> 0
>>>> root_at_lvps176-28-13-94:~# netstat -a | grep "CLOSE_WAIT" | wc -l
>>>> 0
>>>> root_at_lvps176-28-13-94:~# netstat -a | grep "CLOSE_WAIT" | wc -l
>>>> 2
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 2 CLOSE_WAIT
>>>> 72 ESTABLISHED
>>>> 2 FIN_WAIT1
>>>> 39 FIN_WAIT2
>>>> 10 LISTEN
>>>> 2 SYN_RECV
>>>> 110 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 35 ESTABLISHED
>>>> 7 FIN_WAIT2
>>>> 10 LISTEN
>>>> 31 SYN_RECV
>>>> 88 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 1 CLOSE_WAIT
>>>> 70 ESTABLISHED
>>>> 26 FIN_WAIT2
>>>> 1 LAST_ACK
>>>> 10 LISTEN
>>>> 18 SYN_RECV
>>>> 88 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 71 ESTABLISHED
>>>> 2 FIN_WAIT1
>>>> 28 FIN_WAIT2
>>>> 10 LISTEN
>>>> 6 SYN_RECV
>>>> 128 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~#
>>>>
>>>>
>>>> OK, now checking with max request per keep alive connection set to 1:
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 28 ESTABLISHED
>>>> 1 FIN_WAIT1
>>>> 3 FIN_WAIT2
>>>> 10 LISTEN
>>>> 5 SYN_RECV
>>>> 241 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 17 ESTABLISHED
>>>> 2 FIN_WAIT1
>>>> 5 FIN_WAIT2
>>>> 10 LISTEN
>>>> 5 SYN_RECV
>>>> 250 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 1 CLOSING
>>>> 30 ESTABLISHED
>>>> 9 FIN_WAIT2
>>>> 10 LISTEN
>>>> 5 SYN_RECV
>>>> 267 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 29 ESTABLISHED
>>>> 6 FIN_WAIT1
>>>> 15 FIN_WAIT2
>>>> 10 LISTEN
>>>> 5 SYN_RECV
>>>> 221 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 26 ESTABLISHED
>>>> 8 FIN_WAIT1
>>>> 16 FIN_WAIT2
>>>> 10 LISTEN
>>>> 6 SYN_RECV
>>>> 227 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 27 ESTABLISHED
>>>> 3 FIN_WAIT1
>>>> 19 FIN_WAIT2
>>>> 10 LISTEN
>>>> 10 SYN_RECV
>>>> 183 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 1 CLOSE_WAIT
>>>> 21 ESTABLISHED
>>>> 4 FIN_WAIT1
>>>> 21 FIN_WAIT2
>>>> 10 LISTEN
>>>> 10 SYN_RECV
>>>> 196 TIME_WAIT
>>>> root_at_lvps176-28-13-94:~# netstat -an|awk '/tcp/ {print $6}'|sort|uniq
>>>> -c
>>>> 1 CLOSE_WAIT
>>>> 32 ESTABLISHED
>>>> 4 FIN_WAIT1
>>>> 15 FIN_WAIT2
>>>> 10 LISTEN
>>>> 7 SYN_RECV
>>>> 239 TIME_WAIT
>>>>
>>>>
>>>> Looks worser. Around ~200 (oscillating) connections in TIME_WAIT
>>>> state. When not using keep-alive many connections are created and dropped
>>>> and sometimes connection can stay in TIME_WAIT for 240ms. It's easy to go
>>>> out of network sockets, it seems.
>>>>
>>>>
>>>>
>>>>
>>>> Thanks.
>>>>>
>>>>> WBR,
>>>>> Alexey.
>>>>>
>>>>>
>>>>>
>>>>> Hm, I've seen one problem with the Virtuozzo server in the log:
>>>>> quotaugidlimit
>>>>> Number of user/group IDs allowed for the Container internal disk
>>>>> quota. If set to 0, UID/GID quota will not be enabled.
>>>>>
>>>>> This is set to limit 2000, but I don't understand what it has with
>>>>> files, it shall be number of UID/GIDs, and at the moment is has been steady
>>>>> at 43 at VPS.
>>>>> Perhaps these are problem with Virtuozzo setup but it has to be proven
>>>>> somehow.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mladen Adamovic
>>>>> Numbeo
>>>>> Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
>>>>> (Business Registration Number 62612240)
>>>>> Tel. +381-66-058-595
>>>>> email: mladen.adamovic_at_gmail.com
>>>>> web: http://www.numbeo.com
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mladen Adamovic
>>>> Numbeo
>>>> Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
>>>> (Business Registration Number 62612240)
>>>> Tel. +381-66-058-595
>>>> email: mladen.adamovic_at_gmail.com
>>>> web: http://www.numbeo.com
>>>>
>>>
>>>
>>>
>>> --
>>> Mladen Adamovic
>>> Numbeo
>>> Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
>>> (Business Registration Number 62612240)
>>> Tel. +381-66-058-595
>>> email: mladen.adamovic_at_gmail.com
>>> web: http://www.numbeo.com
>>>
>>
>>
>>
>> --
>> Mladen Adamovic
>> Numbeo
>> Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
>> (Business Registration Number 62612240)
>> Tel. +381-66-058-595
>> email: mladen.adamovic_at_gmail.com
>> web: http://www.numbeo.com
>>
>>
>>
>
>
> --
> Mladen Adamovic
> Numbeo
> Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
> (Business Registration Number 62612240)
> Tel. +381-66-058-595
> email: mladen.adamovic_at_gmail.com
> web: http://www.numbeo.com
>
>
>
--
Mladen Adamovic
Numbeo
Drziceva 9, 11120 Belgrade-Zvezdara, Serbia
(Business Registration Number 62612240)
Tel. +381-66-058-595
email: mladen.adamovic_at_gmail.com
web: http://www.numbeo.com