Apologies for the delay getting this information out, but I wanted to
make sure I had as much information as possible. Please see my
response inline.
On May 23, 2006, at 12:51 AM, Nazrul Islam wrote:
> During the GlassFish BOF 0251, few developers asked about the
> quality of the Application Server. From the GlassFish Quality
> Portal, it is not clear how much testing is being done. Could we
> share the "quality matrix" with the community?
Here's some relevant information. We'll post this to the Quality page
shortly.
Platforms
Solaris Sparc 9/10, Sol X86 9/10
Solaris Sparc/X86 64 bit
Win XP, Win 2003, Win 2K
Linux 3.0, 4.0
Mac OS (Tiger)
Databases and Drivers
Oracle, Derby, MS SQL, MySQL, Sybase, DB2, PostGres
Postgres Driver, Datadirect Driver 3.5.26 (Type 4), Oracle Thin
Driver (otn.oracle.com) ( Type 2 ),IBM Driver Type 2,Connector/J
Driver 3.1,Native Derby driver
JDK Versions
JDK 1.5_06, JDK 1.5_04, JDK 1.6 beta, JDK 1.6 beta2
>
> The following questions may be good to answer...
> 1. How may test cases were executed and passed for the recent
> FCS release?
For the last FCS release, there were 33,400 compatibility tests and
17,547 functional tests executed.
> 2. How many bugs were fixed for the recent FCS release? Latest
> bug trend.
1,793 bugs were fixed for this release! We're working hard on
significantly expanding the bug information available. Expect to see
charts, a unified bug list and other dynamic information soon.
> 3. What is the current code coverage?
Code coverage is currently in the planning stages. We'll publish a
schedule as soon as the plan is finalized.
> 4. How many test cases passed on a nightly build?
Quicklook tests = 62
Functional Smoke Tests = 37
Compatibility Smoke Tests = 347
The latest promoted and nightly results are now available on the
quality page.
> 5. How many test cases passed a promoted build?
I'm still in the process of fully tracking this down; I can tell you
that 6476 tests are run for the core functionality alone per promoted
build.
> 6. How can people help?
We're looking for contributors to our unified test framework where we
will be working out architectural issues of the new framework.
We're also looking for web content. If you have tested GlassFish
with an application and you'd like to share your experience please
let us know!
Read our quality mailing list regularly and get involved solving a
problem that piques your interest!
> 7. What are the plans for next version?
Future plans/enhancements will be discussed on the quality mailing
list. If it seems slow this week, many engineers are taking a break
after releasing GlassFish and participating at JavaOne.
> 8. Is is possible for us to some how quantify quality? Do we
> know if quality improved since J2EE 1.4 SDK (8.x code base)?
Here are some metrics that can help quantify GlassFish quality:
- 3,117 new tests were added
- 1,793 bugs fixed
- 25 Apps and Frameworks sanity tested
- 48 builds tested with a passing rate of 98%
Stay tuned for more!
-- Raj
> 9. etc.
>
> --
> Nazrul Islam - (408) 276-6468 - Sun Microsystems, Inc.
>