Raj,
Can we please clarify this? I was told before that the number of tests run
on a nightly and a promoted build are the same, and the nightly can fail
the tests and the promoted must pass them all. Your numbers are different.
thanks,
-marina
Raj Premkumar wrote:
> Apologies for the delay getting this information out, but I wanted to
> make sure I had as much information as possible. Please see my response
> inline.
>
> On May 23, 2006, at 12:51 AM, Nazrul Islam wrote:
>
>> During the GlassFish BOF 0251
>> <https://www28.cplan.com/javaone06_cv_124_1/sessions_catalog.jsp?ilc=124-1&ilg=english&isort=1&is=%3CISEARCH%3E&isession_id=BOF-0251>,
>> few developers asked about the quality of the Application Server. From
>> the GlassFish Quality Portal
>> <http://wiki.java.net/bin/view/Projects/GlassFishQuality>, it is not
>> clear how much testing is being done. Could we share the "quality
>> matrix" with the community?
>
>
> Here's some relevant information. We'll post this to the Quality page
> shortly.
>
> Platforms
> Solaris Sparc 9/10, Sol X86 9/10
> Solaris Sparc/X86 64 bit
> Win XP, Win 2003, Win 2K
> Linux 3.0, 4.0
> Mac OS (Tiger)
>
> Databases and Drivers
> Oracle, Derby, MS SQL, MySQL, Sybase, DB2, PostGres
> Postgres Driver, Datadirect Driver 3.5.26 (Type 4), Oracle Thin Driver
> (otn.oracle.com) ( Type 2 ),IBM Driver Type 2,Connector/J Driver
> 3.1,Native Derby driver
>
> JDK Versions
> JDK 1.5_06, JDK 1.5_04, JDK 1.6 beta, JDK 1.6 beta2
>
>
>>
>> The following questions may be good to answer...
>> 1. How may test cases were executed and passed for the recent FCS
>> release?
>
>
> For the last FCS release, there were 33,400 compatibility tests and
> 17,547 functional tests executed.
>
>> 2. How many bugs were fixed for the recent FCS release? Latest bug
>> trend.
>
>
> 1,793 bugs were fixed for this release! We're working hard on
> significantly expanding the bug information available. Expect to see
> charts, a unified bug list and other dynamic information soon.
>
>> 3. What is the current code coverage?
>
>
> Code coverage is currently in the planning stages. We'll publish a
> schedule as soon as the plan is finalized.
>
>> 4. How many test cases passed on a nightly build?
>
>
> Quicklook tests = 62
> Functional Smoke Tests = 37
> Compatibility Smoke Tests = 347
>
> The latest promoted and nightly results are now available on the quality
> page.
>
>
>> 5. How many test cases passed a promoted build?
>
>
> I'm still in the process of fully tracking this down; I can tell you
> that 6476 tests are run for the core functionality alone per promoted build.
>
>> 6. How can people help?
>
>
> We're looking for contributors to our unified test framework where we
> will be working out architectural issues of the new framework.
>
> We're also looking for web content. If you have tested GlassFish with
> an application and you'd like to share your experience please let us know!
>
> Read our quality mailing list regularly and get involved solving a
> problem that piques your interest!
>
>
>> 7. What are the plans for next version?
>
>
> Future plans/enhancements will be discussed on the quality mailing list.
> If it seems slow this week, many engineers are taking a break
> after releasing GlassFish and participating at JavaOne.
>
>> 8. Is is possible for us to some how quantify quality? Do we know
>> if quality improved since J2EE 1.4 SDK (8.x code base)?
>
>
> Here are some metrics that can help quantify GlassFish quality:
>
> - 3,117 new tests were added
> - 1,793 bugs fixed
> - 25 Apps and Frameworks sanity tested
> - 48 builds tested with a passing rate of 98%
>
> Stay tuned for more!
>
> -- Raj
>
>> 9. etc.
>>
>>--
>>Nazrul Islam - (408) 276-6468 - Sun Microsystems, Inc.
>>
>>
>