Hi Marina,
Yes,the number of test run on a nightly and promoted build are the same,
and all the tests must pass before the build is released.
Once the build is released, the quality teams pick up the build and run
their weekly test cycle (which is the number Raj is quoting below).
Thanks,
Geoff
Marina Vatkina wrote On 06/08/06 15:45,:
> Raj,
>
> Can we please clarify this? I was told before that the number of tests run
> on a nightly and a promoted build are the same, and the nightly can fail
> the tests and the promoted must pass them all. Your numbers are different.
>
> thanks,
> -marina
>
> Raj Premkumar wrote:
>
>>Apologies for the delay getting this information out, but I wanted to
>>make sure I had as much information as possible. Please see my response
>>inline.
>>
>>On May 23, 2006, at 12:51 AM, Nazrul Islam wrote:
>>
>>
>>>During the GlassFish BOF 0251
>>><https://www28.cplan.com/javaone06_cv_124_1/sessions_catalog.jsp?ilc=124-1&ilg=english&isort=1&is=%3CISEARCH%3E&isession_id=BOF-0251>,
>>>few developers asked about the quality of the Application Server. From
>>>the GlassFish Quality Portal
>>><http://wiki.java.net/bin/view/Projects/GlassFishQuality>, it is not
>>>clear how much testing is being done. Could we share the "quality
>>>matrix" with the community?
>>
>>
>>Here's some relevant information. We'll post this to the Quality page
>>shortly.
>>
>>Platforms
>>Solaris Sparc 9/10, Sol X86 9/10
>>Solaris Sparc/X86 64 bit
>>Win XP, Win 2003, Win 2K
>>Linux 3.0, 4.0
>>Mac OS (Tiger)
>>
>>Databases and Drivers
>>Oracle, Derby, MS SQL, MySQL, Sybase, DB2, PostGres
>>Postgres Driver, Datadirect Driver 3.5.26 (Type 4), Oracle Thin Driver
>>(otn.oracle.com) ( Type 2 ),IBM Driver Type 2,Connector/J Driver
>>3.1,Native Derby driver
>>
>>JDK Versions
>>JDK 1.5_06, JDK 1.5_04, JDK 1.6 beta, JDK 1.6 beta2
>>
>>
>>
>>>The following questions may be good to answer...
>>> 1. How may test cases were executed and passed for the recent FCS
>>>release?
>>
>>
>>For the last FCS release, there were 33,400 compatibility tests and
>>17,547 functional tests executed.
>>
>>
>>> 2. How many bugs were fixed for the recent FCS release? Latest bug
>>>trend.
>>
>>
>>1,793 bugs were fixed for this release! We're working hard on
>>significantly expanding the bug information available. Expect to see
>>charts, a unified bug list and other dynamic information soon.
>>
>>
>>> 3. What is the current code coverage?
>>
>>
>>Code coverage is currently in the planning stages. We'll publish a
>>schedule as soon as the plan is finalized.
>>
>>
>>> 4. How many test cases passed on a nightly build?
>>
>>
>>Quicklook tests = 62
>>Functional Smoke Tests = 37
>>Compatibility Smoke Tests = 347
>>
>>The latest promoted and nightly results are now available on the quality
>>page.
>>
>>
>>
>>> 5. How many test cases passed a promoted build?
>>
>>
>>I'm still in the process of fully tracking this down; I can tell you
>>that 6476 tests are run for the core functionality alone per promoted build.
>>
>>
>>> 6. How can people help?
>>
>>
>>We're looking for contributors to our unified test framework where we
>>will be working out architectural issues of the new framework.
>>
>>We're also looking for web content. If you have tested GlassFish with
>>an application and you'd like to share your experience please let us know!
>>
>>Read our quality mailing list regularly and get involved solving a
>>problem that piques your interest!
>>
>>
>>
>>> 7. What are the plans for next version?
>>
>>
>>Future plans/enhancements will be discussed on the quality mailing list.
>>If it seems slow this week, many engineers are taking a break
>>after releasing GlassFish and participating at JavaOne.
>>
>>
>>> 8. Is is possible for us to some how quantify quality? Do we know
>>>if quality improved since J2EE 1.4 SDK (8.x code base)?
>>
>>
>>Here are some metrics that can help quantify GlassFish quality:
>>
>>- 3,117 new tests were added
>>- 1,793 bugs fixed
>>- 25 Apps and Frameworks sanity tested
>>- 48 builds tested with a passing rate of 98%
>>
>>Stay tuned for more!
>>
>>-- Raj
>>
>>
>>> 9. etc.
>>>
>>>--
>>>Nazrul Islam - (408) 276-6468 - Sun Microsystems, Inc.
>>>
>>>
>>