TCK Coverage Document <was> Re: JSR311: Preparing for 1.0 FCS

From: Paul Sandoz <Paul.Sandoz_at_Sun.COM>
Date: Fri, 29 Aug 2008 23:11:57 +0200


Please see below for the current TCK coverage document. RI results
will follow early next week.


TCK Coverage Document for JSR-311 JAX-RS: JavaŠ API for RESTful Web

TCK Components:
- User's Guide
- Compatibility Rules
- Configuration Instructions
- Test Suite
- API Tests
- Signature Tests
- End-to-End Tests
- Framework Code
- JavaTest TM Harness

Terminology of Metrics
- Assertion: A specific statement of functionality or behavior
derived from a specification.
   A testable assertion is one that can be validated in an
implementation by testing.
- Test: A binary application (or script) comprised of one or more
Test Cases.
- Test Case: A single set of test inputs, execution conditions, and
expected results
   developed to verify an implementation's conformance with a
specific assertion.
- Specification Assertion Coverage: Ratio of all assertions tested by
at least one test
   case to the total number of testable assertions defined by the
- API Coverage: Ratio of methods directly exercised by test cases to
the total number
   of methods defined by the specification.

- Total testable assertions derived from JAX-RS Specification and

         96 specification assertions
      + 296 API assertions
        392 testable assertions

- Total testable assertions tested

         29 specification assertions tested
      + 154 API assertions tested
        183 testable assertions tested

   - 30% of testable specification assertions tested
   - 52% of testable API assertions tested
   - 47% of all (specification and API) assertions tested

- Assertions were identified through the use of CTS Tools and hand

- Number of tests in JAX-RS TCK: 200 This total may change slightly
before the TCK ships.

- API Signature Coverage: 100% for all defined public and protected
members and validated
   by the signature tests included with the JAX-RS TCK.

Quality Assurance
- JAX-RS TCK was run using representative configurations of the
Reference Implementation on
   the following platforms, either Java 2 SE 5.0 or Java 2 SE 6.0 used:
    * Solaris 10 sparc
    * Red Hat (RHEL) 4.0
    * Windows XP Professional
    * Mac OS 10.5
- Code quality was demonstrated through the use of code reviews and
- User's Guide was constructed from the standard TCK User's Guide
- Documentation instructions were verified and tested

Justification of Adequacy
The JAX-RS TCK version 1 provides a comprehensive set of tests to ensure
all implementations of the JAX-RS specifications are compatible. As
with all
TCKs it is impossible to provide tests for 100% assertion coverage.
Note, while
the TCK may not have tests for all assertions in the JAX-RS
specification, all
implementations must be compatible with the specification.

Untested assertions are due to the effects of late specification
changes, and the
availability and cost of test development resources.

On Aug 27, 2008, at 10:38 PM, Marc Hadley wrote:

> I just created a new 0.11 snapshot:
> Spec:
> Javadoc:
> SVN tag:
> jsr311-api-0.11/
> The bits are also available in a Maven 2 repo at:
> This is our candidate 1.0 spec and API and we're now making the
> final few tweaks to bring the RI and TCK into line. Here's the plan
> from now:
> - End of this week or start of next (9/2 latest), send the TCK
> coverage report and RI results to this list
> - End of next week or start of following (9/8 latest), submit the
> spec, api, RI and TCK to the JCP to start the EC final approval
> ballot. The ballot is open for two weeks.
> - Week of 9/22 (if we pass the ballot), submit the final materials
> to the JCP for publication
> - Week of 10/6 final spec published.
> We'd hoped to be able to provide access to the TCK before we got to
> this stage but apparently we are the first JSR to request this and
> the legal gears grind too slowly to make it possible. EC members
> will have access to the TCK once the final approval ballot starts.
> Java EE licensees will have access to the TCK once the spec goes
> final and we're working on an eval license for non-EE licensees.
> Marc.
> ---
> Marc Hadley <marc.hadley at>
> CTO Office, Sun Microsystems.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail: