arjan tijms wrote on 10/21/13 3:51 PM:
> Hi,
>
>
> On Mon, Oct 21, 2013 at 8:01 PM, Bill Shannon <bill.shannon_at_oracle.com
> <mailto:bill.shannon_at_oracle.com>> wrote:
>
> We might consider adding tests if
> serious portability issues are discovered, but probably not just to make
> the TCK
> better.
>
>
> That sounds reasonable and this perfectly answers my question: for really
> serious issues an update can be considered, and otherwise it's for the next
> version as per the usual procedure.
>
>
> Note also that expert group members and licensees have access to the TCK
> before
> the spec is finalized, explicitly for the purpose of helping us ensure
> that the
> TCK is correct and complete. Sadly, that almost never happens.
>
>
> Hmmm, that seems like an area where things could be improved really. Due to
> the nature of the TCK it's perhaps not entirely trivial to involve the
> community, but it would be great if something could be done here.
>
> As an actual user I may know for instance that certified product Foo AS
> version 6.3 has a case where something is clearly a spec violation, so I know
> that case is most likely not covered by a TCK test. If I have submitted a JIRA
> issue for this, then I would like to be able to ask that Foo AS version 6.3 is
> used to test the new TCK before the corresponding new spec is finalized. If a
> new test is indeed added to the TCK then this particular test should thus fail
> on that product.
>
> Would something like this possible? Asking the community to submit known
> failure cases and then testing that a new version of the TCK indeed fails on
> those?
Well, sort of. We definitely encourage reporting of such issues, but to the
spec lead and to the vendor with the failing product. Products are required to
adhere to the spec *even if there's no TCK test* for that part of the spec.
Since new substantial tests of this sort would typically only be added to a new
version of the TCK corresponding to a new version of the spec, all existing
products are going to fail that version of the TCK until they're updated to the
new version of the spec.
And through all these discussions about what *could* happen, you have to realize
that our TCK development resources are limited and we have to prioritize which
tests to spend time creating so not every test that you can think of will be
added to the TCK.
> >The confidentiality restriction you refer to was removed in the latest
> version of the JCP, and should not be >present in the current TCK licenses.
> The TCK code is still proprietary, but you can talk about TCK results.
>
> That's good to hear really. If I'm not mistaken there also was a restriction
> that disallowed people with access to the TCK to create their own set of
> TCK-like tests. Was this indeed a restriction, and if so has this been removed
> as well perhaps?
I don't remember if that's still in the TCK license or not; you can read it as
well as I can if you really care.
The purpose of this clause was to prevent people developing competitive test
suites that could cause confusion over which was the definitive test for spec
compliance. Clearly there are things in this area that are reasonable and
things that are not and some judgment is called for. It's difficult to encode
that judgment in a legal document. Intent often matters more than the technical
details.
This is all part of making compatibility binary - either you're compatible or
you're not. We don't want products to be 99.9% compatible, and we don't
products to claim to be "more compatible" than other products. There's plenty
of areas for product differentiation but this is not supposed to be one of them.