On 21 Oct 2013, at 18:11, Bill Shannon <bill.shannon_at_oracle.com> wrote:
> arjan tijms wrote on 10/21/13 3:51 PM:
>> Hi,
>>
>>
>> On Mon, Oct 21, 2013 at 8:01 PM, Bill Shannon <bill.shannon_at_oracle.com> wrote:
>> We might consider adding tests if
>> serious portability issues are discovered, but probably not just to make the TCK
>> better.
>>
>> That sounds reasonable and this perfectly answers my question: for really serious issues an update can be considered, and otherwise it's for the next version as per the usual procedure.
>>
>>
>> Note also that expert group members and licensees have access to the TCK before
>> the spec is finalized, explicitly for the purpose of helping us ensure that the
>> TCK is correct and complete. Sadly, that almost never happens.
>>
>> Hmmm, that seems like an area where things could be improved really. Due to the nature of the TCK it's perhaps not entirely trivial to involve the community, but it would be great if something could be done here.
>>
>> As an actual user I may know for instance that certified product Foo AS version 6.3 has a case where something is clearly a spec violation, so I know that case is most likely not covered by a TCK test. If I have submitted a JIRA issue for this, then I would like to be able to ask that Foo AS version 6.3 is used to test the new TCK before the corresponding new spec is finalized. If a new test is indeed added to the TCK then this particular test should thus fail on that product.
>>
>> Would something like this possible? Asking the community to submit known failure cases and then testing that a new version of the TCK indeed fails on those?
> Well, sort of. We definitely encourage reporting of such issues, but to the spec lead and to the vendor with the failing product. Products are required to adhere to the spec even if there's no TCK test for that part of the spec.
>
> Since new substantial tests of this sort would typically only be added to a new version of the TCK corresponding to a new version of the spec, all existing products are going to fail that version of the TCK until they're updated to the new version of the spec.
>
> And through all these discussions about what could happen, you have to realize that our TCK development resources are limited and we have to prioritize which tests to spend time creating so not every test that you can think of will be added to the TCK.
One idea we had for CDI, which we never actually followed through on, was to release some sort of "TCK addendum", which contained tests that it would be good if an implementation passed, as we knew they were gaps in our initial TCK coverage. Of course, this wouldn't be an official thing at all! But we never got around to doing it.
>
>> >The confidentiality restriction you refer to was removed in the latest version of the JCP, and should not be >present in the current TCK licenses. The TCK code is still proprietary, but you can talk about TCK results.
>>
>> That's good to hear really. If I'm not mistaken there also was a restriction that disallowed people with access to the TCK to create their own set of TCK-like tests. Was this indeed a restriction, and if so has this been removed as well perhaps?
> I don't remember if that's still in the TCK license or not; you can read it as well as I can if you really care.
>
> The purpose of this clause was to prevent people developing competitive test suites that could cause confusion over which was the definitive test for spec compliance. Clearly there are things in this area that are reasonable and things that are not and some judgment is called for. It's difficult to encode that judgment in a legal document. Intent often matters more than the technical details.
>
> This is all part of making compatibility binary - either you're compatible or you're not. We don't want products to be 99.9% compatible, and we don't products to claim to be "more compatible" than other products. There's plenty of areas for product differentiation but this is not supposed to be one of them.
>