arjan tijms wrote on 10/21/13 5:09 PM:
> Hi,
>
>
> On Tue, Oct 22, 2013 at 1:11 AM, Bill Shannon <bill.shannon_at_oracle.com
> <mailto:bill.shannon_at_oracle.com>> wrote:
>
>>
>> Would something like this possible? Asking the community to submit known
>> failure cases and then testing that a new version of the TCK indeed fails
>> on those?
> Well, sort of. [...]
>
> Since new substantial tests of this sort would typically only be added to
> a new version of the TCK corresponding to a new version of the spec, all
> existing products are going to fail that version of the TCK until they're
> updated to the new version of the spec.
>
>
> Indeed, so that's maybe a bit of the issue here.
>
> A TCK is made final before or simultaneously with the new version of the spec,
> isn't it? But there aren't any real products other than the RI then to test it
> against.
Simultaneously with.
> So while most likely every existing product implementing Java EE x is going to
> fail the full TCK for x + 1, what I'd hoped is that it would be possible to
> just run a single test in isolation and test that this test indeed fails on
> the existing product for Java EE x.
You can certainly look at the pass/fail details for individual tests.
> For instance, in Java EE 6 not all implementations call a JASPIC auth module
> for a non-protected resource, while the spec says they should. For Java EE 7 I
> would have liked to be able to ask that the implementation where I know this
> doesn't happen is tested for this particular test. I know that the existing
> implementation could theoretically never pass the whole test suite since some
> requirements have been added for Java EE 7, but I would be interested only in
> an "auth module called for public page" test, and the assurance that it indeed
> failed on the specific implementation that I mentioned.
>
> That isolated test would not be Java EE 7/JASPIC 1.1 specific, since nothing
> specific changed in that area between versions. A test that just requests a
> non-protected resource, then checks if an auth module has indeed been invoked
> makes just as much sense on a Java EE 6 implementation as it does on a Java EE
> 7 one. The only difference would be that the test just wasn't in the Java EE 6
> TCK.
>
> I guess the TCK currently only tests if new products comply? If indeed so, I
> think it might not be a bad idea to also test known (old) failure cases to see
> if the TCK indeed fails correctly when it needs to fail.
What you're asking would require a lot more coordination than we currently do.
Remember that we don't test all the products; each vendor tests their own product.
>
>
> And through all these discussions about what *could* happen, you have to
> realize that our TCK development resources are limited and we have to
> prioritize which tests to spend time creating so not every test that you
> can think of will be added to the TCK.
>
>
> Here it would be beneficial if people could contribute tests I guess, but I
> understand this is not as relatively trivial as say contributing a patch to
> Mojarra or so.
Licensees have always been able to contribute tests. It essentially never happens.
>
>
> I don't remember if that's still in the TCK license or not; you can read
> it as well as I can if you really care.
>
>
> Okay, I'll try to find the license then and see if it's still in there.
The licenses are linked from the JSR pages on jcp.org.