users@javaee-spec.java.net

[javaee-spec users] Re: How can serious TCK issues be addressed?

From: arjan tijms <arjan.tijms_at_gmail.com>
Date: Wed, 23 Oct 2013 17:04:08 +0200

On Wed, Oct 23, 2013 at 3:31 PM, Edward Burns <edward.burns_at_oracle.com>wrote:

> I'm going to come down against the TCK addendum idea. It dilutes the

"compatibility is a boolean property" value proposition that I think is

foundational to everything we do with JCP.


I agree this is indeed a very important value proposition, which is why I
raised this concern and question in the first place.

In practice however compatibility is not always entirely boolean. JSF is a
good example where the two implementations have a very high degree of
compatibility with the specification, but throw Servlet, CDI and Bean
Validation into the mix and the result is that occasionally over at
OmniFaces we have a hard time to keep our library compatible with all
implementations that are out there. See e.g. a compatibility page like
this: https://code.google.com/p/omnifaces/wiki/KnownIssuesCDI

Now it's a question whether it's always strictly a TCK issue. Oftentimes
it's a combination of implementation bugs and spec holes, but occasionally
it really is a TCK problem. Implementation bugs and spec holes can be
easily addressed and most importantly be validated that they have been
handled (this is particularly easy for the open source products). TCK
issues are a lot harder to validate.



> Instead, I propose something similar to what we do already with JSF.
> There, we have a large body of automated tests that are entirely
> implementation agnostic. These tests go well beyond the TCK, and are
> actively developed with TDD. They are, in all ways but name, a TCK
> addendum.


Right, so I wasn't entirely correct in my earlier statement where I said
that it would be impractical to run Mojarra's tests on Myfaces. Indeed, the
agnostic ones (which are the majority) are in fact implementation agnostic.


I've been suggesting to the MyFaces folks for years that they
> take our automated tests and run them against their impl. I don't know
> if they've tried it, but technically there is nothing stopping them from
> doing so.
>

Maybe a community member should step in and just try to run it on Myfaces.
I've executed the Mojarra test suite a couple of times before and its
indeed a very, very comprehensive suite. I'll make a mental note to try
running it on MyFaces in the near future ;)



> I don't want to formalize the TCK addendum idea. Instead, we should:
>
> 1. Support the concept that implementations should be just as open with
> their test harness as they are with their source code.
>
> 2. Encourage TDD in impls.
>
> 3. Reduce the barriers to entry for contributing new tests to the impl's
> test harness.
>

I really agree with this list, although "just as open with their test
harness as they are with their source code" means for WebLogic, WebSphere
and JEUS among others that they will not be open at all with their tests,
as all of them are closed source.



> This has the added benefit of completely sidestepping any legal concerns
> about the TCK because you are only dealing with whe the impl source code
> license.


This is certainly an option, but then to really mean something shouldn't
someone actually run all implementations against it, e.g. like was done
with the original Acid1 test for browsers, and wouldn't a specific
implementation then effectively provide a competing test suite as Bill
called it? I'm really all for this idea, but just wondering about this
entire competing test suite thing.

Kind regards,
Arjan