Martin Matula wrote:
> Hi Kohsuke,
> What do you think are the requirements for an "easy-to-use" test
> harness? IOW, what do you mean by "easy-to-use"? What should the harness
> be able to do?
There are a few workflows that developers follow to interact with test
harness, and "ease-of-use" means making these workflows as easy as possible.
For example, one workflow is to write a test, and this needs to be easy.
Today, if you look at any of JAX-WS tests, there are like at least a
dozen files that need to be filled in, and this is not easy. For
example, the config directory alone contains 6 files in it. By having a
metadata-like description of a test (which should be relatively simple,
as most of the information can be inferred), we should be able to
generate most of those, instead of asking humans to write it.
Another workflow is to run a test, and do so in a variety of
configurations. For example, we often want to run tests by using
multiple transports. So we need to be able to write a test without tying
it to any particular transport (this isn't there today.) It also needs
to be easy for people like Paul to use existing tests with their
transport implementations.
For some kind of transports, servers need to be "deployed", and this
should be also handled inside the harness. For a given transport,
there's often different ways to "deploy" --- for example, for HTTP, you
might want to deploy to an app server running outside the test process
(so that you can test latest Glassfish),or to deploy it inside an
embedded servlet container (so that you can test HTTP transport more
efficiently and in parallel.) Again, it's relatively important for
transport implementations to be able to plug their own "deployment"
mode, so that SOAP/TCP can be tested inside a single VM, for example.
Today's Ant based test harness just doesn't offer this kind of
flexibility. To support multiple web containers, I thought the Cargo
project would be very useful here.
When running a test, it's critically important to be able to run it
inside a debugger, and you need to be able to debug all processes
involved. Not just runtime, but also wsimport, wsgen, etc. Again,
today's Ant script really makes this hard --- you have to manually tweak
the script to do it. This calls for a test mode for running everything
in a single VM. I should be able to just do the following to run a test:
$ java TestRunner <testname> ...
With a lot more options as switches to tweak the behavior. For JAXB test
harness, I developed code to load javac in the VM and so on, which might
be useful here.
Another important part of running tests is to run it from Hudson, so you
need to be able to run multiple tests in one VM (this helps turn-around
time significantly), and also run multiple tests in parallel (so that we
can take advantage of multiple CPUs in the system.) Again, this is not
possible today. I developed parallel-junit for this and used in JAXB.
Furthermore, same tests need to be usable for testing a standalone
JAX-WS, JAX-WS in JWSDP, JAX-WS in Glassfish, etc, so the classpath
configuration needs to be flexible, too. I used Forehead for JAXB test
harness for this (I guess today you should be using Classworlds, probably.)
One more workflow that people do is to watch out for failures (in
particular new failures), and see the failure reports. I'm adding
features related to this in Hudson, but for that to work we need
projects to be able to produce JUnit test reports in an XML format (that
Ant uses.) This allows people to see the newly introduced failure in any
given build, plus its output, etc, etc.
I've done most of these things for JAXB, so I'm reasonably familiar with
how to do these things (hence I offered to do it by myself), but there
are some new aspects of JAX-WS, such as:
- People are building on top of JAX-WS (namely Tango), so
the harness needs to be usable for them, too.
- It's common for those who are building on top of JAX-WS
to hit a problem in the lower stacks. So it's very desirable
for such a team to be able to send a test to another team
for their inspection. If tests are really as easy to run as
"java TestRunner <testname>", then maybe it's OK to ask people
to check out a workspace and do it, but otherwise perhaps we
can come up with a scheme to package up a test and run it
somewhere else.
I've probably missed a few requirements, but this should be a good start.
--
Kohsuke Kawaguchi
Sun Microsystems kohsuke.kawaguchi_at_sun.com