Kohsuke,
Sounds promising, though I don't fully understand the proposal.
I see 3 levels of testing:
- true unit tests that can build and run as part of the normal build
- end-to-end tests closely tied to a particular area of the server (my
AMX tests)
- full-blown "the world" tests that setup clusters etc.
There might be crossover between the latter two.
In the short term my problem is building the AMX tests...I very much
like the automation provided by the src/tests directory, so the
question is whether that same ease of use can be applied to my end to
end tests. This is where my understanding breaks down of your
response below.
Thanks,
Lloyd
On Apr 15, 2008, at 2:12 PM, Kohsuke Kawaguchi wrote:
>
> I think tests like that better live in its own directory with some
> kind of build script --- essentially the same set up as quick look
> test.
>
> Since this is an end-to-end test, I think it's actually more
> valuable that you can take any v3 bundle built somewhere and run the
> tests.
>
> For example, when we modify HK2, Hudson can build a custom GFv3 +
> this bleedging edge HK2, and then it can run your tests, before the
> rest of GFv3 even sees this new build of HK2. In this way, by the
> time HK2 gets integrated, chances of that breaking AMX will be much
> much less.
>
> Also, if the test itself is pushed to the Maven repository, then we
> might be able to hook up execution of tests in several other places,
> such as during the normal build of AMX.
>
>
> I think there's a value in having a good harness that runs end-to-
> end test, building on top of JUnit. Its usefulness goes far beyond
> just AMX.
>
>
> Lloyd L Chambers wrote:
>> I am currently trying to figure out how to port V2 AMX tests to V3.
>> In V3, how can we support tests other than plain vanilla unit tests?
>> - might or might not be true unit tests
>> - might or might not be junit tests
>> - might need special setup to execute.
>> What I would like is formal support for compiling (and optionally
>> executing) test code that lives other than under the test/ directory.
>> More details--
>> In V3 we have a structure that supports unit tests--source code
>> goes under src/main/java and test code goes under src/test/java.
>> The build compiles both automagically--very nice!
>> In V2 I developed a fairly large set of tests for the AMX
>> management API. While the tests are not unit tests in the usual
>> sense, they junit.framework.TestCase tests, and run (in V2) using
>> the junit test framework. The tests do *end-to-end* testing from
>> a client to a running server, and given the inherent nature of
>> what needs to be tested, this is the best approach for this
>> particular case; mock objects offer little value since the goal is
>> to test the end-to-end behavior, not some isolated piece of code.
>> So let me call these "End to End Tests" or EETs. These don't
>> work when placed in the test/ directory, since they require a
>> test environment with a full running server--they'll all just
>> fail. So I can't put my test code under tests/. Annotating them
>> with @Ignore is no good either, because they *must* be run when
>> the right setup is in place.
>> Unfortunately, if the test code does not live under test/, then it
>> has to live elsewhere, along with some kind of build script. That
>> means making that script rely on build products of the regular
>> build, eg duplicating and maintaining knowledge found elsewhere in
>> the build.
>> ---
>> Lloyd L Chambers
>> lloyd.chambers_at_sun.com
>> Sun Microsystems, Inc
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems kohsuke.kawaguchi_at_sun.com
---
Lloyd L Chambers
lloyd.chambers_at_sun.com
Sun Microsystems, Inc