Kohsuke,
First, the test code for amx-impl would depend on amx-impl and amx-api.
I'm happy with any solution that:
- compiles the test code as part of the build for the module (amx-impl)
- maintains the ability to have plain, normal unit tests that run as
today as part of the build
- allows the end-to-end test code to be *not* run (since it won't work
during the build).
- bonus: test code can run as a module, allowing in-process testing
in the DAS
My last suggestion on a module was to have:
amx-api
amx-impl
amx-test <=== a module strictly for testing, would depend on amx-
api, amx-impl, junit
One complication is how module amx-test could ever be started
externally, since nothing could depend on it, hence nothing could
access it.
Lloyd
Lloyd L Chambers wrote:
> Also, debugging failures is more convenient in one JVM than in both
> a client and server JVM...
> Given the optional nature of modules, in this case I'd prefer
> making the AMX tests a module (that would load only when invoked).
Now I'm bit confused because that was my original suggestion, and I
thought you wanted to have test code in src/test/java.
So you changed the mind and now you feel that it should better be its
own module, right?
On Apr 16, 2008, at 12:34 PM, Kohsuke Kawaguchi wrote:
> Lloyd L Chambers wrote:
>> Kohsuke,
>> My main comment is that this disconnects the building of the test
>> code from the building of the source, which will make it easy to
>> forget the test code when developing the main code.
>> So I would really prefer that this test code be in the main source
>> tree (at least), and (preferably) built as part of the build. The
>> test itself would not run as part of the build.
>
I see. OK, yeah, it should be doable. I think the only tricky aspect
is that there are limits in what your test code can depend on, because
the build as a whole cannot have a cyclic dependency.
When you are writing end-to-end test, though, you might want to depend
on some other modules (which in turn depend on AMX) that your main
code doesn't depend on. That won't work.
But otherwise it should be doable.
>
>> On Apr 15, 2008, at 2:55 PM, Kohsuke Kawaguchi wrote:
>>> Lloyd L Chambers wrote:
>>>> Kohsuke,
>>>> Sounds promising, though I don't fully understand the proposal.
>>>> I see 3 levels of testing:
>>>> - true unit tests that can build and run as part of the normal
>>>> build
>>>> - end-to-end tests closely tied to a particular area of the
>>>> server (my AMX tests)
>>>> - full-blown "the world" tests that setup clusters etc.
>>>> There might be crossover between the latter two.
>>>> In the short term my problem is building the AMX tests...I very
>>>> much like the automation provided by the src/tests directory,
>>>> so the question is whether that same ease of use can be applied
>>>> to my end to end tests. This is where my understanding breaks
>>>> down of your response below.
>>>
>>> My proposal is to place this in its own Maven module outside the
>>> main GFv3 build tree, so it won't go into your src/tests directory.
>>>
>>> But with a little bit of Maven mojo development, I think a
>>> similar ease of use can be attained. Something like:
>>>
>>> - you run "mvn test" on your test module to execute it against bits
>>> in your local Maven repository.
>>>
>>> - you run "mvn test -Dhome=path/to/gfv3" to run tests on the
>>> existing
>>> GFv3 installation somewhere on the disk.
>>>
>>> - you run "mvn install" and it pushes the stand-alone runnable
>>> test jar
>>> to the maven repository
>>>
>>> - the stand-alone runnable test jar can be invoked like
>>> "java -jar amx-test.jar path/to/gfv3" to run GFv3.
>>>
>>>
>>>
>>>
>>>> Thanks,
>>>> Lloyd
>>>> On Apr 15, 2008, at 2:12 PM, Kohsuke Kawaguchi wrote:
>>>>>
>>>>> I think tests like that better live in its own directory with
>>>>> some kind of build script --- essentially the same set up as
>>>>> quick look test.
>>>>>
>>>>> Since this is an end-to-end test, I think it's actually more
>>>>> valuable that you can take any v3 bundle built somewhere and
>>>>> run the tests.
>>>>>
>>>>> For example, when we modify HK2, Hudson can build a custom GFv3
>>>>> + this bleedging edge HK2, and then it can run your tests,
>>>>> before the rest of GFv3 even sees this new build of HK2. In
>>>>> this way, by the time HK2 gets integrated, chances of that
>>>>> breaking AMX will be much much less.
>>>>>
>>>>> Also, if the test itself is pushed to the Maven repository,
>>>>> then we might be able to hook up execution of tests in several
>>>>> other places, such as during the normal build of AMX.
>>>>>
>>>>>
>>>>> I think there's a value in having a good harness that runs end-
>>>>> to- end test, building on top of JUnit. Its usefulness goes
>>>>> far beyond just AMX.
>>>>>
>>>>>
>>>>> Lloyd L Chambers wrote:
>>>>>> I am currently trying to figure out how to port V2 AMX tests to
>>>>>> V3.
>>>>>> In V3, how can we support tests other than plain vanilla unit
>>>>>> tests?
>>>>>> - might or might not be true unit tests
>>>>>> - might or might not be junit tests
>>>>>> - might need special setup to execute.
>>>>>> What I would like is formal support for compiling (and
>>>>>> optionally executing) test code that lives other than under
>>>>>> the test/ directory.
>>>>>> More details--
>>>>>> In V3 we have a structure that supports unit tests--source
>>>>>> code goes under src/main/java and test code goes under src/
>>>>>> test/ java. The build compiles both automagically--very nice!
>>>>>> In V2 I developed a fairly large set of tests for the AMX
>>>>>> management API. While the tests are not unit tests in the
>>>>>> usual sense, they junit.framework.TestCase tests, and run
>>>>>> (in V2) using the junit test framework. The tests do *end-
>>>>>> to-end* testing from a client to a running server, and given
>>>>>> the inherent nature of what needs to be tested, this is the
>>>>>> best approach for this particular case; mock objects offer
>>>>>> little value since the goal is to test the end-to-end
>>>>>> behavior, not some isolated piece of code.
>>>>>> So let me call these "End to End Tests" or EETs. These
>>>>>> don't work when placed in the test/ directory, since they
>>>>>> require a test environment with a full running server--
>>>>>> they'll all just fail. So I can't put my test code under
>>>>>> tests/. Annotating them with @Ignore is no good either,
>>>>>> because they *must* be run when the right setup is in place.
>>>>>> Unfortunately, if the test code does not live under test/,
>>>>>> then it has to live elsewhere, along with some kind of
>>>>>> build script. That means making that script rely on build
>>>>>> products of the regular build, eg duplicating and
>>>>>> maintaining knowledge found elsewhere in the build.
>>>>>> ---
>>>>>> Lloyd L Chambers
>>>>>> lloyd.chambers_at_sun.com
>>>>>> Sun Microsystems, Inc
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>>> --
>>>>> Kohsuke Kawaguchi
>>>>> Sun Microsystems kohsuke.kawaguchi_at_sun.com
>>>> ---
>>>> Lloyd L Chambers
>>>> lloyd.chambers_at_sun.com
>>>> Sun Microsystems, Inc
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>>
>>> --
>>> Kohsuke Kawaguchi
>>> Sun Microsystems kohsuke.kawaguchi_at_sun.com
>> ---
>> Lloyd L Chambers
>> lloyd.chambers_at_sun.com
>> Sun Microsystems, Inc
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems kohsuke.kawaguchi_at_sun.com
---
Lloyd L Chambers
lloyd.chambers_at_sun.com
Sun Microsystems, Inc