Lloyd L Chambers wrote:
> Also, debugging failures is more convenient in one JVM than in both a
> client and server JVM...
>
> Given the optional nature of modules, in this case I'd prefer making
> the AMX tests a module (that would load only when invoked).
Now I'm bit confused because that was my original suggestion, and I
thought you wanted to have test code in src/test/java.
So you changed the mind and now you feel that it should better be its
own module, right?
>
> Lloyd
>
>
> On Apr 15, 2008, at 3:36 PM, Lloyd L Chambers wrote:
>> What if the test code itself were a module? This has some advantages:
>> 1. By being a module and therefore running in-process, the tests
>> could be sped up by a huge amount (they can still exercise the
>> client path appropriately).
>> 2. The tests could be invoked via the GUI or command line at any
>> time, say after making config changes.
>> 3. By being a module, QA or whatever would simply invoke the
>> module; no need for extra setup and build, etc.
>>
>> ...
>>
>> My main comment is that the proposal to separate code disconnects
>> the building of the test code from the building of the source, which
>> will make it easy to forget the test code when developing the main
>> code.
>>
>> So I would really prefer that this test code be in the main source
>> tree (at least), and (preferably) built as part of the build. The
>> test itself would not run as part of the build.
>>
>> Lloyd
>>
>>
>> On Apr 15, 2008, at 2:55 PM, Kohsuke Kawaguchi wrote:
>>> Lloyd L Chambers wrote:
>>>> Kohsuke,
>>>> Sounds promising, though I don't fully understand the proposal.
>>>> I see 3 levels of testing:
>>>> - true unit tests that can build and run as part of the normal build
>>>> - end-to-end tests closely tied to a particular area of the server
>>>> (my AMX tests)
>>>> - full-blown "the world" tests that setup clusters etc.
>>>> There might be crossover between the latter two.
>>>> In the short term my problem is building the AMX tests...I very
>>>> much like the automation provided by the src/tests directory, so
>>>> the question is whether that same ease of use can be applied to
>>>> my end to end tests. This is where my understanding breaks down
>>>> of your response below.
>>>
>>> My proposal is to place this in its own Maven module outside the
>>> main GFv3 build tree, so it won't go into your src/tests directory.
>>>
>>> But with a little bit of Maven mojo development, I think a similar
>>> ease of use can be attained. Something like:
>>>
>>> - you run "mvn test" on your test module to execute it against bits
>>> in your local Maven repository.
>>>
>>> - you run "mvn test -Dhome=path/to/gfv3" to run tests on the existing
>>> GFv3 installation somewhere on the disk.
>>>
>>> - you run "mvn install" and it pushes the stand-alone runnable test
>>> jar
>>> to the maven repository
>>>
>>> - the stand-alone runnable test jar can be invoked like
>>> "java -jar amx-test.jar path/to/gfv3" to run GFv3.
>>>
>>>
>>>
>>>
>>>> Thanks,
>>>> Lloyd
>>>> On Apr 15, 2008, at 2:12 PM, Kohsuke Kawaguchi wrote:
>>>>>
>>>>> I think tests like that better live in its own directory with
>>>>> some kind of build script --- essentially the same set up as
>>>>> quick look test.
>>>>>
>>>>> Since this is an end-to-end test, I think it's actually more
>>>>> valuable that you can take any v3 bundle built somewhere and run
>>>>> the tests.
>>>>>
>>>>> For example, when we modify HK2, Hudson can build a custom GFv3
>>>>> + this bleedging edge HK2, and then it can run your tests,
>>>>> before the rest of GFv3 even sees this new build of HK2. In this
>>>>> way, by the time HK2 gets integrated, chances of that breaking
>>>>> AMX will be much much less.
>>>>>
>>>>> Also, if the test itself is pushed to the Maven repository, then
>>>>> we might be able to hook up execution of tests in several other
>>>>> places, such as during the normal build of AMX.
>>>>>
>>>>>
>>>>> I think there's a value in having a good harness that runs end-
>>>>> to- end test, building on top of JUnit. Its usefulness goes far
>>>>> beyond just AMX.
>>>>>
>>>>>
>>>>> Lloyd L Chambers wrote:
>>>>>> I am currently trying to figure out how to port V2 AMX tests to
>>>>>> V3.
>>>>>> In V3, how can we support tests other than plain vanilla unit
>>>>>> tests?
>>>>>> - might or might not be true unit tests
>>>>>> - might or might not be junit tests
>>>>>> - might need special setup to execute.
>>>>>> What I would like is formal support for compiling (and
>>>>>> optionally executing) test code that lives other than under
>>>>>> the test/ directory.
>>>>>> More details--
>>>>>> In V3 we have a structure that supports unit tests--source code
>>>>>> goes under src/main/java and test code goes under src/test/
>>>>>> java. The build compiles both automagically--very nice!
>>>>>> In V2 I developed a fairly large set of tests for the AMX
>>>>>> management API. While the tests are not unit tests in the
>>>>>> usual sense, they junit.framework.TestCase tests, and run (in
>>>>>> V2) using the junit test framework. The tests do *end-to-end*
>>>>>> testing from a client to a running server, and given the
>>>>>> inherent nature of what needs to be tested, this is the best
>>>>>> approach for this particular case; mock objects offer little
>>>>>> value since the goal is to test the end-to-end behavior, not
>>>>>> some isolated piece of code.
>>>>>> So let me call these "End to End Tests" or EETs. These don't
>>>>>> work when placed in the test/ directory, since they require a
>>>>>> test environment with a full running server--they'll all just
>>>>>> fail. So I can't put my test code under tests/. Annotating
>>>>>> them with @Ignore is no good either, because they *must* be
>>>>>> run when the right setup is in place.
>>>>>> Unfortunately, if the test code does not live under test/, then
>>>>>> it has to live elsewhere, along with some kind of build
>>>>>> script. That means making that script rely on build products
>>>>>> of the regular build, eg duplicating and maintaining knowledge
>>>>>> found elsewhere in the build.
>>>>>> ---
>>>>>> Lloyd L Chambers
>>>>>> lloyd.chambers_at_sun.com
>>>>>> Sun Microsystems, Inc
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>>
>>>>>
>>>>> --
>>>>> Kohsuke Kawaguchi
>>>>> Sun Microsystems kohsuke.kawaguchi_at_sun.com
>>>> ---
>>>> Lloyd L Chambers
>>>> lloyd.chambers_at_sun.com
>>>> Sun Microsystems, Inc
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>
>>>
>>> --
>>> Kohsuke Kawaguchi
>>> Sun Microsystems kohsuke.kawaguchi_at_sun.com
>>
>> ---
>> Lloyd L Chambers
>> lloyd.chambers_at_sun.com
>> Sun Microsystems, Inc
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>
> ---
> Lloyd L Chambers
> lloyd.chambers_at_sun.com
> Sun Microsystems, Inc
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
>
--
Kohsuke Kawaguchi
Sun Microsystems kohsuke.kawaguchi_at_sun.com