Also, debugging failures is more convenient in one JVM than in both a
client and server JVM...
Given the optional nature of modules, in this case I'd prefer making
the AMX tests a module (that would load only when invoked).
Lloyd
On Apr 15, 2008, at 3:36 PM, Lloyd L Chambers wrote:
> What if the test code itself were a module? This has some advantages:
> 1. By being a module and therefore running in-process, the tests
> could be sped up by a huge amount (they can still exercise the
> client path appropriately).
> 2. The tests could be invoked via the GUI or command line at any
> time, say after making config changes.
> 3. By being a module, QA or whatever would simply invoke the
> module; no need for extra setup and build, etc.
>
> ...
>
> My main comment is that the proposal to separate code disconnects
> the building of the test code from the building of the source, which
> will make it easy to forget the test code when developing the main
> code.
>
> So I would really prefer that this test code be in the main source
> tree (at least), and (preferably) built as part of the build. The
> test itself would not run as part of the build.
>
> Lloyd
>
>
> On Apr 15, 2008, at 2:55 PM, Kohsuke Kawaguchi wrote:
>> Lloyd L Chambers wrote:
>>> Kohsuke,
>>> Sounds promising, though I don't fully understand the proposal.
>>> I see 3 levels of testing:
>>> - true unit tests that can build and run as part of the normal build
>>> - end-to-end tests closely tied to a particular area of the server
>>> (my AMX tests)
>>> - full-blown "the world" tests that setup clusters etc.
>>> There might be crossover between the latter two.
>>> In the short term my problem is building the AMX tests...I very
>>> much like the automation provided by the src/tests directory, so
>>> the question is whether that same ease of use can be applied to
>>> my end to end tests. This is where my understanding breaks down
>>> of your response below.
>>
>> My proposal is to place this in its own Maven module outside the
>> main GFv3 build tree, so it won't go into your src/tests directory.
>>
>> But with a little bit of Maven mojo development, I think a similar
>> ease of use can be attained. Something like:
>>
>> - you run "mvn test" on your test module to execute it against bits
>> in your local Maven repository.
>>
>> - you run "mvn test -Dhome=path/to/gfv3" to run tests on the existing
>> GFv3 installation somewhere on the disk.
>>
>> - you run "mvn install" and it pushes the stand-alone runnable test
>> jar
>> to the maven repository
>>
>> - the stand-alone runnable test jar can be invoked like
>> "java -jar amx-test.jar path/to/gfv3" to run GFv3.
>>
>>
>>
>>
>>> Thanks,
>>> Lloyd
>>> On Apr 15, 2008, at 2:12 PM, Kohsuke Kawaguchi wrote:
>>>>
>>>> I think tests like that better live in its own directory with
>>>> some kind of build script --- essentially the same set up as
>>>> quick look test.
>>>>
>>>> Since this is an end-to-end test, I think it's actually more
>>>> valuable that you can take any v3 bundle built somewhere and run
>>>> the tests.
>>>>
>>>> For example, when we modify HK2, Hudson can build a custom GFv3
>>>> + this bleedging edge HK2, and then it can run your tests,
>>>> before the rest of GFv3 even sees this new build of HK2. In this
>>>> way, by the time HK2 gets integrated, chances of that breaking
>>>> AMX will be much much less.
>>>>
>>>> Also, if the test itself is pushed to the Maven repository, then
>>>> we might be able to hook up execution of tests in several other
>>>> places, such as during the normal build of AMX.
>>>>
>>>>
>>>> I think there's a value in having a good harness that runs end-
>>>> to- end test, building on top of JUnit. Its usefulness goes far
>>>> beyond just AMX.
>>>>
>>>>
>>>> Lloyd L Chambers wrote:
>>>>> I am currently trying to figure out how to port V2 AMX tests to
>>>>> V3.
>>>>> In V3, how can we support tests other than plain vanilla unit
>>>>> tests?
>>>>> - might or might not be true unit tests
>>>>> - might or might not be junit tests
>>>>> - might need special setup to execute.
>>>>> What I would like is formal support for compiling (and
>>>>> optionally executing) test code that lives other than under
>>>>> the test/ directory.
>>>>> More details--
>>>>> In V3 we have a structure that supports unit tests--source code
>>>>> goes under src/main/java and test code goes under src/test/
>>>>> java. The build compiles both automagically--very nice!
>>>>> In V2 I developed a fairly large set of tests for the AMX
>>>>> management API. While the tests are not unit tests in the
>>>>> usual sense, they junit.framework.TestCase tests, and run (in
>>>>> V2) using the junit test framework. The tests do *end-to-end*
>>>>> testing from a client to a running server, and given the
>>>>> inherent nature of what needs to be tested, this is the best
>>>>> approach for this particular case; mock objects offer little
>>>>> value since the goal is to test the end-to-end behavior, not
>>>>> some isolated piece of code.
>>>>> So let me call these "End to End Tests" or EETs. These don't
>>>>> work when placed in the test/ directory, since they require a
>>>>> test environment with a full running server--they'll all just
>>>>> fail. So I can't put my test code under tests/. Annotating
>>>>> them with @Ignore is no good either, because they *must* be
>>>>> run when the right setup is in place.
>>>>> Unfortunately, if the test code does not live under test/, then
>>>>> it has to live elsewhere, along with some kind of build
>>>>> script. That means making that script rely on build products
>>>>> of the regular build, eg duplicating and maintaining knowledge
>>>>> found elsewhere in the build.
>>>>> ---
>>>>> Lloyd L Chambers
>>>>> lloyd.chambers_at_sun.com
>>>>> Sun Microsystems, Inc
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>>
>>>> --
>>>> Kohsuke Kawaguchi
>>>> Sun Microsystems kohsuke.kawaguchi_at_sun.com
>>> ---
>>> Lloyd L Chambers
>>> lloyd.chambers_at_sun.com
>>> Sun Microsystems, Inc
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems kohsuke.kawaguchi_at_sun.com
>
> ---
> Lloyd L Chambers
> lloyd.chambers_at_sun.com
> Sun Microsystems, Inc
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
---
Lloyd L Chambers
lloyd.chambers_at_sun.com
Sun Microsystems, Inc