"the behavior changes subtly depending on exactly what modules you
have in the system"
Absolutely. But that's not all--
We will (soon) have not just AMXConfig MBeans, but also a variety of
runtime and other monitoring MBeans and possibly support for arbitrary
module-supplied MBeans as well. They also have inter-relationships in
some cases (eg traversing from a config MBean to a monitoring or JSR
77 one and vice versa).
AMX has many unit tests, but one (exhaustive) test ("GenericTest")
verifies the MBeanInfo of every AMX MBean, the ability to get every
Attribute, the ability to identity-set each mutable attribute, etc. A
variety of other validity checks are made as well such as singly-vs-
multiply-fetched access, uniqueness of names, case-sensitivity issues,
etc.
To maximize the value of such generic tests, all possible modules
should be loaded and running, otherwise the associated MBeans won't be
tested. This is why when the AMX tests start, monitoring is set to
HIGH for everything--all possibly MBeans should be present.
Lloyd
On Apr 15, 2008, at 2:50 PM, Kohsuke Kawaguchi wrote:
> Keith Babo wrote:
>> Is there a reason why you couldn't just bootstrap a nucleus
>> distribution of v3 in each unit test? If we added some basic hooks
>> for the test to declare which additional modules are required, you
>> could configure a runtime specifically for the test. If the test
>> itself is packaged inside a bundle, you can drive some very
>> sophisticated end-to-end testing.
>> The above is pretty much how Spring DM integration tests work today
>> and it's quite nice.
>
> I think there are several issues.
>
> The first is that this AMX end-to-end test depends on a whole bunch
> of other modules, so it has to bring up the considerable portion of
> GFv3. So there's a chicken-and-egg problem if you try to do that
> during a AMX build.
>
> The second is that we often want to run tests not just during the
> build, but also when various GFv3 distributions are assembled, as
> the behavior changes subtly depending on exactly what modules you
> have in the system.
>
>
> I think what you are describing is also very useful for some other
> kinds of module-level functionality tests, and that is something
> we'd also like to be able to do, but I think the end-to-end test
> needs a different kind of set up.
>
>
>
>> regards,
>> keith
>> Kohsuke Kawaguchi wrote:
>>> I think tests like that better live in its own directory with some
>>> kind of build script --- essentially the same set up as quick look
>>> test.
>>> Since this is an end-to-end test, I think it's actually more
>>> valuable that you can take any v3 bundle built somewhere and run
>>> the tests.
>>> For example, when we modify HK2, Hudson can build a custom GFv3 +
>>> this bleedging edge HK2, and then it can run your tests, before
>>> the rest of GFv3 even sees this new build of HK2. In this way, by
>>> the time HK2 gets integrated, chances of that breaking AMX will be
>>> much much less.
>>> Also, if the test itself is pushed to the Maven repository, then
>>> we might be able to hook up execution of tests in several other
>>> places, such as during the normal build of AMX.
>>> I think there's a value in having a good harness that runs end-to-
>>> end test, building on top of JUnit. Its usefulness goes far beyond
>>> just AMX.
>>> Lloyd L Chambers wrote:
>>>> I am currently trying to figure out how to port V2 AMX tests to V3.
>>>>
>>>> In V3, how can we support tests other than plain vanilla unit
>>>> tests?
>>>> - might or might not be true unit tests
>>>> - might or might not be junit tests
>>>> - might need special setup to execute.
>>>>
>>>> What I would like is formal support for compiling (and
>>>> optionally executing) test code that lives other than under the
>>>> test/ directory.
>>>>
>>>> More details--
>>>>
>>>> In V3 we have a structure that supports unit tests--source code
>>>> goes under src/main/java and test code goes under src/test/
>>>> java. The build compiles both automagically--very nice!
>>>>
>>>> In V2 I developed a fairly large set of tests for the AMX
>>>> management API. While the tests are not unit tests in the usual
>>>> sense, they junit.framework.TestCase tests, and run (in V2)
>>>> using the junit test framework. The tests do *end-to-end*
>>>> testing from a client to a running server, and given the
>>>> inherent nature of what needs to be tested, this is the best
>>>> approach for this particular case; mock objects offer little
>>>> value since the goal is to test the end-to-end behavior, not
>>>> some isolated piece of code.
>>>>
>>>> So let me call these "End to End Tests" or EETs. These don't
>>>> work when placed in the test/ directory, since they require a
>>>> test environment with a full running server--they'll all just
>>>> fail. So I can't put my test code under tests/. Annotating
>>>> them with @Ignore is no good either, because they *must* be run
>>>> when the right setup is in place.
>>>>
>>>> Unfortunately, if the test code does not live under test/, then
>>>> it has to live elsewhere, along with some kind of build script.
>>>> That means making that script rely on build products of the
>>>> regular build, eg duplicating and maintaining knowledge found
>>>> elsewhere in the build.
>>>>
>>>> ---
>>>> Lloyd L Chambers
>>>> lloyd.chambers_at_sun.com
>>>> Sun Microsystems, Inc
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>>>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
>> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems kohsuke.kawaguchi_at_sun.com
---
Lloyd L Chambers
lloyd.chambers_at_sun.com
Sun Microsystems, Inc