Note that randomized order is what you get with Java SE 1.7.0_02 and later. This is why we are seeing some seemingly random unit test failures when we build with Java SE 7.
Tom
--
Sent from my Kindle Fire with K-9 Mail. Please excuse my brevity.
Andreas Loew <Andreas.Loew_at_oracle.com> wrote:
Sorry,
Am 26.02.2012 00:34, schrieb Bobby Bissett:
> Frankly, if junit tests are really supposed to be order-independent,
> junit ought to randomize them by default so people can't easily write
> buggy tests.
but I would still tend to argue against this approach:
If only two of e.g. twenty test methods in one JUnit test class have an
order dependency (e.g. testAAA() must run before testBBB() for both to
succeed), then what is the likelihood for a random execution mode to
uncover this? If I am not mistaken, only 50% of all runs would fail...
If you are the author of both the code and the test, that should
probably work out fine and should tell you that you have indeed
overlooked something regarding test state independence, but if you
suppose that the original author has already checked in both the code
and the test - because the test was already existing and he was only
doing a slight change, and during the two test runs he did, everything
seemed to work fine for him (let's say he was "lucky" despite random
test case execution...), some other person looking at this might then be
in real trouble, as you have no means in general to know what exactly
might be going on in any more complex scenario: What is to blame for
this intermittent failure? The code to be tested, the test case as such
or things like illegal state sharing between JUnit test methods?
In my opinion, still, non-deterministic code/faults (even when caused by
random execution of test cases) are a nightmare that should be avoided...
Using simple math, you can calculate how many subsequent test runs you
need in order see the failure with at least 90%, 99% or even 99,999%
likelihood, but in theory, you could still miss a pathological case:
Would you indeed want to execute all JUnit tests e.g. five times on
every build, each time calculating a new random order, just to be sure
that you did not overlook an order dependency fault with a likelihood of
96,875%?
I'd rather prefer to try and address such things beforehand: Those
people writing buggy test methods that depend on state created by other
test methods generally are the same people who tend to write buggy
software, because they are lacking insight.
If somebody in your team is indeed lacking this very basic insight in
how JUnit test cases are supposed to work, namely that each test should
run strictly independent and must not share any state with other test
methods, then this person is also more or less likely to be lacking
other insights on how to produce proper Java code, too... :-(
So I think educating developers about how to write proper code in
general - including proper JUnit tests - is the important thing here -
and if in doubt, executing peer code reviews and/or code reviews by the
team leads/software architects before any larger portions of code of
unknown or questionable quality are to be checked in might be more
helpful than executing unit tests in random order...
Just my 2 cents on this...
Best regards,
Andreas
--
Andreas Loew | Senior Java Architect
ACS Principal Service Delivery Engineer
ORACLE Deutschland B.V. & Co. KG