dev@ws-test-harness.java.net

Re: e2e test harness -- now with better error output!

From: Fabian Ritzmann <Fabian.Ritzmann_at_Sun.COM>
Date: Fri, 20 Oct 2006 04:19:25 +0300

Kohsuke Kawaguchi wrote:
> Fabian Ritzmann wrote:
>> Isn't that the perfect excuse to get yourself a Mac. :-)
>
> Tell that to Vella, please.

My pleasure. :-)

>> $ find
>> /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home
>> -follow -name tools.jar
>> $
>>
>> So much about depending on implementation details... :-)
>
> I know. The good news is in Mustang this is fixed.

The bad news for me is that Mustang will likely require to buy a new
version of Mac OS X. :-(

What exactly is fixed in Mustang?

>> There seems to be a classes.jar that contains all com.sun and
>> java/javax classes. You can find it in
>> $JAVA_HOME/../Classes/classes.jar.
>
> Ca you check if it contain javac? But then you say it contains all
> java/javax classes. Is this the equivalent of rt.jar in Unix/Windows?
> If so, it should be in the bootstrap classloader, so all we need to do
> is to relax the error check a bit.

Not sure what I should be looking for but I definitely see a number of
com.sun.tools.javac and sun.tools.javac classes in there. I can't easily
check what's on the bootstrap classpath but I would expect to find it there.

> Does that mean in Mac every JRE comes with javac?

There is only one bundle they call J2SE and it contains a complete JDK.

>>>> Can't we run all tests and then return with a non-0 exit code if
>>>> anything failed?
>>>
>>> That's exactly what we are doing right now. The consequence is that
>>> the build script cannot set failonerror="true", because that would
>>> kill the build as long as any test fail.
>>>
>>> The discussion in this thread was that you don't want that to
>>> happen, but you do want the build to break if there's a catastrophic
>>> errors in running a test, like this one.
>>>
>>> So I suggested that we only return non-0 when there's a catastrophic
>>> failures.
>>
>> Yes, that makes sense. Or could we do it such that the harness
>> returns an error code if there had been a fatal failure or a test
>> failure, then we run the junitreport task and only after that we
>> fail? Something along these lines:
>>
>> <java jar="${test.lib.dir}/harness.jar" fork="true"
>> resultproperty="test.returncode"/>
>> ..
>> <junitreport/>
>> ..
>> <condition property="test.failed">
>> <not>
>> <equals arg1="0" arg2="test.returncode"/>
>> </not>
>> </condition>
>> <fail if="test.failed/>
>
> That could work, too, but with this approach, a catastrophic failure
> would show up with "no test failure" --- it's just that there was no
> test, but I think it could be confusing.

Yes, that would be confusing. Would it make sense to work with different
error codes for fatal failures and for test failures?

Fabian