attached mail follows:
- This document should be used as a guideline when creating new
development unit tests based on TestNG.
- This document is not about the merits of using TestNG or any
other Java based testing framework.
- The proposed framework uses the existing devtests framework
to build on.
- Ease-of-use - slope of the learning curve to add new tests -
is a primary goal.
The directory structure overall would look something like this:
appserv-tests
|
---- config.properties
|
---- common/
| |
| ---- common.xml
| |
| ---- properties.xml
| |
| ...(others like derby.properties etc.)
|
---- devtestsNG/
|
---- build.xml
|
---- common
| |
| ---- util/java/src/....
| |
| ---- build.xml
|
---- mytest-module-1
| |
| ---- build.xml
| |
| ---- build.properties
| |
| ---- testng.xml
| |
| ---- tests
| |
| ---- com/sun/mytests/Test1.java
| ...
|
|---- mytest-module-2
| |
| ---- build.xml
| |
| ---- build.properties
| |
| ---- my-submodule-2-1
| | |
| | ---- (replicate dir structure of module-1 or 2)
| |
| ---- my-submodule-2-2
| |
| ---- (replicate dir structure of module-1 or 2)
|
|----- ... other modules
The following directories are use by TestNG and the devtests framework
and should not used by the devtest authors directly
- test-output
- test-report
- test-tmp
New top-level targets defined in top-level-build.xml and common-build.xml
run-groups
run-failed
report
compile-build-util (currently this compiles the util src files
and jars them up in appserv-tests/lib/myutil.jar - needs to be cleaned up)
The top-level build.xml would look something like this
Features:
- Ability to run test across all modules based on group membership.
This requires standardization of certain group
names like performance-tests, regression-tests, quick-tests etc..
- Ability to only run failed tests from previous test runs
- Ability to generate a drill-down report of the test output
This feature uses the junit library to summarize all module reports
in a single html report.
- Provide an Java API and implementation to programmatically
perform certain services like check if the server is
running or if a certain app is deployed successfully etc.
Essentially these features will provide the ability to check
whether certain pre-conditions that are needed for the tests
to be performed are satisfied are not.
The actual tests themselves can be configured to run only if
the preconditions are satisfied. This feature is work in progress.
Setup instructions:
- Env variable like APS_HOME, S1AS_HOME etc. need to be set
- The common framework facilities can be extended by adding
common target to either common build.xml or programmatically
adding an API in the common/util directory. The existing
common build.xml(as in config/common.xml) has a good number
of common asadmin based tasks.
- Any module/submodule build targets linked with any of the
targets at the top-level(appserv-tests/devtests/) should
have their configuration defined in the top-level
configuration file(common/config.properties or whatever it
turns out that it called)
- Each module/sub-module that contains tests should have a
build.properties file that defines at least the following properties
<property name="module" value="actual-module-name-here"/>
<property name="build.dir" value="${env.APS_HOME}/build/${module}"/>
These values are used as part of the directory path names
during compilation, archive creation etc.
TODO (in no particular order):
- Provide default values commonly used property names
like web.xml, sun-web.xml
These would say default to ${basedir}/descriptors/web.xml
and ${basedir}/descriptors/sun-web.xml respectively
- Clean up properties.xml/common.xml - remove unused targets, standardize
target/property names
- Come up with a standardized set of group names for TestNG
group membership which will enable us to run tests tagged
to specific groups across modules.
- Support running devtests running on a different machine.
(I presume this will still require an installation of glassfish
on the local machine since a lot of the current tests
add ${env.S1AS_HOME}/lib/*.jar to the execution classpath)
- maven'ize the top-level devtest targets so that it
blends in seamlessly with the rest of the glassfish
- IDE support
- Able to run a single test
- Logs
Test Log <=> Server Log
Both logs captured, and marked for each test
- Remote Instances
PE-EE flexibility
- Persistence EE and Java SE containers
- Roadmap should include Tools, i.e. NetBeans integration
- Deployment, jars support
Java EE technology and Non-Java EE Extensions (i.e. MBean, etc)
- Able to run old framework
Top-level build.xml
<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE project [ <!ENTITY configProperties SYSTEM "file:./../../config/properties.xml"> <!ENTITY configCommon SYSTEM "file:./../../config/common.xml"> ... ]> <project name="mytests2" default="usage" basedir="."> &configProperties; &configCommon; ... <target name="all"> <ant dir="web1" target="all"/> <ant dir="web2" target="all"/> ... <antcall target="report"/> </target> <target name="clean"> <ant dir="web1" target="clean"/> <ant dir="web2" target="clean"/> ... <delete dir="test-report"/> </target> <target name="run-groups"> <ant dir="web1" target="run-groups"/> <ant dir="web2" target="run-groups"/> ... <antcall target="report"/> </target> <target name="run-failed"> <ant dir="web1" target="run-failed"/> <ant dir="web2" target="run-failed"/> ... <antcall target="report"/> </target> </project>Leaf-level build.xml
<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE project [ <!ENTITY commonSetup SYSTEM "file:./../../../config/properties.xml"> <!ENTITY commonBuild SYSTEM "file:./../../../config/common.xml"> ... <!ENTITY testproperties SYSTEM "file:./build.properties"> ]> <project name="web1-testng-tests" default="all" basedir="."> &commonSetup; &commonBuild; &testproperties; ... <target name="all" depends="clean, build,deploy,run,report,undeploy"/> <target name="compile" depends="clean"> <antcall target="compile-common"> <param name="src" value="servlet"/> </antcall> </target> <target name="build" depends="compile"> <antcall target="webclient-war-common"> <param name="hasWebclient" value="yes"/> <param name="webclient.war.classes" value="**/*.class"/> </antcall> </target> <target name="deploy" depends="init-common"> <antcall target="deploy-war-common"/> </target> <target name="undeploy" depends="init-common"> <antcall target="undeploy-war-common"/> </target> <target name="clean"> <delete dir="${build.dir}"/> <delete dir="test-output"/> <delete dir="test-report"/> </target> <target name="run-testng-cli" depends="compile-tests"> <delete dir="test-output"/> <testng outputdir="test-output"> <classpath location="${build.dir}"/> <classfileset dir="${build.dir}" includes="*.class"> </classfileset> </testng> </target> <target name="run" depends="compile-tests"> <delete dir="test-output"/> <testng outputdir="test-output"> <classpath location="${build.dir}"/> <xmlfileset dir="." includes="testng.xml"> </xmlfileset> </testng> </target> <target name="run-groups" depends="compile-tests"> <delete dir="test-output"/> <testng outputdir="test-output" groups="${group-names}"> <classpath location="${build.dir}"/> <!-- can't seem to run using the groups option and testng.xml <xmlfileset dir="." includes="testng.xml"> </xmlfileset> --> <classfileset dir="${build.dir}" includes="example/*.class"> </classfileset> </testng> </target> <target name="run-failed"> <antcall target="common-run-failed"/> </target> <!-- this target 'compile-tests' can be parameterized and moved to common.xml. The following parameters would be needed: 1. ${build.dir} 2. ${src.dir} --> <target name="compile-tests" description="compile the test ng example" depends=""> <delete dir="${build.dir}"/> <mkdir dir="${build.dir}"/> <javac debug="true" fork="true" source="1.5" classpathref="path2testng" srcdir="tests" destdir="${build.dir}" /> </target> <target name="tng-compile" description="compile the examples" depends=""> <delete dir="${build.dir}"/> <mkdir dir="${build.dir}"/> <javac debug="true" fork="true" source="1.5" classpathref="path2testng" srcdir="tests" destdir="${build.dir}" /> </target> </project>XML contains targets that would be added to the common build.xml
<path id="path2testng">
<pathelement location="/abc/testng-5.0-jdk15.jar"/>
</path>
<taskdef name="testng"
classpathref="path2testng"
classname="org.testng.TestNGAntTask"/>
<!-- Generate the TestNG report -->
<target name="report">
<delete dir="test-report"/>
<mkdir dir="test-report"/>
<junitreport todir="test-report">
<fileset dir=".">
<include name="**/test-output/**/*.xml"/>
<exclude name="**/testng-failed.xml"/>
</fileset>
<report format="noframes" todir="test-report"/>
</junitreport>
</target>
<target name="common-run-failed">
<delete dir="test-tmp/"/>
<move todir="test-tmp/">
<fileset dir="." includes="**/test-output/**/testng-failed.xml"/>
</move>
<delete dir="test-output/"/>
<testng outputdir="test-output">
<classpath location="${build.dir}"/>
<xmlfileset dir="."
includes="test-tmp/**/testng-failed.xml">
</xmlfileset>
</testng>
</target>
<target name="compile-build-util">
<delete dir="${env.APS_HOME}/build/util"/>
<mkdir dir="${env.APS_HOME}/build/util"/>
<javac srcdir="${env.APS_HOME}/devtests/mytests2/common/src"
destdir="${env.APS_HOME}/build/util"
classpath="${env.S1AS_HOME}/lib/javaee.jar:${env.S1AS_HOME}/lib/appserv-ext.jar">
</javac>
<delete file="${env.APS_HOME}/lib/myutil.jar"/>
<jar destfile="${env.APS_HOME}/lib/myutil.jar"
basedir="${env.APS_HOME}/build/util"/>
</target>
Leaf-level module test
directory structure
mytest-module-1/
|
---- build.xml
|
---- build.properties
|
---- testng.xml
|
---- descriptor/
| |
| ---- web.xml
| |
| ---- sun-web.xml
|
---- servlet/
| |
| ---- com/sun/mywebapp/TestServlet.java
| ...
|
---- tests/
|
---- com/sun/mytests/Test1.java
...
sample build.properties
<property name="module" value="web1"/>
<property name="appname" value="${module}"/>
<property name="web.xml" value="descriptor/web.xml"/>
<property name="sun-web.xml" value="descriptor/sun-web.xml"/>
<property name="contextroot" value="/${appname}"/>
<property name="build.dir" value="${env.APS_HOME}/build/${module}/xyz"/>
...