Thanks, Ashish. Could you write a (can be short) blog about it so we
can spotlight it in TheAquarium?
Thanks,
- eduard/o
Ashish Sahni wrote:
> Hi,
>
> We're planning to propose the following (see attached mail) doc as
> a guideline to develop devtests using the TestNG framework.
> Please feel free to send comments. Your feedback is highly appreciated.
>
> Thanks
> Ashish
>
>
> ------------------------------------------------------------------------
>
> Subject:
> TestNG doc
> From:
> Ashish Sahni <Ashish.Sahni_at_sun.com>
> Date:
> Fri, 04 Aug 2006 15:32:52 -0700
> To:
> Ashish Sahni <Ashish.Sahni_at_sun.com>
>
> To:
> Ashish Sahni <Ashish.Sahni_at_sun.com>
>
>
> - This document should be used as a guideline when creating new
> development unit tests based on TestNG.
> - This document is not about the merits of using TestNG or any
> other Java based testing framework.
> - The proposed framework uses the existing devtests framework
> to build on.
> - Ease-of-use - slope of the learning curve to add new tests -
> is a primary goal.
>
> The directory structure overall would look something like this:
>
> appserv-tests
> |
> ---- config.properties
> |
> ---- common/
> | |
> | ---- common.xml
> | |
> | ---- properties.xml
> | |
> | ...(others like derby.properties etc.)
> |
> ---- devtestsNG/
> |
> ---- build.xml
> |
> ---- common
> | |
> | ---- util/java/src/....
> | |
> | ---- build.xml
> |
> ---- mytest-module-1
> | |
> | ---- build.xml
> | |
> | ---- build.properties
> | |
> | ---- testng.xml
> | |
> | ---- tests
> | |
> | ---- com/sun/mytests/Test1.java
> | ...
> |
> |---- mytest-module-2
> | |
> | ---- build.xml
> | |
> | ---- build.properties
> | |
> | ---- my-submodule-2-1
> | | |
> | | ---- (replicate dir structure of module-1 or 2)
> | |
> | ---- my-submodule-2-2
> | |
> | ---- (replicate dir structure of module-1 or 2)
> |
> |----- ... other modules
>
> The following directories are use by TestNG and the devtests framework
> and should not used by the devtest authors directly
> - test-output
> - test-report
> - test-tmp
>
> New top-level targets defined in top-level-build.xml and common-build.xml
> run-groups
> run-failed
> report
> compile-build-util (currently this compiles the util src files
> and jars them up in appserv-tests/lib/myutil.jar - needs to be cleaned up)
>
>
> The top-level build.xml would look something like this
> <http://javaweb.sfbay.sun.com/%7Eahshishs/testng/testngREADMEsingledoc.html#top-level-build.xml>
>
> A leaf-level build.xml would look something like this
> <http://javaweb.sfbay.sun.com/%7Eahshishs/testng/testngREADMEsingledoc.html#leaf-level-build.xml>
>
> This
> <http://javaweb.sfbay.sun.com/%7Eahshishs/testng/testngREADMEsingledoc.html#new-common-targets.xml>
> XML contains targets that would be added to the common build.xml
> Some targets in the leaf-level-build.xml can be parameterized and moved
> into the commons
> A sample
> <http://javaweb.sfbay.sun.com/%7Eahshishs/testng/testngREADMEsingledoc.html#sample-leaf-level-module.html>
> leaf-level module test directory structure
> A build.properties
> <http://javaweb.sfbay.sun.com/%7Eahshishs/testng/testngREADMEsingledoc.html#sample-build.properties>
> example
>
> Features:
>
> - Ability to run test across all modules based on group membership.
> This requires standardization of certain group
> names like performance-tests, regression-tests, quick-tests etc..
> - Ability to only run failed tests from previous test runs
> - Ability to generate a drill-down report of the test output
> This feature uses the junit library to summarize all module reports
> in a single html report.
> - Provide an Java API and implementation to programmatically
> perform certain services like check if the server is
> running or if a certain app is deployed successfully etc.
> Essentially these features will provide the ability to check
> whether certain pre-conditions that are needed for the tests
> to be performed are satisfied are not.
> The actual tests themselves can be configured to run only if
> the preconditions are satisfied. This feature is work in progress.
>
>
> Setup instructions:
>
> - Env variable like APS_HOME, S1AS_HOME etc. need to be set
> - The common framework facilities can be extended by adding
> common target to either common build.xml or programmatically
> adding an API in the common/util directory. The existing
> common build.xml(as in config/common.xml) has a good number
> of common asadmin based tasks.
> - Any module/submodule build targets linked with any of the
> targets at the top-level(appserv-tests/devtests/) should
> have their configuration defined in the top-level
> configuration file(common/config.properties or whatever it
> turns out that it called)
> - Each module/sub-module that contains tests should have a
> build.properties file that defines at least the following properties
> <property name="module" value="actual-module-name-here"/>
> <property name="build.dir" value="${env.APS_HOME}/build/${module}"/>
> These values are used as part of the directory path names
> during compilation, archive creation etc.
>
>
> TODO (in no particular order):
> - Provide default values commonly used property names
> like web.xml, sun-web.xml
> These would say default to ${basedir}/descriptors/web.xml
> and ${basedir}/descriptors/sun-web.xml respectively
> - Clean up properties.xml/common.xml - remove unused targets, standardize
> target/property names
> - Come up with a standardized set of group names for TestNG
> group membership which will enable us to run tests tagged
> to specific groups across modules.
> - Support running devtests running on a different machine.
> (I presume this will still require an installation of glassfish
> on the local machine since a lot of the current tests
> add ${env.S1AS_HOME}/lib/*.jar to the execution classpath)
> - maven'ize the top-level devtest targets so that it
> blends in seamlessly with the rest of the glassfish
> - IDE support
> - Able to run a single test
> - Logs
> Test Log <=> Server Log
> Both logs captured, and marked for each test
> - Remote Instances
> PE-EE flexibility
> - Persistence EE and Java SE containers
> - Roadmap should include Tools, i.e. NetBeans integration
> - Deployment, jars support
> Java EE technology and Non-Java EE Extensions (i.e. MBean, etc)
> - Able to run old framework
>
>
>
>
> Top-level build.xml
>
> <?xml version="1.0" encoding="ISO-8859-1"?>
>
> <!DOCTYPE project [
> <!ENTITY configProperties SYSTEM "file:./../../config/properties.xml">
> <!ENTITY configCommon SYSTEM "file:./../../config/common.xml">
> ...
> ]>
>
> <project name="mytests2" default="usage" basedir=".">
>
> &configProperties;
> &configCommon;
> ...
>
> <target name="all">
> <ant dir="web1" target="all"/>
> <ant dir="web2" target="all"/>
> ...
> <antcall target="report"/>
> </target>
>
> <target name="clean">
> <ant dir="web1" target="clean"/>
> <ant dir="web2" target="clean"/>
> ...
> <delete dir="test-report"/>
> </target>
>
> <target name="run-groups">
> <ant dir="web1" target="run-groups"/>
> <ant dir="web2" target="run-groups"/>
> ...
> <antcall target="report"/>
> </target>
>
> <target name="run-failed">
> <ant dir="web1" target="run-failed"/>
> <ant dir="web2" target="run-failed"/>
> ...
> <antcall target="report"/>
> </target>
>
>
> </project>
>
> Leaf-level build.xml
>
> <?xml version="1.0" encoding="ISO-8859-1"?>
>
> <!DOCTYPE project [
> <!ENTITY commonSetup SYSTEM "file:./../../../config/properties.xml">
> <!ENTITY commonBuild SYSTEM "file:./../../../config/common.xml">
> ...
> <!ENTITY testproperties SYSTEM "file:./build.properties">
> ]>
>
> <project name="web1-testng-tests" default="all" basedir=".">
>
> &commonSetup;
> &commonBuild;
> &testproperties;
> ...
>
> <target name="all" depends="clean, build,deploy,run,report,undeploy"/>
>
> <target name="compile" depends="clean">
> <antcall target="compile-common">
> <param name="src" value="servlet"/>
> </antcall>
> </target>
>
> <target name="build" depends="compile">
> <antcall target="webclient-war-common">
> <param name="hasWebclient" value="yes"/>
> <param name="webclient.war.classes" value="**/*.class"/>
> </antcall>
> </target>
>
> <target name="deploy" depends="init-common">
> <antcall target="deploy-war-common"/>
> </target>
>
> <target name="undeploy" depends="init-common">
> <antcall target="undeploy-war-common"/>
> </target>
>
> <target name="clean">
> <delete dir="${build.dir}"/>
> <delete dir="test-output"/>
> <delete dir="test-report"/>
> </target>
>
> <target name="run-testng-cli" depends="compile-tests">
> <delete dir="test-output"/>
> <testng outputdir="test-output">
> <classpath location="${build.dir}"/>
> <classfileset dir="${build.dir}"
> includes="*.class">
> </classfileset>
> </testng>
> </target>
>
> <target name="run" depends="compile-tests">
> <delete dir="test-output"/>
> <testng outputdir="test-output">
> <classpath location="${build.dir}"/>
> <xmlfileset dir="."
> includes="testng.xml">
> </xmlfileset>
> </testng>
> </target>
>
> <target name="run-groups" depends="compile-tests">
> <delete dir="test-output"/>
> <testng outputdir="test-output"
> groups="${group-names}">
> <classpath location="${build.dir}"/>
> <!-- can't seem to run using the groups option and testng.xml
> <xmlfileset dir="."
> includes="testng.xml">
> </xmlfileset>
> -->
> <classfileset dir="${build.dir}"
> includes="example/*.class">
> </classfileset>
> </testng>
> </target>
>
> <target name="run-failed">
> <antcall target="common-run-failed"/>
> </target>
>
> <!-- this target 'compile-tests' can be parameterized and moved
> to common.xml. The following parameters would be needed:
> 1. ${build.dir}
> 2. ${src.dir}
> -->
> <target name="compile-tests"
> description="compile the test ng example" depends="">
> <delete dir="${build.dir}"/>
> <mkdir dir="${build.dir}"/>
> <javac debug="true"
> fork="true"
> source="1.5"
> classpathref="path2testng"
> srcdir="tests"
> destdir="${build.dir}"
> />
> </target>
> <target name="tng-compile"
> description="compile the examples" depends="">
> <delete dir="${build.dir}"/>
> <mkdir dir="${build.dir}"/>
> <javac debug="true"
> fork="true"
> source="1.5"
> classpathref="path2testng"
> srcdir="tests"
> destdir="${build.dir}"
> />
> </target>
>
>
>
> </project>
>
> XML contains targets that would be added to the common build.xml
>
> <path id="path2testng">
> <pathelement location="/abc/testng-5.0-jdk15.jar"/>
> </path>
>
> <taskdef name="testng"
> classpathref="path2testng"
> classname="org.testng.TestNGAntTask"/>
>
> <!-- Generate the TestNG report -->
> <target name="report">
> <delete dir="test-report"/>
> <mkdir dir="test-report"/>
> <junitreport todir="test-report">
> <fileset dir=".">
> <include name="**/test-output/**/*.xml"/>
> <exclude name="**/testng-failed.xml"/>
> </fileset>
> <report format="noframes" todir="test-report"/>
> </junitreport>
> </target>
>
> <target name="common-run-failed">
> <delete dir="test-tmp/"/>
> <move todir="test-tmp/">
> <fileset dir="." includes="**/test-output/**/testng-failed.xml"/>
> </move>
> <delete dir="test-output/"/>
> <testng outputdir="test-output">
> <classpath location="${build.dir}"/>
> <xmlfileset dir="."
> includes="test-tmp/**/testng-failed.xml">
> </xmlfileset>
> </testng>
> </target>
>
> <target name="compile-build-util">
> <delete dir="${env.APS_HOME}/build/util"/>
> <mkdir dir="${env.APS_HOME}/build/util"/>
> <javac srcdir="${env.APS_HOME}/devtests/mytests2/common/src"
> destdir="${env.APS_HOME}/build/util"
> classpath="${env.S1AS_HOME}/lib/javaee.jar:${env.S1AS_HOME}/lib/appserv-ext.jar">
> </javac>
> <delete file="${env.APS_HOME}/lib/myutil.jar"/>
> <jar destfile="${env.APS_HOME}/lib/myutil.jar"
> basedir="${env.APS_HOME}/build/util"/>
> </target>
>
>
> Leaf-level module test directory structure
>
> mytest-module-1/
> |
> ---- build.xml
> |
> ---- build.properties
> |
> ---- testng.xml
> |
> ---- descriptor/
> | |
> | ---- web.xml
> | |
> | ---- sun-web.xml
> |
> ---- servlet/
> | |
> | ---- com/sun/mywebapp/TestServlet.java
> | ...
> |
> ---- tests/
> |
> ---- com/sun/mytests/Test1.java
> ...
>
>
>
> sample build.properties
>
> <property name="module" value="web1"/>
> <property name="appname" value="${module}"/>
>
> <property name="web.xml" value="descriptor/web.xml"/>
> <property name="sun-web.xml" value="descriptor/sun-web.xml"/>
>
> <property name="contextroot" value="/${appname}"/>
>
> <property name="build.dir" value="${env.APS_HOME}/build/${module}/xyz"/>
> ...
>
>