I've had so many headaches in the past with QuickLook on Windows that
I hope we can avoid them this time.
Tests are a critical part of quality, and the ease and convenience of
running them should receive the highest level of attention.
General comments, based on those experiences:
1. Don't make it work like QuirkLook, that runs successfully all of
the time with a fresh checkout on one platform (Solaris), and some or
none of the time on others, with innumerable obtuse settings,
warnings, garbage output that I don't want to know about. And make
sure the emailed results aren't (literally) gibberish.
2. Do not require ANY environment variables. There is no need in
the common case, and it's just plain annoying. Get them from current
location by default. Allow, but do not require overrides.
3. Put settings into a well-commented properties file, general
settings in one file, and then *separate files for each module*.
Add quality comments that use proper spelling and grammar and will
make sense to someone just starting for the first time. There should
be no required settings that must be changed prior to running the tests
4. Make a separate ant/maven target for each module get "run-admin-
tests", "run-amx-tests", "run-blah-blan-tests".
5. Pay attention to plurals; eg don't make it "run-test"; make it
"run-tests", etc. Don't make it "run-tsts" or crap like that. Use
dashes and words the same way every time.
6. Designate a person responsible for the quality and accuracy of
the documentation, and put it into html so it can be read easily with
a browser. MAKE SURE there is a "last verified as accurate" date at
the top of the documentation, and provide a live email link for
questions that will be answered in a timely manner.
By doing these things, acceptance and use by developers will be rapid.
Lloyd Chambers
On Aug 4, 2006, at 3:48 PM, Ashish Sahni wrote:
> Hi,
>
> We're planning to propose the following (see attached mail) doc as
> a guideline to develop devtests using the TestNG framework.
> Please feel free to send comments. Your feedback is highly
> appreciated.
>
> Thanks
> Ashish
>
>
> From: Ashish Sahni <Ashish.Sahni_at_sun.com>
> Date: August 4, 2006 3:32:52 PM PDT
> To: Ashish Sahni <Ashish.Sahni_at_sun.com>
> Subject: TestNG doc
>
>
> - This document should be used as a guideline when creating new
> development unit tests based on TestNG.
> - This document is not about the merits of using TestNG or any
> other Java based testing framework.
> - The proposed framework uses the existing devtests framework
> to build on.
> - Ease-of-use - slope of the learning curve to add new tests -
> is a primary goal.
>
> The directory structure overall would look something like this:
>
> appserv-tests
> |
> ---- config.properties
> |
> ---- common/
> | |
> | ---- common.xml
> | |
> | ---- properties.xml
> | |
> | ...(others like derby.properties etc.)
> |
> ---- devtestsNG/
> |
> ---- build.xml
> |
> ---- common
> | |
> | ---- util/java/src/....
> | |
> | ---- build.xml
> |
> ---- mytest-module-1
> | |
> | ---- build.xml
> | |
> | ---- build.properties
> | |
> | ---- testng.xml
> | |
> | ---- tests
> | |
> | ---- com/sun/mytests/Test1.java
> | ...
> |
> |---- mytest-module-2
> | |
> | ---- build.xml
> | |
> | ---- build.properties
> | |
> | ---- my-submodule-2-1
> | | |
> | | ---- (replicate dir structure of module-1 or 2)
> | |
> | ---- my-submodule-2-2
> | |
> | ---- (replicate dir structure of module-1 or 2)
> |
> |----- ... other modules
>
> The following directories are use by TestNG and the devtests framework
> and should not used by the devtest authors directly
> - test-output
> - test-report
> - test-tmp
>
> New top-level targets defined in top-level-build.xml and common-
> build.xml
> run-groups
> run-failed
> report
> compile-build-util (currently this compiles the util src files
> and jars them up in appserv-tests/lib/myutil.jar - needs to be
> cleaned up)
>
> The top-level build.xml would look something like this
> A leaf-level build.xml would look something like this
> This XML contains targets that would be added to the common build.xml
> Some targets in the leaf-level-build.xml can be parameterized and
> moved into the commons
> A sample leaf-level module test directory structure
> A build.properties example
> Features:
>
> - Ability to run test across all modules based on group membership.
> This requires standardization of certain group
> names like performance-tests, regression-tests, quick-tests etc..
> - Ability to only run failed tests from previous test runs
> - Ability to generate a drill-down report of the test output
> This feature uses the junit library to summarize all module reports
> in a single html report.
> - Provide an Java API and implementation to programmatically
> perform certain services like check if the server is
> running or if a certain app is deployed successfully etc.
> Essentially these features will provide the ability to check
> whether certain pre-conditions that are needed for the tests
> to be performed are satisfied are not.
> The actual tests themselves can be configured to run only if
> the preconditions are satisfied. This feature is work in progress.
>
>
> Setup instructions:
>
> - Env variable like APS_HOME, S1AS_HOME etc. need to be set
> - The common framework facilities can be extended by adding
> common target to either common build.xml or programmatically
> adding an API in the common/util directory. The existing
> common build.xml(as in config/common.xml) has a good number
> of common asadmin based tasks.
> - Any module/submodule build targets linked with any of the
> targets at the top-level(appserv-tests/devtests/) should
> have their configuration defined in the top-level
> configuration file(common/config.properties or whatever it
> turns out that it called)
> - Each module/sub-module that contains tests should have a
> build.properties file that defines at least the following properties
>
>
> These values are used as part of the directory path names
> during compilation, archive creation etc.
>
>
> TODO (in no particular order):
> - Provide default values commonly used property names
> like web.xml, sun-web.xml
> These would say default to ${basedir}/descriptors/web.xml
> and ${basedir}/descriptors/sun-web.xml respectively
> - Clean up properties.xml/common.xml - remove unused targets,
> standardize
> target/property names
> - Come up with a standardized set of group names for TestNG
> group membership which will enable us to run tests tagged
> to specific groups across modules.
> - Support running devtests running on a different machine.
> (I presume this will still require an installation of glassfish
> on the local machine since a lot of the current tests
> add ${env.S1AS_HOME}/lib/*.jar to the execution classpath)
> - maven'ize the top-level devtest targets so that it
> blends in seamlessly with the rest of the glassfish
> - IDE support
> - Able to run a single test
> - Logs
> Test Log <=> Server Log
> Both logs captured, and marked for each test
> - Remote Instances
> PE-EE flexibility
> - Persistence EE and Java SE containers
> - Roadmap should include Tools, i.e. NetBeans integration
> - Deployment, jars support
> Java EE technology and Non-Java EE Extensions (i.e. MBean, etc)
> - Able to run old framework
>
>
>
> Top-level build.xml
>
> "file:./../../config/properties.xml">
> "file:./../../config/common.xml">
> ...
> ]>
>
>
>
> &configProperties;
> &configCommon;
> ...
>
>
>
>
> ...
>
>
>
>
>
>
> ...
>
>
>
>
>
>
> ...
>
>
>
>
>
>
> ...
>
>
>
>
>
> Leaf-level build.xml
>
> "file:./../../../config/properties.xml">
> "file:./../../../config/common.xml">
> ...
> "file:./build.properties">
> ]>
>
>
>
> &commonSetup;
> &commonBuild;
> &testproperties;
> ...
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> XML contains targets that would be added to the common build.xml
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Leaf-level module test directory structure
> mytest-module-1/
> |
> ---- build.xml
> |
> ---- build.properties
> |
> ---- testng.xml
> |
> ---- descriptor/
> | |
> | ---- web.xml
> | |
> | ---- sun-web.xml
> |
> ---- servlet/
> | |
> | ---- com/sun/mywebapp/TestServlet.java
> | ...
> |
> ---- tests/
> |
> ---- com/sun/mytests/Test1.java
> ...
>
>
> sample build.properties
>
>
>
>
>
>
>
>
> ...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe_at_glassfish.dev.java.net
> For additional commands, e-mail: dev-help_at_glassfish.dev.java.net