I assume you have downloaded the jtds-1.2.5-dist.zip. Unzip this and copy the jtds-1.2.5.jar to the glassfishv3/glassfish/lib directory.
Start the domain and from the admin console when you try to create a new jdbc connection pool with restype "javax.sql.DataSource" and database vendor "jtds", in the next page, the datasource classname is automatically introspected. This is a feature only in GlassFish V3. If you are using GlassFish V2, you can use the classname "net.sourceforge.jtds.jdbcx.JtdsDataSource".
Hope this helps.
[Message sent by forum member 'sm157516' (shalini.muthukrishnan_at_sun.com)]
http://forums.java.net/jive/thread.jspa?messageID=382081