Which JMS provider are you using? I presume you're using Open Message Queue (which comes with Glassfish).
Which JMS resource adapter are you using to connect it to JBoss?
The property sendUndeliverableMsgsToDMQ is a feature of Glassfish's JMSRA resource adapter, which is for use only with Glassfish and Open Message Queue. So it isn't appropriate for use with JBoss.
Glassfish also includes the GenericJMSRA resource adapter, which is intended to work with any app server and any JMS provider though I don't have specific information on using it with JBoss (others may be able to advise). GenericJMSRA has a similar property SendBadMessagesToDMD which may be what you require.
See
http://docs.sun.com/app/docs/doc/820-4335/6nfqc3qi1?a=view
GenericJMSRA also has its own website:
https://genericjmsra.dev.java.net
There's a third, general purpose, resource adapter from Sun which may also be a possibility: JMSJCA, which is available separately:
https://jmsjca.dev.java.net/
Nigel
[Message sent by forum member 'nigeldeakin' (nigel.deakin_at_sun.com)]
http://forums.java.net/jive/thread.jspa?messageID=368955