I think this error message can also appears with "unexpected" order of conditions (but I don't know why ;-).
Can you try with this other order of conditions:
SELECT OBJECT(cr) FROM CategoryRoot cr, Catalog ca
WHERE ca.catalogId = ?1 AND cr NOT MEMBER OF ca.categoryRoots
[Message sent by forum member 'jmarine' (jmarine)]
http://forums.java.net/jive/thread.jspa?messageID=289437