I found the problem!
It was a classloader issue. The thread that triggered the reload had the
org.glassfish.jersey.internal.spi.AutoDiscoverable on it's path but loaded by a
different webapp classloader!
Once I ensured the same classloader was used for all reloads the problem went away.