An endpoint that is deployed programmatically, e.g. by registering a ServerEndpointConfig, should be excluded from the Servlet container scan. Maybe some containers will do that automatically but since the container scan imposes no limitations, it is capable of detecting all kinds of classes including package private, private inner classes, even anonymous classes. That may make it difficult to deploy via ServerEndpointConfig correctly. Essentially, a ServerApplicationConfig might need to be used to filter out the ServerEndpointConfig types but that's hardly an ideal way and a workaround at best.
One simple change that would go a long way is to define a requirement for any scanned types to be public. There is such a requirement for @ServerEndpoint types, and it makes sense to extend it to all scanned types in any case.
Rossen