dev@glassfish.java.net

Re: Behavior of asadmin stop-instance command

From: Ken Cavanaugh <ken.cavanaugh_at_oracle.com>
Date: Wed, 3 Nov 2010 12:53:07 -0700

On Nov 3, 2010, at 12:15 PM, Byron Nevins wrote:

> The confusing situation has been in place since 3.0 Namely there is a "force" argument. Until now it did nothing at all. Now it does something.
> The choice was to either get rid of force completely and then add it back in in the future when the non-daemon thread problem has been solved. Or have force do what it was always supposed to do. "They" decided on the latter and I implemented it. It is trivially easy to remove the argument completely. Perhaps you should bring it up at the Engineering meeting?
> You must not have tried the command out. It definitely does not turn it into a Zombie. In fact -- you can easily access the HTTP listener in a browser after running 'stop-domain --force false'! Why? [1]
Actually I did try the command during the window when Byron hadn't made --force true the default. I observed
in my IIOP FOLB test that the AppServer instance did not terminate, but the ORB was still alive, and responded to
requests by indicating that the EJB no longer existed. That makes it impossible to detect that the instance is no longer
usable and failover to another instance. That made the instance a Zombie from the IIOP/Remote EJB viewpoint.

Probably something needs to call ORB.shutdown somewhere, and that's not happening yet. I think all of these related
shutdown problems would best be dealt with in the next release.

 I have not tried --force false since I observed the above behavior, which was roughly 10/29-11/1.
>
> [1] The "real" shutdown of GF happens because of the call to GlassFIshRuntime.shutdown(). That is called via a shutdown hook (see core/bootstrapping code). Shutdown Hooks are only called after exiting main() and the last non-daemon thread has stopped. Or if System.exit() is called. In the case of "force=false" neither of these cases is true and no shutdown hooks are called. I suggested that we move this code out of a shutdown hook and call it explicitly (I filed an issue). That issue was immediately closed. So that's not going to happen.

Tom just sent an email saying that force would be a hidden option for 3.1, which (I think?) solves the problem of
the user getting confused about --force false.

Thanks,

Ken.

>
>
> On 11/3/2010 11:36 AM, Ken wrote:
>>
>> Ken Cavanaugh wrote:
>>> Byron Nevins wrote:
>>>> Yes - that is how stop-server commands work. Before the --force option was 100% ignored. Now it is used. If you set --force to false, then the server will not stop. That's because System.exit() is not called and your server has non-daemon threads left running - so it will never stop.
>>>>
>> So what you are saying is, if a user types "stop-instance --force false", the instance does not stop. Worse, the
>> instance is left in a zombie state (e.g. ORB still runs, but EJBs are undeployed). I don't see that this is a useful
>> behavior to expose to the customer. From our meeting this morning, I think the docs and SQE folks agree with
>> this.
>>
>> I can understand that there might be a desire to have a clean shutdown option (e.g. kill vs. kill -9),
>> but if the clean shutdown doesn't work, we shouldn't support it. It's not clear to me what your
>> plan is for MS7 (and FCS) in GF 3.1, but it seems that either stop-instance --force false should
>> result in useful behavior (at least in a "normal" case), or stop-instance --force false should
>> not be supported. If the --force option is needed for other plans (e.g. backward compatibility or
>> future support), I think --force false should produce a warning and do nothing. Otherwise, the
>> --force option should be removed.
>>
>> Another question is: how should SQE test stop-instance --force false? The current behavior is not
>> well-defined, and cannot be tested.
>>
>> My desire here is simple: avoid situations that confuse the user. If the option is not useful, it
>> should not be supported.
>>
>> Thanks,
>>
>> Ken.
>>
>
> --
> Byron Nevins - Oracle Corporation
> Home: 650-359-1290
> Cell: 650-784-4123
> Sierra: 209-295-2188