I was recently stumped by some shutdown issues in a Tomcat-hosted Java webservice. The server just wouldn’t shut down cleanly. After (most of) a day running round in circles, I tracked the issue down to embedded Infinispan - and specifically JGroups.
I’m writing the solution up here in the hopes someone might find it useful.
To ensure a proper resource cleanup during JVM shutdown, Infinispan has built in shutdown hooks. Tomcat (and most other Application Servers) have their own shutdown hooks, so it’s desirable to turn off Infinispan’s built-in shutdown hooks to prevent any contention issues. Via
By unregistering the shutdown hooks (as above), non-daemon JGroups threads are left running inside the Application Server - even after a shutdown call. This prevents the Application Server process from terminating - it needs to be killed. From jstack, the culprit threads are:
- multicast receiver
The question: how do we stop these runaway threads and allow the Application Server to shutdown gracefully?
The fix is fairly simple.
Being a Servlet Container, Tomcat conforms to the Java Servlet API. We can use the ServletContextListener Interface to initiate a graceful Infinispan shutdown when Tomcat shutdown is initiated. Using Spring:
That’s all there is to it!
Moral of the Story
When you unregister Infinispan shutdown hooks via
DONT_REGISTER, always make sure to manually stop the Infinispan caches via