I am using the AWS Kinesis Client Library.
I need a way to shutdown Kinesis Worker thread during deployments so, that I stop at a checkpoint and not in the middle of
I see a shutdown boolean present in
Worker.java but it is made private.
The reason I need is that checkpointing and idempotency is critical to me and I don’t want to kill the process in the middle of a batch.
Thanks to @CaptainMurphy, I noticed that
shutdown() method which safely shuts down the worker and the
LeaseCoordinator. What it doesn’t do is call
shutdown() task in the
IRecordProcessor. It abruptly terminates the
IRecordProcessor without worrying about the state.
I do understand that Idempotency between checkpoints is not guaranteed by the KCL and the developer should make the design fault tolerant but I feel that the
IRecordProcessor should be properly shutdown before
LeaseCoordinator stops irrespective of that.
Since version 1.7.1 (see note below) application can request a graceful shutdown, and record processors that implement the IShutdownNotificationAware will be given a chance to checkpoint before being shutdown.
- make sure record processor implements
IShutdownNotificationAwareinterface in addition to one of
IRecordProcessorinterfaces. Call checkpoint in
shutdownRequested(IRecordProcessorCheckpointer checkpointer)method. Pay attention –
shutdownmethod of IRecordProcessor should call checkpoint only if shutdown reason is TERMINATE
- on application shutdown initiate worker shutdown process
shutdown = worker.requestShutdown();shutdown.get(); // wait for shutdown complete
PS: Kinesis Client before version 1.7.4 contains race condition which prevents correct shutdown. So use version 1.7.4 or later.