Gracefully handle AWS ECS stop task with .NET Core Application

Question 1

I wrote a .NET core console application to run as a task in AWS ECS Fargate. Basically, the console application starts a processing thread and then needs to just keep running (much like a Kestrel web host.)

Initially, I drop in a Console.ReadKey() thinking it will never hit and the app will keep running. Well that leads to .NET exception…

Cannot read keys when either application does not have a console 
or when console input has been redirected. Try Console.Read.

1

2

3

Cannot read keys when either application does not have a console

or when console input has been redirected. Try Console.Read.

So I then replace ReadKey with Console.Read(). The application acts like it isn’t even there, keeps on moving, and the app exits as soon as it starts.

Through research, I discover the Console.CancelKeyPress event. I implement that, and it works in the fact that the console app will keep running.

ManualResetEvent _quitEvent = new ManualResetEvent(false);
Console.CancelKeyPress += (sender, eArgs) => {
                Console.WriteLine("Shutting down.");
                _quitEvent.Set();
                eArgs.Cancel = true;
            };
_quitEvent.WaitOne();

1

2

3

4

5

6

7

8

ManualResetEvent _quitEvent = new ManualResetEvent(false);

Console.CancelKeyPress += (sender, eArgs) => {

Console.WriteLine("Shutting down.");

_quitEvent.Set();

eArgs.Cancel = true;

};

_quitEvent.WaitOne();

But I would like to gracefully handle shutdown of the application when that does occur.

Through additional research, I find that when AWS ECS stops a task, it doesn’t send Ctrl-C (SIGINT) it instead will send SIGTERM. https://docs.aws.amazon.com/cli/latest/reference/ecs/stop-task.html

I then implement what I think is the standard event handling for SIGTERM in .NET (seen below.)

AssemblyLoadContext.Default.Unloading += ctx =>
            {
                Console.WriteLine("Shutting down.");
                _quitEvent.Set();
            };
_quitEvent.WaitOne();

1

2

3

4

5

6

7

AssemblyLoadContext.Default.Unloading += ctx =>

{

Console.WriteLine("Shutting down.");

_quitEvent.Set();

};

_quitEvent.WaitOne();

This compiles and runs fine in ECS. BUT… I hit the “stop” button through AWS ECS console, it stops the task and relaunches a new task. When I look at the log of that stopped task, there is no shutdown log message, no indication of a graceful shutdown, nothing. It appears that it was just force killed.

Am I missing something here? What can I do differently to gracefully catch the stop-task in AWS ECS to gracefully shut down the task?

Question 2

It seems like it has do with your .net code of handling graceful shutdown. I just tested with following code locally, with docker and with ECS Stop-Task. At all places, I received graceful shutdown. I don’t have your full code so I created my own example.

using System;
using System.Runtime.Loader;
using System.Threading;

namespace consoleapp
{
    class Program
    {
        static void Main(string[] args)
        {
            var _quitEvent = new ManualResetEvent(false);
            AssemblyLoadContext.Default.Unloading += ctx =>
            {
                System.Console.WriteLine("Unloding fired");
                _quitEvent.Set();
            };
            System.Console.WriteLine("Waiting for signals");
            _quitEvent.WaitOne();
            System.Console.WriteLine("Received signal gracefully shutting down");
            Console.WriteLine("Hello World!");
        }
    }
}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

using System;

using System.Runtime.Loader;

using System.Threading;

namespace consoleapp

{

class Program

{

static void Main(string[] args)

{

var _quitEvent = new ManualResetEvent(false);

AssemblyLoadContext.Default.Unloading += ctx =>

{

System.Console.WriteLine("Unloding fired");

_quitEvent.Set();

};

System.Console.WriteLine("Waiting for signals");

_quitEvent.WaitOne();

System.Console.WriteLine("Received signal gracefully shutting down");

Console.WriteLine("Hello World!");

}

Full github repo reference if you want to try by yourself.
https://github.com/imran9m/netconsoleapp.git

Sample Logs you should see for container.

Waiting for signals

Unloding fired

Received signal gracefully shutting down

Hello World!

Note – To test SIGTERM signal locally, you will need to run kill -SIGTERM {PID} instead of CTRL+C on dotnet CLI.

Gracefully handle AWS ECS stop task with .NET Core Application

Question:

Answer:

Leave a Reply Cancel reply