I use AWS CloudWatch log agent to push my application log to AWS Cloudwatch.

In the cloudwatchLogs config file inside my EC2 instance, I have this entry:

According to this configuration, all log files in workingdir directory are being sent to cloudwatchLogs in the same stream were the name is the instance Id.

My question is, I want for each log file, create a separate logStream, so that the logs reading can be more fast and parseable. In other words, every time I have a new log file, a new logstream is created automatically.

I thought of doing that by a shell script in a cron job but then I’ll have to change many other configurations in the architecture, so I’m looking for a way to do it in the config file. In the documentation, they say that :

log_stream_name

Specifies the destination log stream. You can use a literal string or
predefined variables ({instance_id}, {hostname}, {ip_address}), or
combination of both to define a log stream name. A log stream is
created automatically if it doesn’t already exist.

The names of the log files can’t be 100% predictible, but they always have this structure though:

Also, another problem is that :

A running agent must be stopped and restarted for configuration
changes to take effect.

How can I set the logStream in this case?

Any ideas or suggestions or workarounds are very appreciated.

I know it’s been almost two years now, but I wanted to do the exact same thing and couldn’t find a way to get it to work.
I resorted to the AWS Support, which then confirmed this cannot be done. We’re limited to the options offered in the documentation, just like you posted. You can, however, have log groups contain the log file path up to the first dot:

log_group_name – Optional. Specifies what to use as the log group name
in CloudWatch Logs. Allowed characters include a-z, A-Z, 0-9, ‘_’
(underscore), ‘-‘ (hyphen), ‘/’ (forward slash), and ‘.’ (period).

We recommend that you specify this field to prevent confusion. If you
omit this field, the file path up to the final dot is used as the log
group name. For example, if the file path is
/tmp/TestLogFile.log.2017-07-11-14, the log group name is
/tmp/TestLogFile.log.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html