Welcome to CloudAffaire and this is Debjeet
In the last blog post, we have discussed CloudWatch Events and also create CloudWatch events on EC2 instance state change.
In this blog post, we are going to discuss CloudWatch Logs. We are also going to configure CloudWatch Log Agent in an EC2 instance hosting Amazon Linux 2 to collect and store system logs in CloudWatch console.
You can use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, AWS CloudTrail, Route 53, and other sources. You can then retrieve the associated log data from CloudWatch Logs.
CloudWatch Logs component:
A log event is a record of some activity recorded by the application or resource being monitored. The log event record that CloudWatch Logs understands contains two properties: the timestamp of when the event occurred, and the raw event message. Event messages must be UTF-8 encoded.
A log stream is a sequence of log events that share the same source. More specifically, a log stream is generally intended to represent the sequence of events coming from the application instance or resource being monitored. For example, a log stream may be associated with an Apache access log on a specific host. When you no longer need a log stream, you can delete it using the aws logs delete-log-stream command. In addition, AWS may delete empty log streams that are over 2 months old.
Log groups define groups of log streams that share the same retention, monitoring, and access control settings. Each log stream has to belong to one log group. For example, if you have a separate log stream for the Apache access logs from each host, you could group those log streams into a single log group called MyWebsite.com/Apache/access_log. There is no limit on the number of log streams that can belong to one log group.
You can use metric filters to extract metric observations from ingested events and transform them to data points in a CloudWatch metric. Metric filters are assigned to log groups, and all of the filters assigned to a log group are applied to their log streams.
Retention settings can be used to specify how long log events are kept in CloudWatch Logs. Expired log events get deleted automatically. Just like metric filters, retention settings are also assigned to log groups, and the retention assigned to a log group is applied to their log streams.
Prerequisite for CloudWatch Log Agent:
- Running EC2 instance.
- Instance configured with AWS CLI and IAM role for CloudWatch Log Agent
We have already created the EC2 instance with proper role and also configured AWS CLI
Next, we are going to install and configure CloudWatch Log Agent.
Step 1: Connect to your EC2 instance and install CloudWatch Log Agent.
sudo yum update -y sudo yum install -y awslogs
Step 2: View CloudWatch Log Agent configuration files
sudo ls -l /etc/awslogs/
Step 3: Configure awscli.conf and awslogs.conf with region and log details
sudo vi /etc/awslogs/awscli.conf sudo vi /etc/awslogs/awslogs.conf
You can referrer below AWS documentation for Log Agent configuration
Step 4: Start the CloudWatch Log Agent
sudo systemctl start awslogsd
Once the Log Agent is started, it will publish logs as per configuration details in the configuration file. You can check the logs in CloudWatch console under Logs.
Hope you have enjoyed this article. With this, we are concluding our introductory series on CloudWatch and will start with a new AWS service from the next blog post.
To get more details on CloudWatch, please refer below AWS documentation