How To Integrate CloudWatch With CloudTrail
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed how to enable insight events in CloudTrail.
https://cloudaffaire.com/how-to-enable-insight-events-in-aws-cloudtrail/
In today’s blog post, we will discuss how to integrate CloudWatch with CloudTrail. You might want to trigger something say a lambda function, a system manager document or any other automation based on a CloudTrail event. There are different ways to achieve that and today I will cover one such way. You can send your CloudTrail events to CloudWatch logs and once your event is in CloudWatch, it opens up endless possibility of automation.
But before that let us cover one basic thing first. Many of us wonders (especially when starting with AWS) why we need CloudTrail when we know there is a tool called CloudWatch where we can monitor our AWS infrastructure and what is the difference between CloudWatch and CloudTrail.
Difference Between CloudWatch And CloudTrail:
CloudWatch: CloudWatch is mainly used to monitor your AWS infrastructure. Using CloudWatch, you can have a detailed report on your resource utilization by AWS services. Store and analyse the logs generated by different AWS service. Create automation triggers based on certain events or threshold.
CloudTrail: CloudTrail on the other hand is an audit tool which keeps record of who does what and when in your AWS infrastructure. Using CloudTrail you get a detailed information in the form of event for all the API actions that were performed by anyone (humans) or anything (computers) on your AWS infrastructure.
How To Integrate CloudWatch With CloudTrail:
Prerequisites:
- AWS CLI installed and configured with proper access. You can use below link to install and configure AWS CLI.
https://cloudaffaire.com/how-to-install-aws-cli/
https://cloudaffaire.com/how-to-configure-aws-cli/
Step 1: Create S3 bucket and bucket policy for CloudTrail logging.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
## Create a S3 bucket to store CloudTrail Logs aws s3api create-bucket \ --bucket s3-for-cloudtrail-logs \ --region ap-south-1 \ --create-bucket-configuration LocationConstraint=ap-south-1 ## Create a bucket policy definition file cat <<'EOF'> bucket_policy.json { "Version": "2012-10-17", "Statement": [ { "Sid": "AWSCloudTrailAclCheck20150319", "Effect": "Allow", "Principal": {"Service": "cloudtrail.amazonaws.com"}, "Action": "s3:GetBucketAcl", "Resource": "arn:aws:s3:::s3-for-cloudtrail-logs" }, { "Sid": "AWSCloudTrailWrite20150319", "Effect": "Allow", "Principal": {"Service": "cloudtrail.amazonaws.com"}, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::s3-for-cloudtrail-logs/*", "Condition": {"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}} } ] } EOF ## Create a S3 bucket policy for CloudTrail aws s3api put-bucket-policy \ --bucket s3-for-cloudtrail-logs \ --policy file://bucket_policy.json |
Step 2: Create a new regional Trail
|
1 2 3 4 5 6 7 |
## Create a new regional trail aws cloudtrail create-trail \ --name RegionalTrail \ --s3-bucket-name s3-for-cloudtrail-logs \ --no-include-global-service-events \ --no-is-multi-region-trail \ --no-enable-log-file-validation |
Step 3: Enable event selector for your Trail.
Note: This is not a prerequisite to integrate CloudTrail with CloudWatch. But without this filter, all the management events (lots of it) will be pushed to our CloudWatch logs which is not a problem, but for simplicity and cost I am limiting only data events and that too for only on specific bucket object operations.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
## Create advance event selector definition cat <<'EOF'> event_selector.json [ { "Name": "Log PutObject and DeleteObject events for two S3 prefixes", "FieldSelectors": [ { "Field": "eventCategory", "Equals": ["Data"] }, { "Field": "resources.type", "Equals": ["AWS::S3::Object"] }, { "Field": "eventName", "Equals": ["GetObject","PutObject","DeleteObject"] }, { "Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::s3-for-cloudtrail-logs/myapp"] } ] } ] EOF ## Create new event selector for your trail aws cloudtrail put-event-selectors \ --trail-name RegionalTrail \ --advanced-event-selectors file://event_selector.json |
Step 4: Create a new log group in CloudWatch to store CloudTrail events.
|
1 2 3 4 5 |
## Create a CloudWatch log group aws logs create-log-group \ --log-group-name CloudTrail/mys3objectlogs Next, we need to provide access to CloudTrail in the above CloudWatch log group to put events. |
Step 5: Create IAM role and policy for CloudTrail to assume.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
## Create an IAM role for CloudTrail to send events to CloudWatch logs ## Create assume role policy definition cat <<'EOF'> myiamasumpolicy.json { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "cloudtrail.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF ## Create IAM role aws iam create-role \ --role-name cloudtrail_iam_role \ --assume-role-policy-document file://myiamasumpolicy.json ## Create IAM policy definition ACN_ID=$(aws sts get-caller-identity | jq -r .Account) && cat < { "Version": "2012-10-17", "Statement": [ { "Sid": "AWSCloudTrailCreateLogStream2014110", "Effect": "Allow", "Action": [ "logs:CreateLogStream" ], "Resource": [ "arn:aws:logs:ap-south-1:$ACN_ID:log-group:CloudTrail/mys3objectlogs:log-stream:${ACN_ID}_CloudTrail_ap-south-1*" ] }, { "Sid": "AWSCloudTrailPutLogEvents20141101", "Effect": "Allow", "Action": [ "logs:PutLogEvents" ], "Resource": [ "arn:aws:logs:ap-south-1:$ACN_ID:log-group:CloudTrail/mys3objectlogs:log-stream:${ACN_ID}_CloudTrail_ap-south-1*" ] } ] } EOF ## Update the IAM role with the above IAM policy aws iam put-role-policy \ --role-name cloudtrail_iam_role \ --policy-name cloudtrail_iam_policy \ --policy-document file://myiampolicy.json |
Now we are ready to update our Trail to send events to CloudWatch logs.
Step 6: Update CloudTrail trail to send events to CloudWatch logs and start CloudTrail logging.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
## Update CloudTrail to send events to CloudWatch Logs IAM_ROLE_ARN=$(aws iam get-role \ --role-name cloudtrail_iam_role | jq -r .Role.Arn) && LOG_GROUP_ARN=$(aws logs describe-log-groups \ --log-group-name-prefix CloudTrail/mys3objectlogs | jq -r .logGroups[0].arn) && aws cloudtrail update-trail \ --name RegionalTrail \ --cloud-watch-logs-log-group-arn $LOG_GROUP_ARN \ --cloud-watch-logs-role-arn $IAM_ROLE_ARN ## Start logging for the trail aws cloudtrail start-logging \ --name RegionalTrail |
Next, we are going to perform some API actions in our S3 bucket (mentioned in event selector definition: step 3) to generate CloudTrail events.
Step 7: Put an object in your S3 bucket to generate CloudTrail event.
|
1 2 3 4 5 6 |
## Put one object to specified location as mentioned in event selector echo "hello world" > object1.txt && aws s3api put-object \ --bucket s3-for-cloudtrail-logs \ --key myapp/object1.txt \ --body object1.txt |
Due to the above action, CloudTrail will generate new events and push to CloudWatch Logs and S3 bucket. Wait for 15 mins and then check the S3 bucket and CloudWatch Logs.
Step 8: Check CloudWatch Logs and S3 for CloudTrail events
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
## Check if any CloudTrail logs are dilivered to the S3 bcuket aws s3api list-objects \ --bucket s3-for-cloudtrail-logs \ --prefix "AWSLogs/$ACN_ID/CloudTrail" ## CloudTrail typically delivers logs within an average of about 15 ## minutes of an API call. This time is not garunteed. ## Check if any CloudTrail events dilivered to CloudWatch Logs LOG_STREAM_NAME=${ACN_ID}_CloudTrail_ap-south-1 && aws logs get-log-events \ --log-group-name CloudTrail/mys3objectlogs \ --log-stream-name $LOG_STREAM_NAME \ --limit 2 |
We have successfully integrated CloudTrail with CloudWatch, which opens up endless possibility of automation. You can now create an event rule on CloudWatch logs or Create a CloudWatch Alarm on matrices using log event filters (Topic for another blog maybe 😊)
Step 9: Clean up.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
## Stop logging for your Trail aws cloudtrail stop-logging \ --name RegionalTrail ## Delete the Trail aws cloudtrail delete-trail \ --name RegionalTrail ## Delete IAM Role & Policy aws iam delete-role-policy \ --role-name cloudtrail_iam_role \ --policy-name cloudtrail_iam_policy && aws iam delete-role \ --role-name cloudtrail_iam_role ## Delete CloufWatch Log Group and Stream aws logs delete-log-stream \ --log-group-name CloudTrail/mys3objectlogs \ --log-stream-name $LOG_STREAM_NAME && aws logs delete-log-group \ --log-group-name CloudTrail/mys3objectlogs ## Delete the S3 bucket with objects (CloudTrail Logs) aws s3 rb \ s3://s3-for-cloudtrail-logs --force |
Hope you have enjoyed this article. To know more about AWS CloudTrail, please refer below official documentation
https://docs.aws.amazon.com/cloudtrail/index.html