How To Configure Access Logs In Application Load Balancer
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed sticky sessions in Application Load Balancer.
https://cloudaffaire.com/how-to-configure-sticky-sessions-in-application-load-balancer/
In this blog post, we will discuss Access Logs in Application Load Balancer.
How To Configure Access Logs In Application Load Balancer:
What is Access Logs?
Elastic Load Balancing provides access logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the time the request was received, the client’s IP address, latencies, request paths, and server responses. You can use these access logs to analyze traffic patterns and troubleshoot issues.
Access logging is an optional feature of Elastic Load Balancing that is disabled by default. After you enable access logging for your load balancer, Elastic Load Balancing captures the logs and stores them in the Amazon S3 bucket that you specify as compressed files. You can disable access logging at any time.
Each access log file is automatically encrypted using SSE-S3 before it is stored in your S3 bucket and decrypted when you access it. You do not need to take any action; the encryption and decryption is performed transparently.
There is no additional charge for access logs. You are charged storage costs for Amazon S3, but not charged for the bandwidth used by Elastic Load Balancing to send log files to Amazon S3.
Access Log Files:
Elastic Load Balancing publishes a log file for each load balancer node every 5 minutes. Log delivery is eventually consistent. The load balancer can deliver multiple logs for the same period. This usually happens if the site has high traffic. The file names of the access logs use the following format:
|
1 |
bucket[/prefix]/AWSLogs/aws-account-id/elasticloadbalancing/region/yyyy/mm/dd/aws-account-id_elasticloadbalancing_region_load-balancer-id_end-time_ip-address_random-string.log.gz |
Access Log Entries:
Elastic Load Balancing logs requests sent to the load balancer, including requests that never made it to the targets. For example, if a client sends a malformed request, or there are no healthy targets to respond to the request, the request is still logged. Note that Elastic Load Balancing does not log health check requests.
Each log entry contains the details of a single request (or connection in the case of WebSockets) made to the load balancer. For WebSockets, an entry is written only after the connection is closed. If the upgraded connection can’t be established, the entry is the same as for an HTTP or HTTPS request.
Bucket Permissions:
When you enable access logging, you must specify an S3 bucket for the access logs. The bucket must meet the following requirements.
Requirements
- The bucket must be located in the same Region as the load balancer.
- The bucket must have a bucket policy that grants Elastic Load Balancing permission to write the access logs to your bucket.
Access Logs Demo:
Step 1: Create a custom VPC for your ALB.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
############################################## ## Access Logs In Application Load Balancer ## ############################################## ## Prerequisite: AWS CLI installed and configured with proper access ## https://cloudaffaire.com/category/aws/aws-cli/ ##---------------------------------- ## Create custom vpc for your alb ## ##---------------------------------- ## Create a directory for this demo mkdir aws_alb_access_logs_demo && cd aws_alb_access_logs_demo ## Create a VPC with DNS hostname enabled AWS_VPC_ID=$(aws ec2 create-vpc \ --cidr-block 10.0.0.0/16 \ --query 'Vpc.{VpcId:VpcId}' \ --output text) && aws ec2 modify-vpc-attribute \ --vpc-id $AWS_VPC_ID \ --enable-dns-hostnames "{\"Value\":true}" ## Create two public subnets AWS_SUBNET_PUBLIC_ONE_ID=$(aws ec2 create-subnet \ --vpc-id $AWS_VPC_ID --cidr-block 10.0.1.0/24 \ --availability-zone ap-south-1a --query 'Subnet.{SubnetId:SubnetId}' \ --output text) && AWS_SUBNET_PUBLIC_TWO_ID=$(aws ec2 create-subnet \ --vpc-id $AWS_VPC_ID --cidr-block 10.0.2.0/24 \ --availability-zone ap-south-1b --query 'Subnet.{SubnetId:SubnetId}' \ --output text) && aws ec2 modify-subnet-attribute \ --subnet-id $AWS_SUBNET_PUBLIC_ONE_ID \ --map-public-ip-on-launch && aws ec2 modify-subnet-attribute \ --subnet-id $AWS_SUBNET_PUBLIC_TWO_ID \ --map-public-ip-on-launch ## Create an Internet Gateway AWS_INTERNET_GATEWAY_ID=$(aws ec2 create-internet-gateway \ --query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \ --output text) && aws ec2 attach-internet-gateway \ --vpc-id $AWS_VPC_ID \ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID ## Create a route table AWS_CUSTOM_ROUTE_TABLE_ID=$(aws ec2 create-route-table \ --vpc-id $AWS_VPC_ID \ --query 'RouteTable.{RouteTableId:RouteTableId}' \ --output text ) && aws ec2 create-route \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \ --destination-cidr-block 0.0.0.0/0 \ --gateway-id $AWS_INTERNET_GATEWAY_ID ## Associate the public subnet with route table AWS_ROUTE_TABLE_ASSOID_ONE=$(aws ec2 associate-route-table \ --subnet-id $AWS_SUBNET_PUBLIC_ONE_ID \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \ --query 'AssociationId' \ --output text) && AWS_ROUTE_TABLE_ASSOID_TWO=$(aws ec2 associate-route-table \ --subnet-id $AWS_SUBNET_PUBLIC_TWO_ID \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \ --query 'AssociationId' \ --output text) ## Create a security group aws ec2 create-security-group \ --vpc-id $AWS_VPC_ID \ --group-name myvpc-security-group \ --description 'My VPC non default security group' ## Get security group ID's AWS_DEFAULT_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups \ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \ --query 'SecurityGroups[?GroupName == `default`].GroupId' \ --output text) && AWS_CUSTOM_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups \ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \ --query 'SecurityGroups[?GroupName == `myvpc-security-group`].GroupId' \ --output text) ## Create security group ingress rules aws ec2 authorize-security-group-ingress \ --group-id $AWS_CUSTOM_SECURITY_GROUP_ID \ --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow SSH"}]}]' && aws ec2 authorize-security-group-ingress \ --group-id $AWS_CUSTOM_SECURITY_GROUP_ID \ --ip-permissions '[{"IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow HTTP"}]}]' |
Step 2: Create two EC2 instances for your Application Load Balancer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
##-------------------------------------------------- ## Create two ec2 instances your alb target group ## ##-------------------------------------------------- ## Get Amazon Linux 2 latest AMI ID AWS_AMI_ID=$(aws ec2 describe-images \ --owners 'amazon' \ --filters 'Name=name,Values=amzn2-ami-hvm-2.0.????????-x86_64-gp2' 'Name=state,Values=available' \ --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \ --output 'text') ## Create a key-pair aws ec2 create-key-pair \ --key-name myvpc-keypair \ --query 'KeyMaterial' \ --output text > myvpc-keypair.pem && chmod 400 myvpc-keypair.pem ## Create user data to configure LAMP stack vi myuserdataone.txt ----------------------- #!/bin/bash sudo yum install -y httpd sudo systemctl start httpd sudo usermod -a -G apache ec2-user sudo chown -R ec2-user:apache /var/www sudo chmod 2775 /var/www sudo find /var/www -type d -exec chmod 2775 {} \; sudo find /var/www -type f -exec chmod 0664 {} \; sudo echo "hello from instance one" > /var/www/html/index.html ----------------------- :wq vi myuserdatatwo.txt ----------------------- #!/bin/bash sudo yum install -y httpd sudo systemctl start httpd sudo usermod -a -G apache ec2-user sudo chown -R ec2-user:apache /var/www sudo chmod 2775 /var/www sudo find /var/www -type d -exec chmod 2775 {} \; sudo find /var/www -type f -exec chmod 0664 {} \; sudo echo "hello from instance two" > /var/www/html/index.html ----------------------- :wq ## Create two EC2 instance in two public subnet AWS_EC2_INSTANCE_ONE_ID=$(aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-type t2.micro \ --key-name myvpc-keypair \ --monitoring "Enabled=false" \ --security-group-ids $AWS_CUSTOM_SECURITY_GROUP_ID \ --subnet-id $AWS_SUBNET_PUBLIC_ONE_ID \ --user-data file://myuserdataone.txt \ --private-ip-address 10.0.1.10 \ --query 'Instances[0].InstanceId' \ --output text) && AWS_EC2_INSTANCE_TWO_ID=$(aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-type t2.micro \ --key-name myvpc-keypair \ --monitoring "Enabled=false" \ --security-group-ids $AWS_CUSTOM_SECURITY_GROUP_ID \ --subnet-id $AWS_SUBNET_PUBLIC_TWO_ID \ --user-data file://myuserdatatwo.txt \ --private-ip-address 10.0.2.10 \ --query 'Instances[0].InstanceId' \ --output text) |
Step 3: Create your Application Load Balancer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
##------------------------------------ ## Create application load balancer ## ##------------------------------------ ## Create the application load balancer AWS_ALB_ARN=$(aws elbv2 create-load-balancer \ --name my-application-load-balancer \ --subnets $AWS_SUBNET_PUBLIC_ONE_ID $AWS_SUBNET_PUBLIC_TWO_ID \ --security-groups $AWS_CUSTOM_SECURITY_GROUP_ID \ --query 'LoadBalancers[0].LoadBalancerArn' \ --output text) ## Check the status of load balancer aws elbv2 describe-load-balancers \ --load-balancer-arns $AWS_ALB_ARN \ --query 'LoadBalancers[0].State.Code' \ --output text ## Once the ALB status is active, get the DNS name for your ALB AWS_ALB_DNS=$(aws elbv2 describe-load-balancers \ --load-balancer-arns $AWS_ALB_ARN \ --query 'LoadBalancers[0].DNSName' \ --output text) && echo $AWS_ALB_DNS ## Create the target group for your ALB AWS_ALB_TARGET_GROUP_ARN=$(aws elbv2 create-target-group \ --name my-alb-targets \ --protocol HTTP --port 80 \ --vpc-id $AWS_VPC_ID \ --query 'TargetGroups[0].TargetGroupArn' \ --output text) ## Register both the instances in the target group aws elbv2 register-targets --target-group-arn $AWS_ALB_TARGET_GROUP_ARN \ --targets Id=$AWS_EC2_INSTANCE_ONE_ID Id=$AWS_EC2_INSTANCE_TWO_ID ## Create a listener for your load balancer with a default rule that forwards requests to your target group AWS_ALB_LISTNER_ARN=$(aws elbv2 create-listener --load-balancer-arn $AWS_ALB_ARN \ --protocol HTTP --port 80 \ --default-actions Type=forward,TargetGroupArn=$AWS_ALB_TARGET_GROUP_ARN \ --query 'Listeners[0].ListenerArn' \ --output text) ## Verify the health of the registered targets for your target group aws elbv2 describe-target-health --target-group-arn $AWS_ALB_TARGET_GROUP_ARN |
Step 4: Configure Access Logs for your Application Load Balancer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
##--------------------------------- ## Configure Access Logs For ALB ## ##--------------------------------- ## Create a S3 bucket to store the access logs ## Bucket name must be globally unique aws s3api create-bucket \ --bucket my-cloudaffaire-alb-access-log-demo-bucket \ --region ap-south-1 \ --create-bucket-configuration LocationConstraint=ap-south-1 ## Create bucket policy to grant ELB permission to write the access logs to your bucket ## Replace aws-account-id and bucket-name in below policy ## bucket-name = Replace my-cloudaffaire-alb-access-log-demo-bucket with your bucket name ## aws-account-id = Elastic Load Balancing Account ID (Check below link for details) ## https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-logging-bucket-permissions vi my_bucket_policy.json --------------------- { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:: }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::my-cloudaffaire-alb-access-log-demo-bucket/*" } ] } --------------------- :wq ## Apply the bucket policy to your S3 bucket ## Replace my-cloudaffaire-alb-access-log-demo-bucket with your bucket name aws s3api put-bucket-policy \ --bucket my-cloudaffaire-alb-access-log-demo-bucket \ --policy file://my_bucket_policy.json ## Enable access logging for your ALB aws elbv2 modify-load-balancer-attributes \ --load-balancer-arn $AWS_ALB_ARN \ --attributes Key=access_logs.s3.enabled,Value=true \ Key=access_logs.s3.bucket,Value=my-cloudaffaire-alb-access-log-demo-bucket ## Open the DNS name of your ALB (below output) in your browser and hit refresh several time echo $AWS_ALB_DNS ## Example entry http 2019-12-29T12:24:25.116077Z app/my-application-load-balancer/3d2f7f8613b8ba30 43.252.249.116:43082 192.168.1.10:80 -1 -1 -1 502 - 132 293 "GET http://my-application-load-balancer-1794743675.ap-south-1.elb.amazonaws.com:80/ HTTP/1.1" "curl/7.29.0" - - arn:aws:elasticloadbalancing:ap-south-1:XXXXXXXXXXXXX:targetgroup/my-alb-targets/ca70844a705eb677 "Root=1-5e089af8-6ece94386b8ee03805536470" "-" "-" 0 2019-12-29T12:24:24.996000Z "forward" "-" "-" "192.168.1.10:80" "-" http 2019-12-29T12:24:21.243742Z app/my-application-load-balancer/3d2f7f8613b8ba30 43.252.249.116:43080 192.168.2.10:80 -1 -1 -1 502 - 132 293 "GET http://my-application-load-balancer-1794743675.ap-south-1.elb.amazonaws.com:80/ HTTP/1.1" "curl/7.29.0" - - arn:aws:elasticloadbalancing:ap-south-1:XXXXXXXXXXXXX:targetgroup/my-alb-targets/ca70844a705eb677 "Root=1-5e089af5-333400966ee6e9c38280f905" "-" "-" 0 2019-12-29T12:24:21.123000Z "forward" "-" "-" "192.168.2.10:80" "-" |
Step 5: Cleanup.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
## Delete the listener and Deregister targets aws elbv2 delete-listener \ --listener-arn $AWS_ALB_LISTNER_ARN && aws elbv2 deregister-targets \ --target-group-arn $AWS_ALB_TARGET_GROUP_ARN \ --targets Id=$AWS_EC2_INSTANCE_ONE_ID Id=$AWS_EC2_INSTANCE_TWO_ID ## Delete target group and ALB aws elbv2 delete-target-group \ --target-group-arn $AWS_ALB_TARGET_GROUP_ARN && aws elbv2 delete-load-balancer \ --load-balancer-arn $AWS_ALB_ARN ## Terminate the ec2 instances aws ec2 terminate-instances \ --instance-ids $AWS_EC2_INSTANCE_ONE_ID && aws ec2 terminate-instances \ --instance-ids $AWS_EC2_INSTANCE_TWO_ID ## Delete key pair aws ec2 delete-key-pair \ --key-name myvpc-keypair ## Delete custom security group (once instances are terminated) aws ec2 delete-security-group \ --group-id $AWS_CUSTOM_SECURITY_GROUP_ID ## Delete internet gateway aws ec2 detach-internet-gateway \ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID \ --vpc-id $AWS_VPC_ID && aws ec2 delete-internet-gateway \ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID ## Disassociate the subnets from custom route table aws ec2 disassociate-route-table \ --association-id $AWS_ROUTE_TABLE_ASSOID_ONE && aws ec2 disassociate-route-table \ --association-id $AWS_ROUTE_TABLE_ASSOID_TWO ## Delete custom route table aws ec2 delete-route-table \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID ## Delete the public subnets aws ec2 delete-subnet \ --subnet-id $AWS_SUBNET_PUBLIC_ONE_ID && aws ec2 delete-subnet \ --subnet-id $AWS_SUBNET_PUBLIC_TWO_ID ## Delete the vpc aws ec2 delete-vpc \ --vpc-id $AWS_VPC_ID ## Empty the S3 bucket and delete the bucket aws s3 rm s3://my-cloudaffaire-alb-access-log-demo-bucket/ --recursive && aws s3api delete-bucket --bucket my-cloudaffaire-alb-access-log-demo-bucket ## Delete the directory used in this demo cd .. && rm -rf aws_alb_access_logs_demo |
Hope you have enjoyed this article, In the next blog post, we will discuss Network Load Balancer in AWS.
All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.
To get more details on AWS ELB, please refer below AWS documentation
https://docs.aws.amazon.com/elasticloadbalancing/index.html
An error occurred (InvalidConfigurationRequest) when calling the ModifyLoadBalancerAttributes operation: Access Denied for bucket: my-solytics-alb-access-log-bucket. Please check S3bucket permission
i am facing this error.