How to backup and restore an S3 bucket using AWS Backup Service?
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we will discuss how to backup and restore an S3 bucket using the AWS Backup service.
Prerequisites:
AWS CLI installed and configured.
How to take backup and restore an S3 bucket using AWS Backup Service?
Step 1: Create a KMS key to encrypt your backups in the AWS backup vault.
|
1 2 3 4 |
## Create a new KMS key KMS_KEY_ARN=$(aws kms create-key \ --tags TagKey=Purpose,TagValue=BackupVault \ --description "Used to encrypt backup vault" | jq -r .KeyMetadata.Arn) |
Step 2: Create a new backup vault.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
## Create a new backup vault aws backup create-backup-vault \ --backup-vault-name myvault \ --encryption-key-arn $KMS_KEY_ARN \ --backup-vault-tags sla=gold ## Create an access policy for backup vault USER_ARN=$(aws sts get-caller-identity | jq -r .Arn) && ACCOUNT_ID=$(aws sts get-caller-identity | jq -r .Account) && cat << EOF > backup_access_policy.json { "Version": "2012-10-17", "Statement": [ { "Sid": "statement ID", "Effect": "Deny", "Principal": "*", "Action": "backup:DeleteRecoveryPoint", "Resource": "*", "Condition": { "StringNotLike": { "aws:PrincipalArn": [ "$USER_ARN", "arn:aws:iam::$ACCOUNT_ID:root" ] } } } ] } EOF ## Add the access policy to the vault aws backup put-backup-vault-access-policy \ --backup-vault-name myvault \ --policy file://backup_access_policy.json |
Step 3: Create an IAM role that will be used by the AWS backup service to backup your S3 bucket.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
## Create assume role policy definition cat <<'EOF'> backup_assume_role_policy.json { "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "backup.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF ## Create IAM role aws iam create-role \ --role-name backup_iam_role \ --assume-role-policy-document file://backup_assume_role_policy.json ## Create an IAM policy file cat << EOF > backup_policy.json { "Version":"2012-10-17", "Statement":[ { "Sid":"Backup", "Action":[ "s3:*", "kms:*", "events:*", "cloudwatch:*", "ssm:*", "dynamodb:*", "ec2:*", "storagegateway:*", "rds:*", "elasticfilesystem:*", "fsx:*", "ds:*", "backup:*", "tag:*" ], "Effect":"Allow", "Resource":"*" } ] } EOF ## Create the IAM policy IAM_POLICY_ARN=$(aws iam create-policy \ --policy-name backup_policy \ --policy-document file://backup_policy.json | jq -r .Policy.Arn) && echo $IAM_POLICY_ARN ## Attach the IAM policy to the IAM role aws iam attach-role-policy \ --role-name backup_iam_role \ --policy-arn $IAM_POLICY_ARN ## Get the role ARN ACCOUNT_ID=$(aws sts get-caller-identity | jq -r .Account) && ACCOUNT_ARN=$(aws sts get-caller-identity | jq -r .Arn) && IAM_ROLE_ARN=arn:aws:iam::$ACCOUNT_ID:role/backup_iam_role && echo $IAM_ROLE_ARN |
Step 4: Create a new S3 bucket with some objects that will be backed up by the AWS backup service.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
## Create an S3 bucket aws s3api create-bucket \ --bucket cloudaffaire-s3-backup-demo \ --region ap-south-1 \ --create-bucket-configuration LocationConstraint=ap-south-1 && S3_BUCKET_ARN='arn:aws:s3:::cloudaffaire-s3-backup-demo' ## Create a s3 bucket policy definition file cat << EOF > bucket_policy.json { "Version": "2012-10-17", "Statement": [ { "Sid": "S3Access", "Principal": {"AWS": ["$ACCOUNT_ARN"]}, "Effect": "Allow", "Action": ["s3:*"], "Resource": ["$S3_BUCKET_ARN/*"] } ] } EOF ## Create a s3 bucket policy aws s3api put-bucket-policy \ --bucket cloudaffaire-s3-backup-demo \ --policy file://bucket_policy.json ## Create an object to the S3 bcuket echo "hello world" > hello.txt && aws s3api put-object \ --bucket cloudaffaire-s3-backup-demo \ --key hello.txt \ --body hello.txt ## Enable S3 bucket versioning aws s3api put-bucket-versioning \ --bucket cloudaffaire-s3-backup-demo \ --versioning-configuration Status=Enabled |
Note: In order for the backup service to take backup of an S3 bucket, versioning must be enabled to the S3 bucket.
Step 5: Take a backup of the S3 bucket using the AWS backup service.
|
1 2 3 4 5 6 7 |
## Create a backup of the s3 bucket aws backup start-backup-job \ --backup-vault-name myvault \ --resource-arn $S3_BUCKET_ARN \ --iam-role-arn $IAM_ROLE_ARN \ --start-window-minutes 60 \ --complete-window-minutes 120 |
Step 6: Get details on the backup.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
## List backup jobs aws backup list-backup-jobs ## Once the "State": "COMPLETED" ## List recovery point by backup vault aws backup list-recovery-points-by-backup-vault \ --backup-vault-name myvault \ --by-resource-arn $S3_BUCKET_ARN ## Get protected resource details PROTECTED_RESOURCE_ARN=$(aws backup list-protected-resources | jq -r .Results[0].ResourceArn) && aws backup describe-protected-resource \ --resource-arn $PROTECTED_RESOURCE_ARN ## Get recovery point details RECOVERY_POINT_ARN=$(aws backup list-recovery-points-by-backup-vault \ --backup-vault-name myvault \ --by-resource-arn $S3_BUCKET_ARN | jq -r .RecoveryPoints[0].RecoveryPointArn) && aws backup describe-recovery-point \ --backup-vault-name myvault \ --recovery-point-arn $RECOVERY_POINT_ARN |
Next, we will delete the S3 bucket and perform a full restore of the S3 bucket along with its content (objects) using the backup taken in step 5.
Step 7: Delete the S3 bucket to simulate a disaster.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
## Delete the S3 bucket with objects aws s3api put-bucket-versioning \ --bucket cloudaffaire-s3-backup-demo \ --versioning-configuration Status=Suspended && VERSION_ID=$(aws s3api list-object-versions \ --bucket cloudaffaire-s3-backup-demo \ --prefix hello.txt \ --query "Versions[].VersionId" \ --output json | jq 'unique' | jq -r '.[]') && aws s3api delete-object \ --bucket cloudaffaire-s3-backup-demo \ --key hello.txt \ --version-id $VERSION_ID && aws s3 rb \ s3://cloudaffaire-s3-backup-demo --force |
Step 8: Restore the S3 bucket along with content using the AWS backup service.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
## Get restore metadata aws backup get-recovery-point-restore-metadata \ --backup-vault-name myvault \ --recovery-point-arn $RECOVERY_POINT_ARN ## Create a restore point metadata definition file cat << EOF > backup_restore_metadata.json { "DestinationBucketName": "cloudaffaire-s3-backup-demo", "ItemsToRestore": "[\"hello.txt\"]", "NewBucket": "true", "CreationToken": "jsfd3sjdgf6532", "Encrypted": "false", "EncryptionType": "original" } EOF ## Start the restore job aws backup start-restore-job \ --recovery-point-arn $RECOVERY_POINT_ARN \ --metadata file://backup_restore_metadata.json \ --iam-role-arn $IAM_ROLE_ARN \ --resource-type S3 |
Note: You can also restore individual objects.
Observe, the S3 bucket has been restored again.
Next, we will create a backup plan for scheduled backup of the S3 bucket using the AWS backup service.
Step 9: Create a backup plan to backup the AWS S3 bucket at regular intervals.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
## Create a new backup plan definition cat << EOF > backup_plan.json { "BackupPlanName": "mybackupplan", "Rules": [ { "RuleName": "DailyBackups", "TargetBackupVaultName": "myvault", "ScheduleExpression": "cron(0 5 ? * * *)", "StartWindowMinutes": 60, "CompletionWindowMinutes": 1400, "CopyActions": [], "EnableContinuousBackup": false, "Lifecycle": { "DeleteAfterDays": 7 } } ] } EOF ## Create a new backup plan aws backup create-backup-plan \ --backup-plan file://backup_plan.json ## Get backup plan details BACKUP_PLAN_ID=$(aws backup list-backup-plans | jq -r .BackupPlansList[0].BackupPlanId) && aws backup get-backup-plan \ --backup-plan-id $BACKUP_PLAN_ID ## Create a backup selection definition cat << EOF > backup_selection.json { "SelectionName": "mybackupselection", "IamRoleArn": "$IAM_ROLE_ARN", "Resources": ["$S3_BUCKET_ARN"], "ListOfTags": [ { "ConditionType": "STRINGEQUALS", "ConditionKey": "Backup", "ConditionValue": "Daily" } ] } EOF ## Create a backup selection for your backup plan aws backup create-backup-selection \ --backup-plan-id $BACKUP_PLAN_ID \ --backup-selection file://backup_selection.json ## Get details on the backup selection BACKUP_SELECTION_ID=$(aws backup list-backup-selections \ --backup-plan-id $BACKUP_PLAN_ID | jq -r .BackupSelectionsList[0].SelectionId) && aws backup get-backup-selection \ --backup-plan-id $BACKUP_PLAN_ID \ --selection-id $BACKUP_SELECTION_ID ## Add a tag to the S3 bucket aws s3api put-bucket-tagging \ --bucket cloudaffaire-s3-backup-demo \ --tagging 'TagSet=[{Key=Backup,Value="Daily"}]' ## Wait for the backup to run (05:00 AM UTC) |
Step 10: Clean up.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
## Delete the backup selection aws backup delete-backup-selection \ --backup-plan-id $BACKUP_PLAN_ID \ --selection-id $BACKUP_SELECTION_ID ## Delete the backup plan aws backup delete-backup-plan \ --backup-plan-id $BACKUP_PLAN_ID ## Delete the recovery point aws backup delete-recovery-point \ --backup-vault-name myvault \ --recovery-point-arn $RECOVERY_POINT_ARN ## Delete the access policy for backup vault aws backup delete-backup-vault-access-policy \ --backup-vault-name myvault ## Delete the backup vault aws backup delete-backup-vault \ --backup-vault-name myvault ## Delete the IAM role aws iam detach-role-policy \ --role-name backup_iam_role \ --policy-arn $IAM_POLICY_ARN && aws iam delete-policy \ --policy-arn $IAM_POLICY_ARN && aws iam delete-role \ --role-name backup_iam_role ## Delete the S3 bucket with objects aws s3api put-bucket-versioning \ --bucket cloudaffaire-s3-backup-demo \ --versioning-configuration Status=Suspended && VERSION_ID=$(aws s3api list-object-versions \ --bucket cloudaffaire-s3-backup-demo \ --prefix hello.txt \ --query "Versions[].VersionId" \ --output json | jq 'unique' | jq -r '.[]') && aws s3api delete-object \ --bucket cloudaffaire-s3-backup-demo \ --key hello.txt \ --version-id $VERSION_ID && aws s3 rb \ s3://cloudaffaire-s3-backup-demo --force |
Hope you have enjoyed this article. To get more details in AWS S3 and backup service, please refer the below documentations
https://docs.aws.amazon.com/s3/index.html
https://docs.aws.amazon.com/aws-backup/index.html