How to Copy Files Between S3 Buckets in 2 Different Accounts Using Boto3?

Introduction

Managing data across different AWS accounts is a common scenario in today’s distributed and multi-account AWS environments. You might need to copy files between S3 buckets located in different accounts, whether for data synchronization, backup, or other purposes. In this post, we’ll dive into how you can achieve this using Boto3, a Python library provided by AWS.

Concepts Used

Amazon S3 (Simple Storage Service)

Amazon S3 is a scalable object storage service in AWS, allowing for the storage and retrieval of any amount of data.

Boto3

Boto3 is the Amazon Web Services (AWS) SDK for Python, enabling Python developers to write software that uses services like Amazon S3.

Cross-Account Access

This refers to accessing resources between different AWS accounts, typically requiring specific permissions and roles.

Step-by-Step Guide: Copying Files Between S3 Buckets in Different Accounts

Step 1: Set Up IAM Roles and Policies

Create IAM roles in both source and destination accounts that have the necessary permissions to read from the source bucket and write to the destination bucket.

Step 2: Assume the Necessary Role in the Source Account

Using Boto3, you can assume the role that has read access to the source bucket:

import boto3

sts_client = boto3.client('sts')
assumed_role_object = sts_client.assume_role(
    RoleArn="arn:aws:iam::SOURCE_ACCOUNT_ID:role/SourceBucketAccessRole",
    RoleSessionName="AssumeRoleSession1"
)

credentials = assumed_role_object['Credentials']

import boto3

sts_client = boto3.client('sts')

assumed_role_object = sts_client.assume_role(

RoleArn="arn:aws:iam::SOURCE_ACCOUNT_ID:role/SourceBucketAccessRole",

RoleSessionName="AssumeRoleSession1"

)

credentials = assumed_role_object['Credentials']

Step 3: Create S3 Client for Source Account

Using the credentials obtained from assuming the role, create an S3 client for the source account:

s3_source_client = boto3.client(
    's3',
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken'],
)

s3_source_client = boto3.client(

's3',

aws_access_key_id=credentials['AccessKeyId'],

aws_secret_access_key=credentials['SecretAccessKey'],

aws_session_token=credentials['SessionToken'],

)

Step 4: Assume the Necessary Role in the Destination Account

Similar to step 2, assume the role for the destination account that has write access to the target bucket.

Step 5: Create S3 Client for Destination Account

Create an S3 client for the destination account, just as you did for the source:

s3_dest_client = boto3.client(
    's3',
    aws_access_key_id=dest_credentials['AccessKeyId'],
    aws_secret_access_key=dest_credentials['SecretAccessKey'],
    aws_session_token=dest_credentials['SessionToken'],
)

s3_dest_client = boto3.client(

's3',

aws_access_key_id=dest_credentials['AccessKeyId'],

aws_secret_access_key=dest_credentials['SecretAccessKey'],

aws_session_token=dest_credentials['SessionToken'],

)

Step 6: Copy the Object Between Buckets

Now that you have both clients, use them to copy the object from the source to the destination bucket:

copy_source = {
    'Bucket': 'my-source-bucket',
    'Key': 'my-file.txt'
}
s3_dest_client.copy_object(Bucket='my-destination-bucket', CopySource=copy_source, Key='my-file.txt')

copy_source = {

'Bucket': 'my-source-bucket',

'Key': 'my-file.txt'

}

s3_dest_client.copy_object(Bucket='my-destination-bucket', CopySource=copy_source, Key='my-file.txt')

Conclusion

Copying files between S3 buckets across different AWS accounts involves assuming appropriate IAM roles and utilizing Boto3. By following the step-by-step guide outlined above, developers can smoothly handle this operation, fulfilling various use cases like data migration, synchronization, or sharing between different parts of an organization. Always ensure that the proper permissions are in place to maintain security and access control.