Question:
Is it possible to copy from one AWS accounts S3 bucket into another AWS accounts Redshift cluster? The way I tried to do it was to log in using SQL Workbench to my AWS Account (Account1) and used a IAM User of (Account2) to copy the file over like this:
|
1 2 3 4 5 |
copy my_table (town,name,number) from 's3://other-s3-account-bucket/fileToCopy.tsv' credentials 'aws_access_key_id= delimiter '\t'; |
I know the other account’s user has s3 permissions after double checking. Do I have share IAM users or setup different permissions in order to do this?
Answer:
You will need to “pull” the data from the other account’s S3 bucket.
- AWS Account A has an S3 bucket called
source-bucket-account-a. - AWS Account B has a Redshift cluser called
TargetCluster. - On bucket
source-bucket-account-a, add a bucket policy allowing AWS Account B to read files.
A sample policy:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
{ "Version": "2012-10-17", "Statement": [ { "Sid": "DelegateS3Access", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:: }, "Action": [ "s3:Get*", "s3:List*" ], "Resource": [ "arn:aws:s3:::source-bucket-account-a", "arn:aws:s3:::source-bucket-account-a/*" ] } ] } |
It’s very similar to the following:
http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example2.html
or the following:
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_policy-examples.html
- Once the bucket policy is in place, you use the credentials for AWS Account B to run the
copycommand because it owns the Redshift cluster. In thecopycommand, you specify the bucket by it’s namesource-bucket-account-a.
The bucket policy has granted read access to AWS Account B so it can “pull” the data into Redshift.