Question:
I have to write a rails task for getting files from s3 bucket but my bucket have more than 1000 object.
|
1 2 3 4 5 6 7 |
. #Connection codes and configures . bucket = AWS::S3::Bucket.find('my_bucket') puts bucket.size # => 1000 |
this code just give me 1000 objects 🙁
how can i get my all objects from s3 bucket ?
Answer:
As stated in the S3 developer documentation:
To manage large result sets, Amazon S3 uses pagination to split them
into multiple responses. Each list keys response returns a page of up
to 1,000 keys with an indicator indicating if the response is
truncated. You send a series of list keys requests until you have
received all the keys.
The response to a REST GET Bucket operation contains the IsTruncated element which plays the role of the above mentioned indicator.
To retrieve the next set of results, using the AWS::S3 library, use the last key from the current page of results as the marker in your next request.