getting more than 1000 object from s3 bucket on rails application


I have to write a rails task for getting files from s3 bucket but my bucket have more than 1000 object.

this code just give me 1000 objects 🙁
how can i get my all objects from s3 bucket ?


As stated in the S3 developer documentation:

To manage large result sets, Amazon S3 uses pagination to split them
into multiple responses. Each list keys response returns a page of up
to 1,000 keys with an indicator indicating if the response is
truncated. You send a series of list keys requests until you have
received all the keys.

The response to a REST GET Bucket operation contains the IsTruncated element which plays the role of the above mentioned indicator.

To retrieve the next set of results, using the AWS::S3 library, use the last key from the current page of results as the marker in your next request.

Leave a Reply