Question:
There are more than 3k objects under the prefix. I use the following code to list all objects to get their names, but the API only retrieve 1000 objects.
s3_client = boto3.client(‘s3’)
|
1 2 3 4 5 6 7 8 9 10 11 12 |
response = s3_client.list_objects( Bucket = "my-bucket", Prefix = "my-prefix", MaxKeys=50000 ) s3 = boto3.resource('s3') bucket = s3.Bucket(S3) print(len(response['Contents'])) # only retrieve 1000 |
Answer:
Use paginators to loop through multiple pages. See: Creating Paginators
|
1 2 3 4 5 6 7 8 9 10 |
import boto3 client = boto3.client('s3') paginator = client.get_paginator('list_objects') operation_parameters = {'Bucket': 'my-bucket', 'Prefix': 'my-prefix'} page_iterator = paginator.paginate(**operation_parameters) for page in page_iterator: print(page['Contents']) |