Question:
What are the options available to schedule or automate refresh of a quicksight SPICE dataset?
Are there any APIs available to automate spice datatset refresh? preferably using python.
Answer:
You have two options,
– Using API services available in the latest version of boto3
Use ‘create_ingestion‘ method to initiate dataset refresh, and use ‘describe_ingestion‘ to check the status of refresh
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import boto3 import time import sys client = boto3.client('quicksight') response = client.create_ingestion(DataSetId=' while True: response = client.describe_ingestion(DataSetId=' if response['Ingestion']['IngestionStatus'] in ('INITIALIZED', 'QUEUED', 'RUNNING'): time.sleep(10) #change sleep time according to your dataset size elif response['Ingestion']['IngestionStatus'] == 'COMPLETED': print("refresh completed. RowsIngested {0}, RowsDropped {1}, IngestionTimeInSeconds {2}, IngestionSizeInBytes {3}".format( response['Ingestion']['RowInfo']['RowsIngested'], response['Ingestion']['RowInfo']['RowsDropped'], response['Ingestion']['IngestionTimeInSeconds'], response['Ingestion']['IngestionSizeInBytes'])) break else: print("refresh failed! - status {0}".format(response['Ingestion']['IngestionStatus'])) sys.exit(1) |
DataSetId of dataset can be found from aws URI or use ‘list_data_sets‘ method to list all datasets and get DataSetId from the field [‘DataSetSummaries’][‘DataSetId’] method call response
IngestionId – set unique id, I used current time in epoch [str(int(time.time()))]
– Schedule refresh using schedule option in quicksight dataset
You can schedule refreshes for ‘hourly’, ‘daily’, ‘weekly’ or ‘monthly’ cadence using schedule option in quicksight-dataset
