Streams in DynamoDB
Hello Everyone
Welcome to CloudAffaire and this is Debjeet
In the last blog post, we have discussed Secondary Indexes in DynamoDB
https://cloudaffaire.com/secondary-indexes-in-dynamodb/
In this blog post, we are going to discuss streams in DynamoDB.
Streams in DynamoDB:
DynamoDB Streams is an optional feature that captures data modification events in DynamoDB tables. The data about these events appear in the stream in near real time, and in the order that the events occurred. Each event is represented by a stream record.
If you enable a stream on a table, DynamoDB Streams writes a stream record whenever one of the following events occurs:
- A new item is added to the table: The stream captures an image of the entire item, including all of its attributes.
- An item is updated: The stream captures the “before” and “after” image of any attributes that were modified in the item.
- An item is deleted from the table: The stream captures an image of the entire item before it was deleted.
Each stream record also contains the name of the table, the event timestamp, and other metadata. Stream records have a lifetime of 24 hours; after that, they are automatically removed from the stream.
Next, we are going to enable stream on a table in our local DynamoDB instance.
Step 1: Create a table named CloudAffaire
1 2 3 4 5 6 7 |
aws dynamodb create-table ^ --table-name CloudAffaire ^ --attribute-definitions AttributeName=id,AttributeType=N AttributeName=ename,AttributeType=S ^ --key-schema AttributeName=id,KeyType=HASH AttributeName=ename,KeyType=RANGE ^ --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 ^ --endpoint-url http://localhost:8000 ^ --output table |
Step 2: Insert a single record
1 2 3 4 |
aws dynamodb put-item ^ --table-name CloudAffaire ^ --item "{""id"":{""N"":""100""},""ename"":{""S"":""Bob""}}" ^ --endpoint-url http://localhost:8000 |
Step 3: Enable stream on the table
1 2 3 4 5 6 |
aws dynamodb update-table ^ --table-name CloudAffaire ^ --attribute-definitions AttributeName=id,AttributeType=N AttributeName=ename,AttributeType=S ^ --stream-specification StreamEnabled=TRUE,StreamViewType=NEW_AND_OLD_IMAGES ^ --endpoint-url http://localhost:8000 --output table |
Note down the LatestStreamARN from the output which will be required to query the stream data.
You can also get the LatestStreamARN from the output of describe-table
1 2 3 4 |
aws dynamodb describe-table ^ --table-name CloudAffaire ^ --endpoint-url http://localhost:8000 ^ --output table |
LatestStreamARN: arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877
You will also need shardId apart from LatestStreamARN to get stream data. To get the shardid, execute describe-stream with –stream-arn as LatestStreamARN
1 2 3 4 |
aws dynamodbstreams describe-stream ^ --stream-arn arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877 ^ --endpoint-url http://localhost:8000 ^ --output text |
shardId : shardId-00000001546238036879-b81e7a3e
Step 4: Insert another record in the CloudAffaire table. This will be captured in stream.
1 2 3 4 |
aws dynamodb put-item ^ --table-name CloudAffaire ^ --item "{""id"":{""N"":""101""},""ename"":{""S"":""Debjeet""}}" ^ --endpoint-url http://localhost:8000 |
To get the stream data using get-record, we need the shard iterator id. Next, we are going to get the shard iterator id using get-shard-iterator AWS CLI
Step 5: Get shard iterator id for last put-item operation
1 2 3 4 5 |
aws dynamodbstreams get-shard-iterator ^ --stream-arn arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877 ^ --shard-id shardId-00000001546238036879-b81e7a3e ^ --shard-iterator-type TRIM_HORIZON ^ --endpoint-url http://localhost:8000 |
Note: Replace stream-arn and shard-id that you get from previous steps.
ShardIterator: 000|arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877|c2hhcmRJZC0wMDAwMDAwMTU0NjIzODAzNjg3OS1iODFlN2EzZXwwMDAwMDAwMDAwMDAwMDAwMDAwMjd8MDAwMDAwMDAwMDAwMDAwMDAxNTQ2MjQ0Njg1NjA1
Step 6: Get stream data using get-record
1 2 3 |
aws dynamodbstreams get-records ^ --shard-iterator "000|arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877|c2hhcmRJZC0wMDAwMDAwMTU0NjIzODAzNjg3OS1iODFlN2EzZXwwMDAwMDAwMDAwMDAwMDAwMDAwMjd8MDAwMDAwMDAwMDAwMDAwMDAxNTQ2MjQ0Njg1NjA1" ^ --endpoint-url http://localhost:8000 |
Note: Stream are captured for any new item, update item and delete item operations.
You can also get the list of streams using list-stream AWS CLI
1 2 |
aws dynamodbstreams list-streams ^ --endpoint-url http://localhost:8000 |
Delete the table:
1 2 |
aws dynamodb delete-table --table-name CloudAffaire ^ --endpoint-url http://localhost:8000 |
Hope you have enjoyed this article. In the next blog post, we will explore the DynamoDB AWS console.
To get more details on DynamoDB, please refer below AWS documentation
https://docs.aws.amazon.com/dynamodb/index.html