Streams in DynamoDB

Hello Everyone

Welcome to CloudAffaire and this is Debjeet

In the last blog post, we have discuss Secondary Indexes in DynamoDB

https://cloudaffaire.com/secondary-indexes-in-dynamodb/

In this blog post we are going to discuss streams in dynamodb.

Streams in DynamoDB

DynamoDB Streams is an optional feature that captures data modification events in DynamoDB tables. The data about these events appear in the stream in near real time, and in the order that the events occurred. Each event is represented by a stream record.

If you enable a stream on a table, DynamoDB Streams writes a stream record whenever one of the following events occurs:

• A new item is added to the table: The stream captures an image of the entire item, including all of its attributes.
• An item is updated: The stream captures the “before” and “after” image of any attributes that were modified in the item.
• An item is deleted from the table: The stream captures an image of the entire item before it was deleted.

Each stream record also contains the name of the table, the event timestamp, and other metadata. Stream records have a lifetime of 24 hours; after that, they are automatically removed from the stream.

Next, we are going to enable stream on a table in our local DynamoDB instance.

Step 1: Create a table named CloudAffaire

aws dynamodb create-table ^
--table-name CloudAffaire ^
--attribute-definitions AttributeName=id,AttributeType=N AttributeName=ename,AttributeType=S ^
--key-schema AttributeName=id,KeyType=HASH AttributeName=ename,KeyType=RANGE ^
--provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 ^
--endpoint-url http://localhost:8000 ^
--output table

Step 2: Insert a single record

aws dynamodb put-item ^
--table-name CloudAffaire ^
--item "{""id"":{""N"":""100""},""ename"":{""S"":""Bob""}}" ^
--endpoint-url http://localhost:8000

Step 3: Enable stream on the table

aws dynamodb update-table ^
--table-name CloudAffaire ^
--attribute-definitions AttributeName=id,AttributeType=N AttributeName=ename,AttributeType=S ^
--stream-specification StreamEnabled=TRUE,StreamViewType=NEW_AND_OLD_IMAGES ^
--endpoint-url http://localhost:8000
--output table

Note down the LatestStreamARN from the output which will be required to query the stream data.

You can also get the LatestStreamARN from the output of describe-table

aws dynamodb describe-table ^
--table-name CloudAffaire ^
--endpoint-url http://localhost:8000 ^
--output table

LatestStreamARN: arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877

You will also need shardId apart from LatestStreamARN to get stream data. To get the shardid, execute describe-stream with –stream-arn as LatestStreamARN

aws dynamodbstreams describe-stream ^
--stream-arn arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877 ^
--endpoint-url http://localhost:8000 ^
--output text

shardId : shardId-00000001546238036879-b81e7a3e

Step 4: Insert another record in the CloudAffaire table. This will be captured in stream.

aws dynamodb put-item ^
--table-name CloudAffaire ^
--item "{""id"":{""N"":""101""},""ename"":{""S"":""Debjeet""}}" ^
--endpoint-url http://localhost:8000

To get the stream data using get-record, we need the shard iterator id. Next, we are going to get the shard iterator id using get-shard-iterator AWS CLI

Step 5: Get shard iterator id for last put-item operation

aws dynamodbstreams get-shard-iterator ^
--stream-arn arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877 ^
--shard-id shardId-00000001546238036879-b81e7a3e ^
--shard-iterator-type TRIM_HORIZON ^
--endpoint-url http://localhost:8000

Note: Replace stream-arn and shard-id that you get from previous steps.

ShardIterator: 000|arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877|c2hhcmRJZC0wMDAwMDAwMTU0NjIzODAzNjg3OS1iODFlN2EzZXwwMDAwMDAwMDAwMDAwMDAwMDAwMjd8MDAwMDAwMDAwMDAwMDAwMDAxNTQ2MjQ0Njg1NjA1

Step 6: Get stream data using get-record

aws dynamodbstreams get-records ^
--shard-iterator "000|arn:aws:dynamodb:ddblocal:000000000000:table/CloudAffaire/stream/2018-12-31T06:33:56.877|c2hhcmRJZC0wMDAwMDAwMTU0NjIzODAzNjg3OS1iODFlN2EzZXwwMDAwMDAwMDAwMDAwMDAwMDAwMjd8MDAwMDAwMDAwMDAwMDAwMDAxNTQ2MjQ0Njg1NjA1" ^
--endpoint-url http://localhost:8000

Note: Stream are captured for any new item, update item and delete item operations.

You can also get the list of streams using list-stream AWS CLI

aws dynamodbstreams list-streams ^
--endpoint-url http://localhost:8000

Delete the table:

aws dynamodb delete-table --table-name CloudAffaire ^
--endpoint-url http://localhost:8000

Hope you have enjoyed this article. In the next blog post we will explore the DynamoDB AWS console.

To get more details on DynamoDB, please refer below AWS documentation

https://docs.aws.amazon.com/dynamodb/index.html