How to put data from server to Kinesis Stream


I am new to Kinesis. Reading out the documentation i found i can create the Kinesis Stream to get data from Producer. Then using KCL will read this data from Stream to further processing. I understand how to write the KCL application by implemeting IRecordProcessor .

However the very first stage as how to put data on Kinesis stream is still not clear to me. Do we have some AWS API which does need implementation to achieve this.

Scenarios: I have an server which is contineously getting data from various sources in the folders. Each folder is containing the text file whose rows are containing the required attributes for furhter analytical work. i have to push all these data to Kinesis Stream.

I need code something as below below class putData method wil be used to out in Kinesis stream

However reading file from the source folder from the server and then what design i should use to call putData to get the record on Kinesis stream. Do i need infinite loop and reading all files and then do this or some framework which will better do this with care of fault tolerance , single point of failure all . Any help would be greatly appreciated.

Briefly: I need a better technique to put regularly generated data to Kinesis Stream the data is generated at regular interval to server.


So it seems you are already using…

Specific method you want is as follows.

You need a stream name, record, and stream key.

But it seems you have all this?

You would then need a program running always tailing your server log file and when ever there is a new line it will push this.

But your data will only sit their for 24 hours. You then need a worker program to consume the data and place it in some other AWS resource.

Leave a Reply