Question:
I need to create a log file in AWS S3 (or any other AWS service that can help here).
AFAIU there is no way to append a line to an existing log file in S3. This means that I would either need to retrieve and resend the whole log each time a new message comes, or that I will need to create a new object per message. The latter option is complicated when retrieving data.
I have lots of log messages and it is important not to lose them so it is not an option to buffer them in my server memory and send to S3 once in a while.
Which AWS service would be my best option (also in terms of saving costs).
Answer:
What you are looking for is called EBS (Elastic Block Store).
You can attach an EBS drive to any server, and start logging there. If your server dies, the EBS disk will still be around and you can look at the data later.
Even better, just create a central syslog server (with an EBS drive), and have all your (stateless) app servers send their logs there.
When your EBS drive fills up, archive the day/week/etc into S3 logs. This lets you store fairly large files in S3 (and even do compression if you want).