Question:
Whats the best, light-weight, way to FPT/SCP files to S3 every few min? So wget/curl could work. Also a custom solution could work, maybe Java with DB to maintain the state. Any ideas would be helpful.
Answer:
Just use s3cmd
or boto
or the S3 command-line tools or… well… quite a few other options.
Basically you’re just posting to the S3 API. You can do it in nearly any language using AWS’s myriad SDKs. s3cmd
, however, is a nice, lightweight little CLI tool that’s probably more than you need.
Your comment suggests you’re looking for something slightly different. Here are a few recommendations based on the source and destination of the files:
- If you’re moving files from EC2 or your localhost to S3, use
s3cmd
. If you need to run it regularly, consider a cron job. The following will run every five minutes.
12*/5 * * * * s3cmd sync --delete-removed my_local_directory/ s3://my-bucket/path/ - If you’re moving files from S3 to EC2 or your localhost, use
s3cmd
but reverse the syntax:
12*/5 * * * * s3cmd sync s3://my-bucket/path/ my_local_directory/ - If you’re moving files from your localhost to EC2, consider
scp
orrsync
.