Whats the best, light-weight, way to FPT/SCP files to S3 every few min? So wget/curl could work. Also a custom solution could work, maybe Java with DB to maintain the state. Any ideas would be helpful.
boto or the S3 command-line tools or… well… quite a few other options.
Basically you’re just posting to the S3 API. You can do it in nearly any language using AWS’s myriad SDKs.
s3cmd, however, is a nice, lightweight little CLI tool that’s probably more than you need.
Your comment suggests you’re looking for something slightly different. Here are a few recommendations based on the source and destination of the files:
- If you’re moving files from EC2 or your localhost to S3, use
s3cmd. If you need to run it regularly, consider a cron job. The following will run every five minutes.
Default12*/5 * * * * s3cmd sync --delete-removed my_local_directory/ s3://my-bucket/path/
- If you’re moving files from S3 to EC2 or your localhost, use
s3cmdbut reverse the syntax:
Default12*/5 * * * * s3cmd sync s3://my-bucket/path/ my_local_directory/
- If you’re moving files from your localhost to EC2, consider