Whats the best way pull files to S3 using FPT/SCP?

Question:

Whats the best, light-weight, way to FPT/SCP files to S3 every few min? So wget/curl could work. Also a custom solution could work, maybe Java with DB to maintain the state. Any ideas would be helpful.

Answer:

Just use s3cmd or boto or the S3 command-line tools or… well… quite a few other options.

Basically you’re just posting to the S3 API. You can do it in nearly any language using AWS’s myriad SDKs. s3cmd, however, is a nice, lightweight little CLI tool that’s probably more than you need.

Your comment suggests you’re looking for something slightly different. Here are a few recommendations based on the source and destination of the files:

  • If you’re moving files from EC2 or your localhost to S3, use s3cmd. If you need to run it regularly, consider a cron job. The following will run every five minutes.
  • If you’re moving files from S3 to EC2 or your localhost, use s3cmd but reverse the syntax:
  • If you’re moving files from your localhost to EC2, consider scp or rsync.

Leave a Reply