I have a script that I need to run once a day that requires a lot of memory. I would like to run it on a dedicated amazon box.
Is there some automated way to build a box, download all required software (like ruby) and then run my script. After the script is ran, I would like to shutdown the box.
The two options I can think of are:
- I am thinking about hacking EMR to do this. (My script is a mapper against an empty directory)
- Chef – This seemed like too much for one simple script.
You can accomplish setting up a new EC2 instance on startup using the official Ubuntu AMIs, the official Amazon Linux AMIs, and any other AMI that supports the concept of a user-data script.
Create a script (bash, Perl, Python,
- whatever) that starts with #!
- Pass this script as the user-data when running the EC2 instance.
- The script will automatically be run as root on the first boot.
Here’s the article where I introduced the concept of a user-data script:
Automate EC2 Instance Setup with user-data Scripts
Your user-data script can install the required software, configure it, install your work script, and set up a cron job that runs the work script once a day.
If the installation script don’t take a long time to run (e.g., under an hour or few) then you don’t even have to run a single dedicated instance 24 hours a day. You can instead use an approach that lets AWS start an instance for you on a regular schedule.
Here’s an article I wrote that provides details on this approach with sample commands:
Running EC2 Instances on a Recurring Schedule with Auto Scaling
The general approach is to use Auto Scaling to start an instance with your user-data script on a regular schedule. Your job will terminate the instance when it has completed. They key is to suspend Auto Scaling’s normal desire to re-start instances that terminate so that you don’t pay for a running instance until the next time your job starts.