Automating Amazon Ebs Snapshots Anyone Have a Good Script or Solution for This on Linux

Automating Amazon EBS snapshots anyone have a good script or solution for this on linux

You can easily script something to do this for you.

1. setup the EC2 commandline API tools
2. set EC2_CERT and EC2_PRIVATE_KEY in order to be able to use the API tools
3. parse the results of ec2-describe-snapshots
4. delete the appropriate snapshots

The results look something like:

SNAPSHOT    snap-xxxxxxxx   vol-xxxxxxxx    completed   2009-08-26T07:39:33+0000    100%

You can then do some parsing of the dates and sorting and start removing the older snapshots.

NOTE: I don't know if we can trust the sort order but I've never seen it sort any other way than oldest to newest.

To delete a snapshot, use ec2-delete-snapshot snap-xxxxxxxx.

The rest I leave to you as a simple Bash script that you can call daily or however often you need from cron.

How to make a daily back up of my ec2 instance?

Hopefully your instance is EBS backed.

If so, you can backup your instance by taking an EBS Snapshot. That can be done through aws.amazon.com (manually), using AWS Command Line Tools (which can be automated and scheduled in cron or Windows Task Scheduler as appropriate) or through the AWS API.

You want to ensure that no changes are made to the state of the database backup files during the snapshot process. When I used this strategy for MySQL running on Ubuntu, I used a script to ensure a consistent snapshot. That script uses a feature of the XFS file system to freeze the filesystem during the snapshot. In that deployment, the snapshot only took 2-3 seconds and was performed at a very off-peak time. Any website visitors would experience a 2-3 second lag. For Windows, if the device can not be rebooted for the snapshot (you have no maintenance window at night), I would instead create a separate EBS device (e.g. a "S:\" device for snapshots), use SQL Server backup tools to create a .bak file on that other device, then create an EBS snapshot of that separate EBS device.

For details on scripting the backup, see this related question:

Automating Amazon EBS snapshots anyone have a good script or solution for this on linux

If you have separate storage mounted e.g. for your database, be sure you back that up too!

UPDATE

To create a snapshot manually,

• Browse to https://console.aws.amazon.com/ec2/home?#s=Volumes
• Right-click on the volume you want to backup (the instance the volume is attached to is in the column named 'Attachment Information')
• Select Create Snapshot

To create an AMI image from the instance and lauch other instances just like it (in instances with more resources or to balance load, etc.):

• Browse to https://console.aws.amazon.com/ec2/home?#s=Instances
• Right-click on the instance you want to create the AMI from
• Select Create Image (EBS AMI)

How to automatically snapshot a volume of an Amazon EC2 instance?

Ok well,

1. The first line where he runs (source). Thats the same as . /etc/environment. Anyways all he's doing is loading a file that has a list of environmental variables that amazon requires. At least this is what i assume.
2. He's making this script much more complicated than it needs to be. He doesn't need to run the ec2-describe-instances command and save the output to a file then grep the output etc....
3. You can put whatever you want for the DESC. You can just replace everything to the right of the = to whatever text you want. Just make sure to put quotes around it.

I would change two things about this script.

1. Get the InstanceId at runtime in the script. Don't hard code it into the script. This line will work no matter where the script is running.

   MY_INSTANCE_ID=`curl http://169.254.169.254/1.0/meta-data/instance-id`
   

2. Instead of calling ec2-describe-volumes and saving the output to a temp file etc... Just use a filter on the command and tell it which instance id you want.

   VOLUME_LIST=(`ec2-describe-volumes --filter attachment.instance-id=$MY_INSTANCE_ID | awk '{ print $2 }'`)
   

How to make EBS volume to be available with zero downtime and attach to EC2 instance when ebs volume crashes?

If you want to improve the availability of the large drive, I would recommend FSx. It's not so trivial - you need to set up Active Directory, and your EC2 needs to join AD - but once you have all these components (and there is additional cost as well) - FSx provides significantly higher availability than EBS assuming you're using a multi-AZ setup.
More information here

Note that load balancer tags are irrelevant to your question - although to address your first situation you probably need load balancer balancing two EC2 instances

Bootstrap Scripts vs Custom AMI for making EC2 easy to rebuild

If you create custom AMI:

• It'll be faster to provision new servers.
• It'll be easy to maintain the unification of the environment.
• It'll be difficult to update the AMI in-case of any modifications.

If you use bootstrapping script:

• It'll be easier to modify in case of any changes.
• It'll be hard to maintain the unification of the environment (e.g. can face versioning issues).
• Slow to provision new servers.

BTW, how about using AMI for things that are less likely to change and then using bootstrap scripts e.g. user data to do things that'll change over time.

How to setup automatic scheduled snapshots for each single AMI/EBS?

You can use the AWS command-line tools to automate EBS snapshots. Just schedule a cron job or similar to run ec2-create-snapshot command at the desired interval on your ebs volume.

You can also make API calls over http to do the same thing, if you don't want to install the command line tools.

See the link for more information on creating EBS snapshots.

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html

Alternative tools for Amazon EC2?

I'm a bit late but I have a solution!

I found the same problems with the Amazon AMI tools. They're a decent reference implementation but very difficult to use particularly when you have more than a couple instances. I wrote a replacement command-line tool as part of another project, called Rudy that answers most of your concerns

The commands are more intuitive than Amazon's AMI tools:

• rudy-ec2 instances -C
• rudy-ec2 groups -A -p 8080 -a 11.22.33.44 group-name
• rudy-ec2 volumes -C -s 100
• rudy-ec2 images
• ...

All configuration is in a single file (~/.rudy/config).

It can output in several formats (yaml, json, csv, tsv, and of course regular text):

rudy-ec2 -f yaml snapshots
--- 
:awsid: snap-2457b24d
:progress: 100%
:created: "2009-05-08T15:24:17.000Z"
:volid: vol-4ee10427
:status: completed

Regarding the private keys, There are no EC2 tools that allow to create private keys for with a password for booting a public instance because the API doesn't support it. However, if you create your own image, you can use your private keys.

Here's more info:

• GitHub Project
• An introduction to rudy-ec2

---

Related Topics