Automating Linux Ebs Snapshots Backup and Clean-Up

Automating Linux EBS snapshots backup and clean-up

Try the following shell-script, I use this to create snapshot for most of my projects and it works well.

https://github.com/rakesh-sankar/Tools/blob/master/AmazonAWS/EBS/EBS-Snapshot.sh

You can give me pull-request/fork the project to add the functionality of cleaning-up the old entries. Also watch for this repo, when I find some time I will update the code to have clean-up functionality.

Automating Amazon EBS snapshots anyone have a good script or solution for this on linux

You can easily script something to do this for you.

1. setup the EC2 commandline API tools
2. set EC2_CERT and EC2_PRIVATE_KEY in order to be able to use the API tools
3. parse the results of ec2-describe-snapshots
4. delete the appropriate snapshots

The results look something like:

SNAPSHOT    snap-xxxxxxxx   vol-xxxxxxxx    completed   2009-08-26T07:39:33+0000    100%

You can then do some parsing of the dates and sorting and start removing the older snapshots.

NOTE: I don't know if we can trust the sort order but I've never seen it sort any other way than oldest to newest.

To delete a snapshot, use ec2-delete-snapshot snap-xxxxxxxx.

The rest I leave to you as a simple Bash script that you can call daily or however often you need from cron.

How to setup automatic scheduled snapshots for each single AMI/EBS?

You can use the AWS command-line tools to automate EBS snapshots. Just schedule a cron job or similar to run ec2-create-snapshot command at the desired interval on your ebs volume.

You can also make API calls over http to do the same thing, if you don't want to install the command line tools.

See the link for more information on creating EBS snapshots.

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/ebs-creating-snapshot.html

Delete old snapshots on AWS

If you want to continue with the same tools you are currently using, i.e. CloudWatch for running scheduled events, you can also create a CloudWatch event that will trigger a lambda function (or a set of).
Using lambda you can write code that will implement what ever logic you choose, for example:
Using an AWS SDK, List all snapshots which are +7 days, and delete them.

Useful Links:

Schedule AWS Lambda Functions Using CloudWatch Events

AWS SDK for JS

AWS SDK for Python

Here is a very nice blog post about accomplishing exactly this - Automating Amazon EBS Snapshot Management with AWS Step Functions and Amazon CloudWatch Events

AWS Difference between a snapshot and AMI

There are two types of AMIs (and corresponding instances):

1. instance-store (sometimes called S3-based). These are less common and I don't recommend them for beginners. An instance-store AMI is a copy of the root instance-store volume plus some metadata, all saved in an S3 bucket in a special format

2. EBS boot. This is probably what you are using. An EBS boot AMI is an EBS snapshot of the EBS root volume plus some metadata like the architecture, kernel, AMI name, description, block device mappings, and more.

You can take a snapshot of an EBS boot volume and turn it into an EBS boot AMI by registering it with the appropriate metadata. The trickiest part of this is specifying the correct AKI id (kernel) so that it boots correctly.

Tricks to make an AWS spot instance persistent?

We ended up finding a solution, and here is what we had to do. I'm going to list this out step-by-step, to make recreating this easier for those who may be looking for a similar type of solution...

1. Create a new spot request instance. Make sure to uncheck "Delete on Termination" for the root device, so that the volume stays behind in the next step. Make sure to note the architecture (we always use x86_64) and the kernel ID that your instance is using (very important!)
2. Now, SSH into your new instance and make a file or something, so you can see the effect of persistence first-hand. After making some changes to the filesystem, go ahead and logout of the SSH connection and terminate the instance.
3. Awesome. Now, go to your EC2 web console and find the new volume that was being used for the instance we just terminated. Right click the volume and select "Create Image". Follow the wizard, making certain to select the same architecture and kernel ID that we noted earlier.
4. Now, start the spot request wizard using your new image. Follow the wizard, again making certain to uncheck "Delete on Termination". Additionally, and this is the easy step to miss, make sure to expand the collapsed section titled 'Advanced Options' and set the correct kernel ID again.

If you follow the above steps to the T, you will have a new instance at the same point that your old instance was at when it was terminated. Therefore, we have achieved some form of persistence.

---

Related Topics