Cloud Zone is brought to you in partnership with:

Sean Hull is a renowned author, speaker and advisor on MySQL & Oracle databases, web performance and scalability, EC2, Linux & Internet Architecture. He is the author of "Oracle & Open Source" O'Reilly, 2001, and speaks widely at conferences and forums. With 20 years of professional experience, he consults with large enterprises and startups in New York City and the San Francisco Bay Area. Sean is a DZone MVB and is not an employee of DZone and has posted 64 posts at DZone. You can read more from them at their website. View Full User Profile

The 5 Point Checklist Backup and Recovery in EC2

05.30.2012
| 6913 views |
  • submit to reddit
Best practices for backups and disaster recovery aren’t tremendously different in the cloud than from a managed hosting environment.  But they are more crucial since cloud servers are less reliable than physical servers.  Also the security aspect may play a heightened role in the cloud.  Here are some points to keep in mind.

1. Perform multiple types of backups
2. Keep non-proprietary backups offsite
3. Test your backups – perform firedrills
4. Encrypt backups in S3
5. Perform Replication Integrity Checks

Perform Multiple Types of Backups

Your database tier is typically your primary datastore, so it’s backups are often the most crucial.  Snapshots of EBS volumes are powerful and fast ways to perform full database backups in the AWS environment.  This involves locking all tables briefly, and running the snapshot command, followed by a release of all those table locks.  Be sure to test this process to ensure that the temporary locks on the database don’t create a pileup on your webservers.

You can find xtrabackup at Percona’s site

Keep Non-proprietary Backups Offsite

The EC2 snapshots are great, but they only work in EC2.  So you’ll also want to perform other types of backups.  Personally I like having a few different options in the event I need to restore.  Logical backups are great for restoring one table, but are slow for restoring the entire database.  Hotbackups are great and fast to restore the whole database, but take a lot of space so may not be as efficient if you just need to restore one table.  So I like to have both.  Percona’s xtrabackup and the associated innobackupex script provide an open-source hotbackup solution for MySQL.  Get it!  Then intersperse those backups with mysqldumps as well.  Alternating days, for example.

Test Your Backups – Perform Firedrills

Any good disaster recovery plan must be thoroughly tested.  Set aside the time to actually run through this from start to finish.  This is where the cloud really excels to your advantage.  Spinup all the servers that makeup your entire environment, load balancer, webservers, database servers, checkout all the source code, and configuration files.  You put your configuration files in version control, right?  Then restore the database.  This firedrill tests your server spinup scripts, your version control of source code and configuration files, and your database backups.  All of these pieces must be in place for the fire drill to succeed.  Lastly running through the whole process forces you to document details, and you find out how long your disaster recovery would actually take.

Encrypt Backups in S3

S3 stores objects as private by default, however it makes sense for particularly sensitive data to also encrypt those backups.  Remember you control access to your encryption keys but not where the data is stored or where it might move around.  So it can’t hurt to be extra cautious.  Here’s an excellent article on the topic.  Using mk-query-digest to checksum

Perform Replication Integrity Checks

A MySQL slave or passive master database can be a great way to offload backups away from the primary database server.  This reduces impact to your customers while backups are running.  But MySQL replication is not bulletproof.  The slaves can drift out of sync with the master silently without throwing errors.  That’s why it’s important to use an integrity checking tool like Maatkit’s mk-table-checksum.  This tool can be set in cron to perform checksums on a slice of your database periodically.

Here’s an excellent article on using the tool.  Ongoing MySQL Integrity Checking with mk-table-checksum

Published at DZone with permission of Sean Hull, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Daniel Slazer replied on Tue, 2012/06/12 - 12:19pm

I'd like to know how hard it would be to pursue this idea.

Let's say I have picture1 and picture2. picture1 is 500x500 and picture2 is 200x20. picture2 might be a small part of picture1, previously cropped. Now, I want to know programmatically, if picture2 is indeed part of picture1.

Any ideas and obstacles doing this?

Uri Wolloch replied on Sun, 2013/01/20 - 7:55am

 Hi,

I agree with most of this post, especially of the need perform drill and making sure your recovery scenario actually works. In my opinion, creating different types of backup is usually needed to serve different purposes. For example, File backup to S3 may be an excellent solution for long term backup/archiving, especially with the new option which allows moving objects to Glacier over time.

For operational/short term backup, the most efficient and powerful way would be to use snapshots. For durability, snapshots are kept in S3 and are therefore on a different infrastructure than the EBS storage itself. Furthermore, AWS allows now to copy snapshots between regions, and by that give an excellent answer to disaster recovery scenarios.

Keeping backups outside of AWS, appealing as it is, is not necessary and can be cumbersome and costly. In case you want t migrate outside of AWS, you can plan your migration separately. your backup solution doesn't need to address that.

Uri Wolloch

CTO - N2W Software

Cloud Protection Manager (CPM) is a comprehensive backup and recovery solution for EC2, and is now offered as a free beta

More details:  www.n2ws.com

Ujas Ujass replied on Fri, 2013/08/23 - 7:09am

Performing Firedrills to test the backups is the best option for every developer, I started using it when I started working and it never failed me. I even made a small program to search for the best products on the internet, I found out I can get affordable Velux blinds at skylightdeals.com in this way.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.