On converting from single-AZ RDS to multi-AZ RDS

On converting from single-AZ RDS to multi-AZ RDS,

published at 8:05pm on 05/04/11

A couple of weeks ago, we were bitten by the Amazon AWS outage that affected so much of the internet. We were lucky enough that the outage only affected our main RDS database instance, we did not lose any data, we were able to recover from a snapshot, and we managed to get some great press out of the deal. That said, we decided that we wanted a bit more reliability from our hosting infrastructure, and while most of our system was designed to fail gracefully, our database was a single point of failure that, in this case, failed.

So, while it may not have helped us in this particular instance (due to the nature of Amazon’s outage), we decided that we should convert our database instance from a single instance to one that is backed by Amazon’s Mutli-Availability Zone failover mechanism. Even though, according to Amazon’s own post-mortem, their Multi-AZ instances didn’t failover correctly in this particular outage, I trust that they’ve learned from their mistakes and that, in the future, the failover will work properly.

So what does this actually mean? It means that there our Mysql database instance has a hot spare always at the ready in case anything catastrophic happens to our master. In case of any sort of failure to our main database, Amazon should detect this and automatically switch us over to the backup. Then, they will bring up a new backup instance for our newly promoted master and we will be protected again from failure. This should all happen transparently and our site should see minimal interruption in the case of this kind of failure. In addition, backups and upgrades are all done on the backup instance, meaning that we should see even more stability in our day-to-day usage. Sounds like a win all around for us.

We had our main database server up and running for about a year I could not for the life of me find any information about how long this conversion would actually take. We have a 30GB large instance living in the US East Availability Zone.

At 9:55 pm EST on a Sunday night I took our site offline and started the conversion process. This consisted solely of clicking a checkbox in the RDS web interface, making sure that the checkbox to apply the changes immediately was checked, and clicked “save.” The conversion process started almost immediately and at 10 minutes later it was done.

I’m not sure if the database was inaccessible that whole time, but I took the site down anyway, just to be safe. All it all, the upgrade was completely transparent to me and required no other intervention on my part.

We’ve seen no adverse affects to this upgrade (other than to our bill – the multi-AZ instances cost twice as much as a non-AZ instance), but for us, the peace of mind is worth it.