Surviving a data centre outage
Hey everyone, quick post from me (Luke) today. The cluster that this blog is hosted on is spread across four data centres, and one of them appears to be experiencing a complete network outage that’s been happening all day today (Sunday).
As you can see from their status page, their core routers are down in their Chicago data centre:
Their tech support phone line is jammed, and there’s been no updates from the data centre so far on Twitter or on their status page, apart from that little red dot.
As you can see from Twitter, there’s a lot of stress and anxiety:
And yet HybridCluster recovered automatically, in less than a minutes, as soon as the data centre went down, recovering all the websites, databases and mailboxes that were hosted there onto other servers in other data centres, and automatically updated our DNS, so I can get on with having a relaxing Sunday, which by the way involves having dinner with these guys.