Back to blog »

Understanding How Facebook Disappeared from the Internet And What You Can Do to Prevent it from Happening to You!

This article appeared on 10.4.21 on Cloudflare. With enhancement by DALIM SOTWARE
10/04/2021 Tom Strickx
 
“Facebook can't be down, can it?”, we thought, for a second.
 
Today at 15:51 UTC, we opened an internal incident entitled "Facebook DNS lookup returning SERVFAIL" because we were worried that something was wrong with our DNS resolver 1.1.1.1. But as we were about to post on ourpublic status page we realized something else more serious was going on.
 
Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had "pulled the cables" from their data centers all at once and disconnected them from the Internet.
 
This wasn't a DNS issue itself, but failing DNS was the first symptom we'd seen of a larger Facebook outage.
 
How's that even possible?
 

Update from Facebook

Facebook has now published a blog post giving some details of what happened internally. Externally, we saw the BGP and DNS problems outlined in this post but the problem actually began with a configuration change that affected the entire internal backbone. That cascaded into Facebook and other properties disappearing and staff internal to Facebook having difficulty getting service going again.
 

So, what can you do to prevent something like this happening to you or in the best case, minimize your downtime?

No one ever wants their site to be unreachable or have downtime which can lead to lose of revenue. Your DALIM SOFTWARE environment provides options to keep you running with some planning. With the right architecture a hybrid environment could help avoid some DNS issues.
 
Yes, it won’t solve the fat finger updating in a routing table but there are some issues that can be avoided with a DALIM SOFTWARE solution both running at your facility and in the cloud.
 
There are environments implemented today doing just that after the storms knocked out power on the U.S. east coast years ago. A cloud solution can give you back up when your environment goes down. Either a hardware or software problem can lead to long downtime.
 
So, reach out to your DALIM SOFTWARE Sales representative today to discuss these options and ease your mind of the issues that can happen to you. 
BOOK A DEMO