Facebook made BGP withdrawals near the time of the incident, however, 2 prefixes covering two of their 4 DNS nameservers (a and b) are still being advertised across the Internet. They are reachable on the Internet but traffic is dropping at Facebook’s network edge.
ThousandEyes tests can confirm that at 15:40 UTC on October 4, the Facebook application became unreachable due to DNS failure. Facebook’s authoritative DNS nameservers became unreachable at that time. The issue is still ongoing as of 17:02 UTC.
The 2 DNS nameservers (a and b) are reachable because covering prefix 129.134.0.0/17 is still being advertised, but this advertisement may not have been designed to support the nameserver service.
The 3 specific prefixes covering a and b nameservers before the incident were 129.134.30.0/23, 129.134.30.0/24, 129.134.31.0/24. The specific routes covering all 4 nameservers (a-d) were withdrawn from the Internet at approximately 15:39 UTC.
We've compiled all of these updates into a blog (including some additional graphics), which we'll be updating with our full analysis as soon as it's available. Check back here:
Update from Facebook Engineering: "[...] We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime."
@thousandeyes
and according to reddit the people with physical access to the routers and the people who need to make the change are two different groups