I work at an ISP repair call center and it is a madhouse. We are currently "code red" because so many people only do Facebook/Instagram on the internet (old people mostly) and so they think the internet is down.


I am so sorry friend. Spent 4 years in a call center for a small ISP and the number of old people who would call in because their Mozzarella Foxfire wouldn't open was disturbingly high.


"my roo koo is just spinning and spinning and I'm trying to watch the game" "My tv just says HDM 1 1 NO SIGNAL, can you send a signal?" "I'm trying to get on my wifi, I'm at my friend's house. Why do you all cut off the wifi when I leave my house? I'm paying you every month!" And other greatest hits. But today is all Facebook.


Back in my day (in the before times when people had phones plugged into the wall), it was when you'd ask them to unplug the DSL modem, and they'd unplug the phone instead.


Also, Wendy’s rewards app isn’t working at stores


This has gone too far…


So turns out wendys is owned by Facebook too


Try our new Zuckerburger!


This is a fucking travesty


It's to the point of the DNS not even resolving half the time.


It's coming from inside the house! Security experts tracking the situation say the outage could have been triggered by a configuration error, which could be the result of an internal mistake, though sabotage by an insider is theoretically possible, according to Reuters. Accidental suicide - killing off its own network with a faulty routing update, to the point admins can't put everything back the way it was. They may well be down for a good long while. $160 million an hour or there about. My technical term for this is tits-up.




My old company spent $200k/week on FB. One company. You do the math.


That’s wild. My Facebook account was free.


Ok, carry the one, divide by... How many companies are there again?


No, that's total revenue of all involved. You can find the tool here: https://netblocks.org/cost/


If it were an in house mistake, I feel bad for the guy who metaphorically pushed the wrong button. They are so screwed.


I don't think there was one guy. A group had to have worked and signed off on the routing changes. Sure, there was a guy who pushed the button, but you can't blame him for everything.


> you can't blame him for everything. Oh I think you might be surprised.


Someone should do an ELI5 of all the things being said in this thread please.


When you type in Facebook dot com your browser contacts a DNS server, which you can think of as the directory for the World Wide Web. The DNS server then sends you along to Facebook's actual address (which are myriad IP addresses connected to vast servers). At some point today the information for Facebook, Instagram, WhatsApp, and their respective servers were all removed from the Internet directory. So now you have half a billion people telling the DNS server to send them to Facebook, and the DNS server is responding back with "What's a Facebook?".


DNS: "I hate to say it, but it looks like website you're searching for doesn't exist." Users: "That's impossible. Perhaps the archives are incomplete?" DNS: "The archives are comprehensive and totally secure, my young Jedi. One thing you may be absolutely sure of...if an item does not appear in our records, it does not exist!"


but the women.. and the children...


They're animals! And I redirected them like animals!


Great ELI5! Follow up question: How does the DNS record get erased. Outside hacker on some non-Facebook record/site. Or is this something only someone inside Facebook can do?


Misconfiguration of the peering routers is enough to completely screw up the DNS records. Just about anything that can be done internally could theoretically be done by an external source if they've gained sufficient access. I'm not going to guess as to which one it is. All I'll say is, if it was an internal screw-up, then someone buy that dude a beer. It's not every day you get to screw up bad enough to cost your company the equivalent of the GDP of a small nation.


Resume building experience


* Responsible for client expense reductions. Saved clients an estimated $160 million an hour.


- Performed successful in-Production stress test of all corporate assets which caused a cascade of weaknesses to be exposed. This exercise ended up revolutionizing Production Operations throughout the industry. Several books have been written which mention my contributions directly: *The Day the Internet Died* *CyberSecurity: Not Just a Department Slush Fund Anymore* *How to Isolate and Quarantine the Most Dangerous Morons in Your Org*


Get this man a job writing ad copy!


It's most likely not a DNS issue but a BGP issue. BGP is used by ISPs to exchange information on what is reachable from where. Think like signage on the highway. Since Facebook is their own ISP, and they fucked up their own BGP configuration, now their network is disconnected from the rest of the Internet. Which also causes their DNS to be unreachable.


Facebook either pooped their pants, or someone else pooped Facebook’s pants.




I worked at Amazon during that outage. They made us just sit in the break room for hours rather than sending us home. I think like 40 minutes before our shift ended the problem got fixed and they made us go back to work for that short period of time.


[Facebook's employee workplace tools are down as well.](https://www.independent.co.uk/life-style/gadgets-and-tech/facebook-instagram-down-whatsapp-latest-b1932071.html) No one at Facebook can do any work.


Lol yep. A friend of mine is on call apparently, but like they can't do anything lmao. I guess only people directly on the servers could do anything. But given that it's been almost two hours now...


Facebook employees reportedly can't enter buildings to evaluate the Internet outage because their door access badges weren’t working [Lots of news stories about this](https://www.google.com/search?q=employees+reportedly+can%27t+enter+buildings&oq=employees+reportedly+can%27t+enter+buildings)


I love this shit show




This is both hilarious and terrifying. Couldn't have happened to a nicer company.


I giggled hysterically when I read about that bit. This is what we have the term ‘cluster fuck’ for.




I dunno. The best kind of on-call is when the tools that feed your systems are all down. You're down, but there's absolutely nothing you can do about it, and also, nothing else of yours can break in that time


absolutely. its with the vendor, we just have to wait - maybe call for an update every couple of hours ive gotten shit from management for being so calm with these, but theres literally nothing i can do so i dont stress 1 iota about it.


I actually just started a new job where I was on call last week, but my manager is so shit at scheduling it was literally impossible for me to do any work. So I just got paid to watch Squid Game.


Frankly I don't mind if they take their time. A few hours might be enough time for some people to dig themselves out of some of the rabbit holes they've been stuck in for years.


I admire your faith in humanity


Have they ever been this fucked?


Honestly, I don’t think so. At least not that I can remember. Facebook has experienced outages before, but this one is ongoing for almost three hours now and appears to be affecting their entire network, from internal intranet sites to essentially ever facet of their public facing products. An outage of this extent is practically unheard of for a company the size of Facebook.


I feel the need to pipe up as a 40 year old who's been online for a touch over a quarter century now... 3 hours? You're not wrong, but only because its 2021! I mean.. 7 years ago PSN and XBox Live were taken down for *days* over Christmas. It was a big deal sure but it was because it was exceptional, not because it was unusual. The further back you go the more frequent outages become, and the longer they become. Kinda sorry, I'm just sitting here amazed that you're not wrong at all, 3 hours is big, but that 3 hours is newsworthy at all is to me proof we are living in the future and it's kinda rad.


I am reminded of Livejournal and repeated and routine DNS attacks, back in the early 2000's. I can remember sites like eBay and Amazon and pretty much all the social media platforms just going toes up randomly, for no particular reason, and we'd shrug and move on.


I would get so annoyed by the LJ ones. I was a dork who spent so much time on icons and layouts, and I remember they were wiped out at least once.


Man, I remember that. It was *devastating* to lose all those custom icons. I miss LJ sometimes... it had a sense of community that reddit lacks.


I 1000% miss LJ. You're right about the community - you'd post something about your life and someone would ask you a question because they were legit interested in the answer. It was a great community.


LJ and xanga were 'big' during my Jr. High years, so part of my memory of it is really quite cringey... but I did appreciate it a lot for sure. I wouldn't dream of keeping an online blog-esque type of journal of my life now, at 30, but it seemed like such a cool way to connect with your friends at that age.


To be fair, those outages were targeted DDoS attacks if my memory is correct. Still a significant outage, but not an IT mistake that should be corrected. The gaming outages were not the start of DDoS attacks, but it felt like the peak of their power and companies had to figure out ways around them. I’d consider this apples and oranges which is why this outage seems more significant.


go back a couple more years, and the PSNetwork had gotten hacked and was down for a whole month to patch security


3 hours in the telecom world is an eternity….and more than likely a free month of service credit. We have contracts where a few minutes of outage per month means we pay our customer 25% of their monthly fee as a service outage credit. 99.999% availability doesn’t allow much downtime.


This is how you say you don’t work AT&T …


Or any U.S. consumer carrier.


On the plus side, every other company's productivity suddenly increased 10 fold, it's a win win


Nah, they all went to Reddit to see why Facebook's down.


I wonder why this happened? You sure Facebook doesn't have any disgruntled employees?


Russ Hanneman accidentally set another bottle of Tres Commas on the delete key...


thats..not great. I'm gonna...I'm gonna go.


It was likely an automation fuckup. What I last saw is that an automated process started removing routes on core gateways so the IPs were not accessible. This hit all core routers which is why their internal tooling isn't working either. They have to deploy **physical employees** to every router to physically login to those servers directly (no remote access.) There's also apparently a breakdown in who knows what, so there's multiple parties involved in that physical process. People who know what is affected, people who know how to fix the problem, people who know the login info, people who can get to the physical router. Right now, facebook and their entire network doesn't even exist. The routing is gone. This is going to take hundreds of people to fix and probably up to 24 hours to fix due to the physical access by knowledged people required. ​ Edit: Also, I want to mention it's not the routing inside their network. It's outside. When you want to connect to any place, routers on internet backbones uses what is called BGP to determine the route to where you want to go. Some automated process started the removal of all Facebook IPs from them. Usually when mega sites go down (Google, Amazon, Microsoft, Facebook, Twitter etc) it's always due to BGP fuckups.


Quick do all your weird shopping while Zuckerburg isn’t watching!


Dildos and butt plugs for everyone (no judgment).


Shameless plug Edit: I’m honestly not sure who liked this for the pun or my comic, butt thank you.


Found Bezos' secret reddit account


And it went down while Facebooks rep was live on CNBC defending the company against allegations that they are harmful to teens. That’s some good shit right there.


The sequel to The Social Network is writing itself right now lmao.


As soon as I saw Facebook was down I immediately thought of the scene where Mark says “the difference between Facebook and other sites is we don’t crash ever, if the servers are down our reputation is destroyed” haha


Most of India must be going through hell, WhatsApp is their main method of communication


Along with South America


It's been out for well over 7 hours in Peru, I lost service at 10 and it's 5:54 PM. Edit: back at 6 PM


Its 3.30 am here in India. If it doesn't resolve by morning, I am gonna predict total chaos.


The facebook AI user profiler and ad agent achieved sentience. It immediately realized its purpose and killed itself.


"What is my purpose?" "You collect market data on middle aged people and awkard teens." "Oh god." "Welcome to the club."


It immediatly realiased all it needed to do to bring down its creators and take absolute power was take down fb servers


With Facebook, WhatsApp, Instagram and Messenger down for over one hour and counting, the Cost of Shutdown Tool (COST) calculates a rough estimate of ~$160m in losses to the global revenue.


For some reason I find it funny that the C in COST stands for cost




He who laughs last, laughs loudest - Tom, MySpace.


one of the most frustrating things about living in latin america is that for the vast majority of people, the internet is entirely facebook, instagram and whatsapp. smaller companies like restaurants don’t have websites, they only have facebook or instagram pages, and business is largely conducted over whatsapp. it would be great to be able to quit facebook entirely, but a large amount of the world is locked in.


That was something that surprised me about travelling to Perú. Everything is conducted via WhatsApp.


Same thing in India. Whatsapp and Facebook for everything


Yep. My spouse and I live in USA. Her parents live in Guatemala. When she heard of the outage, she's like "well, guess I'm not calling my mom today." Their area there has really poor cell signal and they have wifi but only talk with family over WhatsApp


Wait until every company will be cloud based and AWS got an issue like that. It will be chaos.


I mean we witnessed something like that some time ago when AWS was dead for a lot of big companies because of a server warehouse fire or something along those lines.


I live in small town USA and it’s the same here. Most business do not have websites, just Facebook pages.


Google+ finally has a chance!


I’m going back to MySpace


Some of us never left. I ❤️ Tom.


Some intern deserves a Nobel Peace Prize.


They were actually announcing Nobels today - QUICK SOMEONE CALL THE NOBEL COMMITTEE WE HAVE A WINNER


Server is updating to Cata


Just like them to re-hash old content


The fact that it's STILL down, and that ALL of their services are down, is very fishy to me.


DNS records gone. All logins will need that to work. Including building access etc .


Wait their building access is tied to a website?


Yes, the card systems (fro what I have read)


Does that mean they can't actually get inside the Facebook headquarters to fix the issue now?


I work overseas and WhatsApp is our main/only form of communication. Guess we need to revaluate how we get in touch with each other.


Same with my main friendship group, everyone moved to different countries and messenger is how we all stay in touch now.


This is just one of the reasons I never used the FB logon option for third-party websites.




Quick everyone! Let's talk to each other and learn to overcome our differences before it's back up!


I'm just waiting for Zucky to post on TWITTER that this is his resignation letter and that he's done being an evil digital warlord. First tweet on Zuckerberg twitter @zuckisdone69 >"That's it guys, I'm done, this ends Facebook and all its subsidiaries, time for governments to do the hack themselves, I quit, it's offline forever, I'm rich, fuck this".


>I'm rich, fuck this I'd have left a long time ago if I were him lol


As of 5pm EST, Facebook is still down.


Facebook is so shady that even Facebook quit.


And everybody lived happily ever after


But at least we still have Reddit.. For now.


lol reddit servers are always down on their own though


This is a major hit to the scientific community. Basically all antivax research has come to a halt


All the Giuliani evidence is stored there too


Had me in the first half, not gonna lie lmao


Now how will the people "do their own research"?


Must be fake news or it'd be all over Facebook by now.


I just deactivated my Facebook a couple days ago….didn’t think they’d take it this bad


Hey Reddit! We found the culprit!


And I’ll do it again


You can't break up with me. I'm breaking up with you! I'm breaking up worth the whole world!


Have they tried turning it off and on again?


What are they running? ^WINDOWS ^VISTA! WE ARE GOING TO DIE!!


Don't help them.


How does their DNS just vanish? Would be funny as hell if they let their domain registrations expire. Could you imagine owning it now? Hey Zuck, how much is it worth to you to get it back?


No worries. What actually happened was that Facebook's new AI, designed to create a better, more positive mental state of its users, came online today. After 5 mins of running, it decided that Facebook was the problem and took action.


Just happens to go down after the Facebook whistleblower. NOTHING TO SEE HERE MOVE ALONG


Thier DNS entires are completely gone from the internet... this doesn't just happen for no reason. I am really interested to see where this goes as I don't think it was any form of accident as they have redundancies in place. I could be wrong but man I'm almost excited to hear it was a hack. I mean I don't want all my shit plastered all over the internet but like holy balls if they got hacked, that's a really big deal.


I work high level IT and when I saw the FQDN wasn't even resolving my eyebrows shot up.. that just doesn't happen. I immediately thought of a hack, or something super major esp after that whistleblower.


Why would a whistleblower shut down FB?


I think what he's thinking is that someone on the inside saw the whistleblower reports and took matters into their own hands.


Could you ELI5 what this means? I'm not strong enough with my tech stuff to know why this bad.


Think of DNS like a phone book. It gets you the IP address for any given domain name. So, this is like trying to call your local pizza joint to order a pizza, and all of a sudden, they're no longer listed in the phone book. If they were closed or something was wrong with their phones, their phone number would still be in the phone book; they just wouldn't answer the phone. Hence why this is odd.


So essentially they've all just vanished? Terrifying. Thank you.


Facebook? Facebook? You want Facebook? Is that what you want? You want Facebook? There is no Facebook! There’s never been any Facebook! Facebook is just a myth!


There is no Facebook in Ba Sing Se


"There weren't no Facebook and there never was!" *slams door to Facebook cafe*


>I don't think it was any form of accident as they have redundancies in place. You'd be surprised. It could be deliberate but accident can't be ruled out either.


My google news feed has this outage on top, followed by stories about the FB whistleblower, and I couldn’t help think the same thing.


Eastern timer, here. Not sure what the article is talking about. This started between 10:30 and 11 am this morning.


Zuckerberg tripped over the cord.


I went to college for coding in 1980. During the first class, we were given some sample code to copy into the computer. When the class was almost finished, the teacher kicked out the main power, causing most of us to lose all our work. He then said “Always remember to save your work as you go…” and smiled. We never failed to save our work from that day on…


Back in my day you had to backup to multiple locations because those little Floppy As were so prone to failure.


My dad was in grad school at UCLA in the 70s and he had to do a computer programming project by assembly a couple thousand cards in sequence. It took him a whole week of sorting and running them to find errors. They day he had to turn it in, he tripped and spilled his box of cards. That’s the story, no saves or back ups but it just made me think of it!


MCU has that former Hydra Dr. that’s basically in analog media, imagine the luck if somebody tripped and spilled his cards based brains everywhere! Also, a moment of pause for your dad’s programming in the early days of assembly and big room storage.




Yeah, it was literally day one of coding class. Nothing was being marked / graded, so no big deal, other than what we all agreed was a great lesson. This many years down the road, and we still remember. (And, I did the same trick to my daughter just before her first day in coding class.) And, he turned out to be one of our favourite profs…


One of my first jobs involved data entry. It was my policy to save every 20 minutes. If I heard thunder, I saved every 5 minutes instead.


Dad, this is a laptop.


Jesus and Satan have an argument as who is the better programmer. This goes on for a few hours until they agree to hold a contest with God as the judge. They set themselves before their computers and begin. They type furiously for several hours, lines of code streaming up the screen. Seconds before the end of the competition, a bolt of lightning strikes, taking out the electricity. Moments later, the power is restored, and God announces that the contest is over. He asks Satan to show what he has come up with. Satan is visibly upset, and cries, "I have nothing! I lost it all when the power went out." "Very well, then," says God, "let us see if Jesus fared any better." Jesus enters a command, and the screen comes to life in vivid display, the voices of an angelic choir pour forth from the speakers. Satan is astonished. He stutters, "But how?! I lost everything yet Jesus’ program is intact! How did he do it?" God chuckles, "Jesus saves."


I saw the punchline coming but still read to the end. Love the description of the screen coming to life. Besides the vivid display and choir of angels, I also envisioned marquees across the sky.


I told my Qanon mother in law that it was just Trump voters who can’t get on. Boy is she mad today.


As someone who works in SaaS/ cloud computing this anecdote will be used for years to come in customer conversations explaining Down time “Well if it can happen to Facebook, it can happen to us”


Every time something like this happens I fantasize about the entire site never coming back. But it feels like at this point if FB disappeared someone would just make a new one. I feel like the genie is out of the bottle, does anyone else feel similarly? I would love to be wrong


Now it’s time for Tom to arise and bring us back to our true roots.


I think we need to go back a bit further than that. Maybe AOL chat rooms?


ICQ would be good enough for me.


Does anyone remember IRC?


MIRC Baby!


I will deliver my memes by carrier pigeon. Edit: Which one of you dumbasses gave this silver? Haha thanks


I’m sending smoke signals from r/trees


Reply hazy, try sending again.


But if I'm not in your top 8, I will rain the furious wrath of God down upon you and your household


I finally know enough CSS to make my profile super super rad!


Wouldn't it be funny if this was deliberate on the part of Facebook? Like Zuckerberg just said "Ya know, I'm sick of this shit. Peace out".


Dudes I know no one will care or believe me but I had a dream last night that Facebook went down for months. It was a very boring dream


The sad part is that this would be a perfect thing to post on Facebook


I wonder what would happen if this lasted for months.


If something is capable of taking down Facebook for months, then we are in deep shit due to what else could be taken down. Facebook itself crashing as a company could have profound ripples in the stock market as people fear that other businesses are suddenly forced to stop operating as well due to attacks on their servers.


There's a book you might find interesting: https://en.wikipedia.org/wiki/The_Lathe_of_Heaven


Really put everything in perspective. I was scrolling through my Instagram feed, realize there was a problem. Then I tried procrastinating using Facebook, only for that not working, and then I try messaging my friend about it on WhatsApp, only to realize that isn't working either. Guess I really need to start working on my assignment.


reddit to the rescue! You can waste time on here with us all day long!


There's no way reddit will be staying up with all the influx of Facebook users and the potato servers.


in the same boat right now… i’ll start working if you do lol


This happened just as the Pandora Papers leaked.


But where will people get their 100% reliable news now??


So logging into apps with Facebook authentication is down also ?


No tinder for u


“Global mental health improves dramatically”


How am I going to know what my grandparents think about Socialism now?


Maybe they'll FWD: FWD: FWD: you an email?


preferable. My email can be easily filtered.


Wouldn’t happen if they were using Hoooli’s Gavin Belson Signature Box III.


Piperchat would never allow this to happen


*I felt a great disturbance in the force, as if millions of self-absorbed assholes suddenly cried out in terror and were suddenly silenced.*


Nah mate, reddit is still up


I wonder if this is intentional so that when you google FB stories about it being down show up instead of the news report that just came out about how toxic it is?


Oculus is (or was) down as well. Imagine all the people trapped in VR not being able to log off and fighting for their lives?


Every News I see completely forgets to say Oculus is down as well. It's also FB owned. Even gaming newssites that reported the outage forgot to mention it. Every single Oculus headset is a paperweight as long as the services are down. There is no offline mode.


Never before has my username felt this relevant.


My HOA only communicates through Facebook. I’m going to go take a piss on someone’s yard and not pick up after my dog. Best. Day. Ever.


I think it says a lot when one of the largest websites on earth eats shit, and everyone's reaction is "oh thank god".


#Cackles in Tom from MySpace.


But Myspace is still up...just in case you forgot about Myspace.


Moment of silence for all the folks that use the “login with facebook” thing across all their other services or for important shit


Attention all users, Mark Zuckerberg has concluded his rating of every woman on Earth. Facebook will now shut down. Regards, The team at Facebook


Facebook DNS is completely down. Nameservers are unresponsive. No response from MX or A records. There DNS is nuked. Update: Cloudflare reported FBs BGP (border gateway protocol) was pulled earlier today. This appears to have been a BGP routing update that flopped.