Is Facebook down?

[andrewc]

If it’s not affecting you & you have a pretty cynical outlook to companies such as Facebook then yes.  If something like WhatsApp is your main means of communication then it’s a bit of a nightmare.  Some people will be quite worried.   I’ve never used Facebook & my WhatsApp account is used to chat to one friend only, I tried to persuade her to move to Signal but she couldn’t be bothered.


https://twitter.com/agoodfireburns/status/1445135459030548491?s=21

[Kim]

Problem is that when you raise those sort of issues people think you're being a snobby elitist nerd and use Facebook anyway.

I gave up trying to persuade QUILTBAG groups that using a platform with a long history of bait-and-switch privacy settings and associated outings was a bad idea.  If they won't listen, it's unlikely that the average volunteering group will think twice.

And the sort of people for whom Facebook is the default tool for event organisation are unlikely to have experienced the social exclusion or loss of friendships effect of other people switching their online activity to Facebook.

[Jaded]

If it’s not affecting you & you have a pretty cynical outlook to companies such as Facebook then yes.  If something like WhatsApp is your main means of communication then it’s a bit of a nightmare.  Some people will be quite worried.   I’ve never used Facebook & my WhatsApp account is used to chat to one friend only, I tried to persuade her to move to Signal but she couldn’t be bothered.


https://twitter.com/agoodfireburns/status/1445135459030548491?s=21

Well, thats my point. Facebook have spent the last decade persuading people that It is the internet and It is communication. They use this to make money.

A regular refrain on our shit Facebook community page, that tries to make out it is a huge force for good here, is “have you googled that?” The response is usually either that they don't have the internet, or that it is quicker and better to ask a bunch of randomly selected who-are-they people on Fecebook.

It’s like a reverse AOL.

[Jaded]

Problem is that when you raise those sort of issues people think you're being a snobby elitist nerd and use Facebook anyway.

I gave up trying to persuade QUILTBAG groups that using a platform with a long history of bait-and-switch privacy settings and associated outings was a bad idea.  If they won't listen, it's unlikely that the average volunteering group will think twice.

Indeed.

[Kim]

A regular refrain on our shit Facebook community page, that tries to make out it is a huge force for good here, is “have you googled that?” The response is usually either that they don't have the internet[...]

Is this extreme technical ignorance, or are they on one of those dodgy Indian unlimited-data-as-long-as-it's-Facebook phone tariffs^[1]?


_{[1] I vaguely recall this sort of thing wasn't legal in the EU, which probably means it's another brexit bonus.}

[Jaded]

I think it is that they get a phone, and there is an 'f' icon, and it goes from there.

Ignorance.

Actually similar to "I can't find it on my Google?" And people googling "yacf.co.uk" to get to yacf.co.uk and so on.

cf people running out of Petril because they put £10 in every Friday. (and don't understand that price rises might play havoc with that strategy)

[Jaded]

You need to consider that half of the population is below average intelligence.

Many of the others are capable of stupidity too...

[andrewc]

Looks like stuff is slowly coming back.  Some of my WhatsApp messages are now showing as sent.

[hellymedic]

It's back!

[Jaded]

It sounds like they have got the Army in to get it to work.

[quixoticgeek]

Well it's back, so we can put the popcorn away.

I will however leave this here for your enjoyment:

https://twitter.com/pupOnSecurity/status/1445133562911526917

J

[Kim]

Actually similar to "I can't find it on my Google?" And people googling "yacf.co.uk" to get to yacf.co.uk and so on.

I think that ship sailed when the browser authors joined in and made the address bar double as a google (or whatever) search bar.  The ship then sank when on mobile browsers they hide the address bar and insist you type URLs into the search bar on the home page.   

[FifeingEejit]

I used to quietly laugh when people did that, now I also do it as its the least painful way...

Except in IE11 at work, and that still fucking bings internal addresses if you so much as make a single error.

Sent from my BKL-L09 using Tapatalk

[pcolbeck]

That was quite funny. Seems a BGP config update screwed up and Facebook's routers withdrew the routes that advertised the IP of Facebook's DNS servers. So far so normal if embarrassing screw-up.  Been there done that.

The thing is these things are normally tested. Facebook like all large enterprises will be multiply redundant. They have multiple datacentres and multiple DNS servers located in them. Normally you would update the routers in one and make sure everything was OK before doing the rest. Looks like they pushed out an update to all of them at the same time ....
Still annoying and embarrassing but lets just get the support team to remote in and fix it.

Then it seems non of their engineering team could remote connect to the routers. Standard practice is that you have a separate out of band management network to datacentre kit so if you do screw-up the main network you still have an alternate way in to fix it. Given that when they did send people to location to gain physical access to the serial consoles of the routers apparently there passes wouldn't work to let them in I am guessing that the authentication systems for both building access and remote access had a dependency on the same DNS servers that were unreachable. That's just bad disaster planning.

Oops !

I suspect just rebooting the routers wouldn't have fixed it either.Some of these massive tech companies run there networking kit stateless. That is the configuration isn't saved locally the routers and switches download their config from a server on boot rather than from local storage. Might have been an issue resolving the server name to download the config from ....

[aidan.f]

Yeah,  missing the  'what if' ...

[Beardy]

It sounds like they’ve not done any disaster management gaming, or if they have, had come to the conclusion that yesterdays outage was too far left on the probability axis to warrant any serious investment. Idiots.

[ScumOfTheRoad]

I suspect just rebooting the routers wouldn't have fixed it either.Some of these massive tech companies run there networking kit stateless. That is the configuration isn't saved locally the routers and switches download their config from a server on boot rather than from local storage. Might have been an issue resolving the server name to download the config from ....

Aha. If you are using ansible (or something similar) to configure switches on boot then ..errr... if you cant contact the ansible server...

Actually ansible works by push - you dont need an ansible agent running on the switch AFAIK. However the switch wont be able to reverse lookup the ansible server which is trying to ssh in to configure it.

[robgul]

I worry for the mental health of the FB users that have lost 6 or 7 hours of their lives . . .  probably a positive effect?

I use WhatsApp for intra-family stuff and the two cycling clubs - that's it, and will remain so.   [I may this week have to send my one and only annual text*  - the plumber we use only responds to such communication]

*phones are for speaking into and listening to.

[ian]

Honestly, organizations that use FB to announce things bug me. My local swimming pool, which has a perfectly functional webpage that's accessible to anyone on the internet, for instance, will announce any news (like it's closed at the moment due to flooding – I know) on FB.

[Jaded]

You posted that and YACF went down.

Dark forces...

[Kim]

Honestly, organizations that use FB to announce things bug me. My local swimming pool, which has a perfectly functional webpage that's accessible to anyone on the internet, for instance, will announce any news (like it's closed at the moment due to flooding – I know) on FB.

About half the time they remember to make the facebook group public.  Which would be a lot more useful if half the screen wasn't taken up by a "Login or create an account" popup.

The problem's getting worse too.  Used to be that this sort of thing was pure laziness: Social media had a nice friendly interface for posting content, whereas updating the website was slightly more involved.  But we've now got the additional problem that many otherwise normal people over the age of 30 live on their smartphones and may not even have a proper computer, which makes them seriously disinclined towards competent editing and curation of content.

[Mr Larrington]

My grate frend Al, who is as near as makes no odds the same age as me and therefore Should Know Better, had to be roundly castigated for shooting video in portrait and posting it on Farcebok.  The word “barbarian” may have been involved.

[Kim]

I'm seeing massive latency/packetloss on my EE backup LTE connection.  Squinting at smokeping, it started at the same time that Facebook stopped responding to pings...

This appears to be because I'm pinging one of EE's DNS servers, on the basis that it  a) responds to pings  and  b) is actually within EE's network, rather than farmed out to some CDN.  The connection itself was presumably fine.

It turns out that Facebook disappearing causes a massive increase in DNS traffic, as everything tries to connect to InstaFaceApp.

[Afasoas]

* utters something about walled gardens and walks away in dispair *

[pcolbeck]

I'm seeing massive latency/packetloss on my EE backup LTE connection.  Squinting at smokeping, it started at the same time that Facebook stopped responding to pings...

This appears to be because I'm pinging one of EE's DNS servers, on the basis that it  a) responds to pings  and  b) is actually within EE's network, rather than farmed out to some CDN.  The connection itself was presumably fine.

It turns out that Facebook disappearing causes a massive increase in DNS traffic, as everything tries to connect to InstaFaceApp.

It also massively slowed down a lot of the Internet's DNS servers due to the way Facebook withdrew the routes to the their DNS servers rather than dropping the A records. It meant that rather than just returning an error saying "cant find an address for that url" the other DNS servers went into massive cycles of retrying Facebook requests thus devouring their CPU cycles.