21 ms·
Facebook employees badges aren't working, unable to enter buildings
- cbtacy 5y ago"And sometimes the universe says, 'go get a different job already.'"
- throwawaymanbot 5y agoSuspicious. What timing eh!
- gmiller123456 5y agoSeems odd. There is NO ONE that can let people in? Every building I've worked in has required badges, but there's always security inside that can check IDs and let people in. And methods of security getting in even if there's a power failure.
- mc32 5y agoThe local readers should have a cache of all known active badges as of their last update (whatever their cycle is). It's not supposed to require a live connection to auth people in, as far as I know.
- tmp_anon_22 5y agoThe current system seems to prioritize de-activation of employee badges versus ease of use for existing employees. A local cache would need an invalidation mechanism that was networked.
- gmiller123456 5y agoThat's not the way a security system should work, it should "fail secure" and not let anyone in that it can't verify 100%. There are lots of reasons to lock all of the doors, or even lock out a single person. And, since such systems can fail frequently, there always needs to be a backup plan in place, almost always involving humans.
- deleted 5y ago[deleted]
- manquer 5y agoThat would be bad, if you need to suspend access effective immediately, either cause the badge is lost or ex-employee is hostile etc. Any sort of cache would give a window of opportunity to an attacker who has access to the badge.
- mc32 5y agoTrue. If the network were down, yes, it should fail secure. But if the network is up it should not go to the server(s) for every swipe/auth. Also not all perimeters have to be treated the same.
- manquer 5y agoIf it didn't go to server for every swipe then that window of opportunity would exist. i.e. the time where server revoked the badge abd the device/cache considers it valid. Even a few minutes could be risky with an ex-employee who knows what he is doing. There are InfoSec reasons why some companies will revoke access as soon or even before telling an employee he is being let go and Security escort the employee off premise. While it is very poor way to handle human relations, it is sound from InfoSec perspective. You are right, as you point out not all perimeters are same, typically more sensitive areas have lesser exceptions though a security guard will not be able to swipe/override you in to a DC as he would in normal office building. Someone else mentioned that they had to trigger Fire Alarms to get all the engineers they needed fast enough at a DC during a down time at eBay.
- daRealDodo 5y agoThe market have never been more ripe for a new player.
- katmannthree 5y agoCan't help but feel that the world would be a slightly better place if it stayed this way.
- ferdowsi 5y agoAgreed. The theoretical fat-fingering SRE who stopped all access to FB has probably done more to benefit humanity than any other single engineer in recent memory.
- aroman 5y agoMany pockets of humanity are materially worse off today because of these outages. The (perhaps sad) reality is that tens of thousands of small businesses are completely dependent on WhatsApp and Instagram, especially in developing countries. This outage will cause lots of lost business by people who don't have massive cash reserves to ride out the lost revenue, unlike Facebook itself.
- cma 5y agoIf it helps them break free of the facebook ecosystem/tax authority it could still be good even if expensive in the short-term.
- caballeto 5y agoBusinesses which rely on Facebook, as their sole distribution channel could get out of business if Facebook goes out. A lot of economical value will be lost.
- ForHackernews 5y agoAllowing copyrights to expire on pre-1950 creative works would also destroy billions of dollars of "economic value", but that doesn't mean it would be bad for humanity. We should be careful to not take our values from a spreadsheet.
- 5y ago
- deadalus 5y agomassive whistleblower leak surfaces about Facebook public opinion starts to change about fb Zuck fears people will abandon his platforms shuts all of the down and claims hax0rs facebook goes down divert the story to Russian and Chinese hackers people instantly forget about whistleblower facebooks stock balances out after a week
- loa_in_ 5y agoPeople wouldn't abandon Facebook et al. even if Zuckerberg was outed as a creature from outer space. Many people just wouldn't care.
- enriquto 5y ago"Well, actually", as someone who has religiously avoided Facebook services for almost 15 years... I'd happily join FB if Zuck turned out to be an alien!
- micromacrofoot 5y agoI put my tinfoil hat on and came up with this early on, but the outage has gone on for far too long for the price to be worth it now. They're losing millions and millions of dollars from this. I'm now leaning towards possible inside job related to the recent string of bad press. If this is an external hack it's incredible. If this is a simple mistake... Facebook seems dangerously incompetent.
- asdff 5y agoFB is in the spy, stock is practically unsinkable at this point what with everyones retirement plan buying and holding it.
- ThePadawan 5y agoHeard a story once about the badge readers at a European Google office being suspiciously slow. As in, "the duration of a ping to the US and back" slow. Yep, every badge in and out in the whole office had to go cross-continental (before the door unlocked).
- kevincox 5y agoThis is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would preload the readers with the access list or at least have a copy on site. Of course it isn't a no-brainer: - It is still a non-trivial amount of data for an embedded device. - You want to be able to revoke quickly. - You still want to log the access if offline (and how big should that buffer be?).
- TillE 5y agoNetworked (as they are now) but with a complete local cache seems pretty straightforward. If you've already got an embedded device doing network requests, it's not much of a stretch to stick in a 1GB SD card or whatever.
- jandrese 5y agoThere is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. You could set the system up to push data out to all of the badge readers whenever there is a change, but that's a big messy operation. A more complex but probably workable solution is to have them readers cache the data but also have the server maintain a list of those caches, and when someone is changed you invalidate all of the readers that the person used within the last cache timeout period. Even then it is still messy, if a reader has temporarily lost network connectivity their cache could be out of sync. It's one of those problems that gets really hairy once you're down in the weeds working out the details.
- deeblering4 5y ago
- deleted 5y ago[deleted]
- EpicDavi 5y agoI would say this is a bit misleading. Personally, I was able to access my FB building this morning (after the outage started) and have continued to be able to use my badge with no problem since. Not saying that there are no employees experiencing this issue, but it is not affecting all Facebook employees.
- hinkley 5y agoGiven it's a routing problem, doesn't it matter where your office is on the network with respect to the servers that control the door locks? You could be in a zone of stability that's not affected by the overall chaos.
- asdff 5y agoHonestly having buildings automatically lock down like this sounds like an absolute worst practice. FB headquarters is in california. What happens when the big one hits? server is killed and no one can get inside and look for survivors? Shouldn't need a functional network connection to be able to open the door of a building, but here we are I guess.
- nicoburns 5y agoApparently this issue is effecting specifically some of the people who are trying to fix the outage https://news.ycombinator.com/item?id=28749244 https://news.ycombinator.com/item?id=28749244
- 5y ago
- coolspot 5y agoTo reconfigure misconfigured routers one needs physical access -> routers are in a high-security area of a secure DC -> physical access systems do not let anyone in due to network outage.
- notyourday 5y agoThe first rule of netops is to have a separate, externally accessible, not controlled by the managed routers network of serial consoles with the entry points connected to IP over LTE or completely dedicated network of a different provider. We have been doing this since 1996
- packetslave 5y ago...and then you go to use them during an outage ...and discover that the POTS into the modem in Finland is flaky so you can't dial-in in from the US. ...and the OOB network access requires a smartcard that's kept in a safe... and the safe's combination is in the secrets tool that requires network access. ...and then you cry. (for sanity's sake: NOT referring to today's FB outage. Don't quote me)
- notyourday 5y agoI'm just saying we have been doing this for decades using KISS principal. It is not sexy and we dont write fancy blog posts about our super smart BGP software that we wrote, but it worked. We had outages. Nasty, nasty outages, where the long distance fiber was down and we got our network partitioned and our neteng was not able to get into the routers via standard means but our OOB worked... because it was dumb and dumb allowed us to recover from really bad problems within 10-15 minutes.
- packetslave 5y agoAnd of course you test your OOB weekly to make sure every device in every POP is accessible all the time, automatically and systematically, right? Otherwise, Murphy's Law ensures that when you desperately need to access the peering router in 60 Hudson, or Equinix Chicago, or 1 Wilshire to fix an outage, that will be the modem that doesn't answer the phone, or the console cable that got pinched in a door. (again, not referring to today's outage)
- intricatedetail 5y agoI hope that all their workers quit and services never come back up. One can only dream...
- suyash 5y agoMost sophisticated and biggest hack. They really thought about all use-cases.
- rvz 5y agoMaybe it is for everyone's own good. The Facebook mafia needed to be shutdown.
- huetius 5y agoWould you (or someone else in-the-know) help me, an out-of-the-loop person, understand why Facebook is reviled so much more intensely on this site than other social media platforms?
- tayistay 5y agoSimply because the average reader of *Hacker News* knows more about the tech industry than the average user of social media platforms.
- colinmhayes 5y agoFacebook's algorithm prioritizes divisive political content that is commonly disinformation(propaganda) because they figured out it increases engagement.
- jspaetzel 5y agoWhat other social media platforms?
- huetius 5y agoIdk, any of the big ones. Reddit, Twitter, TikTok, e.g.
- skinnymuch 5y agoMore reviled than their big tech contemporaries too. Amazon, Apple, Google. Snapchat, Pinterest, Discord (sort of), Youtube (sort of) can round out that social media sites list. Probably a few more not being thought of.
- Apocryphon 5y agoYou forgot Twitter, which imo beats FB on toxicity at times.
- ImBanned 5y agoNow Facebook can see what it's like being banned for 30 days.
- suyash 5y agoYou win the best comment award!! Or what is like to be banned indefinitely!
- Jcowell 5y agoWhat I wonder if this isn’t a scheme on Facebook part to show the world what it feels like without it. At first you might think it’ll “wake up” the addicts and whatnot but you can’t just cut off an addict from their supply and expect them to just be ok with it. Instead they’ll relapse harder without help and might dig in deeper, never wanting to feel the withdraw again. I wonder if there’s a phenomenon called Social Withdrawn Symptoms.
- cblconfederate 5y agoInternet of Things finally being useful
- theHIDninja 5y agoGood. Facebook should be forcefully and globally taken offline for the crimes its sites commit on a daily basis. All Facebook employees should be arrested and put into solitary confinement until we get to the bottom of the crimes they are individually and collectively guilty of. Terroristic actions taken against these corporations are entirely justified. This holds true for Twitter, Google, Apple, and Amazon.
- deleted 5y ago[deleted]
- panarky 5y agoTwitter, Google, Apple and Amazon are not equivalent to Facebook. Facebook is 100x as malevolent as those four combined.
- theHIDninja 5y agoGoogle, Apple and Amazon, maybe. Twitter actively participates in and profits from "cancel" culture so they may as well be direct political rivals to the West.
- sennight 5y agoThis isn't how badge readers work. Every controlled door is wired to a relay panel. That panel connects to a access control db - but it always maintains a cache that is very slow to expire... because the manufacturers weren't clueless morons.
- kamaal 5y ago>>cache that is very slow to expire... People have been WFH/Remote working for a while now. Like 1.5+ years. Besides when everything runs fine it all feels magic. You discover how things can break in a situation like this. Im pretty sure they are now rushing to restore access for every FB employee to their buildings.
- cududa 5y agoExcept some of the readers were made in house. I’ve never been into a Facebook conference room that didn’t use one of their custom touch screen/ card readers.
- sennight 5y agoThat isn't access control, that is a an intern project. Those readers aren't wired to devices that could trap you in a burning building, they simply provide a rube goldberg solution to the traditional sign-in sheet.
- cromka 5y ago> Every controlled door is wired to a relay panel. That panel connects to a access control db Absolutely not. E.g. HID is a well-documented protocol and you can do whatever you want using the RS232 interface they come with (depending on the model). I personally wrote a gateway that would dynamically and directly check the access with my own software.
- sennight 5y agolol, California fire code makes no exemptions for python projects... this whole story has a "virgins talking about sex" vibe.
- footlose_3815 5y agoLocking the doors to shred the evidence! /s But seriously, such odd timing with Pandora Papers and with the Whistleblower.
- mysterydip 5y ago[conference room, some years ago, maybe]: technician: "But if we authenticate using the facebook DNS, how will we do our jobs if it goes down?" boss: "The only way our DNS servers are going down is if facebook closes, in which case you're out of a job anyway."
- leokennis 5y agoAlso possible: technician: "But if we authenticate using the facebook DNS, how will we do our jobs if it goes down?" boss: "That's a good point but not a blocker for go live, let's remember to add a story to our backlog to revisit this."
- InsomniacL 5y agoLots of people drawing conclusions between recent leaks and some sort of attempt to cover them up. I'd suggest more likely is a disgruntled engineer performing some sort of sabotage because of the content of the leaks.
- eointierney 5y agoWhat utterly wretched management at every level, inexcusably idiotic, and suggestive of so much worse about which we don't know. That this company is one of the greatest revenue streams on the planet tells us everything about how badly we evaluate reality. C'mon Humans, let's do better than this. You're challenged!
- MDRobidoux9507 5y agoplump phone