7 ms·

Heard a story once about the badge readers at a European Google office being suspiciously slow. As in, "the duration of a ping to the US and back" slow. Yep,

Heard a story once about the badge readers at a European Google office being suspiciously slow. As in, "the duration of a ping to the US and back" slow. Yep, every badge in and out in the whole office had to go cross-continental (before the door unlocked).

kevincox 5y ago
This is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would preload the readers with the access list or at least have a copy on site. Of course it isn't a no-brainer: - It is still a non-trivial amount of data for an embedded device. - You want to be able to revoke quickly. - You still want to log the access if offline (and how big should that buffer be?).
- TillE 5y ago
  Networked (as they are now) but with a complete local cache seems pretty straightforward. If you've already got an embedded device doing network requests, it's not much of a stretch to stick in a 1GB SD card or whatever.
  - jandrese 5y ago
    There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. You could set the system up to push data out to all of the badge readers whenever there is a change, but that's a big messy operation. A more complex but probably workable solution is to have them readers cache the data but also have the server maintain a list of those caches, and when someone is changed you invalidate all of the readers that the person used within the last cache timeout period. Even then it is still messy, if a reader has temporarily lost network connectivity their cache could be out of sync. It's one of those problems that gets really hairy once you're down in the weeds working out the details.
    - deeblering4 5y ago
      Check every time, but read from cache upon failure and update cache on success.
      - jandrese 5y ago
        That leaves you open to cache attacks still, and doesn't buy you very much.
        deeblering4 5y ago
        Well yeah it’s still a cache, but it buys you the ability to tolerate intermittent directory failures without failing fully closed or open.
    - johannes1234321 5y ago
      > There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. Luckily this isn't a pure IT system, but bound to physical bodies moving around. Some delay is acceptable. Most likely the person is already in anyways.
    - TheAceOfHearts 5y ago
      You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. Why is it a requirement to be able to instantly revoke a person's badge access eeverywhere in real time anyway?
      - zamadatix 5y ago
        > You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. That seems like the system GP just described, including the hairiness you can get into. > Why is it a requirement to be able to instantly revoke a person's badge access everywhere in real time anyway? With 140,000 employees and hundreds of locations across the globe it becomes a question of dealing with hundreds of local databases and dealing with people that need to be in more than 1 or 1 central database (be it truly centralized or cache coherent). Also think beyond the "you're fired and standing right in front of me" use case of revocation for why it's a requirement it be instant. A lost/stolen/replicated card can have traveled the world in the time it takes to be realized compromised.
- tyingq 5y ago
  If only they had a database that could, uh, "span" geographic distances.
- skunkworker 5y ago
  I'm surprised they haven't pushed ranges to datacenters located near the last access. You could maintain strong consistency while having fast local reads, once you have entered a building in a country it pulls you into the local range. So you're only slow when switching.
  - elzbardico 5y ago
    DNS was down, it wouldn’t make a difference if you can’t find the local servers
    - skunkworker 5y ago
      The parent comment above wasn't about DNS, it was about using a centralized authorization mechanism where each check needed to travel from the EU to the United States in order to approve/deny a badge check.
      - elzbardico 5y ago
        And even a decentralized system would be affected by a dns failure, unless you kept a copy of the database on the premises themselves
marginalia_nu 5y ago
Unrelated anecdote, I once worked in an office where the light switches were powered by the cloud. If you pressed the on and off button at the same time, there would be some sort of a race condition and you'd get blinking disco lighting as two threads presumably went while (!bright) { increaseBrightnessBy10%(); sleep(100ms); } and while (!dark) { decreaseBrightnessBy10%(); sleep(100ms); } Then someone had to log into a server and reboot a process before the blinking stopped.
asdff 5y ago
At what point is this an occupational safety hazard akin to blocking a fire door? Ridiculous that something like access to a building requires a network at all, but leave it to technology companies to overengineer a solution and underthink about the implications.
- bellyfullofbac 5y ago
  Upcoming tweet: "Smoke could be seeen at FB data center, but employees said it's nothing to worry about, it's a workaround to get the fire alarm to trigger and override all doors to open.". Did Nest smoke detectors have to go to the cloud?
  - quickthrowman 5y ago
    Exiting a space is always possible by using a request to exit button/sensor or crash bar. Fire codes do not allow for Triangle Shirtwaist situations anymore.
    - anticensor 5y ago
      Crisis coordination centres are exempt from that rule, and Facebook HQ may well qualify as one given how most people use Facebook.
      - quickthrowman 5y ago
        Would you mind providing a code reference from the NFPA or IFC (or equivalent life safety code in your jurisdiction)
- cesarb 5y ago
  Exit from a building is usually not limited by access control. Either there's a button near the door on the inside which releases the door (cutting the current to the maglock, or energizing the solenoid, or even mechanically releasing the latch), or there's an "emergency use only" fire exit nearby with a crash bar which opens only from the inside (and triggers an alarm when opened), or both. So no, slow badge readers are usually not going to be a safety hazard similar to blocking a fire door.
mc32 5y ago
That should not be the case. Any existing user should be in the local cache. Most companies will have a controller in some control room somewhere which manages the door readers. New users yes. Former users should also need a connection. But existing users should not need a round-trip ping.
- deleted 5y ago
  [deleted]
MDRobidoux9507 5y ago
plumps phone

kevincox 5y ago

This is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would preload the readers with the access list or at least have a copy on site. Of course it isn't a no-brainer: - It is still a non-trivial amount of data for an embedded device. - You want to be able to revoke quickly. - You still want to log the access if offline (and how big should that buffer be?).

TillE 5y ago
Networked (as they are now) but with a complete local cache seems pretty straightforward. If you've already got an embedded device doing network requests, it's not much of a stretch to stick in a 1GB SD card or whatever.
- jandrese 5y ago
  There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. You could set the system up to push data out to all of the badge readers whenever there is a change, but that's a big messy operation. A more complex but probably workable solution is to have them readers cache the data but also have the server maintain a list of those caches, and when someone is changed you invalidate all of the readers that the person used within the last cache timeout period. Even then it is still messy, if a reader has temporarily lost network connectivity their cache could be out of sync. It's one of those problems that gets really hairy once you're down in the weeds working out the details.
  - deeblering4 5y ago
    Check every time, but read from cache upon failure and update cache on success.
    - jandrese 5y ago
      That leaves you open to cache attacks still, and doesn't buy you very much.
      - deeblering4 5y ago
        Well yeah it’s still a cache, but it buys you the ability to tolerate intermittent directory failures without failing fully closed or open.
  - johannes1234321 5y ago
    > There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. Luckily this isn't a pure IT system, but bound to physical bodies moving around. Some delay is acceptable. Most likely the person is already in anyways.
  - TheAceOfHearts 5y ago
    You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. Why is it a requirement to be able to instantly revoke a person's badge access eeverywhere in real time anyway?
    - zamadatix 5y ago
      > You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. That seems like the system GP just described, including the hairiness you can get into. > Why is it a requirement to be able to instantly revoke a person's badge access everywhere in real time anyway? With 140,000 employees and hundreds of locations across the globe it becomes a question of dealing with hundreds of local databases and dealing with people that need to be in more than 1 or 1 central database (be it truly centralized or cache coherent). Also think beyond the "you're fired and standing right in front of me" use case of revocation for why it's a requirement it be instant. A lost/stolen/replicated card can have traveled the world in the time it takes to be realized compromised.
tyingq 5y ago
If only they had a database that could, uh, "span" geographic distances.
skunkworker 5y ago
I'm surprised they haven't pushed ranges to datacenters located near the last access. You could maintain strong consistency while having fast local reads, once you have entered a building in a country it pulls you into the local range. So you're only slow when switching.
- elzbardico 5y ago
  DNS was down, it wouldn’t make a difference if you can’t find the local servers
  - skunkworker 5y ago
    The parent comment above wasn't about DNS, it was about using a centralized authorization mechanism where each check needed to travel from the EU to the United States in order to approve/deny a badge check.
    - elzbardico 5y ago
      And even a decentralized system would be affected by a dns failure, unless you kept a copy of the database on the premises themselves

marginalia_nu 5y ago

Unrelated anecdote, I once worked in an office where the light switches were powered by the cloud. If you pressed the on and off button at the same time, there would be some sort of a race condition and you'd get blinking disco lighting as two threads presumably went while (!bright) { increaseBrightnessBy10%(); sleep(100ms); } and while (!dark) { decreaseBrightnessBy10%(); sleep(100ms); } Then someone had to log into a server and reboot a process before the blinking stopped.

asdff 5y ago

At what point is this an occupational safety hazard akin to blocking a fire door? Ridiculous that something like access to a building requires a network at all, but leave it to technology companies to overengineer a solution and underthink about the implications.

bellyfullofbac 5y ago
Upcoming tweet: "Smoke could be seeen at FB data center, but employees said it's nothing to worry about, it's a workaround to get the fire alarm to trigger and override all doors to open.". Did Nest smoke detectors have to go to the cloud?
- quickthrowman 5y ago
  Exiting a space is always possible by using a request to exit button/sensor or crash bar. Fire codes do not allow for Triangle Shirtwaist situations anymore.
  - anticensor 5y ago
    Crisis coordination centres are exempt from that rule, and Facebook HQ may well qualify as one given how most people use Facebook.
    - quickthrowman 5y ago
      Would you mind providing a code reference from the NFPA or IFC (or equivalent life safety code in your jurisdiction)
cesarb 5y ago
Exit from a building is usually not limited by access control. Either there's a button near the door on the inside which releases the door (cutting the current to the maglock, or energizing the solenoid, or even mechanically releasing the latch), or there's an "emergency use only" fire exit nearby with a crash bar which opens only from the inside (and triggers an alarm when opened), or both. So no, slow badge readers are usually not going to be a safety hazard similar to blocking a fire door.

mc32 5y ago

That should not be the case. Any existing user should be in the local cache. Most companies will have a controller in some control room somewhere which manages the door readers. New users yes. Former users should also need a connection. But existing users should not need a round-trip ping.

deleted 5y ago
[deleted]

MDRobidoux9507 5y ago

plumps phone