4 ms·

This is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would pre

This is true and surprised me. Even with 100K employees and a separate 4096bit RSA key per badge that is only 50MiB of data. You would think that they would preload the readers with the access list or at least have a copy on site. Of course it isn't a no-brainer: - It is still a non-trivial amount of data for an embedded device. - You want to be able to revoke quickly. - You still want to log the access if offline (and how big should that buffer be?).

TillE 5y ago
Networked (as they are now) but with a complete local cache seems pretty straightforward. If you've already got an embedded device doing network requests, it's not much of a stretch to stick in a 1GB SD card or whatever.
- jandrese 5y ago
  There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. You could set the system up to push data out to all of the badge readers whenever there is a change, but that's a big messy operation. A more complex but probably workable solution is to have them readers cache the data but also have the server maintain a list of those caches, and when someone is changed you invalidate all of the readers that the person used within the last cache timeout period. Even then it is still messy, if a reader has temporarily lost network connectivity their cache could be out of sync. It's one of those problems that gets really hairy once you're down in the weeds working out the details.
  - deeblering4 5y ago
    Check every time, but read from cache upon failure and update cache on success.
    - jandrese 5y ago
      That leaves you open to cache attacks still, and doesn't buy you very much.
      - deeblering4 5y ago
        Well yeah it’s still a cache, but it buys you the ability to tolerate intermittent directory failures without failing fully closed or open.
  - johannes1234321 5y ago
    > There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. Luckily this isn't a pure IT system, but bound to physical bodies moving around. Some delay is acceptable. Most likely the person is already in anyways.
  - TheAceOfHearts 5y ago
    You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. Why is it a requirement to be able to instantly revoke a person's badge access eeverywhere in real time anyway?
    - zamadatix 5y ago
      > You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. That seems like the system GP just described, including the hairiness you can get into. > Why is it a requirement to be able to instantly revoke a person's badge access everywhere in real time anyway? With 140,000 employees and hundreds of locations across the globe it becomes a question of dealing with hundreds of local databases and dealing with people that need to be in more than 1 or 1 central database (be it truly centralized or cache coherent). Also think beyond the "you're fired and standing right in front of me" use case of revocation for why it's a requirement it be instant. A lost/stolen/replicated card can have traveled the world in the time it takes to be realized compromised.
      - account42 5y ago
        A couple of seconds delay for the blacklist to propagate is not going to make a difference for a stolen card.
  - marginalia_nu 5y ago
    Why is this a requirement? The instantly part. Any office that uses physical keys will fail to meet this requirement, and we've had that system for quite a long time and it seems to have worked alright.
  - forgetfulness 5y ago
    Turns out that the risk of an employee fired 1 minute ago running full speed, jumping walls and somersaulting into a now-restricted area is a way lower than locking out whole divisions due to network failure. And I can't imagine why would anyone think otherwise.
    - jandrese 5y ago
      And yet this is the requirement. Some companies will also have security show up when it is time to fire someone so they can be escorted directly out of the building.
      - forgetfulness 5y ago
        And that's something that you can implement without creating a huge single point of failure. This was pure hubris, whether it emanated from engineering or elsewhere. Of course, if the employee was indeed a ninja, your guards wouldn't be able to do much, but in all other cases, it will suffice. We all derided the company that made "smart" pet feeders that wouldn't maintain a cache of the feeding schedule, but rather phone home to check; when their servers were down, pets went without food. It's not difficult to think of an scenario like this.
tyingq 5y ago
If only they had a database that could, uh, "span" geographic distances.
skunkworker 5y ago
I'm surprised they haven't pushed ranges to datacenters located near the last access. You could maintain strong consistency while having fast local reads, once you have entered a building in a country it pulls you into the local range. So you're only slow when switching.
- elzbardico 5y ago
  DNS was down, it wouldn’t make a difference if you can’t find the local servers
  - skunkworker 5y ago
    The parent comment above wasn't about DNS, it was about using a centralized authorization mechanism where each check needed to travel from the EU to the United States in order to approve/deny a badge check.
    - elzbardico 5y ago
      And even a decentralized system would be affected by a dns failure, unless you kept a copy of the database on the premises themselves

TillE 5y ago

Networked (as they are now) but with a complete local cache seems pretty straightforward. If you've already got an embedded device doing network requests, it's not much of a stretch to stick in a 1GB SD card or whatever.

jandrese 5y ago
There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. You could set the system up to push data out to all of the badge readers whenever there is a change, but that's a big messy operation. A more complex but probably workable solution is to have them readers cache the data but also have the server maintain a list of those caches, and when someone is changed you invalidate all of the readers that the person used within the last cache timeout period. Even then it is still messy, if a reader has temporarily lost network connectivity their cache could be out of sync. It's one of those problems that gets really hairy once you're down in the weeds working out the details.
- deeblering4 5y ago
  Check every time, but read from cache upon failure and update cache on success.
  - jandrese 5y ago
    That leaves you open to cache attacks still, and doesn't buy you very much.
    - deeblering4 5y ago
      Well yeah it’s still a cache, but it buys you the ability to tolerate intermittent directory failures without failing fully closed or open.
- johannes1234321 5y ago
  > There is a requirement to be able to instantly revoke a person's badge access, so you can't cache the data. It has to check every time. Luckily this isn't a pure IT system, but bound to physical bodies moving around. Some delay is acceptable. Most likely the person is already in anyways.
- TheAceOfHearts 5y ago
  You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. Why is it a requirement to be able to instantly revoke a person's badge access eeverywhere in real time anyway?
  - zamadatix 5y ago
    > You could maintain a local cache which is updated less frequently and a blacklist which gets pushed everywhere when updated. That seems like the system GP just described, including the hairiness you can get into. > Why is it a requirement to be able to instantly revoke a person's badge access everywhere in real time anyway? With 140,000 employees and hundreds of locations across the globe it becomes a question of dealing with hundreds of local databases and dealing with people that need to be in more than 1 or 1 central database (be it truly centralized or cache coherent). Also think beyond the "you're fired and standing right in front of me" use case of revocation for why it's a requirement it be instant. A lost/stolen/replicated card can have traveled the world in the time it takes to be realized compromised.
    - account42 5y ago
      A couple of seconds delay for the blacklist to propagate is not going to make a difference for a stolen card.
- marginalia_nu 5y ago
  Why is this a requirement? The instantly part. Any office that uses physical keys will fail to meet this requirement, and we've had that system for quite a long time and it seems to have worked alright.
- forgetfulness 5y ago
  Turns out that the risk of an employee fired 1 minute ago running full speed, jumping walls and somersaulting into a now-restricted area is a way lower than locking out whole divisions due to network failure. And I can't imagine why would anyone think otherwise.
  - jandrese 5y ago
    And yet this is the requirement. Some companies will also have security show up when it is time to fire someone so they can be escorted directly out of the building.
    - forgetfulness 5y ago
      And that's something that you can implement without creating a huge single point of failure. This was pure hubris, whether it emanated from engineering or elsewhere. Of course, if the employee was indeed a ninja, your guards wouldn't be able to do much, but in all other cases, it will suffice. We all derided the company that made "smart" pet feeders that wouldn't maintain a cache of the feeding schedule, but rather phone home to check; when their servers were down, pets went without food. It's not difficult to think of an scenario like this.

tyingq 5y ago

If only they had a database that could, uh, "span" geographic distances.

skunkworker 5y ago

I'm surprised they haven't pushed ranges to datacenters located near the last access. You could maintain strong consistency while having fast local reads, once you have entered a building in a country it pulls you into the local range. So you're only slow when switching.

elzbardico 5y ago
DNS was down, it wouldn’t make a difference if you can’t find the local servers
- skunkworker 5y ago
  The parent comment above wasn't about DNS, it was about using a centralized authorization mechanism where each check needed to travel from the EU to the United States in order to approve/deny a badge check.
  - elzbardico 5y ago
    And even a decentralized system would be affected by a dns failure, unless you kept a copy of the database on the premises themselves