5 ms·

Everyone was locked out in a building am staying at (40 something stories) for several hours. When I asked the concierge if I can have a look at the system, it

by csomar 9mo ago

Everyone was locked out in a building am staying at (40 something stories) for several hours. When I asked the concierge if I can have a look at the system, it turns out they had none. The whole thing communicated with AWS for some subscription SaaS that provided them with a front-end to register/block cards. And every tap anywhere (elevators/doors/locks) in the building communicated back with this system hosted on AWS. Absolute nightmare.

exikyut 9mo ago
I wonder what happened to the building when us-east-1 went down.
- csomar 9mo ago
  This is in SEA. They probably operate from ap-southeast-1 or 2. But yeah, if the internet goes down, the provider service goes down or AWS goes down they are cooked.
- Ekaros 9mo ago
  Now I am waiting for time when they move us-east-1 physical security to run in us-east-1... Thus locking themselves out when needing some physical intervention on servers to get backup.
  - proctorg76 9mo ago
    Facebook already got bit by this when their BGP setup pooped its pants on Oct 4, 2021
  - iamtedd 9mo ago
    They could ask facebook for advice. https://skeptics.stackexchange.com/questions/52448/were-facebook-employees-unable-to-enter-their-own-building-to-fix-router-problem https://skeptics.stackexchange.com/questions/52448/were-face...
- blitzar 9mo ago
  I wonder what happened to the building when the internet went down. How do you get into the room to reboot the router?
  - mystifyingpoi 9mo ago
    There is probably a break-glass procedure for such cases, like, break the literal window.
    - blitzar 9mo ago
      Is that why there is a brick next to the procedure manual?
      - lazide 9mo ago
        That’s the emergency escape brick.
    - bluGill 9mo ago
      A lot of modern glass is hard to break. In many cases this is a safety feature (if you can't break the glass you can't get shoved out the window in a fight...)
  - stephen_g 9mo ago
    There’s usually a back door with a physical key. The problem can be getting ahold of one of the people with that key though!
- Someone 9mo ago
  As the parent said: “Everyone was locked out in a building am staying at (40 something stories) for several hours.”
sofixa 9mo ago
> Absolute nightmare. Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.
- Nextgrid 9mo ago
  > with redundant power, cooling, etc The doors the system controls don't have any of this. Hell, the whole building doesn't have any of this. And it definitely doesn't have redundant internet connections to the cloud-based control plane. This is fear-mongering when a passive PC running a container image on boot will suffice plenty. For updates a script that runs on boot and at regular intervals that pulls down the latest image with a 30s timeout if it can't reach the server.
  - lazide 9mo ago
    Those devices can be trivially power cycled, and won’t have as many issues with dodgy power. Some PC somewhere with storage is a bigger problem.
    - Nextgrid 9mo ago
      > Some PC somewhere with storage is a bigger problem Both an embedded microcontroller and a PC have storage. The reason you can power-cycle a microcontroller at will is because that storage is read-only and only a specific portion dedicated to state is writable (and the device can be reset if that ever gets corrupted). Use a buildroot/yocto image on the PC with read-only partitions and a separate state partition that the system can rebuild on boot if it gets corrupted and you'll have something that can be power-cycled with no issues. Network hardware is internally often Linux-based and manages to do fine for exactly this reason.
      - lazide 9mo ago
        PCs are orders of magnitude more complex, with a lot more to break. Sounds like a whole lot of work for… what? Assuming the internet connection and AWS work of course. Which they won’t always, then oops.

by csomar 9mo ago

exikyut 9mo ago

I wonder what happened to the building when us-east-1 went down.

csomar 9mo ago
This is in SEA. They probably operate from ap-southeast-1 or 2. But yeah, if the internet goes down, the provider service goes down or AWS goes down they are cooked.
Ekaros 9mo ago
Now I am waiting for time when they move us-east-1 physical security to run in us-east-1... Thus locking themselves out when needing some physical intervention on servers to get backup.
- proctorg76 9mo ago
  Facebook already got bit by this when their BGP setup pooped its pants on Oct 4, 2021
- iamtedd 9mo ago
  They could ask facebook for advice. https://skeptics.stackexchange.com/questions/52448/were-facebook-employees-unable-to-enter-their-own-building-to-fix-router-problem https://skeptics.stackexchange.com/questions/52448/were-face...
blitzar 9mo ago
I wonder what happened to the building when the internet went down. How do you get into the room to reboot the router?
- mystifyingpoi 9mo ago
  There is probably a break-glass procedure for such cases, like, break the literal window.
  - blitzar 9mo ago
    Is that why there is a brick next to the procedure manual?
    - lazide 9mo ago
      That’s the emergency escape brick.
  - bluGill 9mo ago
    A lot of modern glass is hard to break. In many cases this is a safety feature (if you can't break the glass you can't get shoved out the window in a fight...)
- stephen_g 9mo ago
  There’s usually a back door with a physical key. The problem can be getting ahold of one of the people with that key though!
Someone 9mo ago
As the parent said: “Everyone was locked out in a building am staying at (40 something stories) for several hours.”

sofixa 9mo ago

> Absolute nightmare. Yes, but still probably a million times easier for both the building management and the software vendor to have a SaaS for that, than having to buy hardware to put somewhere in the building (with redundant power, cooling, etc.), and have someone deploy, install, manage, update, etc. all of that.

Nextgrid 9mo ago
> with redundant power, cooling, etc The doors the system controls don't have any of this. Hell, the whole building doesn't have any of this. And it definitely doesn't have redundant internet connections to the cloud-based control plane. This is fear-mongering when a passive PC running a container image on boot will suffice plenty. For updates a script that runs on boot and at regular intervals that pulls down the latest image with a 30s timeout if it can't reach the server.
- lazide 9mo ago
  Those devices can be trivially power cycled, and won’t have as many issues with dodgy power. Some PC somewhere with storage is a bigger problem.
  - Nextgrid 9mo ago
    > Some PC somewhere with storage is a bigger problem Both an embedded microcontroller and a PC have storage. The reason you can power-cycle a microcontroller at will is because that storage is read-only and only a specific portion dedicated to state is writable (and the device can be reset if that ever gets corrupted). Use a buildroot/yocto image on the PC with read-only partitions and a separate state partition that the system can rebuild on boot if it gets corrupted and you'll have something that can be power-cycled with no issues. Network hardware is internally often Linux-based and manages to do fine for exactly this reason.
    - lazide 9mo ago
      PCs are orders of magnitude more complex, with a lot more to break. Sounds like a whole lot of work for… what? Assuming the internet connection and AWS work of course. Which they won’t always, then oops.