4 ms·

Having worked on 25K machines, I can assure you that it never deployed to every single machine and failed to do so in interesting ways all the time.

postalrat 3y ago
As a frontend web developer I'm constantly deploying software to many thousands of machines. And you know what? It's pretty damn simple.
- rokkitmensch 3y ago
  [flagged]
- drdaeman 3y ago
  I used to wear your shoes in IE6/7 ages (no longer, I gave up during the "framework of the week" race and went all-backend), and it wasn't simple at all. Browser compatibility with all their rendering nuances, individual system oddities and all sort of fragile stuff. And fortunately, no one bats an eye at a slightly broken site, but everyone hates even a slightly broken vehicle.
- jrumbut 3y ago
  It's simple because we tolerate certain limitations in the web platform. If you had a hard requirement that a page load could never take more than 100ms, regardless of network conditions, you'd have quite a challenge on your hands.
  - onion2k 3y ago
    The laws of physics are definitely very challenging. If you've got a solution please write a blog post.
    - jrumbut 3y ago
      No blog post required! You just install the whole app on a dedicated piece of hardware on site. But then deployment becomes more challenging ;)
      - postalrat 3y ago
        Those deployments are called PWAs.
- onion2k 3y ago
  I'm not really a frontend dev any more but I was for a long time. I can assure you that the only reason you think your code works is because no one tells you it's broken. If you use an error logging or telemetry service (Sentry, Rollbar, New Relic, etc) you will be aware that errors happen in frontend code all the time. It's just that most of the time bugs don't crash the app, and the user doesn't know what to expect so they see a broken feature and think it's meant to be like that.
- uw_rob 3y ago
  I don't think it's fair to consider the updaters for either Chrome or the OS to be simple.
latchkey 3y ago
It always deployed. It was eventually consistent. Any failure would automatically be resolved after a period of time.
- psychlops 3y ago
  Interesting. At any point in time, I had errors from hardware, software and networking. Even the racks would be getting overwhelmed at certain times. Simply being able to ssh into every host wasn't guaranteed. I'm not sure how you did it.
  - kuchenbecker 3y ago
    +1 to this, we have a 0.1% hardware failure rate every time we do a rolling restart (40-50k nodes). Some just never come back, in the best case, but actively misbehave in the worst. If the node is unresponsive we remove it from the cluster and fix it async.
    - latchkey 3y ago
      If the daemon was running, it would ping a central server on a schedule and report its status, the response from the server was if there was a new version available (with the binary in the response), or not. This combined ping/update service really cut down on the overall traffic, and failures. If the machine had crashed, it would start up, start my daemon, and that daemon would start the ping/update process all over again. A large portion of the machines were iPXE booted... so, just reboot was one option and it would all start from scratch again. Yes, some of the boxes had flaky power supplies or would fail an ssd, and that would cause a technician to go out and manually fix things. I found it was critical to think of everything as eventually consistent because my hardware was boxes with 12 GPUs and they were flaky and would crash the whole box randomly. I got used to boxes rebooting hundreds of times. My process would also auto-tune the GPU for stability too, changing clock/power settings until the individual cards would become stable and stop the crashing. The only time I had problems was when the daemon was dead. I had a dashboard where I could see which machines hadn't reported their status. It was easy to pick those off by hand.

postalrat 3y ago

As a frontend web developer I'm constantly deploying software to many thousands of machines. And you know what? It's pretty damn simple.

rokkitmensch 3y ago
[flagged]
drdaeman 3y ago
I used to wear your shoes in IE6/7 ages (no longer, I gave up during the "framework of the week" race and went all-backend), and it wasn't simple at all. Browser compatibility with all their rendering nuances, individual system oddities and all sort of fragile stuff. And fortunately, no one bats an eye at a slightly broken site, but everyone hates even a slightly broken vehicle.
jrumbut 3y ago
It's simple because we tolerate certain limitations in the web platform. If you had a hard requirement that a page load could never take more than 100ms, regardless of network conditions, you'd have quite a challenge on your hands.
- onion2k 3y ago
  The laws of physics are definitely very challenging. If you've got a solution please write a blog post.
  - jrumbut 3y ago
    No blog post required! You just install the whole app on a dedicated piece of hardware on site. But then deployment becomes more challenging ;)
    - postalrat 3y ago
      Those deployments are called PWAs.
onion2k 3y ago
I'm not really a frontend dev any more but I was for a long time. I can assure you that the only reason you think your code works is because no one tells you it's broken. If you use an error logging or telemetry service (Sentry, Rollbar, New Relic, etc) you will be aware that errors happen in frontend code all the time. It's just that most of the time bugs don't crash the app, and the user doesn't know what to expect so they see a broken feature and think it's meant to be like that.
uw_rob 3y ago
I don't think it's fair to consider the updaters for either Chrome or the OS to be simple.

latchkey 3y ago

It always deployed. It was eventually consistent. Any failure would automatically be resolved after a period of time.

psychlops 3y ago
Interesting. At any point in time, I had errors from hardware, software and networking. Even the racks would be getting overwhelmed at certain times. Simply being able to ssh into every host wasn't guaranteed. I'm not sure how you did it.
- kuchenbecker 3y ago
  +1 to this, we have a 0.1% hardware failure rate every time we do a rolling restart (40-50k nodes). Some just never come back, in the best case, but actively misbehave in the worst. If the node is unresponsive we remove it from the cluster and fix it async.
  - latchkey 3y ago
    If the daemon was running, it would ping a central server on a schedule and report its status, the response from the server was if there was a new version available (with the binary in the response), or not. This combined ping/update service really cut down on the overall traffic, and failures. If the machine had crashed, it would start up, start my daemon, and that daemon would start the ping/update process all over again. A large portion of the machines were iPXE booted... so, just reboot was one option and it would all start from scratch again. Yes, some of the boxes had flaky power supplies or would fail an ssd, and that would cause a technician to go out and manually fix things. I found it was critical to think of everything as eventually consistent because my hardware was boxes with 12 GPUs and they were flaky and would crash the whole box randomly. I got used to boxes rebooting hundreds of times. My process would also auto-tune the GPU for stability too, changing clock/power settings until the individual cards would become stable and stop the crashing. The only time I had problems was when the daemon was dead. I had a dashboard where I could see which machines hadn't reported their status. It was easy to pick those off by hand.