3 ms·

I work in gaming and have experienced the opposite side of this: many of our services have more than one "kind" of update, each with its own caveats and gotchas

I work in gaming and have experienced the opposite side of this: many of our services have more than one "kind" of update, each with its own caveats and gotchas, so that it takes an expert in the whole system (meaning really almost ALL of our systems) to determine which would be the least impactful possible one if nothing goes wrong. Not only is there a lot of complexity and lost productivity in managing this process ("Are we sure this change is zero downtime-able?" "Does it need a schema reload?" etc) but we often get it wrong. The result is that, in practice, anything even remotely questionable gets done during a full downtime where we kick players out. It's sometimes helpful to have the option to just restart one little corner of the full system, to minimize impact, but it is helpful to customer experience (if we don't screw it up) and very much the opposite for developer experience (it's crippling to velocity to need to discuss each change with multiple experts and determine the appropriate type of release).

rozap 2y ago
No doubt that traditional deployments are much better for dev experience at (sometimes) the cost of customer experience.
- ElevenLathe 2y ago
  One thing that would help both is deployment automation that could examine the desired changes and work out the best way to deploy them without human input. For distributed systems, this would require rock-solid contracts between individual services for all relevant scenarios, and would also require each update to be specified completely in code (or at least something machine readable), ideally in one commit. This is a level of maturity that seems elusive in gaming.
- toast0 2y ago
  I disagree. Hot loading means I can have a very short cycle on an issue, and move onto something else. Having to think about the implications of hot loading is worth it for the rapid cycle time and not having to hold as many changes in my mind at once.

rozap 2y ago

No doubt that traditional deployments are much better for dev experience at (sometimes) the cost of customer experience.

ElevenLathe 2y ago
One thing that would help both is deployment automation that could examine the desired changes and work out the best way to deploy them without human input. For distributed systems, this would require rock-solid contracts between individual services for all relevant scenarios, and would also require each update to be specified completely in code (or at least something machine readable), ideally in one commit. This is a level of maturity that seems elusive in gaming.
toast0 2y ago
I disagree. Hot loading means I can have a very short cycle on an issue, and move onto something else. Having to think about the implications of hot loading is worth it for the rapid cycle time and not having to hold as many changes in my mind at once.