7 ms·

You build it, you run it [pdf]

39 points by kiyanwang 5y ago

ggeorgovassilis 5y ago
I'll run it if you let me build it my way
- bob1029 5y ago
  I do believe that is the entire point of this. It can be a trap if you don't start with clear requirements, which is usually always.
- MikeDelta 5y ago
  True, I've seen cases where DevOps is that you have to build it their way, but all problems are yours.
kristianov 5y ago
I built it, I run it. I quit, your startup fails.
bodge5000 5y ago
By the looks of things, written by two people who will neither build it or run it
outside1234 5y ago
I think actually the middle way is the best. This is when you have a central platform that is operated by a Platform Team that App Teams can use to deploy their workloads -- but that the App Teams are expected to observe and monitor their own app to make sure it is healthy. This way each individual App Team doesn't have to become a PhD at Kubernetes and the Platform Team (aka "Ops") doesn't have to become a PhD at 50 different apps.
- pram 5y ago
  I agree. Provide a managed pattern that’s good enough for most things. An “on-road” way to deploy and operate. The devs don’t need to worry about designing and janitoring anything but their application. Of course… the “platform” can and probably will become crap over the long run. Or it’s too doctrinaire and opinionated inflexible and new people don’t like it. Then you end up with alternative platforms popping up over the years, and now there’s multiple ops teams working in different silos (legacy and new etc) and eternal migrations. The circle of life!
  - dikei 5y ago
    Yeah, it's kinda inevitable that "the platform" will need rebuilding again and again, because who want to touch that N-1 version that's all yucky and icky.
- lkschubert8 5y ago
  My fear with this is always that the platform team stops building a platform that is particularly useful. "The Platform" becomes the important thing to them instead of the product it delivers. It's also a good way for the platform team to get viewed as a cost center.
  - outside1234 5y ago
    All valid points.
tkiolp4 5y ago
I don’t want to run it… unless you pay me two salaries: one for knowing all the stuff about building modular and maintainable systems, and another for knowing all the stuff about infrastructure topics (conf. management, security, monitoring, scalability, networking, etc.) As a dev, I do know a bit about infra topics (just because I’m curious) but not enough to be oncall on Sunday mornings.
- muxator 5y ago
  Building a _runnable_ system is yet another ability, though. This may not apply to you specifically, but - for example - think about: - clear error messages (with the actual data that caused it, minus privacy-sensitive information) - possibility of reloading configuration without having to restart the whole application (configure-once-at-construction-time is not enough; not every application is a web application with a php-like lifecycle) - atomic check pointing and restoring of long running operations, without leaving a trail of partially truncatd temporary files - security: you do not know how to write secure software until someone hacks your software in an unexpected way. This includes knowing how actual infrastructure configuration affects the running software (symlinks, permissions, capabilities) - performance: unless "scale" a software means blindly increasing the number of replicas, there are so many architectural lessons to be learned studying flame graphs of actually running software that it is hard to imagine to acquire those expertise without making your hands dirty. At that point, it is even possible that your software works so well it no longer needs so much hardware
  - klysm 5y ago
    I think dynamic reconfiguration leads to less runnable software - make everything as immutable as possible and it’s easy to reason about.
    - muxator 5y ago
      This is indeed true in many cases, but not always. If your application is a client-side GUI you may not want to restart it on reconfiguration. This applies to other things as well: think about having to restart your reverse proxy/load balancer each time you add a new route, or rebooting your OS each time you reconfigure a systemd unit, or reinitializing your embedded HW because something changes. The possible incarnations of SW are very much varied, and not everything can be solved in the same way.
joeldg 5y ago
dikei 5y ago
If you want the dev team to monitor their own apps, you'd better give them good tools to do it, i.e don't expect them to restart the server if you don't give them the power button. I've seen this far too many times.
rektide 5y ago
Major lol, because engineers quit. Who runs it then? A bunch of sorry chump ass latecomer teams who are still run by useless ignorant product chumps who cant understand shit & just want features features features. Oh sorry, everything is shit & everyone who built this shitpile split, but you cant really grok any of that can you? Making the engineers the sole & pertpetually responsible ones is a shit show. Common and trash & so typically ignorant.

39 points by kiyanwang 5y ago

ggeorgovassilis 5y ago

I'll run it if you let me build it my way

bob1029 5y ago
I do believe that is the entire point of this. It can be a trap if you don't start with clear requirements, which is usually always.
MikeDelta 5y ago
True, I've seen cases where DevOps is that you have to build it their way, but all problems are yours.

kristianov 5y ago

I built it, I run it. I quit, your startup fails.

bodge5000 5y ago

By the looks of things, written by two people who will neither build it or run it

outside1234 5y ago

I think actually the middle way is the best. This is when you have a central platform that is operated by a Platform Team that App Teams can use to deploy their workloads -- but that the App Teams are expected to observe and monitor their own app to make sure it is healthy. This way each individual App Team doesn't have to become a PhD at Kubernetes and the Platform Team (aka "Ops") doesn't have to become a PhD at 50 different apps.

pram 5y ago
I agree. Provide a managed pattern that’s good enough for most things. An “on-road” way to deploy and operate. The devs don’t need to worry about designing and janitoring anything but their application. Of course… the “platform” can and probably will become crap over the long run. Or it’s too doctrinaire and opinionated inflexible and new people don’t like it. Then you end up with alternative platforms popping up over the years, and now there’s multiple ops teams working in different silos (legacy and new etc) and eternal migrations. The circle of life!
- dikei 5y ago
  Yeah, it's kinda inevitable that "the platform" will need rebuilding again and again, because who want to touch that N-1 version that's all yucky and icky.
lkschubert8 5y ago
My fear with this is always that the platform team stops building a platform that is particularly useful. "The Platform" becomes the important thing to them instead of the product it delivers. It's also a good way for the platform team to get viewed as a cost center.
- outside1234 5y ago
  All valid points.

tkiolp4 5y ago

I don’t want to run it… unless you pay me two salaries: one for knowing all the stuff about building modular and maintainable systems, and another for knowing all the stuff about infrastructure topics (conf. management, security, monitoring, scalability, networking, etc.) As a dev, I do know a bit about infra topics (just because I’m curious) but not enough to be oncall on Sunday mornings.

muxator 5y ago
Building a _runnable_ system is yet another ability, though. This may not apply to you specifically, but - for example - think about: - clear error messages (with the actual data that caused it, minus privacy-sensitive information) - possibility of reloading configuration without having to restart the whole application (configure-once-at-construction-time is not enough; not every application is a web application with a php-like lifecycle) - atomic check pointing and restoring of long running operations, without leaving a trail of partially truncatd temporary files - security: you do not know how to write secure software until someone hacks your software in an unexpected way. This includes knowing how actual infrastructure configuration affects the running software (symlinks, permissions, capabilities) - performance: unless "scale" a software means blindly increasing the number of replicas, there are so many architectural lessons to be learned studying flame graphs of actually running software that it is hard to imagine to acquire those expertise without making your hands dirty. At that point, it is even possible that your software works so well it no longer needs so much hardware
- klysm 5y ago
  I think dynamic reconfiguration leads to less runnable software - make everything as immutable as possible and it’s easy to reason about.
  - muxator 5y ago
    This is indeed true in many cases, but not always. If your application is a client-side GUI you may not want to restart it on reconfiguration. This applies to other things as well: think about having to restart your reverse proxy/load balancer each time you add a new route, or rebooting your OS each time you reconfigure a systemd unit, or reinitializing your embedded HW because something changes. The possible incarnations of SW are very much varied, and not everything can be solved in the same way.

joeldg 5y ago

dikei 5y ago

If you want the dev team to monitor their own apps, you'd better give them good tools to do it, i.e don't expect them to restart the server if you don't give them the power button. I've seen this far too many times.

rektide 5y ago

Major lol, because engineers quit. Who runs it then? A bunch of sorry chump ass latecomer teams who are still run by useless ignorant product chumps who cant understand shit & just want features features features. Oh sorry, everything is shit & everyone who built this shitpile split, but you cant really grok any of that can you? Making the engineers the sole & pertpetually responsible ones is a shit show. Common and trash & so typically ignorant.