3 ms·

This is a compilation of gotcha-discovery-reports, distributed across the surface area of K8s (which, since K8s is huge, covers many nooks and crannies). This i

• tempest_ 6y ago
  Which is why postmortem blog posts are always infinity more enlightening than the "How we build X" posts
  • barrkel 6y ago
    "All happy families are alike; each unhappy family is unhappy in its own way." https://en.wikipedia.org/wiki/Anna_Karenina_principle https://en.wikipedia.org/wiki/Anna_Karenina_principle
    • WJW 6y ago
      Not to knock on Tolstoy but there are many ways in which unhappy families can be grouped together. You have the alcoholic parents, the living-vicariously-through-their-children folks, the abusive parents, etc etc etc. To tie it back to Kubernetes, you have the scheduler-does-not-schedule lessons, the ingress-not-consistent lessons, the autoscaler-killed-all-my-pods lessons, the moving-stateful-services-with-60TB-storage-size-took-20-minutes lessons and probably many more. It's like Churchills explanation of democracy: terrible, but better than all the alternatives. (at scale, at least)
      • teach 6y ago
        I guess it depends on your definition of "at scale" but IMO with Hashicorp's stuff and maybe something like Backstage you get all the benefits k8s but in a package that's much simpler to reason about and manage.
        • corty 6y ago
          By building a neat box with few buttons on the outside and still the same or more complexity on the inside. If something goes wrong, you still have to deal with all the inside parts. The only hope is that the few outside buttons only trigger happy paths without complications. "At scale" this seems unlikely, simplifications usually only work in smaller, simpler use cases.
          • jen20 6y ago
            Nomad very definitely does not beget the same gordion knot of complexity of Kubernetes, while working at substantially larger cluster sizes than Kubernernetes is capable of.
      • MichaelMoser123 6y ago
        it's the too many moving parts and black boxes blues.
      • nyellin 6y ago
        The problem is that everyone needs to learn those lessons on their own. (But I'm biased: I'm working on a troubleshooting tool for K8s which has remediation rules for all those common cases.)
        • amatix 6y ago
          The tool sounds interesting — can you share any more details?
      • wildermuthn 6y ago
        I think you inadvertently explained why Tolstoy is right: while there are many categories of failure, within those categories each failure is it’s own big, and even for the same bug there are often contextual reasons why the bug manifests in one scenario but not another. Yes, there are alcoholics, but every alcoholic has their own lonely story of descent into addiction. There are infinite ways to fail, and only finite ways to succeed. That relative difference between infinite and finite is what Tolstoy is getting at. And this is a helpful perspective, because then success becomes less about trying to prevent infinite failure-points and more about doing a few finite things very well. This is why you see advice from YC like — “don’t worry about competition”, and “talk to users, talk to users, talk to users.”
• hansvm 6y ago
  > There's an old saying that you learn more from failure than from success OT: That was easily the hardest lesson to instill in my math students, and the most impactful once internalized. Being comfortable experimenting with ideas you don't yet fully understand is critical to the learning process.
  • arthur_pryor 6y ago
    it's a hard thing to keep in mind and execute on, even if you know it consciously. i think people in general can often have too much of a move fast and break things attitude, and i tend to be the opposite, but my default tendency toward risk aversion can definitely go to far. balance is important. i think explicitly reminding myself to think about the realistic cost of failure can be helpful. e.g. 15 minutes or an hour starting in an uncertain approach to implementing a software feature, or writing something, or trying to sketch out a proof probably isn't a huge deal. 15 minutes or an hour trying some novel approach to fixing a production bug that's writing bad data when there's some known tedious thing that'd staunch the bleeding in a few minutes, or sinking many hours and/or much money into an uncertain hobby or activity, those might be less worth the risk =) (but i didn't get the impression you were including that sort of thing, just rambling)
    • nyellin 6y ago
      I'm working on a project that lets you script those tedious responses really easily. Would you use something like that?
• nullcat 6y ago
  At the same time your initial reaction must at the very least say something about how k8 is different from other technologies where you won't have this reaction.
  • gen220 6y ago
    Eh, I'd say most popular technologies suffer from the same problem. There's always an accretion disc of "X is garbage/panacea, here's 10 reasons why" content around ubiquitous tools. In fact, I think I can count on on hand, the number industry-standard tools that don't have these wakes. Can you think of any?