3 ms·

If we are all giving out advice let me join in. It’s not wise to dismiss the technical decisions made by developers as “playing with the latest toys”. Because

If we are all giving out advice let me join in. It’s not wise to dismiss the technical decisions made by developers as “playing with the latest toys”. Because you are ignorant. You have no understanding of the business requirements, NFRs etc that have led to the decision. So I prefer to give developers the benefit of the doubt. Lambda architectures have trade-offs like every architecture and they are useful in certain situations. Building everything in one service and scaling it when the components have different scaling behaviours usually isn’t ideal.

hinkley 3y ago
> The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Address this part, or you’re just deflecting. Comments? Suggestions?
- threeseed 3y ago
  Lambdas trade developer experience for operational simplicity. And they suit use cases where they are independent from each other and can be tested in isolation eg transcoding. In the early days of AWS Lambda I saw teams trying to apply older techniques to lambdas and get stuck with things like distributed transactions but industry has learnt a lot since then. So as always right tool for the right job.
  - hinkley 3y ago
    If it doesn’t work for the developers, then eventually it won’t work for anybody. Trading developer experience for just about anything does not scale - up or down. It’s a false economy.
    - threeseed 3y ago
      And if it can’t scale and run reliably in production then it won’t work at all. Writing some code and throwing it over the fence to some Ops person to manage is not workable long term. It’s why DevOps exist because developers have to take some responsibility for operating their own code. And Lambda architectures help with this.
      - hinkley 3y ago
        This whole conversation isn’t about using Lambda. It’s about using an exorbitant amount of lambda. Which means you’re using it to the exclusion of other things. Once a project is segregated to this degree, invariably discovery has gone out the window. You have a couple of people who really understood the system and little hope of replacing or duplicating them. It’s just another form of job security. Ignore Kernighan and make a system even you barely understand.
    - alex_lav 3y ago
      > Trading developer experience for just about anything does not scale - up or down. Statements made confidently while also being totally untrue.
      - zaphirplane 3y ago
        Nothing to do with the interaction above. Hacker news is like that, the person you reply to could be between the person that invented and built the tech discussed or a kid replying quickly before going to do their chores. Always fun to see either way
      - hinkley 3y ago
        That line is companion to the previous: > If it doesn’t work for the developers, then eventually it won’t work for anybody. Don’t conflate technical challenge with logistical challenge. Picking between two algorithms for condensing data is technically challenging. Picking between a solution that is transparent and one that is opaque are not the same kind of challenging. As an old mentor used to say, “I hate the term ‘tech debt’. This [situation] isn’t tech debt, it’s wear and tear.” Opaque is magic. Hiring people who understand it is expensive. Building people who understand it is part luck and part patience. And the opportunity costs, which we almost never account for, let alone properly, are one of the biggest consequences. What else could you do with that time and, more importantly, energy?
- jppittma 3y ago
  Logging/Observability, and distribution of responsibility among teams. The question is then, "did my component behave correctly for the input parameters." Observability and logging gives you the parameters and the behavior, the complaint gives you the output.
GiorgioG 3y ago
Can we stop pretending we all have Google-scale problems? That’s how we got into the current K8s mess we’re in.
- mlhpdx 3y ago
  I dropped a jaw at the $1,500 per month for the cluster. My mind says 15,000 Lambdas costs basically zero dollars… until they start getting run/used. I think the key thing was the request rate, or maybe that combined with the raw number of functions? I can’t imagine using something that was so expensive just to house functions rather than run them at high frequency.
- mbakke 3y ago
  Ah, the irony of scale! As far as I know, Google does not actually use k8s for their public facing services (or anything at all?). I would love to be corrected on this. Google was able to scale quickly back in the day by keeping things as simple as possible. "1 hard drive per server" level simple. Center everything around one massive job scheduler (Borg). Of course a lot has changed since that time, but I can't see Google upending a established infrastructure by making k8s plugins for their network requirements, tying k8s into Borg, etc. It would give less visibility and debugability for little gain. (But again I have zero inside knowledge here) At scale there is no one size fits all. Kubernetes can fit a lot, but will always fall short of a tailor made solution. For more unsolicited advice from an armchair infrastructure hyperscale expert, please subscribe to my newsletter.

hinkley 3y ago

> The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Address this part, or you’re just deflecting. Comments? Suggestions?

threeseed 3y ago
Lambdas trade developer experience for operational simplicity. And they suit use cases where they are independent from each other and can be tested in isolation eg transcoding. In the early days of AWS Lambda I saw teams trying to apply older techniques to lambdas and get stuck with things like distributed transactions but industry has learnt a lot since then. So as always right tool for the right job.
- hinkley 3y ago
  If it doesn’t work for the developers, then eventually it won’t work for anybody. Trading developer experience for just about anything does not scale - up or down. It’s a false economy.
  - threeseed 3y ago
    And if it can’t scale and run reliably in production then it won’t work at all. Writing some code and throwing it over the fence to some Ops person to manage is not workable long term. It’s why DevOps exist because developers have to take some responsibility for operating their own code. And Lambda architectures help with this.
    - hinkley 3y ago
      This whole conversation isn’t about using Lambda. It’s about using an exorbitant amount of lambda. Which means you’re using it to the exclusion of other things. Once a project is segregated to this degree, invariably discovery has gone out the window. You have a couple of people who really understood the system and little hope of replacing or duplicating them. It’s just another form of job security. Ignore Kernighan and make a system even you barely understand.
  - alex_lav 3y ago
    > Trading developer experience for just about anything does not scale - up or down. Statements made confidently while also being totally untrue.
    - zaphirplane 3y ago
      Nothing to do with the interaction above. Hacker news is like that, the person you reply to could be between the person that invented and built the tech discussed or a kid replying quickly before going to do their chores. Always fun to see either way
    - hinkley 3y ago
      That line is companion to the previous: > If it doesn’t work for the developers, then eventually it won’t work for anybody. Don’t conflate technical challenge with logistical challenge. Picking between two algorithms for condensing data is technically challenging. Picking between a solution that is transparent and one that is opaque are not the same kind of challenging. As an old mentor used to say, “I hate the term ‘tech debt’. This [situation] isn’t tech debt, it’s wear and tear.” Opaque is magic. Hiring people who understand it is expensive. Building people who understand it is part luck and part patience. And the opportunity costs, which we almost never account for, let alone properly, are one of the biggest consequences. What else could you do with that time and, more importantly, energy?
jppittma 3y ago
Logging/Observability, and distribution of responsibility among teams. The question is then, "did my component behave correctly for the input parameters." Observability and logging gives you the parameters and the behavior, the complaint gives you the output.

GiorgioG 3y ago

Can we stop pretending we all have Google-scale problems? That’s how we got into the current K8s mess we’re in.

mlhpdx 3y ago
I dropped a jaw at the $1,500 per month for the cluster. My mind says 15,000 Lambdas costs basically zero dollars… until they start getting run/used. I think the key thing was the request rate, or maybe that combined with the raw number of functions? I can’t imagine using something that was so expensive just to house functions rather than run them at high frequency.
mbakke 3y ago
Ah, the irony of scale! As far as I know, Google does not actually use k8s for their public facing services (or anything at all?). I would love to be corrected on this. Google was able to scale quickly back in the day by keeping things as simple as possible. "1 hard drive per server" level simple. Center everything around one massive job scheduler (Borg). Of course a lot has changed since that time, but I can't see Google upending a established infrastructure by making k8s plugins for their network requirements, tying k8s into Borg, etc. It would give less visibility and debugability for little gain. (But again I have zero inside knowledge here) At scale there is no one size fits all. Kubernetes can fit a lot, but will always fall short of a tailor made solution. For more unsolicited advice from an armchair infrastructure hyperscale expert, please subscribe to my newsletter.