5 ms·

The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a st

The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a stateless edge runtime was a rookie mistake, and the pain they describe was entirely predictable.

nougati 1y ago
The takeaway isn't that they didn't understand, it's that they are sharing information which you agree is valuable
- kburman 1y ago
  What's valuable about rediscovering that stateless architectures requiring network round-trips for state access are slower than in-memory state? This isn't new information, it's a predictable consequence of their architecture choice that anyone with distributed systems experience could have told them on day zero.
  - chronark_ 1y ago
    Not everyone is born with experience in distributed systems
    - sgarland 1y ago
      Sure, but there are some fundamentals about latency that any programmer should know [0] (absolute values outdated, but still useful as relative comparisons), like “network calls are multiple orders of magnitude slower than IPC.” I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics. 0: https://gist.github.com/hellerbarde/2843375 https://gist.github.com/hellerbarde/2843375
      - chronark_ 1y ago
        I cofounded it yeah And yeah you’re right in hindsight it was a terrible idea to begin with I thought it could work but didn’t benchmark it enough and didn’t plan enough. It all looked great in early POCs and all of these issues cropped up as we built it
    - kburman 1y ago
      That's fair, but then the framing matters. The article criticizes serverless architecture rather than acknowledging an evaluation failure. "Serverless was fighting us" vs "We didn't understand serverless tradeoffs" - one is a learning experience, the other is misdirected criticism.
      - chronark_ 1y ago
        Yeah that’s fair
    - lossolo 1y ago
      You don't need experience and there is not really a lot to know about "distributed systems" in this case, that's basic CS knowledge about networks, latency and what "serverless" actually is, you can read about it. To be honest, to me it reads like people who don't understand the problem they're solving, haven't acquired the necessary knowledge to solve it (either by learning themselves or by asking/hiring people who have it), and seeing such an amateurish mistake doesn't inspire confidence for the future. You should either hire people that know what they are doing or upgrade your knowledge about systems you are using before making decisions to use them.
      - nougati 1y ago
        Sometimes I see a post about sorting algorithms online. Some people seem to benefit from reading about these things, but often, I find there isn't much new information for me. That's OK, because I know somebody somewhere benefits from knowing this. It is your decision to make this a circlejerk of musings about how the company must be run by amateurs. Whatever crusade you're fighting in vividly criticising them is not valuable at all. People need to learn and share so we can all improve, stop distracting from that point.
        lossolo 1y ago
        [dead]
  - bcrosby95 1y ago
    Uh, no, 95% of our architectures are stateless and its fine because RTT isn't dogshit, unlike AWS lambda.
- ramraj07 1y ago
  Bo Burmham said, "self awareness does not absolve anyone of anything" But here I dont think they (or their defenders) are still aware of the real lesson here. Theres literally zero information thats valuable here. Its like saying "we used an 18 wheeler as our family car and then we switched over to a regular camry and solved all our problems." What is the lesson to be learned in that statement? The real interesting post mortem would be if they go, "god in retrospect what a stupid decision we took; what were we thinking? Why did we not take a step back earlier and think, why are we doing it this way?" If they wrote a blog post that way, that would likely have amazing takeaways.
  - chronark_ 1y ago
    I can assure you that was pretty close to the internal conversation lol Not sure what the different takeaways would be though?
    - ramraj07 1y ago
      What did your internal discussion conclude for the question "Why did we not take a step back earlier and think, why are we doing it this way?" Im genuinely curious because this is not singling out your team or org, this is a very common occurrence among modern engineering teams, and I've often found myself on the losing end of such arguments. So I am all ears to hear at least one such team telling what goes on in their mind when they make terrible architecture decisions and if they learned anything philosophical that would prevent a repeat.
      - hrimfaxi 1y ago
        I have had CTOs (two in my career) tell me we had to use our AWS credits since they were going to expire worthless. Both experiences were at vc-backed startups.
      - chronark_ 1y ago
        Oh we had it coming for quite some time and knew we would need to rebuild it, we just didn’t have the capacity to do it unfortunately. I was working on it on and off moving one endpoint at a time but it was very slow until we hired someone who was able to focus on it. It didn’t feel good at all. We knew the product had massive flaws due to the latency but couldn’t address it quickly. Especially cause we he to build more workarounds as time went on. Workarounds we knew would be made redundant by the reimplementation. I think we had that discussion if “wtf are we doing here” pretty early, but we didn’t act on it in the beginning, instead we tried different approaches to make it work within the serverless constraints cause that’s what we knew well.
czhu12 1y ago
> Putting a latency-critical API on a stateless edge runtime Isn’t this the whole point of serverless edge? It’s understood to be more complex, with more vendor lockin, and more expensive. Trade off is that it’s better supported and faster by being on the edge. Why would anyone bother to learn a proprietary platform for non critical, latency agnostic service?
- kburman 1y ago
  You're confusing network proximity with application architecture. Edge deployment helps connection latency. Stateless runtime destroys it by forcing every cache access through the network. The whole point of edge is NOT to make latency-critical APIs with heavy state requirements faster. It's to make stateless operations faster. Using it for the former is exactly the mismatch I'm describing. Their 30ms+ cache reads vs sub-10ms target latency proves this. Edge proximity can't save you when your architecture adds 3x your latency budget per cache hit.
  - osigurdson 1y ago
    Realistically, they should be able to do sub ms cache hits which land in the same datacenter. I know cloudflare doesn't have "named" datacenters like other providers but at the end of the day, there are servers somewhere and if your lambda runs twice in the same one there is no reason why a pull-through cache can't experience a standard intra data-center latency hit. I wonder if there is anything other than good engineering getting in the way of this and even sub us intra-process pull through caches for busy lambda functions. After all, if my lambda is getting called 1000X per second from the same point of presence, why wouldn't they keep the process in memory?
  - deleted 1y ago
    [deleted]
- whynotmaybe 1y ago
  On serverless, whenever you call your code, it has to be executed but first the infrastructure has to find a place to run it and sometimes if there's no running instance available, it must fire up a new instance to run your code. That's hot start VS cold start.
torginus 1y ago
My personal experience is that if you want guaranteed anything (quick scaling, latency, CPU, disk or network throughput), your best bet is to manually provision EC2 instances (or use some API that does). Once you give up control hoping to gain performance for free, you usually end up with an unfixable bottleneck.
- randomtoast 1y ago
  If you're looking for a middle ground between VMs and serverless, ECS Fargate is a good option. Because a container is always running, you won't experience any cold start times.
  - sgarland 1y ago
    Yes, though unless you’re provisioning your own EC2s for them to run on, you have no guarantee about the server generation, and IME AWS tends to provision older stuff for Fargate. This may or may not matter to you depending on your application’s needs, but there is a significant performance difference between, say, an m4 family (Haswell / Broadwell) and an m7i family (Sapphire Rapids) - literally a decade of hardware improvements. Memory performance in particular can be a huge hit for latency-sensitive applications.
    - raw_anon_1111 1y ago
      https://aws.amazon.com/blogs/aws/announcing-amazon-ecs-managed-instances-for-containerized-applications/ https://aws.amazon.com/blogs/aws/announcing-amazon-ecs-manag...
  - evantbyrne 1y ago
    ECS is good, just expensive and still requires more devops than it should. Docker Swarm is an easy way to run production container services on VMs. I built a free golang tool called Rove that provisions fresh Ubuntu VMs in one command and diffs updates. It's also easy-enough to use Swarm directly.
    - 1y ago
Esophagus4 1y ago
I’ve found this to be true, with one caveat. Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been avoided with a more thoughtful design. It tends to be a people problem, not a tool problem. However, in my experience cloud vendors sell the snot out of their offerings, and the documentation is closer to marketing than truthful technical documentation. Their products’ genuine performance is a closely guarded proprietary secret, and the only way to find out… e.g. whether Lambdas are fast enough for your use case, or whether AWS RDS cross-region replication is good enough for you… is to run your own performance testing. I’ve been burned enough times by AWS making it difficult to figure out exactly how performant their services are, and I’ve learned to test everything myself for the workloads I’ll be running.
- Danjoe4 1y ago
  This is exactly why I'd rather get a fat VPS from a reputable provider. As long as the bandwidth is sufficient the only limitation is vertical scaling.
  - dlisboa 1y ago
    I'm partial to this, the only thing I've found that is harder to achieve is the "edge" part of cloud services. Having a server at each continent is enough for most needs but having users route to the closest one is not as clear to me. I know about Anycast but not how to make it operational for dynamic web products (not like CDN static assets). Any tips on this?
    - deleted 1y ago
      [deleted]
    - whstl 1y ago
      Someone correct me if I’m wrong but: DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly. One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL. EDIT: Back to Anycast, there are also some providers who allow you BGP configuration, like those: https://www.virtua.cloud/features/your-ip-space https://www.virtua.cloud/features/your-ip-space - https://us.ovhcloud.com/network/byoip https://us.ovhcloud.com/network/byoip - https://docs.hetzner.com/robot/colocation/pricing/ https://docs.hetzner.com/robot/colocation/pricing/ ... However you still need to get the IPs by yourself, by dealing with your Regional Registry (RIPE in my case, in Europe)
ochronus 1y ago
But but it's webscale!
bunderbunder 1y ago
I would not assume this was a "rookie mistake". I've been here once or twice, and a common story is that engineers don't want to do it a certain way, but management overrules them for some vague hand-wavy reason like, "This way is more modern." Another common story is that you know you're not choosing the most [scalable|robust|performant|whatever] design, but ancillary constraints like time and money push you into a "worse is better" decision. Or maybe the original implementation team really didn't know what they were doing. But I'd rather give them the benefit of the doubt. Either way, I appreciate them sharing these observations because sharing these kinds of stories is how we collectively get better as a professional community.
- sunrunner 1y ago
  > but management overrules them for some vague hand-wavy reason like, "This way is more modern." This matches my experience. It's very difficult to argue against costly and/or inappropriate technical decisions in environments where the 'Senior Tech Leadership' team are just not that technical but believe they are, and so are influenced by every current industry trend masquerading as either 'scalable', 'modern' or (worst of all) 'best practice'.
  - bunderbunder 1y ago
    What's even more dangerous is when senior tech leadership used to be technical but haven't actually got their hands dirty in 5 or 10 years, and don't realize that this means they aren't actually holding all the cards when they try to dictate these kinds of tactical, detail-oriented technical decisions. I see this a lot in startups that grew big before they had a chance to grow up.
    - sunrunner 1y ago
      > used to be technical And to add, this rarely indicates anything about the depth and/or breadth of the 'used to' experience. A lot of the strongest individual contributors I see want to stay in that track and use that experience to make positive and sensible change, while the ones that move into the management tracks don't always have such motivations. There's no gatekeeping intended here, just an observation that the ones that are intrinsically motivated by the detailed technical work naturally build that knowledge base through time spent hands-on in those areas and are best able to make more impactful systemic decisions. People in senior tech leadership also are not often exposed to the direct results of their decisions too (if they even stay in the company for long enough to see the outcome of longer-term decisions, which itself is rare). While it's not impossible to find the folk that do have breadth of experience and depth of knowledge but are comfortable and want to be in higher-level decision making places, it's frustratingly rare. And in a lot of cases, the really good ones that speak truth to power end up in situations where 'Their last day was yesterday, we wish them all the best in their future career endeavours.' It's hardly surprising that it's a game that the most capable technical folks just don't want to play, even if they're the ones that should be playing it. This all could just be anecdata from a dysfunctional org, of course...
compiler-guy 1y ago
This is basically criticizing them for admitting to being one of today's 10,000. https://xkcd.com/1053/ https://xkcd.com/1053/ Personally, I appreciate the info and the admission.
smrtinsert 1y ago
Agreed. Wondering what sort of discovery or design phase their legacy arch went thru.

nougati 1y ago

The takeaway isn't that they didn't understand, it's that they are sharing information which you agree is valuable

kburman 1y ago
What's valuable about rediscovering that stateless architectures requiring network round-trips for state access are slower than in-memory state? This isn't new information, it's a predictable consequence of their architecture choice that anyone with distributed systems experience could have told them on day zero.
- chronark_ 1y ago
  Not everyone is born with experience in distributed systems
  - sgarland 1y ago
    Sure, but there are some fundamentals about latency that any programmer should know [0] (absolute values outdated, but still useful as relative comparisons), like “network calls are multiple orders of magnitude slower than IPC.” I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics. 0: https://gist.github.com/hellerbarde/2843375 https://gist.github.com/hellerbarde/2843375
    - chronark_ 1y ago
      I cofounded it yeah And yeah you’re right in hindsight it was a terrible idea to begin with I thought it could work but didn’t benchmark it enough and didn’t plan enough. It all looked great in early POCs and all of these issues cropped up as we built it
  - kburman 1y ago
    That's fair, but then the framing matters. The article criticizes serverless architecture rather than acknowledging an evaluation failure. "Serverless was fighting us" vs "We didn't understand serverless tradeoffs" - one is a learning experience, the other is misdirected criticism.
    - chronark_ 1y ago
      Yeah that’s fair
  - lossolo 1y ago
    You don't need experience and there is not really a lot to know about "distributed systems" in this case, that's basic CS knowledge about networks, latency and what "serverless" actually is, you can read about it. To be honest, to me it reads like people who don't understand the problem they're solving, haven't acquired the necessary knowledge to solve it (either by learning themselves or by asking/hiring people who have it), and seeing such an amateurish mistake doesn't inspire confidence for the future. You should either hire people that know what they are doing or upgrade your knowledge about systems you are using before making decisions to use them.
    - nougati 1y ago
      Sometimes I see a post about sorting algorithms online. Some people seem to benefit from reading about these things, but often, I find there isn't much new information for me. That's OK, because I know somebody somewhere benefits from knowing this. It is your decision to make this a circlejerk of musings about how the company must be run by amateurs. Whatever crusade you're fighting in vividly criticising them is not valuable at all. People need to learn and share so we can all improve, stop distracting from that point.
      - lossolo 1y ago
        [dead]
- bcrosby95 1y ago
  Uh, no, 95% of our architectures are stateless and its fine because RTT isn't dogshit, unlike AWS lambda.
ramraj07 1y ago
Bo Burmham said, "self awareness does not absolve anyone of anything" But here I dont think they (or their defenders) are still aware of the real lesson here. Theres literally zero information thats valuable here. Its like saying "we used an 18 wheeler as our family car and then we switched over to a regular camry and solved all our problems." What is the lesson to be learned in that statement? The real interesting post mortem would be if they go, "god in retrospect what a stupid decision we took; what were we thinking? Why did we not take a step back earlier and think, why are we doing it this way?" If they wrote a blog post that way, that would likely have amazing takeaways.
- chronark_ 1y ago
  I can assure you that was pretty close to the internal conversation lol Not sure what the different takeaways would be though?
  - ramraj07 1y ago
    What did your internal discussion conclude for the question "Why did we not take a step back earlier and think, why are we doing it this way?" Im genuinely curious because this is not singling out your team or org, this is a very common occurrence among modern engineering teams, and I've often found myself on the losing end of such arguments. So I am all ears to hear at least one such team telling what goes on in their mind when they make terrible architecture decisions and if they learned anything philosophical that would prevent a repeat.
    - hrimfaxi 1y ago
      I have had CTOs (two in my career) tell me we had to use our AWS credits since they were going to expire worthless. Both experiences were at vc-backed startups.
    - chronark_ 1y ago
      Oh we had it coming for quite some time and knew we would need to rebuild it, we just didn’t have the capacity to do it unfortunately. I was working on it on and off moving one endpoint at a time but it was very slow until we hired someone who was able to focus on it. It didn’t feel good at all. We knew the product had massive flaws due to the latency but couldn’t address it quickly. Especially cause we he to build more workarounds as time went on. Workarounds we knew would be made redundant by the reimplementation. I think we had that discussion if “wtf are we doing here” pretty early, but we didn’t act on it in the beginning, instead we tried different approaches to make it work within the serverless constraints cause that’s what we knew well.

czhu12 1y ago

> Putting a latency-critical API on a stateless edge runtime Isn’t this the whole point of serverless edge? It’s understood to be more complex, with more vendor lockin, and more expensive. Trade off is that it’s better supported and faster by being on the edge. Why would anyone bother to learn a proprietary platform for non critical, latency agnostic service?

kburman 1y ago
You're confusing network proximity with application architecture. Edge deployment helps connection latency. Stateless runtime destroys it by forcing every cache access through the network. The whole point of edge is NOT to make latency-critical APIs with heavy state requirements faster. It's to make stateless operations faster. Using it for the former is exactly the mismatch I'm describing. Their 30ms+ cache reads vs sub-10ms target latency proves this. Edge proximity can't save you when your architecture adds 3x your latency budget per cache hit.
- osigurdson 1y ago
  Realistically, they should be able to do sub ms cache hits which land in the same datacenter. I know cloudflare doesn't have "named" datacenters like other providers but at the end of the day, there are servers somewhere and if your lambda runs twice in the same one there is no reason why a pull-through cache can't experience a standard intra data-center latency hit. I wonder if there is anything other than good engineering getting in the way of this and even sub us intra-process pull through caches for busy lambda functions. After all, if my lambda is getting called 1000X per second from the same point of presence, why wouldn't they keep the process in memory?
- deleted 1y ago
  [deleted]
whynotmaybe 1y ago
On serverless, whenever you call your code, it has to be executed but first the infrastructure has to find a place to run it and sometimes if there's no running instance available, it must fire up a new instance to run your code. That's hot start VS cold start.

torginus 1y ago

My personal experience is that if you want guaranteed anything (quick scaling, latency, CPU, disk or network throughput), your best bet is to manually provision EC2 instances (or use some API that does). Once you give up control hoping to gain performance for free, you usually end up with an unfixable bottleneck.

randomtoast 1y ago
If you're looking for a middle ground between VMs and serverless, ECS Fargate is a good option. Because a container is always running, you won't experience any cold start times.
- sgarland 1y ago
  Yes, though unless you’re provisioning your own EC2s for them to run on, you have no guarantee about the server generation, and IME AWS tends to provision older stuff for Fargate. This may or may not matter to you depending on your application’s needs, but there is a significant performance difference between, say, an m4 family (Haswell / Broadwell) and an m7i family (Sapphire Rapids) - literally a decade of hardware improvements. Memory performance in particular can be a huge hit for latency-sensitive applications.
  - raw_anon_1111 1y ago
    https://aws.amazon.com/blogs/aws/announcing-amazon-ecs-managed-instances-for-containerized-applications/ https://aws.amazon.com/blogs/aws/announcing-amazon-ecs-manag...
- evantbyrne 1y ago
  ECS is good, just expensive and still requires more devops than it should. Docker Swarm is an easy way to run production container services on VMs. I built a free golang tool called Rove that provisions fresh Ubuntu VMs in one command and diffs updates. It's also easy-enough to use Swarm directly.
  - 1y ago

Esophagus4 1y ago

I’ve found this to be true, with one caveat. Most cloud pain people experience is from a misunderstanding / abuse of solutions architecture and could have been avoided with a more thoughtful design. It tends to be a people problem, not a tool problem. However, in my experience cloud vendors sell the snot out of their offerings, and the documentation is closer to marketing than truthful technical documentation. Their products’ genuine performance is a closely guarded proprietary secret, and the only way to find out… e.g. whether Lambdas are fast enough for your use case, or whether AWS RDS cross-region replication is good enough for you… is to run your own performance testing. I’ve been burned enough times by AWS making it difficult to figure out exactly how performant their services are, and I’ve learned to test everything myself for the workloads I’ll be running.

Danjoe4 1y ago
This is exactly why I'd rather get a fat VPS from a reputable provider. As long as the bandwidth is sufficient the only limitation is vertical scaling.
- dlisboa 1y ago
  I'm partial to this, the only thing I've found that is harder to achieve is the "edge" part of cloud services. Having a server at each continent is enough for most needs but having users route to the closest one is not as clear to me. I know about Anycast but not how to make it operational for dynamic web products (not like CDN static assets). Any tips on this?
  - deleted 1y ago
    [deleted]
  - whstl 1y ago
    Someone correct me if I’m wrong but: DIY Anycast is probably beyond most people’s reach, as you need to deal with BGP directly. One cool trick is using GeoDNS to route the same domain to a different IP depending on the location of the user, but there are some caveats of course due to caching and TTL. EDIT: Back to Anycast, there are also some providers who allow you BGP configuration, like those: https://www.virtua.cloud/features/your-ip-space https://www.virtua.cloud/features/your-ip-space - https://us.ovhcloud.com/network/byoip https://us.ovhcloud.com/network/byoip - https://docs.hetzner.com/robot/colocation/pricing/ https://docs.hetzner.com/robot/colocation/pricing/ ... However you still need to get the IPs by yourself, by dealing with your Regional Registry (RIPE in my case, in Europe)

ochronus 1y ago

But but it's webscale!

bunderbunder 1y ago

I would not assume this was a "rookie mistake". I've been here once or twice, and a common story is that engineers don't want to do it a certain way, but management overrules them for some vague hand-wavy reason like, "This way is more modern." Another common story is that you know you're not choosing the most [scalable|robust|performant|whatever] design, but ancillary constraints like time and money push you into a "worse is better" decision. Or maybe the original implementation team really didn't know what they were doing. But I'd rather give them the benefit of the doubt. Either way, I appreciate them sharing these observations because sharing these kinds of stories is how we collectively get better as a professional community.

sunrunner 1y ago
> but management overrules them for some vague hand-wavy reason like, "This way is more modern." This matches my experience. It's very difficult to argue against costly and/or inappropriate technical decisions in environments where the 'Senior Tech Leadership' team are just not that technical but believe they are, and so are influenced by every current industry trend masquerading as either 'scalable', 'modern' or (worst of all) 'best practice'.
- bunderbunder 1y ago
  What's even more dangerous is when senior tech leadership used to be technical but haven't actually got their hands dirty in 5 or 10 years, and don't realize that this means they aren't actually holding all the cards when they try to dictate these kinds of tactical, detail-oriented technical decisions. I see this a lot in startups that grew big before they had a chance to grow up.
  - sunrunner 1y ago
    > used to be technical And to add, this rarely indicates anything about the depth and/or breadth of the 'used to' experience. A lot of the strongest individual contributors I see want to stay in that track and use that experience to make positive and sensible change, while the ones that move into the management tracks don't always have such motivations. There's no gatekeeping intended here, just an observation that the ones that are intrinsically motivated by the detailed technical work naturally build that knowledge base through time spent hands-on in those areas and are best able to make more impactful systemic decisions. People in senior tech leadership also are not often exposed to the direct results of their decisions too (if they even stay in the company for long enough to see the outcome of longer-term decisions, which itself is rare). While it's not impossible to find the folk that do have breadth of experience and depth of knowledge but are comfortable and want to be in higher-level decision making places, it's frustratingly rare. And in a lot of cases, the really good ones that speak truth to power end up in situations where 'Their last day was yesterday, we wish them all the best in their future career endeavours.' It's hardly surprising that it's a game that the most capable technical folks just don't want to play, even if they're the ones that should be playing it. This all could just be anecdata from a dysfunctional org, of course...

compiler-guy 1y ago

This is basically criticizing them for admitting to being one of today's 10,000. https://xkcd.com/1053/ https://xkcd.com/1053/ Personally, I appreciate the info and the admission.

smrtinsert 1y ago

Agreed. Wondering what sort of discovery or design phase their legacy arch went thru.