12 ms·

Leaving serverless led to performance improvement and a simplified architecture

480 points by vednig 1y ago

pjmlp 1y ago
Their problem isn't serverless, rather Cloudflare Workers and WebAssembly. All major cloud vendors have serveless solutions based on containers, with longer managed lifetimes between requests, and naturally the ability to use properly AOT compiled languages on the containers.
- OvervCW 1y ago
  Agree, it seems like they decided to use Cloudflare Workers and then fought them every step of the way instead of going back and evaluating if it actually fit the use case properly. It reminds me of the companies that start building their application using a NoSQL database and then start building their own implementation of SQL on top of it.
  - zaphirplane 1y ago
    Hey! Bet I can guess who
  - CuriouslyC 1y ago
    Ironically, I really like cloudflare but actively dislike workers and avoid them when possible. R2/KV/D1 are all fantastic and being able to shard customer data via DOs is huge, but I find myself fighting workers when I use them for non-trivial cases. Now that Cloudflare has containers I'm pushing people that way.
- keyle 1y ago
  You're saying serverless can have really low latency and fast 24/7? Isn't serverless at the base the old model, of shared vms, except with a ton of people? I'm old school I guess, baremetal for days...
  - pjmlp 1y ago
    Yes, check Cloud Run, AWS Lambda, Azure Functions with containers.
- fabian2k 1y ago
  At that point, why should I use serverless at all? If I have to think about the lifetime of the servers running my serverless functions?
  - OvervCW 1y ago
    Serverless only makes sense if the lifetime doesn't matter to your application, so if you find that you need to think about your lifetime then serverless is simply not the right technology for your use case.
  - pjmlp 1y ago
    Because it is still less management effort than taking full control of the whole infrastructure. Usually a decision factor between more serverless, or more DevOps salaries.
    - fabian2k 1y ago
      I would doubt that this is categorically true. Serverless inherently makes the whole architecture more complex with more moving parts in most cases compared to classical web applications.
      - pjmlp 1y ago
        Depends pretty much where those classical web applications are hosted, how big is the infrasture taking care of security, backups, scalability, failovers, and the amount of salaries being paid, including on-call bonus.
      - 9rx 1y ago
        > Serverless inherently makes the whole architecture more complex with more moving parts Why's that? Serverless is just the generic name for CGI-like technologies, and CGI is exactly how classical web application were typically deployed historically, until Rails became such a large beast that it was too slow to continue using CGI, and thus running your application as a server to work around that problem in Rails pushed it to become the norm across the industry — at least until serverless became cool again. Making your application the server is what is more complex with more moving parts. CGI was so much simpler, albeit with the performance tradeoff. Perhaps certain implementations make things needlessly complex, but it is not clear why you think serverless must fundamentally be that way.
- CuriouslyC 1y ago
  Cloudflare has containers now too, and having used AppRunner and Cloud Run, it's much easier to work with. Once they get rid of the container caps and add more flexibility in terms of container resources, I would never go back to the big cloud containers, the price and ease of use of Cloudflare's containers just destroy them.
  - pjmlp 1y ago
    I doubt that the bill would be that much cheaper, nonetheless thanks for making me aware they are a thing now.
    - CuriouslyC 1y ago
      They're much cheaper, they're just DOs, and they get billed as such. They also have faster cold start times and automatic multi-region support.
      - OvervCW 1y ago
        What does DO mean in this context?
        CuriouslyC 1y ago
        Durable Object
- iainmerrick 1y ago
  In that scenario, how do you keep cold startup as fast as possible? The nice thing about JS workers is that they can start really fast from cold. If you have low or irregular load, but latency is important, Cloudflare Workers or equivalent is a great solution (as the article says towards the end). If you really need a full-featured container with AOT compiled code, won't that almost certainly have a longer cold startup time? In that scenario, surely you're better off with a dedicated server to minimise latency (assuming you care about latency). But then you lose the ability to scale down to zero, which is the key advantage of serverless.
  - pjmlp 1y ago
    Apparently not nice enough, given that they rewrote the application in Go. Serverless with containers is basically managed Kubernetes, where someone else has the headache to keep the whole infrastructure running.
- Quarrel 1y ago
  Indeed. They get to the bottom of the post and drop: > Fargate handles scaling for us without the serverless constraints They dropped workers for containers.
muragekibicho 1y ago
Interesting writeup. The serverless approach helped with GTM. (I speculate) raising capital afforded them extra devs who noticed the cache latency.
- saidinesh5 1y ago
  > The serverless approach helped with GTM Unlikely? They could've just as well deployed their single go binary to a vm from day 1 and it would've been smooth sailing for their use case, while they acquire customers. The cloudflare workers they chose aren't really suited for latency critical, high throughput APIs they were designing.
seethishat 1y ago
Linux servers running Go apps? Would be nice to see server cost and specs, backup strategy, etc.
- wltr 1y ago
  Backup strategy? What do you mean by that?
  - seethishat 1y ago
    Servers go down. What is the plan to get them "backup" and running ;)
    - fabian2k 1y ago
      They probably don't need one for the application servers. And they probably already have a backup strategy for their DBs.
    - stackskipton 1y ago
      Most server outages are caused by hardware failures which EC2 MOSTLY abstracts from you. Also, if it's just Golang, point Ansible or whatever deploys at new server and trigger a deploy.
    - wltr 1y ago
      But is this not needed with the so-called cloud systems?
      - hunterpayne 1y ago
        Only if that system is stateless. If you have any sort of internal memory that sticks around between requests, then either you face a cold start problem (because of empty caches) or you somehow need to persist that state somewhere. And persisting that state either means you need a backup solution or your latency is terrible because you are hitting network for something that only needs to hit RAM.
  - 1y ago
1GZ0 1y ago
Somewhere in Denmark, DHH is smiling
- noir_lord 1y ago
  Gives him a break from writing out of touch screeds about countries he knows nothing about I guess.
- Sammi 1y ago
  I often don't know what to make of DHH. He's a living contradiction. On one hand he will continually rant about how bad the overhead and waste of cloud services is, and on the other hand he will staunchly defend the most inefficient programming language that is regularly used for backend development, as well as defend the enourmous overfetching that active record leads to. Really I think DHH just likes to tell others what he likes.
  - hunterpayne 1y ago
    In all fairness, the performance penalty for virtualization is 4x and the penalty for interpreted code is 1.5x. So he comes out ahead, but its more in a broken watch is right twice a day sort of way.
hshdhdhehd 1y ago
30ms P99 does not a cache make. Source work somewhere where you easily get 1ms cached relational DB reads from outside the service. 30ms makes me suspect it went cross region.
kunley 1y ago
For a best price-to-performance ratio create your instances and do whatever is needed on them. Software stacks are not that complicated to delegate everything to the Wizards of Cloud Overcharging.
deleted 1y ago
[deleted]
kburman 1y ago
The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a stateless edge runtime was a rookie mistake, and the pain they describe was entirely predictable.
- nougati 1y ago
  The takeaway isn't that they didn't understand, it's that they are sharing information which you agree is valuable
  - kburman 1y ago
    What's valuable about rediscovering that stateless architectures requiring network round-trips for state access are slower than in-memory state? This isn't new information, it's a predictable consequence of their architecture choice that anyone with distributed systems experience could have told them on day zero.
    - chronark_ 1y ago
      Not everyone is born with experience in distributed systems
      - sgarland 1y ago
        Sure, but there are some fundamentals about latency that any programmer should know [0] (absolute values outdated, but still useful as relative comparisons), like “network calls are multiple orders of magnitude slower than IPC.” I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics. 0: https://gist.github.com/hellerbarde/2843375 https://gist.github.com/hellerbarde/2843375
        chronark_ 1y ago
        I cofounded it yeah And yeah you’re right in hindsight it was a terrible idea to begin with I thought it could work but didn’t benchmark it enough and didn’t plan enough. It all looked great in early POCs and all of these issues cropped up as we built it
chronark_ 1y ago
Author of that blog here, happy to answer any questions :)
- flerchin 1y ago
  Really great writeup. The charts tell the story beautifully, and the latency gains are surely a win for your company and customers. I always wonder about the tradeoffs. Is there a measurable latency difference for your non-colocated customers? What does maintenance look like for your Go servers? I assume that your Cloudflare costs dropped?
  - chronark_ 1y ago
    It’s faster for non-colocated customers too weirdly I think cause connections can be reused more often. Cloud flare workers are really prone to doing a lot of TLS handshakes cause they spin up new ones constantly Right now were just hang aws far hate for the go servers, so there really isn’t much maintenance at all. We’ll be moving that into eks soon though cause we are starting to add more stuff and need k8s anyways
- wiether 1y ago
  Not a question: thanks for the writeup and for the honesty of saying that serverless is not inherently bad, just not the right fit for your usecase! Unfortunately too many comments here are quick to come to the wrong conclusion, based only on the title. Not a reason to change it though!
  - chronark_ 1y ago
    Thanks It’s totally fair criticism that the title and wording is a bit clickbaity But that’s ok
- Sammi 1y ago
  Do you have a clearer picture of what use-cases you would use serverless functions for in the future (if any)?
- 1y ago
torginus 1y ago
I think someone should make a timeline of software technology eras, each beginning with 'why XYZ is the future' and ending with articles like this.
yilugurlu 1y ago
These two have resonated with me deeply. - Eliminated complex caching workarounds and data pipeline overhead - Simplified architecture from distributed system to straightforward application We, as developers/engineers (put whatever title you want), tend to make things complex for no reason sometimes. Not all systems have to follow state-of-the-art best practices. Many times, secure, stable, durable systems outperform these fancy techs and inventions. Don't get me wrong, I love to use all of these technologies and fancy stuff, but sometimes that old, boring, monolithic API running on an EC2 solves 98% of your business problems, so no need to introduce ECS, K8S, Serverless, or whatever. Anyway, I guess I'm getting old, or I understand the value of a resilient system, and I'm trying to find peace xD.
- ramraj07 1y ago
  But when were serverless systems like lambda and cloud workers "best practices" for low latency apis?
  - hedora 1y ago
    According to their marketing material, when they started supporting running in edge pop's, they became the best option for low-latency APIs.
    - daxfohl 1y ago
      Last I heard (~5 years ago), lambda@edge doesn't actually run on edge POPs anyway; they're just hooks that you can put in your edge configs that execute logic in the nearest region before/after running your edge config. But it's definitely a datacenter round-trip to invoke them. Adding that much compute to an edge POP is a big lift; even firecracker gets heavy at scale. And security risk for executing arbitrary code since these POPs don't have near the physical security of a datacenter, small scale makes more vulnerable to timing attacks, etc.
1-6 1y ago
I think this is what is being said: "Down with serverless! Long live serverless!"
voodooEntity 1y ago
As someone who worked with serverless for multiple years (mostly amazon lambda but others too) i can absolutely apporove the authors points. While it "takes away" some work from you, it adds this work on other points to solve the "artificial induced problems". Another example i hit was a hard upload limit. Ported an application to a serverless variant, had an import API for huge customer exports. Shouldnt be a problem right? Just setup an ingest endpoint and some background workers to process the data. Tho than i learned : i cant upload more than 100mb at a time through the "api gateway" (basically their proxy to invoke your code) and when asking if i could change it somehow i just was told to tell our customers to upload smaller file chunks. While from a "technical" perspective this sounds logical, our customers not gonne start exchanging all their software so we get a "nicer upload strategy". For me this is comparable with "it works in a vacuum" type of things. Its cool in theory, but as soon it hits reality you will realice quite fast that the time and money you safed on changing from permanent running machines to serverless, you will spend in other ways to solve the serverless specialities.
- akdev1l 1y ago
  The way to work around this issue is to provide a presigned S3 url Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something) I agree this is annoying and maybe I’ve been in AWS ecosystem for too long. However having an API that accepts an unbounded amount of data is a good recipe for DoS attacks, I suppose the 100MB is outdated as internet has gotten faster but eventually we do need some limit
  - voodooEntity 1y ago
    Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also. In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more. And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs. But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place. Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.
tacker2000 1y ago
Incredible that these kinds of services were hosted like this. I guess they never came out of MVP, which could warrant using serverless, but in the end it makes 0 sense to use some slow solution like this for the service they are offering. Why didnt they go with a self hosted backend right away? Its funny how nowadays most devs are too scared to roll their own and just go with the cloud offerings that cost them tech debt and actual money down the road.
- chronark_ 1y ago
  We did initially but thought cloud flare was a better solution for scalability and latency. We believed their docs/marketing without doing extensive benchmarks, which is on us. The appeal was also to use the same typescript stack across everything, which was nice to work with
  - ramraj07 1y ago
    Where did their marketing or documentation say this service is perfect for low latency APIs?
    - chronark_ 1y ago
      I doubt they literally said “perfect for low latency APIs” but their messaging is definitely trying to convince you that they’re fast globally, just look at the workers.ckoudflare.com page
K0IN 1y ago
After building my first Serverless/Cloudflare worker app, this is why I migrated to Deno. Deno enables you to run the same codebase in deno (self-hosted/local) and in deno deploy (serverless platform from deno). I wanted my app to be self-hostable as well, and Cloudflare worker is a hard ecosystem lock to their platform, which makes it undesirable (imo). Here is a link to my reasoning from back then: https://github.com/K0IN/Notify/pull/77#issuecomment-1677607057 https://github.com/K0IN/Notify/pull/77#issuecomment-16776070...
- scottydelta 1y ago
  I ported my worker project into Django since cloudflare workers wouldn’t allow selection of region for hosting workers which is generally required due to data compliances. This is something all cloud providers provide from day one yet cloudflare made it an enterprise feature. Also the vendor lock-in doesn’t help with durable objects and D2 instead of simply doing what supabase and others are doing by providing Postgres or standard SQLite as a service.
gloomyday 1y ago
I think developers are drowning in tools to make things "easy", when in truth many problems are already easy with the most basic stuff in our tool belt (a compiler, some bash scripts, and some libraries). You can always build up from there. This tooling fetish hurts both companies and developers.
- bamboozled 1y ago
  Excerpt AWS lambda is stupidly cheap!
  - Esophagus4 1y ago
    For certain workloads :) And that is actually the advantage of serverless, in my mind. For some low-traffic workloads, you can host for next to nothing. Per invocation, it is expensive, but if you only have a few invocations of a workload that isn't very latency sensitive, you can run an entirely serverless architecture for pennies per month. Where people get burned is moving high traffic volumes to serverless... then they look at their bill and go, "Oh my god, what have I done!?" Or they try to throw all sorts of duct tape at serverless to make it highly performant, which is a fool's errand.
    - cogman10 1y ago
      Exactly. I've always found that how people want to use lambda is the exact opposite of how to use it cost effectively. I've seen a lot of people want to use lambdas as rest endpoints and effectively replace their entire API with a cluster of lambdas. But that's about the most expensive way to use a lambda! 1 request, one lambda. Where these things are useful is when you say "I have this daily data pull and ETL that I need to do." Then all the sudden the cost is pretty dang competitive.
      - hvb2 1y ago
        > Where these things are useful All the backend processing and just general 'glue' in your architectures
codegeek 1y ago
"Self-Hosting : Being tied to Cloudflare's runtime meant our customers couldn't self-host Unkey. While the Workers runtime is technically open source, getting it running locally (even in dev mode) is incredibly difficult. With standard Go servers, self-hosting becomes trivial:" A key point that I always make. Serverless is good if you want a simple periodic task to run intermittently without worrying about a full time server. The moment things get more complex than that (which in real world it almost always is), you need a proper server.
jjice 1y ago
Like the article says, I think serverless has it's place, but I don't think it's for most applications. I can't see myself _ever_ using serverless services as a core part of my application for pretty much any startup, if I can avoid it. The infrastructure overhead is actually worse, IMO. Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to test, and the abstraction layers that exist (unless this has changed recently) always had pitfalls, since there are no true standards. I'd much rather have a docker image as my deliverable "executable" so I can test, but still abstract away some stuff, like environment setup. Giving me a minimal Linux environment and filesystem feels like the most comfortable level of abstraction for me to develop well and also deploy and run in production effectively. I can also run that on demand or (most commonly) run that as a server that sits and waits for requests.
- wredcoll 1y ago
  Let me tell you about all the fun I'm having trying to execute my amazon lambda app locally so I can test before deploying...
  - renlo 1y ago
    Localstack makes that pretty easy. Before Localstack I had a pre-staging environment (dev) target I would deploy to. Their free/community offering includes a Lambda environment; you deploy your dev "Lambda" locally to docker, using the same terraform / deploy flow you'd normally use but pointed at a Localstack server which mimics the AWS API instead. Some of their other offerings require you to pay though (Cloudfront, ECS, others) and I don't use those, yet at least.
    - meekins 1y ago
      Developer-specific sandbox environments with hot code reload is the golden standard here but Localstack is great if you can't do that due to (usually IT deparment-related, not technical) reasons.
    - discomrobertul8 1y ago
deleted 1y ago
[deleted]
kordlessagain 1y ago
If they use Cloudflare that automatically disqualifies them from me reading whatever they wrote, or caring about it.
rvitorper 1y ago
We all love a vendor lock-in, don’t we? Until it backstabs us and we go back to VMs
daxfohl 1y ago
I don't think "serverless is bad" is necessarily the full lesson here. The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuitively make the e2e experience slower, not faster. Prefer building physically near your dependencies. If that's not fast enough, then you have to figure out how to move or sync all your dependencies closer to the client, which except in very simple cases, is almost always a huge can of worms.
- brap 1y ago
  I don’t know if this is a good rule of thumb, I think it really depends on what you use the dependencies for, how often you need them, etc. Consider for example a single DB dependency. Should the server be close to the DB or the client? It depends. How often does the client need the server? How often does the server need the DB? Which usecases are expected to be fast and which can be sacrificed as slow? What can be cached in the server? What can be cached in the client? etc etc. And then of course you can split and do some things on the server and some in the edge…
  - marcosdumay 1y ago
    Oh, it's a very good rule of thumb. It's probably not universal, but it's really close to it. the problem is that nobody designs the dependencies flexible enough to let them run without fine-control. And the main application always wants to change the way it uses the dependencies, so it always needs further flexibility. You can build an exception to the rule if you explicitly try. But I'm not sure one appears naturally. The natural way to migrate your server into the edge is by migrating entire workloads, dependencies included. You can split the work like you said, you just can't split single endpoints.
    - daxfohl 1y ago
      Well, I guess one can take one more step back and say this is all merely an example of "premature optimization is the root of all evil". Unless you know a-priori that you have some very hard latency requirements, start with something simple and low-maintenance. If low-latency requirements come in later, then design that holistically, not just looking at your component. Make sure you're measuring the right things; OOTB metrics often miss the e2e experience. And IME most latency issues come from unexpected places; I know I've spent weeks optimizing services to get an extra percent or two out of them, only to realize there's a config setting that reduced latency by half. So generally, simplicity is your friend when it comes to latencies (among other things). Fewer things to cause long-tail spikes, more simple things you can try out that don't break the whole system, whereas if you start with a highly-optimized thing up-front, fixing some unexpected long-tail issue may require a complete rewrite. Also, check with your PM or end users as to whether latency is even important. If the call to your service is generally followed up to a call to some ten-second process, users aren't going to notice the 20ms improvement to your own thing.
aeldidi 1y ago
I think the "local maximum" we've gotten stuck at for application hosting is having a docker container as the canonical environment/deliverable, and injecting secrets when needed. That makes it easy to run and test locally, but still provides most of the benefits I think (infrastructure-as-code setups, reproducibility, etc). Serverless goes a little too far for most applications (in my opinion), but I have to admit some apps work really well under that model. There's a nearly endless number of simple/trivial utilities which wouldn't really gain anything from having their own infrastructure and would work just fine in a shared or on-demand hosting environment, and a massively scaled stateless service would thrive under a serverless environment much more than it would on a traditional server. That's not to say that I think serverless is somehow only for simple or trivial use cases though, only that there's an impedance mismatch between the "classic web app" model, and what these platforms provide.
- daitangio 1y ago
  You are ready for misterio: https://github.com/daitangio/misterio https://github.com/daitangio/misterio A tiny layer around stareless docker cluster. I created it for my homelab and it gone wild
  - aeldidi 1y ago
    That's really interesting, I might actually use that for mine too. Thanks for sharing.
- mattmanser 1y ago
  Docker is much like microservices. Appropriate for a subset of apps and yet touted as being 'the norm' when it shouldn't be. There are drawbacks to using docker, such as security patching and operational overhead. And if you're blindly putting it into every project, how are you mitigating the risks it introduces? Worse, the big reason it was useful, managing dependency hell, has largely been solved by making developers default to not installing dependencies globally. We don't really need Docker anywhere near like we used to, and yet it persists as the default, unassailable. Of course hosting companies must LOVE it, docker containers must increase their margins by 10% at least! Someone else down thread has mentioned a tooling fetish, I feel Docker is part of that fetish.
mannyv 1y ago
It sounds like you picked the wrong platform because you didn’t understand what you were doing. That’s not a technology issue.
nine_k 1y ago
One thing I could not find in the write-up is the change in the expense. Did serverless save any money, compared to always-up VMs? Did much of their load run under the free tier limits? Serverless shines when the load is very spiky, and you can afford high long-tail latency. Then you don't pay for all that time when your server would be idling. (This is usually not the case for auth APIs, unless they auth other infrequently invoked operations.)
hinkley 1y ago
"Taking hand off of boiling kettle decreased anxiety and increased focus in 97% of study participants." Stephen King's Dark Tower series never resonated with me and I got stuck in book two. But it has one of my favorite philosophical insults of all time: "Those who [X] have forgotten the faces of their fathers." I feel like there's a collective amnesia just beginning to wear off as people remember the Fallacies of Distributed Computing and basic facts about multitasking. And that amnesia absolutely feels to me as if everyone has forgotten the faces of their fathers. <waves cane threateningly>
ThinkBeat 1y ago
As a soon to be graybeard think this has been fairly obvious from the start. And outside of specific workflows, you are adding unneeded complexity to a system that does not need it. In general, an anti-pattern. but it does have valid use in some cases. On the last project I worked on that came to involve serverless, it made no sense at all other than it was "the fad". For this system we had an excellent knowledge of what the theoretical limit of users and connections as. This apparently needed to be done container, serverless, kafka, blah blah. Annoyed with the whole thing I took a few nights to tear logic out from the micro servers or nano services, and wrapped the whole thing into a Frankenstein monolith. AT least 60% of the code all had to do with dealing solely with the code needed to pass information around to different services so it was easier to maintain. Well my hacked together moonlight was not a great start for anything but a demo. I installed Postgres on my laptop, ran the monolith on it, took 3 servers each pushing the theoretical maximum load we would have, and what do you know the performance was fine. But the architecture was the architecture decided upon.
danpalmer 1y ago
30ms p99 for a cache read! Serverless might have been a problem, but I'm not sure it was the problem. In my experience a p99 of 2ms is more typical - 30ms is the sort of time I'd expect for p99 on a database query in production serving. You don't need process-local caches to get the sort of performance they're looking for, and there are good reasons why most teams avoid stateful processing, it's much harder to get right and has bad failure modes.
- hunterpayne 1y ago
  Fun fact, the overhead for a DB query (not including the network latency) is about 250 microseconds (or 0.25ms). And a typical DB kernel can do at least 16MB/core second of throughput. So that 30ms query I would expect is scanning several GB of data to find its results. Try doing that yourself and your latency would probably grow into seconds. Systems programming is hard... PS moving from stateful processing to something with a distributed cache is moving from RAM to network. That's at least a 10x drop off in throughput.
jrm4 1y ago
This really is confirming my theory that the problem here is that "serverless" is so ill-defined that even it's own name is nonsensical. Like, there are still servers. Appears just about as intelligent as calling them "electricity-less." I mean, yes, I no longer think about electricity when deploying things, but that doesn't tell me anything meaningful about what's going on here.
- ljm 1y ago
  It's more like cgi-bin as a service.
pluto_modadic 1y ago
da rulez: - if you don't understand your concept/market, build in a VPS. you can get away with scaling a VPS for a while. - if you intend to be netflix, rent at the edge (mid game) and eventually own the edge servers in POPs (otherwise, "edge" compute isn't worth it). before that, start with a beefy VPS cluster with HA and SQLproxy. - if you spin to zero, use lambdas (for things like codebuild or fractional needs of a computer). before that, build things in VPSes. - if you spin up/down but not to zero, use container platforms? - once you have a reliably steady understanding of your infrastructure, buy physical servers in a colo
Havoc 1y ago
If it’s got to be serverless then use a PaaS that is docker based as input format. That at least gives some level of platform portability if you need to shift it later But yeah in this case 10ms requirement doesn’t leave a lot of room for elaborate anything
jppope 1y ago
I would love to know the net result to their financials for this move. I have no doubt they were able to improve their performance. I'm just wondering if the juice was worth the squeeze, especially if they could have been building other features that customers would want. I didn't read anything about the opportunity cost in the article or even the consideration.
jmg_ 1y ago
> We built chproxy specifically because ClickHouse doesn't like thousands of tiny inserts. It's a Go service that buffers events and sends them in large batches. Each Cloudflare Worker would send individual analytics events to chproxy, which would then aggregate and send them to ClickHouse. While I understand how this isn't the only thing that needed to be buffered, for Clickhouse data specifically I'd be curious why they built a separate service rather than use asynchronous inserts: https://clickhouse.com/docs/optimize/asynchronous-inserts https://clickhouse.com/docs/optimize/asynchronous-inserts
mikeiz404 1y ago
Hey "vercel security checkpoint", I'm repeatedly getting a code 99 "Failed to verify your browser" on an iphone with a VPN exits in California and Canada via proton VPN and Firefox Focus. What gives? ``` sfo1::1760587368-8k6JCK3uO27oMpuTbnS4Hb3X2K9bVsc ```
scuff3d 1y ago
Serverless seems to me like the tech version of the classic sales technique. You see it every Black Friday, get them in the door with the cheap alluring offers, then upsell the shit out of them. And in the case of serverless, lock them with as many provider specific services as possible. I feel like we're headed the same direction with AI right now. It creates as many problems as it solves, and the solution is always to pay them for the newer faster model and use more tokens. With all the new AI security threats coming to light we'll start seeing them offer solutions for that too. They'll sell you the threats and the solutions, and the entire developer community will thank them for it.
TZubiri 1y ago
Seems like serverless is ideal for that <10$/mo budget sweetspot when you can't yet afford a VPS.
- WJW 1y ago
  That mythical sweetspot where you can afford extra development work, but cannot afford 10$/month?
  - TZubiri 1y ago
    Not only is this the case for solo devs, tech partnerships, and spend-averse people who don't do money to time conversions. But if you have low cash on a company and have excess workforce, you might not necessarily get to a point where you have to fire, but you can spend those, already paid for, man-hours, in migrations and infrastructure to save some cash. As an added bonus you have tasks to give your otherwise idle workforce to keep them warmed up
ForHackernews 1y ago
Many organizations would benefit from just running a well-architected monolith. Amazon pushed microservices hard internally, but they face scale challenges most companies do not have and will never have.
KaiserPro 1y ago
about 5 years ago we made an API that took an image and gave you a lat/lon in return (basically like a visual GPS) The front end was entirely in python AWS lambdas, with a message queue that talked to the (slow) GPU backend. The "lambda tax" was about 100ms on average. given that an average request was around 4 seconds it seemed ok.
callamdelaney 1y ago
I once got a job for a company whose entire stack was aws lambda. Not like a normal 'hey we have this flask application which handles 100 endpoints and we just use lambda to handle a call' - more like 'every singe route or function is its own lambda function'. I left inside 2 weeks.
- whstl 1y ago
  Saw this with an infamous 996 AI startup where the founders vibe coded their app. Their app had like 3 or 4 pages, that was all, but every endpoint was a Lambda call. Every frontend log entry was a Lambda call. Every job execution was a Lambda call that called a dozen others. That was their idea of an architecture. Even with <100 customers they were already paying >100k dollars a month on cloud, more than their entire dev team cost. The constant firefighting, downtimes and billing issues were enough to keep the team busy instead of making new features. I was freelancing but I didn't take the job after the "CTO" gave me a tour.
  - naasking 1y ago
    In principle, serverless should be very efficient and cheap if all code is running the same VM. The heterogeneous architecture must make it expensive. Maybe serverless using WASM will one day make it cheap.

480 points by vednig 1y ago

Their problem isn't serverless, rather Cloudflare Workers and WebAssembly. All major cloud vendors have serveless solutions based on containers, with longer managed lifetimes between requests, and naturally the ability to use properly AOT compiled languages on the containers.

OvervCW 1y ago
Agree, it seems like they decided to use Cloudflare Workers and then fought them every step of the way instead of going back and evaluating if it actually fit the use case properly. It reminds me of the companies that start building their application using a NoSQL database and then start building their own implementation of SQL on top of it.
- zaphirplane 1y ago
  Hey! Bet I can guess who
- CuriouslyC 1y ago
  Ironically, I really like cloudflare but actively dislike workers and avoid them when possible. R2/KV/D1 are all fantastic and being able to shard customer data via DOs is huge, but I find myself fighting workers when I use them for non-trivial cases. Now that Cloudflare has containers I'm pushing people that way.
keyle 1y ago
You're saying serverless can have really low latency and fast 24/7? Isn't serverless at the base the old model, of shared vms, except with a ton of people? I'm old school I guess, baremetal for days...
- pjmlp 1y ago
  Yes, check Cloud Run, AWS Lambda, Azure Functions with containers.
fabian2k 1y ago
At that point, why should I use serverless at all? If I have to think about the lifetime of the servers running my serverless functions?
- OvervCW 1y ago
  Serverless only makes sense if the lifetime doesn't matter to your application, so if you find that you need to think about your lifetime then serverless is simply not the right technology for your use case.
- pjmlp 1y ago
  Because it is still less management effort than taking full control of the whole infrastructure. Usually a decision factor between more serverless, or more DevOps salaries.
  - fabian2k 1y ago
    I would doubt that this is categorically true. Serverless inherently makes the whole architecture more complex with more moving parts in most cases compared to classical web applications.
    - pjmlp 1y ago
      Depends pretty much where those classical web applications are hosted, how big is the infrasture taking care of security, backups, scalability, failovers, and the amount of salaries being paid, including on-call bonus.
    - 9rx 1y ago
      > Serverless inherently makes the whole architecture more complex with more moving parts Why's that? Serverless is just the generic name for CGI-like technologies, and CGI is exactly how classical web application were typically deployed historically, until Rails became such a large beast that it was too slow to continue using CGI, and thus running your application as a server to work around that problem in Rails pushed it to become the norm across the industry — at least until serverless became cool again. Making your application the server is what is more complex with more moving parts. CGI was so much simpler, albeit with the performance tradeoff. Perhaps certain implementations make things needlessly complex, but it is not clear why you think serverless must fundamentally be that way.
CuriouslyC 1y ago
Cloudflare has containers now too, and having used AppRunner and Cloud Run, it's much easier to work with. Once they get rid of the container caps and add more flexibility in terms of container resources, I would never go back to the big cloud containers, the price and ease of use of Cloudflare's containers just destroy them.
- pjmlp 1y ago
  I doubt that the bill would be that much cheaper, nonetheless thanks for making me aware they are a thing now.
  - CuriouslyC 1y ago
    They're much cheaper, they're just DOs, and they get billed as such. They also have faster cold start times and automatic multi-region support.
    - OvervCW 1y ago
      What does DO mean in this context?
      - CuriouslyC 1y ago
        Durable Object
iainmerrick 1y ago
In that scenario, how do you keep cold startup as fast as possible? The nice thing about JS workers is that they can start really fast from cold. If you have low or irregular load, but latency is important, Cloudflare Workers or equivalent is a great solution (as the article says towards the end). If you really need a full-featured container with AOT compiled code, won't that almost certainly have a longer cold startup time? In that scenario, surely you're better off with a dedicated server to minimise latency (assuming you care about latency). But then you lose the ability to scale down to zero, which is the key advantage of serverless.
- pjmlp 1y ago
  Apparently not nice enough, given that they rewrote the application in Go. Serverless with containers is basically managed Kubernetes, where someone else has the headache to keep the whole infrastructure running.
Quarrel 1y ago
Indeed. They get to the bottom of the post and drop: > Fargate handles scaling for us without the serverless constraints They dropped workers for containers.

muragekibicho 1y ago

Interesting writeup. The serverless approach helped with GTM. (I speculate) raising capital afforded them extra devs who noticed the cache latency.

saidinesh5 1y ago
> The serverless approach helped with GTM Unlikely? They could've just as well deployed their single go binary to a vm from day 1 and it would've been smooth sailing for their use case, while they acquire customers. The cloudflare workers they chose aren't really suited for latency critical, high throughput APIs they were designing.

seethishat 1y ago

Linux servers running Go apps? Would be nice to see server cost and specs, backup strategy, etc.

wltr 1y ago
Backup strategy? What do you mean by that?
- seethishat 1y ago
  Servers go down. What is the plan to get them "backup" and running ;)
  - fabian2k 1y ago
    They probably don't need one for the application servers. And they probably already have a backup strategy for their DBs.
  - stackskipton 1y ago
    Most server outages are caused by hardware failures which EC2 MOSTLY abstracts from you. Also, if it's just Golang, point Ansible or whatever deploys at new server and trigger a deploy.
  - wltr 1y ago
    But is this not needed with the so-called cloud systems?
    - hunterpayne 1y ago
      Only if that system is stateless. If you have any sort of internal memory that sticks around between requests, then either you face a cold start problem (because of empty caches) or you somehow need to persist that state somewhere. And persisting that state either means you need a backup solution or your latency is terrible because you are hitting network for something that only needs to hit RAM.
- 1y ago

1GZ0 1y ago

Somewhere in Denmark, DHH is smiling

noir_lord 1y ago
Gives him a break from writing out of touch screeds about countries he knows nothing about I guess.
Sammi 1y ago
I often don't know what to make of DHH. He's a living contradiction. On one hand he will continually rant about how bad the overhead and waste of cloud services is, and on the other hand he will staunchly defend the most inefficient programming language that is regularly used for backend development, as well as defend the enourmous overfetching that active record leads to. Really I think DHH just likes to tell others what he likes.
- hunterpayne 1y ago
  In all fairness, the performance penalty for virtualization is 4x and the penalty for interpreted code is 1.5x. So he comes out ahead, but its more in a broken watch is right twice a day sort of way.

hshdhdhehd 1y ago

30ms P99 does not a cache make. Source work somewhere where you easily get 1ms cached relational DB reads from outside the service. 30ms makes me suspect it went cross region.

kunley 1y ago

For a best price-to-performance ratio create your instances and do whatever is needed on them. Software stacks are not that complicated to delegate everything to the Wizards of Cloud Overcharging.

deleted 1y ago

[deleted]

kburman 1y ago

The takeaway here isn’t that serverless doesn’t work, it’s that the authors didn’t understand what they were building on. Putting a latency-critical API on a stateless edge runtime was a rookie mistake, and the pain they describe was entirely predictable.

nougati 1y ago
The takeaway isn't that they didn't understand, it's that they are sharing information which you agree is valuable
- kburman 1y ago
  What's valuable about rediscovering that stateless architectures requiring network round-trips for state access are slower than in-memory state? This isn't new information, it's a predictable consequence of their architecture choice that anyone with distributed systems experience could have told them on day zero.
  - chronark_ 1y ago
    Not everyone is born with experience in distributed systems
    - sgarland 1y ago
      Sure, but there are some fundamentals about latency that any programmer should know [0] (absolute values outdated, but still useful as relative comparisons), like “network calls are multiple orders of magnitude slower than IPC.” I’m assuming you’re an employee of the company based on your comments, so please don’t take this poorly - I applaud any and all public efforts to bring back sanity to modern architecture, especially with objective metrics. 0: https://gist.github.com/hellerbarde/2843375 https://gist.github.com/hellerbarde/2843375
      - chronark_ 1y ago
        I cofounded it yeah And yeah you’re right in hindsight it was a terrible idea to begin with I thought it could work but didn’t benchmark it enough and didn’t plan enough. It all looked great in early POCs and all of these issues cropped up as we built it

chronark_ 1y ago

Author of that blog here, happy to answer any questions :)

flerchin 1y ago
Really great writeup. The charts tell the story beautifully, and the latency gains are surely a win for your company and customers. I always wonder about the tradeoffs. Is there a measurable latency difference for your non-colocated customers? What does maintenance look like for your Go servers? I assume that your Cloudflare costs dropped?
- chronark_ 1y ago
  It’s faster for non-colocated customers too weirdly I think cause connections can be reused more often. Cloud flare workers are really prone to doing a lot of TLS handshakes cause they spin up new ones constantly Right now were just hang aws far hate for the go servers, so there really isn’t much maintenance at all. We’ll be moving that into eks soon though cause we are starting to add more stuff and need k8s anyways
wiether 1y ago
Not a question: thanks for the writeup and for the honesty of saying that serverless is not inherently bad, just not the right fit for your usecase! Unfortunately too many comments here are quick to come to the wrong conclusion, based only on the title. Not a reason to change it though!
- chronark_ 1y ago
  Thanks It’s totally fair criticism that the title and wording is a bit clickbaity But that’s ok
Sammi 1y ago
Do you have a clearer picture of what use-cases you would use serverless functions for in the future (if any)?
1y ago

torginus 1y ago

I think someone should make a timeline of software technology eras, each beginning with 'why XYZ is the future' and ending with articles like this.

yilugurlu 1y ago

These two have resonated with me deeply. - Eliminated complex caching workarounds and data pipeline overhead - Simplified architecture from distributed system to straightforward application We, as developers/engineers (put whatever title you want), tend to make things complex for no reason sometimes. Not all systems have to follow state-of-the-art best practices. Many times, secure, stable, durable systems outperform these fancy techs and inventions. Don't get me wrong, I love to use all of these technologies and fancy stuff, but sometimes that old, boring, monolithic API running on an EC2 solves 98% of your business problems, so no need to introduce ECS, K8S, Serverless, or whatever. Anyway, I guess I'm getting old, or I understand the value of a resilient system, and I'm trying to find peace xD.

ramraj07 1y ago
But when were serverless systems like lambda and cloud workers "best practices" for low latency apis?
- hedora 1y ago
  According to their marketing material, when they started supporting running in edge pop's, they became the best option for low-latency APIs.
  - daxfohl 1y ago
    Last I heard (~5 years ago), lambda@edge doesn't actually run on edge POPs anyway; they're just hooks that you can put in your edge configs that execute logic in the nearest region before/after running your edge config. But it's definitely a datacenter round-trip to invoke them. Adding that much compute to an edge POP is a big lift; even firecracker gets heavy at scale. And security risk for executing arbitrary code since these POPs don't have near the physical security of a datacenter, small scale makes more vulnerable to timing attacks, etc.

1-6 1y ago

I think this is what is being said: "Down with serverless! Long live serverless!"

voodooEntity 1y ago

As someone who worked with serverless for multiple years (mostly amazon lambda but others too) i can absolutely apporove the authors points. While it "takes away" some work from you, it adds this work on other points to solve the "artificial induced problems". Another example i hit was a hard upload limit. Ported an application to a serverless variant, had an import API for huge customer exports. Shouldnt be a problem right? Just setup an ingest endpoint and some background workers to process the data. Tho than i learned : i cant upload more than 100mb at a time through the "api gateway" (basically their proxy to invoke your code) and when asking if i could change it somehow i just was told to tell our customers to upload smaller file chunks. While from a "technical" perspective this sounds logical, our customers not gonne start exchanging all their software so we get a "nicer upload strategy". For me this is comparable with "it works in a vacuum" type of things. Its cool in theory, but as soon it hits reality you will realice quite fast that the time and money you safed on changing from permanent running machines to serverless, you will spend in other ways to solve the serverless specialities.

akdev1l 1y ago
The way to work around this issue is to provide a presigned S3 url Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something) I agree this is annoying and maybe I’ve been in AWS ecosystem for too long. However having an API that accepts an unbounded amount of data is a good recipe for DoS attacks, I suppose the 100MB is outdated as internet has gotten faster but eventually we do need some limit
- voodooEntity 1y ago
  Well i partly agree, and if i would be the one building the counterpart, i prolly had used presigned s3 urls also. In this specific case im getting oldschool file upload request from software that was partly written before the 2000s - noones gonne adjust anything any more. And ye, just accepting giant size uploads is far from good in terms of "Security" like DoS - but ye we talking about stupidly somewhere between 100 and 300mb CSV files (called them "huge" because in terms of product data 200-300mb text include quite alot) - not great but well we try to satisfy our customers needs. But ye like all the other points - everything is solvable somehow - just needs us to spend more time to solve something that technickly wasn't a real problem in first place. Edit: Another funny example. In a similar process on another provider i downloaded files in a similar size range from S3 to parse them - which died again and again. After contacting the hoster, because their logs litearlly just stopped no error tracing nothing) they told me that basically their setup only allows for 10mb local storing - and the default (in this case aws s3 adapter for PHP) always downloads it even if you tell it to "stream". So i build a solution that used HTTP ranged requests to "fake stream" the file into memory in smaller chunks so i could process it afterwards without completely download it. Just another example of : yes its solvable, but annoying.

tacker2000 1y ago

Incredible that these kinds of services were hosted like this. I guess they never came out of MVP, which could warrant using serverless, but in the end it makes 0 sense to use some slow solution like this for the service they are offering. Why didnt they go with a self hosted backend right away? Its funny how nowadays most devs are too scared to roll their own and just go with the cloud offerings that cost them tech debt and actual money down the road.

chronark_ 1y ago
We did initially but thought cloud flare was a better solution for scalability and latency. We believed their docs/marketing without doing extensive benchmarks, which is on us. The appeal was also to use the same typescript stack across everything, which was nice to work with
- ramraj07 1y ago
  Where did their marketing or documentation say this service is perfect for low latency APIs?
  - chronark_ 1y ago
    I doubt they literally said “perfect for low latency APIs” but their messaging is definitely trying to convince you that they’re fast globally, just look at the workers.ckoudflare.com page

K0IN 1y ago

After building my first Serverless/Cloudflare worker app, this is why I migrated to Deno. Deno enables you to run the same codebase in deno (self-hosted/local) and in deno deploy (serverless platform from deno). I wanted my app to be self-hostable as well, and Cloudflare worker is a hard ecosystem lock to their platform, which makes it undesirable (imo). Here is a link to my reasoning from back then: https://github.com/K0IN/Notify/pull/77#issuecomment-1677607057 https://github.com/K0IN/Notify/pull/77#issuecomment-16776070...

scottydelta 1y ago
I ported my worker project into Django since cloudflare workers wouldn’t allow selection of region for hosting workers which is generally required due to data compliances. This is something all cloud providers provide from day one yet cloudflare made it an enterprise feature. Also the vendor lock-in doesn’t help with durable objects and D2 instead of simply doing what supabase and others are doing by providing Postgres or standard SQLite as a service.

gloomyday 1y ago

I think developers are drowning in tools to make things "easy", when in truth many problems are already easy with the most basic stuff in our tool belt (a compiler, some bash scripts, and some libraries). You can always build up from there. This tooling fetish hurts both companies and developers.

bamboozled 1y ago
Excerpt AWS lambda is stupidly cheap!
- Esophagus4 1y ago
  For certain workloads :) And that is actually the advantage of serverless, in my mind. For some low-traffic workloads, you can host for next to nothing. Per invocation, it is expensive, but if you only have a few invocations of a workload that isn't very latency sensitive, you can run an entirely serverless architecture for pennies per month. Where people get burned is moving high traffic volumes to serverless... then they look at their bill and go, "Oh my god, what have I done!?" Or they try to throw all sorts of duct tape at serverless to make it highly performant, which is a fool's errand.
  - cogman10 1y ago
    Exactly. I've always found that how people want to use lambda is the exact opposite of how to use it cost effectively. I've seen a lot of people want to use lambdas as rest endpoints and effectively replace their entire API with a cluster of lambdas. But that's about the most expensive way to use a lambda! 1 request, one lambda. Where these things are useful is when you say "I have this daily data pull and ETL that I need to do." Then all the sudden the cost is pretty dang competitive.
    - hvb2 1y ago
      > Where these things are useful All the backend processing and just general 'glue' in your architectures

codegeek 1y ago

"Self-Hosting : Being tied to Cloudflare's runtime meant our customers couldn't self-host Unkey. While the Workers runtime is technically open source, getting it running locally (even in dev mode) is incredibly difficult. With standard Go servers, self-hosting becomes trivial:" A key point that I always make. Serverless is good if you want a simple periodic task to run intermittently without worrying about a full time server. The moment things get more complex than that (which in real world it almost always is), you need a proper server.

jjice 1y ago

Like the article says, I think serverless has it's place, but I don't think it's for most applications. I can't see myself _ever_ using serverless services as a core part of my application for pretty much any startup, if I can avoid it. The infrastructure overhead is actually worse, IMO. Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to test, and the abstraction layers that exist (unless this has changed recently) always had pitfalls, since there are no true standards. I'd much rather have a docker image as my deliverable "executable" so I can test, but still abstract away some stuff, like environment setup. Giving me a minimal Linux environment and filesystem feels like the most comfortable level of abstraction for me to develop well and also deploy and run in production effectively. I can also run that on demand or (most commonly) run that as a server that sits and waits for requests.

wredcoll 1y ago
Let me tell you about all the fun I'm having trying to execute my amazon lambda app locally so I can test before deploying...
- renlo 1y ago
  Localstack makes that pretty easy. Before Localstack I had a pre-staging environment (dev) target I would deploy to. Their free/community offering includes a Lambda environment; you deploy your dev "Lambda" locally to docker, using the same terraform / deploy flow you'd normally use but pointed at a Localstack server which mimics the AWS API instead. Some of their other offerings require you to pay though (Cloudfront, ECS, others) and I don't use those, yet at least.
  - meekins 1y ago
    Developer-specific sandbox environments with hot code reload is the golden standard here but Localstack is great if you can't do that due to (usually IT deparment-related, not technical) reasons.
  - discomrobertul8 1y ago

deleted 1y ago

[deleted]

kordlessagain 1y ago

If they use Cloudflare that automatically disqualifies them from me reading whatever they wrote, or caring about it.

rvitorper 1y ago

We all love a vendor lock-in, don’t we? Until it backstabs us and we go back to VMs

daxfohl 1y ago

I don't think "serverless is bad" is necessarily the full lesson here. The bigger lesson is when a service has dependencies, moving that service closer to the client (without also moving those dependencies) will counterintuitively make the e2e experience slower, not faster. Prefer building physically near your dependencies. If that's not fast enough, then you have to figure out how to move or sync all your dependencies closer to the client, which except in very simple cases, is almost always a huge can of worms.

brap 1y ago
I don’t know if this is a good rule of thumb, I think it really depends on what you use the dependencies for, how often you need them, etc. Consider for example a single DB dependency. Should the server be close to the DB or the client? It depends. How often does the client need the server? How often does the server need the DB? Which usecases are expected to be fast and which can be sacrificed as slow? What can be cached in the server? What can be cached in the client? etc etc. And then of course you can split and do some things on the server and some in the edge…
- marcosdumay 1y ago
  Oh, it's a very good rule of thumb. It's probably not universal, but it's really close to it. the problem is that nobody designs the dependencies flexible enough to let them run without fine-control. And the main application always wants to change the way it uses the dependencies, so it always needs further flexibility. You can build an exception to the rule if you explicitly try. But I'm not sure one appears naturally. The natural way to migrate your server into the edge is by migrating entire workloads, dependencies included. You can split the work like you said, you just can't split single endpoints.
  - daxfohl 1y ago
    Well, I guess one can take one more step back and say this is all merely an example of "premature optimization is the root of all evil". Unless you know a-priori that you have some very hard latency requirements, start with something simple and low-maintenance. If low-latency requirements come in later, then design that holistically, not just looking at your component. Make sure you're measuring the right things; OOTB metrics often miss the e2e experience. And IME most latency issues come from unexpected places; I know I've spent weeks optimizing services to get an extra percent or two out of them, only to realize there's a config setting that reduced latency by half. So generally, simplicity is your friend when it comes to latencies (among other things). Fewer things to cause long-tail spikes, more simple things you can try out that don't break the whole system, whereas if you start with a highly-optimized thing up-front, fixing some unexpected long-tail issue may require a complete rewrite. Also, check with your PM or end users as to whether latency is even important. If the call to your service is generally followed up to a call to some ten-second process, users aren't going to notice the 20ms improvement to your own thing.

aeldidi 1y ago

I think the "local maximum" we've gotten stuck at for application hosting is having a docker container as the canonical environment/deliverable, and injecting secrets when needed. That makes it easy to run and test locally, but still provides most of the benefits I think (infrastructure-as-code setups, reproducibility, etc). Serverless goes a little too far for most applications (in my opinion), but I have to admit some apps work really well under that model. There's a nearly endless number of simple/trivial utilities which wouldn't really gain anything from having their own infrastructure and would work just fine in a shared or on-demand hosting environment, and a massively scaled stateless service would thrive under a serverless environment much more than it would on a traditional server. That's not to say that I think serverless is somehow only for simple or trivial use cases though, only that there's an impedance mismatch between the "classic web app" model, and what these platforms provide.

daitangio 1y ago
You are ready for misterio: https://github.com/daitangio/misterio https://github.com/daitangio/misterio A tiny layer around stareless docker cluster. I created it for my homelab and it gone wild
- aeldidi 1y ago
  That's really interesting, I might actually use that for mine too. Thanks for sharing.
mattmanser 1y ago
Docker is much like microservices. Appropriate for a subset of apps and yet touted as being 'the norm' when it shouldn't be. There are drawbacks to using docker, such as security patching and operational overhead. And if you're blindly putting it into every project, how are you mitigating the risks it introduces? Worse, the big reason it was useful, managing dependency hell, has largely been solved by making developers default to not installing dependencies globally. We don't really need Docker anywhere near like we used to, and yet it persists as the default, unassailable. Of course hosting companies must LOVE it, docker containers must increase their margins by 10% at least! Someone else down thread has mentioned a tooling fetish, I feel Docker is part of that fetish.

mannyv 1y ago

It sounds like you picked the wrong platform because you didn’t understand what you were doing. That’s not a technology issue.

nine_k 1y ago

One thing I could not find in the write-up is the change in the expense. Did serverless save any money, compared to always-up VMs? Did much of their load run under the free tier limits? Serverless shines when the load is very spiky, and you can afford high long-tail latency. Then you don't pay for all that time when your server would be idling. (This is usually not the case for auth APIs, unless they auth other infrequently invoked operations.)

hinkley 1y ago

"Taking hand off of boiling kettle decreased anxiety and increased focus in 97% of study participants." Stephen King's Dark Tower series never resonated with me and I got stuck in book two. But it has one of my favorite philosophical insults of all time: "Those who [X] have forgotten the faces of their fathers." I feel like there's a collective amnesia just beginning to wear off as people remember the Fallacies of Distributed Computing and basic facts about multitasking. And that amnesia absolutely feels to me as if everyone has forgotten the faces of their fathers. <waves cane threateningly>

ThinkBeat 1y ago

As a soon to be graybeard think this has been fairly obvious from the start. And outside of specific workflows, you are adding unneeded complexity to a system that does not need it. In general, an anti-pattern. but it does have valid use in some cases. On the last project I worked on that came to involve serverless, it made no sense at all other than it was "the fad". For this system we had an excellent knowledge of what the theoretical limit of users and connections as. This apparently needed to be done container, serverless, kafka, blah blah. Annoyed with the whole thing I took a few nights to tear logic out from the micro servers or nano services, and wrapped the whole thing into a Frankenstein monolith. AT least 60% of the code all had to do with dealing solely with the code needed to pass information around to different services so it was easier to maintain. Well my hacked together moonlight was not a great start for anything but a demo. I installed Postgres on my laptop, ran the monolith on it, took 3 servers each pushing the theoretical maximum load we would have, and what do you know the performance was fine. But the architecture was the architecture decided upon.

danpalmer 1y ago

30ms p99 for a cache read! Serverless might have been a problem, but I'm not sure it was the problem. In my experience a p99 of 2ms is more typical - 30ms is the sort of time I'd expect for p99 on a database query in production serving. You don't need process-local caches to get the sort of performance they're looking for, and there are good reasons why most teams avoid stateful processing, it's much harder to get right and has bad failure modes.

hunterpayne 1y ago
Fun fact, the overhead for a DB query (not including the network latency) is about 250 microseconds (or 0.25ms). And a typical DB kernel can do at least 16MB/core second of throughput. So that 30ms query I would expect is scanning several GB of data to find its results. Try doing that yourself and your latency would probably grow into seconds. Systems programming is hard... PS moving from stateful processing to something with a distributed cache is moving from RAM to network. That's at least a 10x drop off in throughput.

jrm4 1y ago

This really is confirming my theory that the problem here is that "serverless" is so ill-defined that even it's own name is nonsensical. Like, there are still servers. Appears just about as intelligent as calling them "electricity-less." I mean, yes, I no longer think about electricity when deploying things, but that doesn't tell me anything meaningful about what's going on here.

ljm 1y ago
It's more like cgi-bin as a service.

pluto_modadic 1y ago

da rulez: - if you don't understand your concept/market, build in a VPS. you can get away with scaling a VPS for a while. - if you intend to be netflix, rent at the edge (mid game) and eventually own the edge servers in POPs (otherwise, "edge" compute isn't worth it). before that, start with a beefy VPS cluster with HA and SQLproxy. - if you spin to zero, use lambdas (for things like codebuild or fractional needs of a computer). before that, build things in VPSes. - if you spin up/down but not to zero, use container platforms? - once you have a reliably steady understanding of your infrastructure, buy physical servers in a colo

Havoc 1y ago

If it’s got to be serverless then use a PaaS that is docker based as input format. That at least gives some level of platform portability if you need to shift it later But yeah in this case 10ms requirement doesn’t leave a lot of room for elaborate anything

jppope 1y ago

I would love to know the net result to their financials for this move. I have no doubt they were able to improve their performance. I'm just wondering if the juice was worth the squeeze, especially if they could have been building other features that customers would want. I didn't read anything about the opportunity cost in the article or even the consideration.

jmg_ 1y ago

> We built chproxy specifically because ClickHouse doesn't like thousands of tiny inserts. It's a Go service that buffers events and sends them in large batches. Each Cloudflare Worker would send individual analytics events to chproxy, which would then aggregate and send them to ClickHouse. While I understand how this isn't the only thing that needed to be buffered, for Clickhouse data specifically I'd be curious why they built a separate service rather than use asynchronous inserts: https://clickhouse.com/docs/optimize/asynchronous-inserts https://clickhouse.com/docs/optimize/asynchronous-inserts

mikeiz404 1y ago

Hey "vercel security checkpoint", I'm repeatedly getting a code 99 "Failed to verify your browser" on an iphone with a VPN exits in California and Canada via proton VPN and Firefox Focus. What gives? ``` sfo1::1760587368-8k6JCK3uO27oMpuTbnS4Hb3X2K9bVsc ```

scuff3d 1y ago

Serverless seems to me like the tech version of the classic sales technique. You see it every Black Friday, get them in the door with the cheap alluring offers, then upsell the shit out of them. And in the case of serverless, lock them with as many provider specific services as possible. I feel like we're headed the same direction with AI right now. It creates as many problems as it solves, and the solution is always to pay them for the newer faster model and use more tokens. With all the new AI security threats coming to light we'll start seeing them offer solutions for that too. They'll sell you the threats and the solutions, and the entire developer community will thank them for it.

TZubiri 1y ago

Seems like serverless is ideal for that <10$/mo budget sweetspot when you can't yet afford a VPS.

WJW 1y ago
That mythical sweetspot where you can afford extra development work, but cannot afford 10$/month?
- TZubiri 1y ago
  Not only is this the case for solo devs, tech partnerships, and spend-averse people who don't do money to time conversions. But if you have low cash on a company and have excess workforce, you might not necessarily get to a point where you have to fire, but you can spend those, already paid for, man-hours, in migrations and infrastructure to save some cash. As an added bonus you have tasks to give your otherwise idle workforce to keep them warmed up

ForHackernews 1y ago

Many organizations would benefit from just running a well-architected monolith. Amazon pushed microservices hard internally, but they face scale challenges most companies do not have and will never have.

KaiserPro 1y ago

about 5 years ago we made an API that took an image and gave you a lat/lon in return (basically like a visual GPS) The front end was entirely in python AWS lambdas, with a message queue that talked to the (slow) GPU backend. The "lambda tax" was about 100ms on average. given that an average request was around 4 seconds it seemed ok.

callamdelaney 1y ago

I once got a job for a company whose entire stack was aws lambda. Not like a normal 'hey we have this flask application which handles 100 endpoints and we just use lambda to handle a call' - more like 'every singe route or function is its own lambda function'. I left inside 2 weeks.

whstl 1y ago
Saw this with an infamous 996 AI startup where the founders vibe coded their app. Their app had like 3 or 4 pages, that was all, but every endpoint was a Lambda call. Every frontend log entry was a Lambda call. Every job execution was a Lambda call that called a dozen others. That was their idea of an architecture. Even with <100 customers they were already paying >100k dollars a month on cloud, more than their entire dev team cost. The constant firefighting, downtimes and billing issues were enough to keep the team busy instead of making new features. I was freelancing but I didn't take the job after the "CTO" gave me a tour.
- naasking 1y ago
  In principle, serverless should be very efficient and cheap if all code is running the same VM. The heterogeneous architecture must make it expensive. Maybe serverless using WASM will one day make it cheap.