4 ms·

Saving another 100TB of RAM

• johnnyApplePRNG 8d ago
  [flagged]
• go_elmo 8d ago
  [flagged]
  • readthenotes1 8d ago
    I expect a another post in a year where they get a performance improvement by reducing the number of calls to create integers from bytes.
  • Maxion 8d ago
    I think it's more important now than it was last year to differentiate pure yolo vibecoding from "AI assisted engineering" or "AI engineering", I.e. deliberate and careful use of AI to speed up coding but without creating too much slop. Maybe aineering?
    • kawogi 8d ago
      AIded development?
      • derwiki 8d ago
        Just like the original harness, aider.chat
    • wild_pointer 8d ago
      Nah, AI code is black or white, and whether the code is good depends on your religion.
  • n738 8d ago
    go_elmo is unimpressed everyone. Pack it up. Time to go home.
• officialchicken 8d ago
  [flagged]
  • agosta 8d ago
    We can tell you didn't read the post because it is definitively NOT garbage. Very interesting write up by the Cloudflare team - the man literally did calculus to improve something. When's the last time any of us did Calculus to improve anything? Bang up job Kevin and everyone!!
  • cyberpunk 8d ago
    Anyone have an idea how it behaves differently from google's jump hash algorithm? The cool thing about google's one is it's so short I can include it in a HN comment: int32_t JumpConsistentHash(uint64_t key, int32_t num_buckets) { int64_t b = 1, j = 0; while (j < num_buckets) { b = j; key = key * 2862933555777941757ULL + 1; j = (b + 1) * (double(1LL << 31) / double((key >> 33) + 1)); } return b; } https://arxiv.org/pdf/1406.2294 https://arxiv.org/pdf/1406.2294
    • cyberpunk 8d ago
      Well I looked it up; nginx, apparently, uses ketama -- it's a ring-style hash probably works better for web backends than the jch above, as when given [0,1,2,3] and replacing the server in slot 1 you're going to have a lot of hash moves. With ketama, you'd only have the '1' hashes moving. You can't really beat google's for brevity, though.
    • prirun 8d ago
      I have used Google's jump hash. As I recall, one of the main differences is that jump hash doesn't have a mechanism to remove targets, eg, a server dies and you don't want to route requests to it. Traditional consistent hashing can do that. I guess if you had 4 servers, server #4 dies, then you can go back to 3 servers by just changing num_buckets from 4 to 3. But if server 1 dies, you can't. Jump hash does allow adding more targets and preserves the property that most request targets stay the same when adding a new target, so if you had 3 targets and add a fourth, ~8% of the requests that would have been sent to targets 1-3 are sent to target 4, evenly chosen from servers 1-3.
• agosta 8d ago
  Bang up article! As someone who doesn't get to do enough (almost any) calculus in my daily programming assignments, I thoroughly enjoyed reading about Kevin's dive into that derivation (linked in the supplemental article). All the people being negative here can swallow raisins
  • terabyteoff 8d ago
    Thanks! Maybe dial it back or people are going to think I paid you
• proc0 8d ago
  The only Rust section is the one on storage improvements about the struct that stores the hash, but do they really need that many hashes that 2 bytes makes that big of a difference? Article doesn't expand, but I guess it's a hash for every task on every computer, so maybe yes.
  • agosta 8d ago
    That's exactly his point/the area of cost saving - that they didn't actually need as many hashes as they had started with. The trick was in finding out how many hashes they could cull without degrading load balance.
• ricardobeat 8d ago
  These optimizations are impressive, but it gets me thinking: at what point does a company become a collection of impenetrable siloes, where nothing really does what you expect? Maybe know with AI this is less of an issue as exploring a codebase is also much faster.
  • simonjgreen 8d ago
    My intuition around larger companies is they are already impenetrable silos, and AI makes it worse
  • BobbyTables2 8d ago
    I feel like any company whose products have RESTful interfaces are already there… One wants to turn on an indicator on a remote device. A simple Boolean value. But we need networking, TLS, authentication plugins, certificate validation, distributed logging, containers, orchestration, HTTP client/server, interprocess communication, daemon dependency management, … Sure, one can say each of these layers and abstractions has an important and justifiable purpose. But one can also step back and start wondering - what the hell are we really doing??? At some level, it seems like each layer of abstraction has to manage others, only simply because they exist. Imagine the simplicity of 1800s telegraph signaling - no software! Too often we build systems with Fortune-50 style hierarchies when a 5-person team could do the whole job.
    • pixl97 8d ago
      Build a system as simple as possible but no simpler. An 1800s telegraph system doesnt work in the modem world, there is far too much communication and the system would just collapse into molten slag. All those things you've listed are because we live in an adversarial world and I'd steal all your money off the telegraph wire if you tried it.
      • deleted 7d ago
        [deleted]
      • HPsquared 7d ago
• jamesforestwest 8d ago
  [dead]
• dr_dshiv 8d ago
  Cloudflare is truly amazing, they have made so much possible for my main side-project at a price and performance that I can’t really take credit for (http://sourcelibrary.org http://sourcelibrary.org), I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
  • davidbarker 8d ago
    This is pleasant coincidence. Really like your site and it's queued to send in my newsletter in the morning! Just happened to see your comment here while I was reading. Great work.
    • dr_dshiv 7d ago
      Oh super — I appreciate that!
  • ChoosesBarbecue 8d ago
    > I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm… Tried out the first 1000 words in Pangram, and it seemed happy it was human written. Not surprised either, it has been some of the better writing I've seen out of Cloudflare recently.
    • terabyteoff 8d ago
      An AI would have known that saying, “Hi, mom” in a professional post was a bad idea.
      • AdamN 7d ago
        Great - now you've trained the next set of AI to put things like "Hi, mom" into their text as a tell that they're human ...
• kingleopold 8d ago
  anyone remember 100tb hosting company?
• sroussey 8d ago
  Someone really needed a few hundred TB to waste on inference and went looking under the rugs…
  • why_only_15 8d ago
    CPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.
    • sroussey 8d ago
      Those machines with GPUs still need RAM of their own, and they generally want large caches to avoid SSD penalties. You even see this spill out in the form of costs for KV cache in <1min, 5m, 1hr rates etc.
      • why_only_15 5d ago
        yeah there's some KV cache offload
    • halJordan 8d ago
      The majority of inference actually does happen in cpu.
      • why_only_15 5d ago
        This is just wrong unless you're using some confusing definition. Notice that companies trying to do lots of inference aren't looking for CPUs, they're looking for GPUs/TPUs.
• parallax_error 8d ago
  I definitely enjoyed this writing style more than a lot of the recent cf blog posts. Cool article!
• swe_dima 8d ago
  does this mean RAM prices can go down now? Please?
• Fordec 8d ago
  This sort of thing makes me thing that we're about to enter an era where software development is going to be where most of the jobs fallout will be. You can't one-shot vibe code your way to this. But for proper Software Engineering, those jobs are safe where more and more problems are going to actually need solving by creatively using math because all the problems individuals deliver are just going to be larger. People are just mourning the loss of the low hanging fruit.
  • killingtime74 8d ago
    I think you're speaking like a software engineer, which is understandable, and not like a historian or economist. There's no reason to believe math based jobs would survive. The models regularly do well on math problems. You can auto-research loop ways to optimize memory usage for any particular program.
    • Fordec 8d ago
      The point isn't that "doing math" is safe. Auto-research solves one target variable in one system, doing it at scale where say one developer is SME for the agentically manged 200 microservices down the line, heh I mean you certainly can, but good luck with that token cost of auto-research when that problem space is O(microservice^2). I point at that example yesterday of that optimized database memory with the comments pointing out that the specific problem fit in memory, over optimized and didn't generalize. The problem isn't the work, but the rework. A historian should know that new solutions to problems doesn't lead to "no problems ever again" but only problems with barriers that the new solution doesn't solve.
      • adrianN 7d ago
        The argument is probably that LLMs can find those optimizations cheaper than a human expert. Since LLM cost at fixed capability seems to be going down you either expect humans to be completely replaced or human wages to be lowered by LLMs.
        • 7d ago
• zer0x4d 8d ago
  Incredibly happy to see this series of CF articles. I was always so proud of devs back in the days where RAM and processing were scarce and who had to get creative to fit even the most basic stuff in the budget. It seemed to me that after RAM and processing became abundant, most gave up on optimization and focused on shipping instead which meant now that even with several cores, a basic notepad or music player failed to work. In a way, RAM becoming more expensive has ushered in a new era of forced optimizations, which I'm really happy for
  • jfengel 8d ago
    I don't remember those days with a ton of fondness. Yes, the challenge was fun, but I really wanted to ship it and get my product in the hands of customers. Now I can spend more time thinking about what they want and less time about what the computer wants.
    • JungleGymSam 7d ago
      [dead]
    • switchbak 7d ago
      Yes, I remember those too. The costs of manual memory management were real and were not low. But costs on the cloud are real too, especially now. I’ve been living in JVM land for a very long time, but now it’s especially clear how important lean services are. Especially now that the bar for writing lean code is so much lower: let the borrow checker figure it out, etc. I just spent a couple days wringing out more performance/memory efficiency for our services. Nice gains to be sure, but it’s still so immensely wasteful compared to something well written running native. If it was my money, I’d be going native for sure.
      • locknitpicker 7d ago
        > But costs on the cloud are real too, especially now. I’ve been living in JVM land for a very long time, but now it’s especially clear how important lean services are. Especially now that the bar for writing lean code is so much lower: let the borrow checker figure it out, etc. I don't think even Cloudflare bothers with this waste of time. If they did, they would certainly not have built their global infrastructure on JavaScript running on V8. They'd have done what Google and old-time Facebook did and built their whole infrastructure on low-level system languages, and hiring the world's leading minds on the subject to milk the last drop of performance from their hardware. Even Google stopped to look at the problem and came up with Go. Not V8.
• aaron695 8d ago
  [dead]
• opsnotes80 8d ago
  [flagged]
• variety8675 8d ago
  It’s nice to see Cloudflare is letting humans write the blog posts again after all the fallout from their LLM slop blogs
• opsnotes80 8d ago
  [flagged]
• jiggawatts 7d ago
  I'm surprised to see no mention of hierarchical rendezvous hashing in either the article or the comments here. It is purpose-designed for exactly this type of proxy/cache load-balancing scenario!
• vlovich123 7d ago
  I would get rid of consistent hashing and ketama for a better system which works save an additional 600TiB. You use the first N bits of your key hash to pick the server partition so it’s a reasonable number (eg 128 servers per partition). Then use high quality precomputed hashes (first 64 bits of sha256) for the server name as N in H(K + N). Use wymum from wyhash as the H so that you do o(n) integer multiplications while retaining a result that’s still a good hash statistically. Now you’re using a tournament hash, the small N means O(N) vs O(N log N) doesn’t matter, and also this O(N) is also going to be much less CPU than computing 160 hashes per key as they do now, so much less latency added per request.
  • MakersF 7d ago
    I think they do only a hash per request. The 160*weight hashes were done per server (per feature set), to partition the hash space. Per request you do a single hash and then a lower_bound on a sorted map to find the serving server (again, on the ring appropriate for the features required by the request, so likely a hash map lookup first)
• sfink 7d ago
  Um. I read the article thinking it would make for a great brain puzzle, but I quickly decided there's something wrong with the question setup because the initial solution didn't make sense. I assumed it was just missing a constraint that would be revealed later, but I'm still not seeing it -- the article just kept patching up the flaws in the wrong solution, the one that is more complicated than the straightforward one. I'm probably still missing something obvious? It's probably something to do with "...in a way that does not require large changes when servers are added or removed." But let's start with the problem as initially posed: you have an infinite stream of tasks and you need to deterministically assign them to N servers. (Perhaps you have to shard the collections of servers, so not every load balancer knows about all of them? But no, that would break the solution in the article.) Ok, then hash the task request (I assume that you hash it, the article doesn't explicitly say, but that's how you'd get determinism) and take that hash mod N, that's your server index. Why hash the servers too? If you roll 6 dice, and then another one to choose which die to use, you're not getting any more randomness. You're matching up two sides, the tasks on one side and the servers on the other; no need to randomize both. Ooh, but that's not a perfect distribution? Ok, if the hash value is large enough to be in the at most N-1 slop values at the top of UINT_MAX, then roll again (compute another hash). But CF is happy with 8% unevenness, there should be no problem with this 0.1% or whatever. Also, how do they find the nearest server hash to a task hash? Surely it's not a log(n) binary search through sorted server hashes, I hope? Weights break this scheme. Now each server has some number of tickets. So you compute hash % T (where T=total tickets) and have to figure out what server that is. There's probably a more clever way, but you could make a big array of (2-byte!) server indexes, one per ticket, and just fill them in and look up at index hash % T. That's 2 bytes per ticket, which feels uncomfortably wasteful if weights can be large. That's where things get more complicated for me: since the tasks are hashed, it doesn't matter what order a server's indexes come in relative to other servers', so sort them by descending weight. [I'm starting to suspect I'm making a fool of myself here by missing something obvious with the whole setup...] Now you can make an array of indexes for servers with the highest weight, then the next lower, then the next. Record the number of servers of each weight. Then you can take the hash % T and figure out which array it's in, then divide by the weight to give the index within that array. To reduce the number of per-weight arrays, you can restrict the weights allowed. If you restrict weights to be powers of two, you can eliminate a division by using a shift. If you really want more flexible weights, you can allow servers to be in more than one of the arrays. Let the arrays be powers of two, and then add an entry to each array corresponding to 1 bits in the binary representation of the weights. That increases the total memory usage of the arrays, so you could somewhat restrict the allowed weights by rounding to the nearest number with, say, 2 or 3 "on" bits at most. With at most 2 bits, that means weights are 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 16, 17, .... The error really isn't bad. And this should all be easily doable without any branches, I'm pretty sure. As long as you statically cap the max weight. Anyway, that's just plowing through with the straightforward approach, and I still think I'm probably missing something major here. I imagine with large numbers of servers, some go down, so fast deletions are probably important. You can get by a little while by marking dead servers and if you "roll" one, just roll again. (Yes, deterministically, assuming other load balancers agree that the server is down.) But when more than some number of servers go down, you'd want to kick off a background task to rebuild a new set of tables -- so that's a factor 2 in size usage to have them both in memory during the rebuild. Adding is trickier, you'd probably want to do a 2-level structure where first you use the hash to decide whether it's in the old set that the table is built for or the set of servers that hasn't been incorporated yet (you'd collect these over time, and empty them out on the next table rebuild.) It's a little weird, because the load balancers' outputs would only agree when the added and deleted sets agreed, but I don't see how to do better than that. (I think you could set up some kind of synchronization scheme so that the old sets would agree, which would make them usually agree on which of the old set of machines gets it.) Somebody, feel free to tell me I'm being stupid! I'm sure there's a constraint that I'm missing, given that my understanding of the initial problem doesn't require any memory at all except for the servers' info. (Or if not, I'll let you know where I'd like to receive shipment of 1% of the memory I've saved...)
• schobi 7d ago
  I can imagine the other internal teams looking at this.. "100 TB gets you attention? Hold my beer.. we will try that as well!"
• videocompressde 7d ago
  I've hit the same thing at smaller scale — once you know the real cardinality, shaving a few bytes per entry beats the clever stuff that never got profiled.
• MisterMunchkin 7d ago
  I like that they have a performance team that actually tries to improve their products over time. I also really appreciate the fact that this is human-written and not just AI slop. It’s refreshing to actually read English instead of Claudelish.
• goodpoint 7d ago
  TLDR: the existing implementation was poorly designed. They packed 2 integers better and saved memory.
• cloudengineer94 7d ago
  Every single optimization stories I read such as these make me super happy
• christina97 7d ago
  I’m not sure why folks are finding this so revolutionary. There are teams of scientists at big techs with PhDs working on all kinds of optimization across compute fleets. This seems cute but its exposition of math is more along the lines of “look at how cool I am that I could do a bit of calculus”, and that makes me question the technical depth at CF. I found the motivation pretty lackluster: nowhere does it actually explain why you use consistent hashing (dividing the item space naively/regularly would actually cause much more than 1/n items to move, which is unintuitive) and how you actually use it. That said, it got me to spend a few minutes studying this and got me to understand the key bit I was missing.
• Sevii 7d ago
  It's crazy to realize 100TB of RAM isn't that much anymore. 10TB server racks are already in production. Before long we'll have 100TB racks serving one instance of an LLM.
• nopurpose 7d ago
  Do I understand correctly, that they spent memory storing largish N hash values per server, so that request hash determines which server to send request to using closest higher value of all server hashes? That in effect boils down to consistently selecting server S with probability P, where P is function of weight and total number of servers? Surely there must be better way to select server with a given probability without storing a massive lookup table of hashes? Randevouz hashing of some sorts
  • varispeed 7d ago
    Who cares if you can buy all the RAM available. To hell with small business and working class who now cannot afford it.
  • QuaternionsBhop 7d ago
    Plus it's limited to 65k entries. Perhaps a btree where parent nodes sum the weights of child nodes would work well. Using the input hash scaled by total weight, a binary search lookup would compute the partial sums for comparison on the fly. Adding/removing a node would only update the ~8 parents when the btree order is 4. Eytzinger layout and struct-of-arrays could be used to improve cache locality during lookup. This does mean an add/remove could drastically change the overall mapping, perhaps that's why consistent hashing is used instead.
• varispeed 7d ago
  Stop hoarding RAM. Are they going to offload it through ebay? Unlikely.
• sharktheone 7d ago
  So interesting to see. Though I think if Cloudflare really wanted to safe more memory, there are even more strategies. Just starting with pointer compression, even if that is not an option, other techniques exist in rust. Assuming some numbers in from the previous post, with 250GB per byte wasted, you could shave off 2bytes of every pointer or 4 if you can fit them into a context related memory cage. And even that is just a beginning here.