10 ms·

In my 3 years old laptop, system memory (dual channel DDR4-3200) delivers about 50 GB / second. You have measured 50M elements per second, with 8 bytes per elem

In my 3 years old laptop, system memory (dual channel DDR4-3200) delivers about 50 GB / second. You have measured 50M elements per second, with 8 bytes per element translates to 400 MB / second. If your hardware is similar, your implementation did less than 1% of theoretical bottleneck. When your data is indeed big and the performance actually matters, consider doing something completely different.

sgarland 2y ago
In the real world, your application is running in a container among hundreds or thousands of other containers. The system's resources are also probably being managed by a hypervisor. The CPU is shared among N tenants _and_ is overcommitted. It's not that much to ask to optimize where you can, when it isn't unreasonably difficult. More importantly, this attitude is precisely why software sucks today. "[CPU, Memory, Disk] is cheap, engineering time isn't." Fuck that. Bad engineering time is _incredibly_ expensive. This is an excuse to not spend time learning the ins and outs of your language, and your hardware. It also frustrates me to no end that people are so deeply incurious. This seems to only happen in tech, which is baffling considering how incredibly broad and deep the industry is. Instead, everyone clamors to learn the new Javascript abstraction that lets them get even further away from the reality of what the magic boxes are doing.
- kmarc 2y ago
  Agree with the sentiment. However it's hard to stay curious, even harder to stay up-to-date. I liked fiddling with storage for a while, got really into it, deepened my knowledge about it. A couple years later I realized everything else (networking, architectures, languages) developed so much, mot of my (non-basic) knowledge was obsolete. Picking up where I left off with all technologies is incredibly hard and caused fatigue. Now I'm at a point where I have the feeling I don't know nothing about anything. It's factually not true, but my gut feeling tells this. Would I be younger, this would trigger a lot of anxiety. Thankfully I can janfle this by now.
  - sgarland 2y ago
    That's understandable. I'm into databases (both professionally and personally), so I get to touch a wide variety of things. Following a query the whole way is pretty neat. Typed query --> client --> wire protocol --> network --> server --> DB frontend --> DB planner --> DB storage engine --> OS --> Memory / Disk. `blktrace` is super interesting to watch commands being sent to the disk.
    - buran77 2y ago
      When you are deeply involved in tech both personally and professionally you are probably very passionate about this and makes sense that you'd only look closely at this field and think "people are so deeply incurious. This seems to only happen in tech". Tech is also one of the (if not the) most dynamic and fast evolving field a normal person will ever touch. Curiosity in tech can drain every single bit of free time and energy you have and you will hardly keep up with the progress, maybe barely scratch the surface. But people's available free time and energy wanes and curiosity is a collateral victim. I've painfully gone through the entire cycle of this, including the bit of resurgence later on when you have a combination of free time but less energy. What I can say is that this absolutely does not happen just in tech. If anything tech is flooded with people with more curiosity than almost any other field.
      - sgarland 2y ago
        > When you are deeply involved in tech both personally and professionally you are probably very passionate about this and makes sense that you'd only look closely at this field and think "people are so deeply incurious. This seems to only happen in tech". Good point. I commented in a sibling post to the same effect. I've certainly felt the personal strain of time sink and procrastination in my homelab. It's currently running k3os, which has been dead for about four years now, because everything I want is still running, and I never seem have the motivation on the weekend to yell at my laptop when I could be playing board games. > including the bit of resurgence later on when you have a combination of free time but less energy. I'm guessing that will happen in another decade or so, when my kids are grown.
- akira2501 2y ago
  > "[CPU, Memory, Disk] is cheap, engineering time isn't." Fuck that. It is. It's absurdly cheap. I ensure I check the amount of time it would take for me to make a performance improvement against the runtime costs of my functions. It's rarely worth the extra effort. Seriously, until you get into the millions of records per second level, you're almost never benefited. You may make your function 2x faster, at a cost of additional complexity, but you never run it enough in a year for it to pay itself back. > Bad engineering time is _incredibly_ expensive. Engineering time is expensive. Period. It speaks to the need to minimize it. > This is an excuse to not spend time learning the ins and outs of your language, and your hardware. All of which will change in a few years, which is fine, if you're also committing to keeping _all that code_ up to date right along with it. Otherwise you end up with an obscure mess that you have to unwind 5 years of context to understand and fix again. Complexity and available mental contexts are forgotten costs. If your language even has that many "ins and outs" to begin with you may want to reconsider that.
  - sgarland 2y ago
    > You may make your function 2x faster, at a cost of additional complexity, but you never run it enough in a year for it to pay itself back. I'm not talking about increased complexity, I'm talking about extremely basic things that take zero extra time, like using the correct data structure. For example, in Python: In [8]: a = array("i", (x for x in range(1_000_000))) ...: l = [x for x in range(1_000_000)] ...: d = deque(l) ...: for x in (a, l, d): ...: print(f"{sys.getsizeof(x) / 2**20} MiB") ...: 3.902385711669922 MiB 8.057334899902344 MiB 7.868537902832031 MiB Very similar structures, with very different memory requirements and access speeds. I can count on one hand with no fingers the number of times I've seen an array used. Or knowing that `random.randint` is remarkably slow compared to `random.random()`, which can matter in a hot loop: In [10]: %timeit math.floor(random.random() * 1_000_000) 31.9 ns ± 0.138 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) In [11]: %timeit random.randint(0, 1_000_000) 163 ns ± 0.653 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) > All of which will change in a few years, which is fine, if you're also committing to keeping _all that code_ up to date right along with it. With the exception of list comprehension over large ranges slowing down from 3.11 --> now, I don't think there's been much in Python that's become dramatically worse such that you would need to refactor it later (I gather the Javascript community does this ritual every quarter or so). Anything being deprecated has years of warning.
- tonyarkles 2y ago
  > In the real world, your application is running in a container among hundreds or thousands of other containers I mean, that’s an engineering decision too. In my day job we’re capturing, pre-processing, running inference on, and post-processing about 500Mpx/s worth of live image data at about 80ms/frame end-to-end at the edge. The processor SoM costs about $3000/unit and uses about 50W running flat out. The retail cost of our overall product is two orders of magnitude more than what the processor is worth but it incurs zero recurring costs for us. Edit: and it’s got 64GB of Unified RAM that I’ve got all to myself :)
  - sgarland 2y ago
    I was wondering if someone from a different sub-industry would disagree here :D That sounds like a very interesting job, with quite different requirements and constraints from what I'm used to. One day, I'll get a job where application latency is critical, and optimizations matter deeply. Undoubtedly I'll find something else to be upset about, but at least it'll be a new complaint.
    - tonyarkles 2y ago
      > Undoubtedly I’ll find something else to be upset about Vendor SDKs. You’ll be upset about that I guarantee :)
- ddtaylor 2y ago
  I can somewhat echo some of the statements here and provide my own experience that is similar. I spend a decent amount of time writing decent C++ code. My friends in other parts of the industry are writing certainly better C++ code than me because they are doing it in environments that are more constricted in various ways. In either case, I do spend my time catching up a bit and would consider myself a competent C++21 programmer in some ways. My experience and my conversations lead me to understand there is so much left on the table with even the most basic implementations. When I implement it correctly in C++ we get close to some of the theoretical limits for the hardware for some workloads, compared to something that is literally 1% as fast running in NodeJS. Wit that said, for so many situations I cannot justify the time and complexity to use C++ for many projects. At least for the stage most projects are in. In theory this optimization can happen later, but it never really does because the horizontal (or sometimes even vertical) scaling kicks in and we're all just willing to throw a few more dollars at the problem instead of re-engineering it. Sure, some of the really big companies like Netflix find a decent reason from time to time to invest the engineering time squeeze out those numbers, but it's becoming the exception and not the rule.
  - Thorrez 2y ago
    >C++21 There's C++20 and C++23. No C++21.
    - ddtaylor 2y ago
      Sorry, I meant to type C++11
  - spookie 2y ago
    I've also found that not having some optimization mindset from the get go limits your product to achieve only a local maxima of performance in the end. It might not even be a good local maxima. It's best to have at least some optimization considerations from the start. I'm saying some because too much is a problem.
- dakiol 2y ago
  It’s hard to learn the ins and outs of dozens of programming languages. One doesn’t usually just use one or two PL over their entire career. I have worked professionally with at least PHP, Java, Python, JS, Go, and Ruby. That without taking into account the respective libraries and frameworks (and without taking into account as well the myriad of other components like dbs, web servers, caches, etc.) It sounds like an excuse, I know. The true is I just don’t have that much time.
- christophilus 2y ago
  > people are so deeply incurious. This seems to only happen in tech It happens everywhere. If anything, techies are more curious than the average Joe. How many fellow programmers can you nerd-snipe with a comment that makes them say, “Well, that’s strange…”
- duckmysick 2y ago
  > It also frustrates me to no end that people are so deeply incurious. This seems to only happen in tech This doesn't match my observations. In many fields, training is limited and hides the details. We train workers to repeat specific tasks and they excel at them. But they don't have conceptual understanding. Any explanation of why things are done the way they are is surface-level. You can see it when a procedure fails for some reason. They way to deal with is to 1) do basic checks 2) try again and if it still fails 3) delegate to someone else. Nobody is going to troubleshoot or optimize tasks unless that's their main job. It happens in construction, in the kitchens, on the assembly lines, and in the offices. It happens because it gets the job done.
  - sgarland 2y ago
    You're right. My wording was flat-out wrong, and I apologize. A more accurate sentiment (IMO) would be "that this happens in tech is baffling, considering..."
  - marcosdumay 2y ago
    Yes, but in most fields, conceptual understanding doesn't matter for most tasks. That's the problem with software. A brick-layer doesn't need to understand structural stability, but in software every task is structure validation, and people expect brick-layers to be productive.
    - phatskat 2y ago
      > A brick-layer doesn't need to understand structural stability Maybe a “junior” brick layer doesn’t need to, as much as a junior engineer doesn’t need to understand the ins and outs of their language. But a senior brick layer, or an architect, needs to understand more of the details so that they can set out the plan for the junior.
      - marcosdumay 2y ago
        Eh... What language do brick layers work with? You mean their spoken language? Also: "a senior brick layer, or an architect" those are complete different professions.
- bulatb 2y ago
  Some apple-pickers think the point of picking apples is to prove how good you are at climbing trees. They'll never not be mad at pluckers who just pluck the nearest apple, and the people who reward them for the apples, and the world that doesn't understand the pluckers picked their apples wrong, and didn't even earn them, and they're not even real apples.
  - pif 2y ago
    Sir, your comment is poetry. I commend you.
  - sgarland 2y ago
    It's one thing if you know the optimal (or at least approaching it) solution, and deliberately use something else for other reasons. At least thought went into it. I'm not upset at this simply for purity's sake; it directly impacts me in my day-to-day job. A hundred devs individually decide that good enough is good enough (deliberately or otherwise), and suddenly my infrastructure is dealing with traffic and query patterns it shouldn't have to, and then I have to explain what network bandwidth limits are, and why it's absurd that we hit them in the first place. In general, what I'm railing against is the continued push towards simplification and abstraction, while not requiring the understanding of what lies beneath. The hyperscalers certainly aren't trying to foster that attitude in-house – someone has to build the abstractions in the first place, after all, and keep datacenters humming. Yet they're happily shoveling it out, because it's a win-win for them: fewer people have the skills necessary to compete (or even to simply not use their services), and more people are able to use their services.
    - jebarker 2y ago
      > it directly impacts me in my day-to-day job It directly impacts anyone that uses software everyday. Most people don't seem to understand and/or care how poorly the software they use runs.
      - yetihehe 2y ago
        Most people don't care about their job, they just want to put in minimum viable effort, get paid and go home to their family. Typically they do this because putting more effort didn't bring them anything (not even some recognition of efforts) or even backfired. It's applicable to all jobs, not only in tech.
- killingtime74 2y ago
  Life is very different for many people and I think we just need to build empathy for people who treat a job as just a job. If they deliver and are not unkind about it there's nothing wrong about not going above and beyond the bare minimum, which is what they are paid for.
  - sgarland 2y ago
    As I commented above, a large part of my umbrage stems from the impact these decisions have on my job. I dislike having to work harder because others didn't want to go above the bare minimum. This isn't unique to any one company, nor my personal experience. At my last job, my team was initially human triaging practically all alerts, because we had the rare ability to read logs. I spoke to someone at Slack once who was stuck doing the same. That's an absurdly low expected level of competence.
    - badpun 2y ago
      You don't _have_ to work harder, as evidenced by those people who do the bare minimum. You just care about your work more than people who pay you for it (or the manager hired to manage you), which is the cause of your frustration here IMO.
      - sgarland 2y ago
        I care about my teammates. Letting them down by not working to my full potential makes me feel bad. I think this is embedded into my pysche from years of military experience and then shift work. I enjoy work, but I care about people.
- jrochkind1 2y ago
  > It also frustrates me to no end that people are so deeply incurious. This seems to only happen in tech, I mean software engineering and computer science from the start is built on abstraction, it makes sense that it attracts people who find taking the abstraction as reality and ignoring the underlying "real" layers to be attractive. (Also it's not "real" until you get down to... voltages? Electrons? I don't know... the whole thing about our whole endeavor being built of abstraction is there are so many layers. Nobody thinks everyone has to routinely go ALL THE WAY down, but I get your point that being more curious than many tend to be about a couple layers down is helpful, with how far down a couple layers is depending on the nature of your work. I'm not saying it's not helpful, I'm suggesting possibly why -- because we select for people who swim happily in abstractions, who love em even. And the success of all of CS and software engineering is based on the ability for it to often work. nobody really has to be curious about voltages when writing a python script to read and write from storage)
  - brewtide 2y ago
    It is interesting that at the lowest levels everything in fact is analog, only turning 'digital' on subjective decisions defining as such.
- crabbone 2y ago
  Nope. If a program is meant to run on the entire CPU, and if it's meant to use all the memory, then that's how it will run. This isn't a question of modernity or availability of tools or programmers' salary. Or, to put it differently, using more or less memory / CPU isn't really an indication of anything about a program. Sometimes you mean to use all that's available, other times you mean to use as little as you can. There's no way to tell which one should be done w/o knowing your goals. For example, the entire free memory on Linux is a free real estate for filesystem caching. So that memory is not wasted, if the system can help it. So, having your entire memory used (to support filesystem cache) wouldn't be really a performance problem (quite the contrary, it would look like the system is doing a good job at caching whatever reads your applications are making).
- tharkun__ 2y ago
  In the real world, your application is running in a container among hundreds or thousands of other containers. The system's resources are also probably being managed by a hypervisor. The CPU is shared among N tenants _and_ is overcommitted. When I read this, I thought the rest of your post would go entirely differently. As in, I immediately agreed with you, only for you to "turn this 180" (according to how I think about this at least :) ) It's not that much to ask to optimize where you can, when it isn't unreasonably difficult. You take the above to mean we should optimize such that L2 cache is used as per the post as much as possible. Optimize the wazoo out of things. But how does that even help in any way, when the CPU this runs on is like you said shared among N tenants and your carefully optimized L2 cache access is still going to be miss because another tenant got "your" CPU in between? If you're paying for bare metal i.e. have the whole instance for yourself, by all means, if your domain actually requires you to use the system in such a way (things like high frequency trading come to mind), then optimize like that! If you're running on seventeen levels of hypervisors that destroy any careful optimization in a random fashion anyway, then what's the point even? (non rhetorical question!)
  - kbelder 2y ago
    It's a version of the tragedy of the commons. Yes, your software bloat isn't the main factor keeping you slow, and cleaning it up might be barely perceptible. But... all the other software makes the same decision. Then, suddenly, everything on the computer is running at 10% efficiency. In a complex, multitasking environment, keeping your code clean benefits other tasks as much or more than your own. It should be considered a responsibility.
    - tharkun__ 2y ago
      I would agree with the sentiment but not necessarily the conclusion. Like, don't use (the equivalent of) Stooge Sort (for your particular situation). But unless you are in a very particular situation, it should be OK for everyone to "just use your language's built-in sort function" (hopefully that's a thing and you don't use something where you need to roll your own even in 2024) assuming that it uses a sane default like quicksort or merge sort that will work perfectly fine for most regular common situations. Another example might be to not stupidly build slow algorithms saying "this won't see much data anyhow" (yes, I've seen that in PRs unfortunately) when a very simple hash lookup will ensure it's fast all the time. But it should be totally fine to assume that your language's hash table implementation is fine for common situations and you don't need to optimize anything unless you're a very special case.
- TacticalCoder 2y ago
  > In the real world, your application is running in a container among hundreds or thousands of other containers. The system's resources are also probably being managed by a hypervisor. The CPU is shared among N tenants _and_ is overcommitted. It's not that much to ask to optimize where you can, when it isn't unreasonably difficult. And the real irony is that those programmers not giving a crap about performances and justifying it are resting on the shoulders of giants who went out of their way to have these kernels, hypervisors, containers, passthroughs, memory usage, etc. be as performant as they can. Same for security. Then you get the "I'm not in the business of writing fast code" and "I'm not a security company, I'm selling X and Y". But they all, on a daily basis, use actual proper software written by people who know about these things.
purplesyringa 2y ago
That's good food for thought. My hardware is quite old, but I agree that the numbers seem somewhat suspicious. I believe that RAM can only be RMW'd in cache lines, so modifying just 8 bytes still requires 64 bytes to be transmitted. I'm assuming the 50 GB/s throughput is half-duplex, and 25 GB/s over 64 bytes is ~400 Melems/s, somewhat closer to my result. I tried using non-temporal stores in the straightforward algorithm, but to my surprise, this led to a significant decrease of performance across all input lengths. > When your data is indeed big and the performance actually matters, consider doing something completely different. I'm not sure what you mean by this. Scaling across machines is just ignoring the problem. What do you mean by "something completely different"?
- saagarjha 2y ago
  Non-temporal stores will not help you if all you do are those accesses. The point of using them is that you don't want them pulled into the caches and pushing out everything else.
- bobmcnamara 2y ago
  > I believe that RAM can only be RMW'd in cache lines, so modifying just 8 bytes still requires 64 bytes to be transmitted. Ages ago I worked with several different memory controllers, and it depends on the memory controller, cache, and MMU configuration. Plenty of systems do require cacheline updates, then modifying 8B requires reading one or two cachelines, updating them, and writing them out eventually. Some caches track cacheline validity with a bit per byte. This enables a CPU to set a book out in memory without fetching the cacheline. The cache may then try to burst read that line from the memory controller, but if it doesn't get around to it before deciding to flush that cacheline, it may issue a single byte write to the controller. The controller can then issue a makes DRAM write to the memory, which will update only certain bytes in the DRAM column. However, this still takes about as long as sending the full cacheline but it offloads the read-modify. Validity per byte is also useful to implement hit under miss. I bet on newer, bigger systems tricks like this are less useful since the memory busses are faster and wider today.
- Const-me 2y ago
  > What do you mean by "something completely different"? I mean compute the exact thing you need to compute, optimizing for performance. I’m pretty sure your current code is spending vast majority of time doing malloc / memcpy while moving input elements from the input array into per-bucket collections. To aggregate the data the way you want, you do not have to do any of that. You only need a single number per bucket to compute the minimum. Updating that accumulator is very cheap as there’s no memory allocations involved: just load, compare, and store the updated number. Moreover, that algorithm is easily to parallelize. Compute an array with these per-bucket minimums on different threads, wait for completion of all tasks, then aggregate multiple arrays computed by each thread into a single array with global minimums for each bucket. Then compute the final sum.
wkat4242 2y ago
And my GPU delivers 1TB/s. Massive difference <3 I wish we could get those kind of speeds on system RAM.
- winter_blue 2y ago
  To/from what sort of devices could the GPU read or write at 1TB/sec, besides main memory? The fastest consumer SSDs top out at several GB/sec (I guess with massive hardware RAID they could be faster, but not sure if they’d be 1TB/sec fast). Even a network adapter that does 10 Gbit/sec is only recently becoming slightly more common for the average consumer. Not sure if any consumer adapters in the 10 or 1Tbit/sec range exist at all.
  - chasd00 2y ago
    > Not sure if any consumer adapters in the 10 or 1Tbit/sec range exist at all. Further, what exactly is a consumer going to plug a 1Tbit/sec adapter into? Your little home ATT fiber connection isn’t going to come close to using that available bandwidth.
    - simoncion 2y ago
      > Further, what exactly is a consumer going to plug a 1Tbit/sec adapter into? Another similarly-equipped machine on the LAN, and the switch(es) between them.
  - semi-extrinsic 2y ago
    Consumer? Even in the "if-you-need-to-ask-what-it-costs-you-can't-afford-it" world of frontier HPC systems, were only getting teasers of 0.8 Tbit/sec NICs this year. As you say, only the GPU, maybe RAM, and the NIC will be able to churn data at these speeds. There is a reason why Mellanox (Nvidia) has developed GPUDirect RDMA, so the GPU and NIC can talk directly to each other. https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/ https://www.servethehome.com/this-is-the-next-gen-nvidia-con...
  - simoncion 2y ago
tc4v 2y ago
cache misses are slow because of latency, not because of throughput.
- geysersam 2y ago
  Isn't the point of the person you replied to that the article author wasn't able to eliminate latency because if they were they'd be constrained by throughput but they are not?
- eddd-ddde 2y ago
  cache misses can absolutely screw throughput because you may need to load more cache lines
toast0 2y ago
> In my 3 years old laptop, system memory (dual channel DDR4-3200) delivers about 50 GB / second. That's almost certainly for (mostly) sequential access. When you just want a couple bytes here and there, and access isn't pipelined and prefetch doesn't accelerate your use case, the real world bandwidth is going to be significantly less.
- Const-me 2y ago
  The input data is sequential. I don’t understand Rust but if the code is doing what’s written there, “simple multiplicative hash and perform a simple analysis on the buckets – say, compute the sum of minimums among buckets” it’s possible to improve substantially. They don’t need to move these megabytes of elements between collections. I would split the input across a few CPU cores, compute per-bucket minimums in parallel on each core, when all completed aggregate minimums across all cores, then compute sum of the results. Multiplicative hash and the final sum should be possible to vectorize on most computers. Updating per-bucket minimum is probably impossible to vectorize (technically AVX512 set has the required instructions but I’m not sure these are efficient) but there’re much fewer buckets than input numbers, which means arrays with per-bucket minimums are likely to fit in caches.

sgarland 2y ago

In the real world, your application is running in a container among hundreds or thousands of other containers. The system's resources are also probably being managed by a hypervisor. The CPU is shared among N tenants _and_ is overcommitted. It's not that much to ask to optimize where you can, when it isn't unreasonably difficult. More importantly, this attitude is precisely why software sucks today. "[CPU, Memory, Disk] is cheap, engineering time isn't." Fuck that. Bad engineering time is _incredibly_ expensive. This is an excuse to not spend time learning the ins and outs of your language, and your hardware. It also frustrates me to no end that people are so deeply incurious. This seems to only happen in tech, which is baffling considering how incredibly broad and deep the industry is. Instead, everyone clamors to learn the new Javascript abstraction that lets them get even further away from the reality of what the magic boxes are doing.

kmarc 2y ago
Agree with the sentiment. However it's hard to stay curious, even harder to stay up-to-date. I liked fiddling with storage for a while, got really into it, deepened my knowledge about it. A couple years later I realized everything else (networking, architectures, languages) developed so much, mot of my (non-basic) knowledge was obsolete. Picking up where I left off with all technologies is incredibly hard and caused fatigue. Now I'm at a point where I have the feeling I don't know nothing about anything. It's factually not true, but my gut feeling tells this. Would I be younger, this would trigger a lot of anxiety. Thankfully I can janfle this by now.
- sgarland 2y ago
  That's understandable. I'm into databases (both professionally and personally), so I get to touch a wide variety of things. Following a query the whole way is pretty neat. Typed query --> client --> wire protocol --> network --> server --> DB frontend --> DB planner --> DB storage engine --> OS --> Memory / Disk. `blktrace` is super interesting to watch commands being sent to the disk.
  - buran77 2y ago
    When you are deeply involved in tech both personally and professionally you are probably very passionate about this and makes sense that you'd only look closely at this field and think "people are so deeply incurious. This seems to only happen in tech". Tech is also one of the (if not the) most dynamic and fast evolving field a normal person will ever touch. Curiosity in tech can drain every single bit of free time and energy you have and you will hardly keep up with the progress, maybe barely scratch the surface. But people's available free time and energy wanes and curiosity is a collateral victim. I've painfully gone through the entire cycle of this, including the bit of resurgence later on when you have a combination of free time but less energy. What I can say is that this absolutely does not happen just in tech. If anything tech is flooded with people with more curiosity than almost any other field.
    - sgarland 2y ago
      > When you are deeply involved in tech both personally and professionally you are probably very passionate about this and makes sense that you'd only look closely at this field and think "people are so deeply incurious. This seems to only happen in tech". Good point. I commented in a sibling post to the same effect. I've certainly felt the personal strain of time sink and procrastination in my homelab. It's currently running k3os, which has been dead for about four years now, because everything I want is still running, and I never seem have the motivation on the weekend to yell at my laptop when I could be playing board games. > including the bit of resurgence later on when you have a combination of free time but less energy. I'm guessing that will happen in another decade or so, when my kids are grown.
akira2501 2y ago
> "[CPU, Memory, Disk] is cheap, engineering time isn't." Fuck that. It is. It's absurdly cheap. I ensure I check the amount of time it would take for me to make a performance improvement against the runtime costs of my functions. It's rarely worth the extra effort. Seriously, until you get into the millions of records per second level, you're almost never benefited. You may make your function 2x faster, at a cost of additional complexity, but you never run it enough in a year for it to pay itself back. > Bad engineering time is _incredibly_ expensive. Engineering time is expensive. Period. It speaks to the need to minimize it. > This is an excuse to not spend time learning the ins and outs of your language, and your hardware. All of which will change in a few years, which is fine, if you're also committing to keeping _all that code_ up to date right along with it. Otherwise you end up with an obscure mess that you have to unwind 5 years of context to understand and fix again. Complexity and available mental contexts are forgotten costs. If your language even has that many "ins and outs" to begin with you may want to reconsider that.
- sgarland 2y ago
  > You may make your function 2x faster, at a cost of additional complexity, but you never run it enough in a year for it to pay itself back. I'm not talking about increased complexity, I'm talking about extremely basic things that take zero extra time, like using the correct data structure. For example, in Python: In [8]: a = array("i", (x for x in range(1_000_000))) ...: l = [x for x in range(1_000_000)] ...: d = deque(l) ...: for x in (a, l, d): ...: print(f"{sys.getsizeof(x) / 2**20} MiB") ...: 3.902385711669922 MiB 8.057334899902344 MiB 7.868537902832031 MiB Very similar structures, with very different memory requirements and access speeds. I can count on one hand with no fingers the number of times I've seen an array used. Or knowing that `random.randint` is remarkably slow compared to `random.random()`, which can matter in a hot loop: In [10]: %timeit math.floor(random.random() * 1_000_000) 31.9 ns ± 0.138 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) In [11]: %timeit random.randint(0, 1_000_000) 163 ns ± 0.653 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each) > All of which will change in a few years, which is fine, if you're also committing to keeping _all that code_ up to date right along with it. With the exception of list comprehension over large ranges slowing down from 3.11 --> now, I don't think there's been much in Python that's become dramatically worse such that you would need to refactor it later (I gather the Javascript community does this ritual every quarter or so). Anything being deprecated has years of warning.
tonyarkles 2y ago
> In the real world, your application is running in a container among hundreds or thousands of other containers I mean, that’s an engineering decision too. In my day job we’re capturing, pre-processing, running inference on, and post-processing about 500Mpx/s worth of live image data at about 80ms/frame end-to-end at the edge. The processor SoM costs about $3000/unit and uses about 50W running flat out. The retail cost of our overall product is two orders of magnitude more than what the processor is worth but it incurs zero recurring costs for us. Edit: and it’s got 64GB of Unified RAM that I’ve got all to myself :)
- sgarland 2y ago
  I was wondering if someone from a different sub-industry would disagree here :D That sounds like a very interesting job, with quite different requirements and constraints from what I'm used to. One day, I'll get a job where application latency is critical, and optimizations matter deeply. Undoubtedly I'll find something else to be upset about, but at least it'll be a new complaint.
  - tonyarkles 2y ago
    > Undoubtedly I’ll find something else to be upset about Vendor SDKs. You’ll be upset about that I guarantee :)
ddtaylor 2y ago
I can somewhat echo some of the statements here and provide my own experience that is similar. I spend a decent amount of time writing decent C++ code. My friends in other parts of the industry are writing certainly better C++ code than me because they are doing it in environments that are more constricted in various ways. In either case, I do spend my time catching up a bit and would consider myself a competent C++21 programmer in some ways. My experience and my conversations lead me to understand there is so much left on the table with even the most basic implementations. When I implement it correctly in C++ we get close to some of the theoretical limits for the hardware for some workloads, compared to something that is literally 1% as fast running in NodeJS. Wit that said, for so many situations I cannot justify the time and complexity to use C++ for many projects. At least for the stage most projects are in. In theory this optimization can happen later, but it never really does because the horizontal (or sometimes even vertical) scaling kicks in and we're all just willing to throw a few more dollars at the problem instead of re-engineering it. Sure, some of the really big companies like Netflix find a decent reason from time to time to invest the engineering time squeeze out those numbers, but it's becoming the exception and not the rule.
- Thorrez 2y ago
  >C++21 There's C++20 and C++23. No C++21.
  - ddtaylor 2y ago
    Sorry, I meant to type C++11
- spookie 2y ago
  I've also found that not having some optimization mindset from the get go limits your product to achieve only a local maxima of performance in the end. It might not even be a good local maxima. It's best to have at least some optimization considerations from the start. I'm saying some because too much is a problem.
dakiol 2y ago
It’s hard to learn the ins and outs of dozens of programming languages. One doesn’t usually just use one or two PL over their entire career. I have worked professionally with at least PHP, Java, Python, JS, Go, and Ruby. That without taking into account the respective libraries and frameworks (and without taking into account as well the myriad of other components like dbs, web servers, caches, etc.) It sounds like an excuse, I know. The true is I just don’t have that much time.
christophilus 2y ago
> people are so deeply incurious. This seems to only happen in tech It happens everywhere. If anything, techies are more curious than the average Joe. How many fellow programmers can you nerd-snipe with a comment that makes them say, “Well, that’s strange…”
duckmysick 2y ago
> It also frustrates me to no end that people are so deeply incurious. This seems to only happen in tech This doesn't match my observations. In many fields, training is limited and hides the details. We train workers to repeat specific tasks and they excel at them. But they don't have conceptual understanding. Any explanation of why things are done the way they are is surface-level. You can see it when a procedure fails for some reason. They way to deal with is to 1) do basic checks 2) try again and if it still fails 3) delegate to someone else. Nobody is going to troubleshoot or optimize tasks unless that's their main job. It happens in construction, in the kitchens, on the assembly lines, and in the offices. It happens because it gets the job done.
- sgarland 2y ago
  You're right. My wording was flat-out wrong, and I apologize. A more accurate sentiment (IMO) would be "that this happens in tech is baffling, considering..."
- marcosdumay 2y ago
  Yes, but in most fields, conceptual understanding doesn't matter for most tasks. That's the problem with software. A brick-layer doesn't need to understand structural stability, but in software every task is structure validation, and people expect brick-layers to be productive.
  - phatskat 2y ago
    > A brick-layer doesn't need to understand structural stability Maybe a “junior” brick layer doesn’t need to, as much as a junior engineer doesn’t need to understand the ins and outs of their language. But a senior brick layer, or an architect, needs to understand more of the details so that they can set out the plan for the junior.
    - marcosdumay 2y ago
      Eh... What language do brick layers work with? You mean their spoken language? Also: "a senior brick layer, or an architect" those are complete different professions.
bulatb 2y ago
Some apple-pickers think the point of picking apples is to prove how good you are at climbing trees. They'll never not be mad at pluckers who just pluck the nearest apple, and the people who reward them for the apples, and the world that doesn't understand the pluckers picked their apples wrong, and didn't even earn them, and they're not even real apples.
- pif 2y ago
  Sir, your comment is poetry. I commend you.
- sgarland 2y ago
  It's one thing if you know the optimal (or at least approaching it) solution, and deliberately use something else for other reasons. At least thought went into it. I'm not upset at this simply for purity's sake; it directly impacts me in my day-to-day job. A hundred devs individually decide that good enough is good enough (deliberately or otherwise), and suddenly my infrastructure is dealing with traffic and query patterns it shouldn't have to, and then I have to explain what network bandwidth limits are, and why it's absurd that we hit them in the first place. In general, what I'm railing against is the continued push towards simplification and abstraction, while not requiring the understanding of what lies beneath. The hyperscalers certainly aren't trying to foster that attitude in-house – someone has to build the abstractions in the first place, after all, and keep datacenters humming. Yet they're happily shoveling it out, because it's a win-win for them: fewer people have the skills necessary to compete (or even to simply not use their services), and more people are able to use their services.
  - jebarker 2y ago
    > it directly impacts me in my day-to-day job It directly impacts anyone that uses software everyday. Most people don't seem to understand and/or care how poorly the software they use runs.
    - yetihehe 2y ago
      Most people don't care about their job, they just want to put in minimum viable effort, get paid and go home to their family. Typically they do this because putting more effort didn't bring them anything (not even some recognition of efforts) or even backfired. It's applicable to all jobs, not only in tech.
killingtime74 2y ago
Life is very different for many people and I think we just need to build empathy for people who treat a job as just a job. If they deliver and are not unkind about it there's nothing wrong about not going above and beyond the bare minimum, which is what they are paid for.
- sgarland 2y ago
  As I commented above, a large part of my umbrage stems from the impact these decisions have on my job. I dislike having to work harder because others didn't want to go above the bare minimum. This isn't unique to any one company, nor my personal experience. At my last job, my team was initially human triaging practically all alerts, because we had the rare ability to read logs. I spoke to someone at Slack once who was stuck doing the same. That's an absurdly low expected level of competence.
  - badpun 2y ago
    You don't _have_ to work harder, as evidenced by those people who do the bare minimum. You just care about your work more than people who pay you for it (or the manager hired to manage you), which is the cause of your frustration here IMO.
    - sgarland 2y ago
      I care about my teammates. Letting them down by not working to my full potential makes me feel bad. I think this is embedded into my pysche from years of military experience and then shift work. I enjoy work, but I care about people.
jrochkind1 2y ago
> It also frustrates me to no end that people are so deeply incurious. This seems to only happen in tech, I mean software engineering and computer science from the start is built on abstraction, it makes sense that it attracts people who find taking the abstraction as reality and ignoring the underlying "real" layers to be attractive. (Also it's not "real" until you get down to... voltages? Electrons? I don't know... the whole thing about our whole endeavor being built of abstraction is there are so many layers. Nobody thinks everyone has to routinely go ALL THE WAY down, but I get your point that being more curious than many tend to be about a couple layers down is helpful, with how far down a couple layers is depending on the nature of your work. I'm not saying it's not helpful, I'm suggesting possibly why -- because we select for people who swim happily in abstractions, who love em even. And the success of all of CS and software engineering is based on the ability for it to often work. nobody really has to be curious about voltages when writing a python script to read and write from storage)
- brewtide 2y ago
  It is interesting that at the lowest levels everything in fact is analog, only turning 'digital' on subjective decisions defining as such.
crabbone 2y ago
Nope. If a program is meant to run on the entire CPU, and if it's meant to use all the memory, then that's how it will run. This isn't a question of modernity or availability of tools or programmers' salary. Or, to put it differently, using more or less memory / CPU isn't really an indication of anything about a program. Sometimes you mean to use all that's available, other times you mean to use as little as you can. There's no way to tell which one should be done w/o knowing your goals. For example, the entire free memory on Linux is a free real estate for filesystem caching. So that memory is not wasted, if the system can help it. So, having your entire memory used (to support filesystem cache) wouldn't be really a performance problem (quite the contrary, it would look like the system is doing a good job at caching whatever reads your applications are making).
tharkun__ 2y ago
In the real world, your application is running in a container among hundreds or thousands of other containers. The system's resources are also probably being managed by a hypervisor. The CPU is shared among N tenants _and_ is overcommitted. When I read this, I thought the rest of your post would go entirely differently. As in, I immediately agreed with you, only for you to "turn this 180" (according to how I think about this at least :) ) It's not that much to ask to optimize where you can, when it isn't unreasonably difficult. You take the above to mean we should optimize such that L2 cache is used as per the post as much as possible. Optimize the wazoo out of things. But how does that even help in any way, when the CPU this runs on is like you said shared among N tenants and your carefully optimized L2 cache access is still going to be miss because another tenant got "your" CPU in between? If you're paying for bare metal i.e. have the whole instance for yourself, by all means, if your domain actually requires you to use the system in such a way (things like high frequency trading come to mind), then optimize like that! If you're running on seventeen levels of hypervisors that destroy any careful optimization in a random fashion anyway, then what's the point even? (non rhetorical question!)
- kbelder 2y ago
  It's a version of the tragedy of the commons. Yes, your software bloat isn't the main factor keeping you slow, and cleaning it up might be barely perceptible. But... all the other software makes the same decision. Then, suddenly, everything on the computer is running at 10% efficiency. In a complex, multitasking environment, keeping your code clean benefits other tasks as much or more than your own. It should be considered a responsibility.
  - tharkun__ 2y ago
    I would agree with the sentiment but not necessarily the conclusion. Like, don't use (the equivalent of) Stooge Sort (for your particular situation). But unless you are in a very particular situation, it should be OK for everyone to "just use your language's built-in sort function" (hopefully that's a thing and you don't use something where you need to roll your own even in 2024) assuming that it uses a sane default like quicksort or merge sort that will work perfectly fine for most regular common situations. Another example might be to not stupidly build slow algorithms saying "this won't see much data anyhow" (yes, I've seen that in PRs unfortunately) when a very simple hash lookup will ensure it's fast all the time. But it should be totally fine to assume that your language's hash table implementation is fine for common situations and you don't need to optimize anything unless you're a very special case.
TacticalCoder 2y ago
> In the real world, your application is running in a container among hundreds or thousands of other containers. The system's resources are also probably being managed by a hypervisor. The CPU is shared among N tenants _and_ is overcommitted. It's not that much to ask to optimize where you can, when it isn't unreasonably difficult. And the real irony is that those programmers not giving a crap about performances and justifying it are resting on the shoulders of giants who went out of their way to have these kernels, hypervisors, containers, passthroughs, memory usage, etc. be as performant as they can. Same for security. Then you get the "I'm not in the business of writing fast code" and "I'm not a security company, I'm selling X and Y". But they all, on a daily basis, use actual proper software written by people who know about these things.

purplesyringa 2y ago

That's good food for thought. My hardware is quite old, but I agree that the numbers seem somewhat suspicious. I believe that RAM can only be RMW'd in cache lines, so modifying just 8 bytes still requires 64 bytes to be transmitted. I'm assuming the 50 GB/s throughput is half-duplex, and 25 GB/s over 64 bytes is ~400 Melems/s, somewhat closer to my result. I tried using non-temporal stores in the straightforward algorithm, but to my surprise, this led to a significant decrease of performance across all input lengths. > When your data is indeed big and the performance actually matters, consider doing something completely different. I'm not sure what you mean by this. Scaling across machines is just ignoring the problem. What do you mean by "something completely different"?

saagarjha 2y ago
Non-temporal stores will not help you if all you do are those accesses. The point of using them is that you don't want them pulled into the caches and pushing out everything else.
bobmcnamara 2y ago
> I believe that RAM can only be RMW'd in cache lines, so modifying just 8 bytes still requires 64 bytes to be transmitted. Ages ago I worked with several different memory controllers, and it depends on the memory controller, cache, and MMU configuration. Plenty of systems do require cacheline updates, then modifying 8B requires reading one or two cachelines, updating them, and writing them out eventually. Some caches track cacheline validity with a bit per byte. This enables a CPU to set a book out in memory without fetching the cacheline. The cache may then try to burst read that line from the memory controller, but if it doesn't get around to it before deciding to flush that cacheline, it may issue a single byte write to the controller. The controller can then issue a makes DRAM write to the memory, which will update only certain bytes in the DRAM column. However, this still takes about as long as sending the full cacheline but it offloads the read-modify. Validity per byte is also useful to implement hit under miss. I bet on newer, bigger systems tricks like this are less useful since the memory busses are faster and wider today.
Const-me 2y ago
> What do you mean by "something completely different"? I mean compute the exact thing you need to compute, optimizing for performance. I’m pretty sure your current code is spending vast majority of time doing malloc / memcpy while moving input elements from the input array into per-bucket collections. To aggregate the data the way you want, you do not have to do any of that. You only need a single number per bucket to compute the minimum. Updating that accumulator is very cheap as there’s no memory allocations involved: just load, compare, and store the updated number. Moreover, that algorithm is easily to parallelize. Compute an array with these per-bucket minimums on different threads, wait for completion of all tasks, then aggregate multiple arrays computed by each thread into a single array with global minimums for each bucket. Then compute the final sum.

wkat4242 2y ago

And my GPU delivers 1TB/s. Massive difference <3 I wish we could get those kind of speeds on system RAM.

winter_blue 2y ago
To/from what sort of devices could the GPU read or write at 1TB/sec, besides main memory? The fastest consumer SSDs top out at several GB/sec (I guess with massive hardware RAID they could be faster, but not sure if they’d be 1TB/sec fast). Even a network adapter that does 10 Gbit/sec is only recently becoming slightly more common for the average consumer. Not sure if any consumer adapters in the 10 or 1Tbit/sec range exist at all.
- chasd00 2y ago
  > Not sure if any consumer adapters in the 10 or 1Tbit/sec range exist at all. Further, what exactly is a consumer going to plug a 1Tbit/sec adapter into? Your little home ATT fiber connection isn’t going to come close to using that available bandwidth.
  - simoncion 2y ago
    > Further, what exactly is a consumer going to plug a 1Tbit/sec adapter into? Another similarly-equipped machine on the LAN, and the switch(es) between them.
- semi-extrinsic 2y ago
  Consumer? Even in the "if-you-need-to-ask-what-it-costs-you-can't-afford-it" world of frontier HPC systems, were only getting teasers of 0.8 Tbit/sec NICs this year. As you say, only the GPU, maybe RAM, and the NIC will be able to churn data at these speeds. There is a reason why Mellanox (Nvidia) has developed GPUDirect RDMA, so the GPU and NIC can talk directly to each other. https://www.servethehome.com/this-is-the-next-gen-nvidia-connectx-8-supernic-for-800gbps-networking/ https://www.servethehome.com/this-is-the-next-gen-nvidia-con...
- simoncion 2y ago

tc4v 2y ago

cache misses are slow because of latency, not because of throughput.

geysersam 2y ago
Isn't the point of the person you replied to that the article author wasn't able to eliminate latency because if they were they'd be constrained by throughput but they are not?
eddd-ddde 2y ago
cache misses can absolutely screw throughput because you may need to load more cache lines

toast0 2y ago

> In my 3 years old laptop, system memory (dual channel DDR4-3200) delivers about 50 GB / second. That's almost certainly for (mostly) sequential access. When you just want a couple bytes here and there, and access isn't pipelined and prefetch doesn't accelerate your use case, the real world bandwidth is going to be significantly less.

Const-me 2y ago
The input data is sequential. I don’t understand Rust but if the code is doing what’s written there, “simple multiplicative hash and perform a simple analysis on the buckets – say, compute the sum of minimums among buckets” it’s possible to improve substantially. They don’t need to move these megabytes of elements between collections. I would split the input across a few CPU cores, compute per-bucket minimums in parallel on each core, when all completed aggregate minimums across all cores, then compute sum of the results. Multiplicative hash and the final sum should be possible to vectorize on most computers. Updating per-bucket minimum is probably impossible to vectorize (technically AVX512 set has the required instructions but I’m not sure these are efficient) but there’re much fewer buckets than input numbers, which means arrays with per-bucket minimums are likely to fit in caches.