7 ms·

RAM is the new disk – and how to measure its performance (2015)

75 points by tanelpoder 6y ago

vlovich123 6y ago
I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software architecture theory we learn is based on existing HW paradigms without much thought being given to how we can change HW paradigms. By nature HW is massively parallel but where physical distance from compute = latency (vs the ultimately serial execution nature of traditional CPUs that can process all data at blistering speed but only one at a time with some SIMD exceptions). There are real-world benefits to this kind of design - memory is cheap and simple to manufacture and abundantly available. The downside though is that the CPU is sitting doing nothing but waiting for memory most of the time, especially when processing large data sets. Imagine how efficient a GC algorithm would be if it could compute a result in the background just doing a concurrent mark and sweep, perhaps as part of a DRAM refresh cycle so that you could even choose to stop refreshing that RAM because your application no longer needs that row. The power and performance savings are pretty enticing.
- tanelpoder 6y ago
  Oracle (Sun) latest CPUs support something called DAX (data analytics extension, not the same thing as the Intel DAX - direct access extensions). It's a coprocessor that allows offloading simpler operations closer to RAM, apparently: "DAX is an integrated co-processor which provides a specialized set of instructions that can run very selective functionality – Scan, Extract, Select, Filter, and Translate – at fast speeds. The multiple DAX share the same memory interface with the processors cores so the DAX can take full advantage of the 140-160 GB/sec memory bandwidth of the SPARC M7 processor." https://blogs.oracle.com/bestperf/accelerating-spark-sql-using-sparc-with-dax%2c-16x-faster-per-core-over-and-above-spark-210-with-tungsten https://blogs.oracle.com/bestperf/accelerating-spark-sql-usi...
- cogman10 6y ago
  Interestingly enough, this is why Simultaneous multithreading [1] exists! The revelation that "The CPU could be doing useful work while stalled waiting on a load" has lead to CPU designers "faking" the number of cores available to the OS to allow the CPU to do more useful work while different "threads" are paused waiting on memory to come back with the data they need. [1] https://en.wikipedia.org/wiki/Simultaneous_multithreading https://en.wikipedia.org/wiki/Simultaneous_multithreading
  - tanelpoder 6y ago
    And all kinds of manual prefetching instructions built into the CPUs (for compilers to use) and prefetching algorithms & predictors built into the CPUs too!
  - nomel 6y ago
    I think the point is that this still eats into CPU <-> memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the stall is from memory access anyways.
    - geocar 6y ago
      Linked-list chasing (as in vlovich123's example) isn't bandwidth-limited as much as it is latency: SMT helps here because you're able to enqueue multiple wait-states "simultaneously".
      - jeffbee 6y ago
        Anything you can do with SMT you can do without SMT using ILP instead. SMT does not grant a CPU extra cache-filling resources.
        geocar 6y ago
        SMT is easier to program. Walking a single linked-list is nothing; A garbage collector walks lots of them that tend to fan out for a bit.
- lalaithion 6y ago
  Even just "go to memory location X+Y, load the value there, set X = that value, repeat N times" would allow fast linked list traversal, as well as doing various high-level pointer chasing faster.
- 55873445216111 6y ago
  There are some memories supporting basic in-memory operations. For example: https://mosys.com/products/blazar-family/be3rmw-bandwidth-engine-3-rmw/ https://mosys.com/products/blazar-family/be3rmw-bandwidth-en.... This supports operations like read-modify-write within the memory device itself. (I have no affiliation with this company.) The barrier to adoption of this is not technical, it's economic. Memory industry has focused on making the highest capacity and lowest cost/bit products. This drives high manufacturing volume which drives economies of scale. Memory products with integrated functions are inherently niche, and therefore do not have anywhere near the market size and economy of scale. Designers have decided (historically) that it is cheaper at the system level to keep the logic operations within the CPU and use a "dumb" commodity memory, even though this necessitates more bandwidth usage. (It's a complex engineering trade-off.) With logic performance continuing to scale faster than memory bandwidth, at some point an architecture that reduces the required memory bandwidth (such as computing in-memory) might start to make sense economically.
  - vlovich123 6y ago
    I think there are niche use-cases that would warrant the cost/complexity trade-off. Namely cloud infrastructure for databases where you might be processing large amounts and transferring all that data first to the CPU is pretty ridiculous. I think the complexity problem is solvable. If you can build such a thing for memory, you can reuse the same general concept for storage like NVME/SSD/spinning disk. It’s entirely possible this is warranted even in consumer devices as quite a bit of OS operations deal with modifying/querying memory whereas doing offload can win you some serious wins (making the machine feel way more snappy/interactive + more powerful to execute things locally). I don’t have hope for traditional CPU designers so the question is whether someone can both design a memory system with this power AND a computational model that makes this easy to adopt in major languages while offering a perf win (given that a memory fetch costs ~100ns for ~256 bytes). It’s challenging and unlikely to come from x86 land where back Hw compatibility is extremely important. The innovation may eventually come from mobile land (which is where Apple is coming from) where memory controllers are custom designed and part of the SoC anyway, changing with each revision, so the hard part remains again how you make this fast, efficient, able to handle multiple concurrent programs (or at least have the OS control coarsest which program’s memory accesses were being prioritized), and language integration (so you can hand off an algorithm and it would be executed without upending existing software design knowledge). The OS integration could be even more intelligent - if the process/thread is spending its time processing memory without returning results to the CPU, just put it to sleep until the result is available (separating memory access and CPU utilization, resulting in drastically better utilization of cycles rather than naively waiting for each memory stall one at a time). So tldr I agree with you totally. This is not going to happen if memory manufacturers continue to go for the “cheap and large” memory route. I do see hope that such concepts may be explored if we get more innovation in the CPU space.
- wcerfgba 6y ago
  This reminds me of the Connection Machine architecture [1] > Each CM-1 microprocessor has its own 4 kilobits of random-access memory (RAM), and the hypercube-based array of them was designed to perform the same operation on multiple data points simultaneously, i.e., to execute tasks in single instruction, multiple data (SIMD) fashion. The CM-1, depending on the configuration, has as many as 65,536 individual processors, each extremely simple, processing one bit at a time. [1] https://en.wikipedia.org/wiki/Connection_Machine https://en.wikipedia.org/wiki/Connection_Machine
- d_tr 6y ago
  A few months ago I wanted to take a look at the Gen-Z fabric specifications, but unfortunately they still have a lame members-only download request form in place.
- dragontamer 6y ago
  > Right now we have to fetch everything into the CPU before we can make decisions. Think about virtual memory. The memory at location #6000 is NOT at where the program thinks is at memory location #6000. In fact, the program might be reading / writing to memory location 0x08004000 (or close to that, whatever the magic start address was on various OS like Linux / Windows). And then your CPU virtually-translates that address to memory-stick #0 column#4000 or whatever. Because of virtual memory: all memory operations must be translated by the CPU before you actually go to RAM (and that translation may require a page-table walk in the worst case)
  - ben509 6y ago
    Well, all memory operations must be translated by an MMU. So can the MMU be a distributed beast that lives in the CPU, the memory and the IO?
    - dragontamer 6y ago
      > So can the MMU be a distributed beast that lives in the CPU, the memory and the IO? That "blah = blah->next" memory operation could be: * Going to swap thanks to swap / overcommit behavior on Linux/Windows. * Or going to a file thanks to mmap * Going to Ethernet and to another computer thanks to RDMA So... no. I'm pretty sure the CPU is the proper place for that kind of routing.
      - freeone3000 6y ago
        why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded) moreover, if we have support for mmap at the MMU level, we can cut the CPU bottleneck for disk access entirely. the disk controller can already handle DMA, but there's simply no way for things that aren't the CPU to trigger it. DirectStorage is an effort for GPUs to trigger it, but what if we could also trigger it by other means?
- BenoitP 6y ago
  > HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU Well, that seems to be the exact definition of what UPMEM is doing: https://www.upmem.com/upmem-announces-silicon-based-processing-in-memory-benchmarks/ https://www.upmem.com/upmem-announces-silicon-based-processi... Between the M1, GPUs, TPUs, RISC-V interesting times are coming in hardware. I blame the physical limits, which are putting the Duke Nukem development method to an end (you promise the client 2x the same performance in 18 months, then play Duke Nukem the whole time). The only way to better performance now is through hardware specialization. That and avoiding Electron.
  - WJW 6y ago
    You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too. There's still loads of performance to be gained just by being better programmers.
    - Arelius 6y ago
      > Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. I feel like this story needs an ending? Did somebody re-implement it using a heap and found significant performance wins? Or was the sorted array used on purpose to take advantage of some specific cache constraints and actually end up being a huge win?
      - WJW 6y ago
        The story ends (so far) with me filing a GH issue and offering a PR, my own benchmarking getting an average 3x speedup over a wide range of inputs. I happen to know that the maintainer has been on vacation for the last week and only got back a few days ago so no further discussion has been had yet.
- charlesdaniels 6y ago
  This is indeed an idea that has been coming up from time to time for over 25 years at least. I think the earliest publication in this space was Gockhale's Terasys (DOI 10.1109/2.375174). It is a good idea in principle, but it's never really taken off to my knowledge. Parallel programming is hard. Deeply embedded programming is hard. The languages we have are mostly bad at both. If you want to search for more, the keyword is "processor in memory" or "processing in memory" (PIM). Also "in memory computing" is commonly used as well. A number of groups are working on this right now. The new buss is "memristors" (memristive computing). Whether or not any of it actually ends up working or being commercially viable outside of a lab remains to be seen.
- monocasa 6y ago
  There are NICs that more or less run BPF for exactly those reasons.
  - hinkley 6y ago
    Give me SQLite running directly on an nvme card.
    - monocasa 6y ago
      I've actually heard rumors of SSD controller firmware that uses SQLite internally. Not quite what you're asking for, for sure.
      - hinkley 6y ago
        To track cell health? Or to queue writes? Or both?
- totalZero 6y ago
  Perhaps one problem with that is the modularity with various processors, which would have to know when they are configured with memory to which certain operations can be farmed out, and when they are configured with traditional memory that cannot manage those in-place operations.
  - vlovich123 6y ago
    SoCs are a good vector of this. Once a technology like this makes it into a space it’s unlikely to ever get removed if it’s actually useful (ie good perf/cost/power wins) as there’s a competitive advantage to having a better one. The longer it’s around, the more second order-effects you have (eg if you need SW integration, tooling etc) entrenching the tech (the same forces that make it difficult to get off the ground in the first place). Apple is a great example of being a company that can pull this off as they have the vertical integration to get it off the ground AND the commitment to ensure a consistent application within a vertical. It may not be within Apple’s business interests if this is primarily useful for “big data”. That seems unlikely to me though so you’d need other applications they might see as strategically useful. SQLite is used heavily at Apple, so accelerating such workloads may be useful. A good chunk of what an OS does overall is largely manipulate memory, so something like the Os scheduler could be bifurcated to pick the next task and have that ready, including modifying the various lists, so that the CPU is doing even less (although unlikely to be a good example as I suspect this is a relatively short operation since it happens so frequently that there’s not really any room for a win).
rektide 6y ago
meanwhile, disk is potentially getting to be as fast as ram, throughput wise. 128 lanes of pcie 4.0 is 256GBps iirc. epyc's 8 channel ddr4-3200 otoh is good for 208GBps. Let's Encrypt stopped a little short, using 24x nvme disks (it fits in a 2U though so that's nice)[1]. that could be up to 96 of 128 pcie links in use. with the right ssds, working on large data-objects, that'd be somewhere a bit under 192GBps versus the max 208GBps of their ram. in truth, ram's random access capabilities are far better, there's much less overhead (although nvme is pretty good). and i'm not sure i've ever seen anyone try to confirm that those 128 lanes of pcie on epyc aren't oversubscribed, that devices really can push that much data around. note that this doesn't necessarily even have to mean using the cpu; pci p2p is where it's at for in-the-know folks doing nvme, network, and gpu data-pushing; epyc's io-die is acting like a data-packet switch in these conditions, rather than having the cpu process/crunch these peripheral's data. [1] https://news.ycombinator.com/item?id=25861422 https://news.ycombinator.com/item?id=25861422
- tanelpoder 6y ago
  Author (of the RAM article) here: Indeed, you can go further, but got to plan for other bandwidth needed by other peripherals, data movement and inter-CPU bandwidth (NUMA) and intra-CPU-core bandwidth limitations too (AMD's infinity fabric is point to point between chiplets, but intel has some ring-bus architecture for moving bits between CPU cores). I got my Lenovo ThinkStation P620 workstation (with AMD Zen-2 ThreadRipper Pro WX, 8-memory channels like EPYC) to scan 10 x PCIe 4.0 SSDs at 66 GB/s (I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe <-> CPU data transfer bottleneck. And even with doing I/O through 3 PCIe root complexes (out of 4 connected to that CPU), I seem to be hitting some inter-CPU-core bandwidth limitation. The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs. Planning to publish some blog entries about these I/O tests but a teaser tweet is here (11M IOPS with a single-socket ThreadRipper workstation - it's not even a NUMA server! :-) https://twitter.com/TanelPoder/status/1352329243070504964 https://twitter.com/TanelPoder/status/1352329243070504964
  - MayeulC 6y ago
    That bandwidth limitation could be due to Infinity Fabric, which seems to be rated at 42GBps (x2 as full duplex, though)? https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric
dang 6y ago
If curious see also 2017 - same title different article https://news.ycombinator.com/item?id=14888360 https://news.ycombinator.com/item?id=14888360

75 points by tanelpoder 6y ago

I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software architecture theory we learn is based on existing HW paradigms without much thought being given to how we can change HW paradigms. By nature HW is massively parallel but where physical distance from compute = latency (vs the ultimately serial execution nature of traditional CPUs that can process all data at blistering speed but only one at a time with some SIMD exceptions). There are real-world benefits to this kind of design - memory is cheap and simple to manufacture and abundantly available. The downside though is that the CPU is sitting doing nothing but waiting for memory most of the time, especially when processing large data sets. Imagine how efficient a GC algorithm would be if it could compute a result in the background just doing a concurrent mark and sweep, perhaps as part of a DRAM refresh cycle so that you could even choose to stop refreshing that RAM because your application no longer needs that row. The power and performance savings are pretty enticing.

tanelpoder 6y ago
Oracle (Sun) latest CPUs support something called DAX (data analytics extension, not the same thing as the Intel DAX - direct access extensions). It's a coprocessor that allows offloading simpler operations closer to RAM, apparently: "DAX is an integrated co-processor which provides a specialized set of instructions that can run very selective functionality – Scan, Extract, Select, Filter, and Translate – at fast speeds. The multiple DAX share the same memory interface with the processors cores so the DAX can take full advantage of the 140-160 GB/sec memory bandwidth of the SPARC M7 processor." https://blogs.oracle.com/bestperf/accelerating-spark-sql-using-sparc-with-dax%2c-16x-faster-per-core-over-and-above-spark-210-with-tungsten https://blogs.oracle.com/bestperf/accelerating-spark-sql-usi...
cogman10 6y ago
Interestingly enough, this is why Simultaneous multithreading [1] exists! The revelation that "The CPU could be doing useful work while stalled waiting on a load" has lead to CPU designers "faking" the number of cores available to the OS to allow the CPU to do more useful work while different "threads" are paused waiting on memory to come back with the data they need. [1] https://en.wikipedia.org/wiki/Simultaneous_multithreading https://en.wikipedia.org/wiki/Simultaneous_multithreading
- tanelpoder 6y ago
  And all kinds of manual prefetching instructions built into the CPUs (for compilers to use) and prefetching algorithms & predictors built into the CPUs too!
- nomel 6y ago
  I think the point is that this still eats into CPU <-> memory bandwidth. Offloading could let the CPU use that memory for better purposes, especially since the stall is from memory access anyways.
  - geocar 6y ago
    Linked-list chasing (as in vlovich123's example) isn't bandwidth-limited as much as it is latency: SMT helps here because you're able to enqueue multiple wait-states "simultaneously".
    - jeffbee 6y ago
      Anything you can do with SMT you can do without SMT using ILP instead. SMT does not grant a CPU extra cache-filling resources.
      - geocar 6y ago
        SMT is easier to program. Walking a single linked-list is nothing; A garbage collector walks lots of them that tend to fan out for a bit.
lalaithion 6y ago
Even just "go to memory location X+Y, load the value there, set X = that value, repeat N times" would allow fast linked list traversal, as well as doing various high-level pointer chasing faster.
55873445216111 6y ago
There are some memories supporting basic in-memory operations. For example: https://mosys.com/products/blazar-family/be3rmw-bandwidth-engine-3-rmw/ https://mosys.com/products/blazar-family/be3rmw-bandwidth-en.... This supports operations like read-modify-write within the memory device itself. (I have no affiliation with this company.) The barrier to adoption of this is not technical, it's economic. Memory industry has focused on making the highest capacity and lowest cost/bit products. This drives high manufacturing volume which drives economies of scale. Memory products with integrated functions are inherently niche, and therefore do not have anywhere near the market size and economy of scale. Designers have decided (historically) that it is cheaper at the system level to keep the logic operations within the CPU and use a "dumb" commodity memory, even though this necessitates more bandwidth usage. (It's a complex engineering trade-off.) With logic performance continuing to scale faster than memory bandwidth, at some point an architecture that reduces the required memory bandwidth (such as computing in-memory) might start to make sense economically.
- vlovich123 6y ago
  I think there are niche use-cases that would warrant the cost/complexity trade-off. Namely cloud infrastructure for databases where you might be processing large amounts and transferring all that data first to the CPU is pretty ridiculous. I think the complexity problem is solvable. If you can build such a thing for memory, you can reuse the same general concept for storage like NVME/SSD/spinning disk. It’s entirely possible this is warranted even in consumer devices as quite a bit of OS operations deal with modifying/querying memory whereas doing offload can win you some serious wins (making the machine feel way more snappy/interactive + more powerful to execute things locally). I don’t have hope for traditional CPU designers so the question is whether someone can both design a memory system with this power AND a computational model that makes this easy to adopt in major languages while offering a perf win (given that a memory fetch costs ~100ns for ~256 bytes). It’s challenging and unlikely to come from x86 land where back Hw compatibility is extremely important. The innovation may eventually come from mobile land (which is where Apple is coming from) where memory controllers are custom designed and part of the SoC anyway, changing with each revision, so the hard part remains again how you make this fast, efficient, able to handle multiple concurrent programs (or at least have the OS control coarsest which program’s memory accesses were being prioritized), and language integration (so you can hand off an algorithm and it would be executed without upending existing software design knowledge). The OS integration could be even more intelligent - if the process/thread is spending its time processing memory without returning results to the CPU, just put it to sleep until the result is available (separating memory access and CPU utilization, resulting in drastically better utilization of cycles rather than naively waiting for each memory stall one at a time). So tldr I agree with you totally. This is not going to happen if memory manufacturers continue to go for the “cheap and large” memory route. I do see hope that such concepts may be explored if we get more innovation in the CPU space.
wcerfgba 6y ago
This reminds me of the Connection Machine architecture [1] > Each CM-1 microprocessor has its own 4 kilobits of random-access memory (RAM), and the hypercube-based array of them was designed to perform the same operation on multiple data points simultaneously, i.e., to execute tasks in single instruction, multiple data (SIMD) fashion. The CM-1, depending on the configuration, has as many as 65,536 individual processors, each extremely simple, processing one bit at a time. [1] https://en.wikipedia.org/wiki/Connection_Machine https://en.wikipedia.org/wiki/Connection_Machine
d_tr 6y ago
A few months ago I wanted to take a look at the Gen-Z fabric specifications, but unfortunately they still have a lame members-only download request form in place.
dragontamer 6y ago
> Right now we have to fetch everything into the CPU before we can make decisions. Think about virtual memory. The memory at location #6000 is NOT at where the program thinks is at memory location #6000. In fact, the program might be reading / writing to memory location 0x08004000 (or close to that, whatever the magic start address was on various OS like Linux / Windows). And then your CPU virtually-translates that address to memory-stick #0 column#4000 or whatever. Because of virtual memory: all memory operations must be translated by the CPU before you actually go to RAM (and that translation may require a page-table walk in the worst case)
- ben509 6y ago
  Well, all memory operations must be translated by an MMU. So can the MMU be a distributed beast that lives in the CPU, the memory and the IO?
  - dragontamer 6y ago
    > So can the MMU be a distributed beast that lives in the CPU, the memory and the IO? That "blah = blah->next" memory operation could be: * Going to swap thanks to swap / overcommit behavior on Linux/Windows. * Or going to a file thanks to mmap * Going to Ethernet and to another computer thanks to RDMA So... no. I'm pretty sure the CPU is the proper place for that kind of routing.
    - freeone3000 6y ago
      why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded) moreover, if we have support for mmap at the MMU level, we can cut the CPU bottleneck for disk access entirely. the disk controller can already handle DMA, but there's simply no way for things that aren't the CPU to trigger it. DirectStorage is an effort for GPUs to trigger it, but what if we could also trigger it by other means?
BenoitP 6y ago
> HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU Well, that seems to be the exact definition of what UPMEM is doing: https://www.upmem.com/upmem-announces-silicon-based-processing-in-memory-benchmarks/ https://www.upmem.com/upmem-announces-silicon-based-processi... Between the M1, GPUs, TPUs, RISC-V interesting times are coming in hardware. I blame the physical limits, which are putting the Duke Nukem development method to an end (you promise the client 2x the same performance in 18 months, then play Duke Nukem the whole time). The only way to better performance now is through hardware specialization. That and avoiding Electron.
- WJW 6y ago
  You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too. There's still loads of performance to be gained just by being better programmers.
  - Arelius 6y ago
    > Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. I feel like this story needs an ending? Did somebody re-implement it using a heap and found significant performance wins? Or was the sorted array used on purpose to take advantage of some specific cache constraints and actually end up being a huge win?
    - WJW 6y ago
      The story ends (so far) with me filing a GH issue and offering a PR, my own benchmarking getting an average 3x speedup over a wide range of inputs. I happen to know that the maintainer has been on vacation for the last week and only got back a few days ago so no further discussion has been had yet.
charlesdaniels 6y ago
This is indeed an idea that has been coming up from time to time for over 25 years at least. I think the earliest publication in this space was Gockhale's Terasys (DOI 10.1109/2.375174). It is a good idea in principle, but it's never really taken off to my knowledge. Parallel programming is hard. Deeply embedded programming is hard. The languages we have are mostly bad at both. If you want to search for more, the keyword is "processor in memory" or "processing in memory" (PIM). Also "in memory computing" is commonly used as well. A number of groups are working on this right now. The new buss is "memristors" (memristive computing). Whether or not any of it actually ends up working or being commercially viable outside of a lab remains to be seen.
monocasa 6y ago
There are NICs that more or less run BPF for exactly those reasons.
- hinkley 6y ago
  Give me SQLite running directly on an nvme card.
  - monocasa 6y ago
    I've actually heard rumors of SSD controller firmware that uses SQLite internally. Not quite what you're asking for, for sure.
    - hinkley 6y ago
      To track cell health? Or to queue writes? Or both?
totalZero 6y ago
Perhaps one problem with that is the modularity with various processors, which would have to know when they are configured with memory to which certain operations can be farmed out, and when they are configured with traditional memory that cannot manage those in-place operations.
- vlovich123 6y ago
  SoCs are a good vector of this. Once a technology like this makes it into a space it’s unlikely to ever get removed if it’s actually useful (ie good perf/cost/power wins) as there’s a competitive advantage to having a better one. The longer it’s around, the more second order-effects you have (eg if you need SW integration, tooling etc) entrenching the tech (the same forces that make it difficult to get off the ground in the first place). Apple is a great example of being a company that can pull this off as they have the vertical integration to get it off the ground AND the commitment to ensure a consistent application within a vertical. It may not be within Apple’s business interests if this is primarily useful for “big data”. That seems unlikely to me though so you’d need other applications they might see as strategically useful. SQLite is used heavily at Apple, so accelerating such workloads may be useful. A good chunk of what an OS does overall is largely manipulate memory, so something like the Os scheduler could be bifurcated to pick the next task and have that ready, including modifying the various lists, so that the CPU is doing even less (although unlikely to be a good example as I suspect this is a relatively short operation since it happens so frequently that there’s not really any room for a win).

rektide 6y ago

meanwhile, disk is potentially getting to be as fast as ram, throughput wise. 128 lanes of pcie 4.0 is 256GBps iirc. epyc's 8 channel ddr4-3200 otoh is good for 208GBps. Let's Encrypt stopped a little short, using 24x nvme disks (it fits in a 2U though so that's nice)[1]. that could be up to 96 of 128 pcie links in use. with the right ssds, working on large data-objects, that'd be somewhere a bit under 192GBps versus the max 208GBps of their ram. in truth, ram's random access capabilities are far better, there's much less overhead (although nvme is pretty good). and i'm not sure i've ever seen anyone try to confirm that those 128 lanes of pcie on epyc aren't oversubscribed, that devices really can push that much data around. note that this doesn't necessarily even have to mean using the cpu; pci p2p is where it's at for in-the-know folks doing nvme, network, and gpu data-pushing; epyc's io-die is acting like a data-packet switch in these conditions, rather than having the cpu process/crunch these peripheral's data. [1] https://news.ycombinator.com/item?id=25861422 https://news.ycombinator.com/item?id=25861422

tanelpoder 6y ago
Author (of the RAM article) here: Indeed, you can go further, but got to plan for other bandwidth needed by other peripherals, data movement and inter-CPU bandwidth (NUMA) and intra-CPU-core bandwidth limitations too (AMD's infinity fabric is point to point between chiplets, but intel has some ring-bus architecture for moving bits between CPU cores). I got my Lenovo ThinkStation P620 workstation (with AMD Zen-2 ThreadRipper Pro WX, 8-memory channels like EPYC) to scan 10 x PCIe 4.0 SSDs at 66 GB/s (I had to move SSD cards around so they'd use separate PCIe root complexes to avoid a PCIe <-> CPU data transfer bottleneck. And even with doing I/O through 3 PCIe root complexes (out of 4 connected to that CPU), I seem to be hitting some inter-CPU-core bandwidth limitation. The throughput differs depending on which specific CPU cores happen to run the processes doing I/Os against different SSDs. Planning to publish some blog entries about these I/O tests but a teaser tweet is here (11M IOPS with a single-socket ThreadRipper workstation - it's not even a NUMA server! :-) https://twitter.com/TanelPoder/status/1352329243070504964 https://twitter.com/TanelPoder/status/1352329243070504964
- MayeulC 6y ago
  That bandwidth limitation could be due to Infinity Fabric, which seems to be rated at 42GBps (x2 as full duplex, though)? https://en.wikichip.org/wiki/amd/infinity_fabric https://en.wikichip.org/wiki/amd/infinity_fabric

dang 6y ago

If curious see also 2017 - same title different article https://news.ycombinator.com/item?id=14888360 https://news.ycombinator.com/item?id=14888360