4 ms·

There's some really interesting little details here. Linux, by default, is making the very reasonable assumption that the marginal cost of converting empty phy

by mjb 3y ago

There's some really interesting little details here. Linux, by default, is making the very reasonable assumption that the marginal cost of converting empty physical memory into caches and buffers is very near zero. This is fundamentally reasonable, because the cost of converting empty memory into used memory isn't really any cheaper than converting a clean cached page into used memory. It's a little more subtle when you take accounting into account, or when you think about dirty pages (which need to be written back to clear memory), or think about caches, but the core assumption is a very reasonable one. Except for on some multi-tenant infrastructure. Here, "empty" pages don't really exist. There's mostly not an empty page of memory kicking around waiting (like there is on client devices). Instead, nearly all the memory on the box is allocated, but each individual guest kernel doesn't know the full allocation. In this world, the assumption that the marginal cost of converting empty to full is zero is no longer true. There's some real cost. Projects like DAMON https://sjp38.github.io/post/damon/ https://sjp38.github.io/post/damon/ exist to handle this case, and similar cases where keeping empty memory rather than low-value cache is worse for the overall system. These kinds of systems aren't super common, especially on the client side, but aren't unusual in large-scale cloud services.

mjb 3y ago
The other interesting detail here is the memory sizing problem. If I can consume all my RAM with caches and buffers, how much RAM do I need? The answer (as always) depends on what you're optimizing for. For performance, bigger is better. For cost, energy, etc you're going to want some way to calculate whether adding more RAM (and so having bigger caches) is worth the cost, heat, power, etc. Gray and Putzolu's classic "The 5 minute rule for trading memory for disc accesses" (https://dl.acm.org/doi/pdf/10.1145/38713.38755 https://dl.acm.org/doi/pdf/10.1145/38713.38755) from 1987 is probably one of the most important CS systems papers of all time. In it, they lay out a way of thinking about memory and cache sizing by comparing the cost of holding cache to the cost of access (this isn't the first use of that line of thinking, but is a very influential statement of it). Back then, they found that storing 4kB in RAM for 5 minutes costs about the same as reading it back from storage. So if you're going to access something again within 5 minutes you should keep it around. The constants have change a lot (RAM is way cheaper, IOs are way cheaper, block sizes are typically bigger) since then, but the logic and way of thinking are largely timeless. The 5 minute rule is a quantitative way of thinking about the size of the working set, an idea that dates back at least to 1968 and Denning's "The working set model for program behavior" (https://dl.acm.org/doi/10.1145/363095.363141 https://dl.acm.org/doi/10.1145/363095.363141). Back to marginal costs - the marginal cost of converting empty RAM to cache is zero in the minute, but only because the full cost has been borne up front when the machine is purchased. It's not zero, just pre-paid.
- GTP 3y ago
  > Back to marginal costs - the marginal cost of converting empty RAM to cache is zero in the minute, but only because the full cost has been borne up front when the machine is purchased. It's not zero, just pre-paid. . Or, in other words, you get to fully use what you paid for.
  - teruakohatu 3y ago
    I think the OPs point was that people tend to buy more RAM than they actually need because they have no idea how much RAM they actually need, because it's always used, and so err on the side of caution
    - Arrath 3y ago
      Well you don't have to call me out for running 64gb of ram in my home desktop like that.
- LeifCarrotson 3y ago
  Huh, never heard of that before. An interesting paper! Running the numbers - assuming 4k record size instead of 1k, ignoring data size changes, ignoring cache, ignoring electricity and rack costs, selecting a $60 Samsung 980 with 4xPCIe and a $95 set of 2x16GB DDR5-6400 DIMMs...I get $0.003/disk access/second/year and $0.0000113 for 4k of RAM, a ratio of 264. That is remarkably close to the original paper's ratio of 400, even though their disks only got 15 random reads per second, not 20,000, and cost $15,000, and their memory cost $1000/MB not $0.002/MB. I'm not sure the "Spend 10 bytes of memory to save 1 instruction per second" works equally well, especially given that processors are now multi-core pipelined complex beasts, but working naively, you could multiply price, frequency, and core count to calculate ~$0.01/MIP (instead of $50k). $0.01 is about the cost of 3 MB of RAM. Dividing both by a million you should spend 3 bytes, not 10 bytes, to save 1 instruction per second.
  - justsomehnguy 3y ago
    > $60 Samsung 980 If this is a Hetzner machine then yes, but enterprise SSDs costs more, especially from enterprise vendors. But this only drives the storage cost up. More so, if you tend to send some big amount of data every 5 minutes and you are somewhat constrained by memory (32 / 1000 x 100 = 3.2%) then it would be easier to just read it from the storage again. If you are not constrained by storage bandwidth, of course. And by the way, the latest gaming consoles (at least PlayStation?) is designed around this concept - they trade having big amount of RAM (which in case of PS5 is shared between GPU and the OS) to just loading assets from the storage extremely fast 'just in time'. Which works fine for games.
dist-epoch 3y ago
> but each individual guest kernel doesn't know the full allocation I was under the impression that at least in some virtual machine types the guest kernel is collaborating with the host kernel through vm drivers to avoid this problem.
- mjb 3y ago
  Well, yeah. But (DAMON and friends aside), Linux doesn't handle that non-zero marginal memory cost well today.

by mjb 3y ago

mjb 3y ago

The other interesting detail here is the memory sizing problem. If I can consume all my RAM with caches and buffers, how much RAM do I need? The answer (as always) depends on what you're optimizing for. For performance, bigger is better. For cost, energy, etc you're going to want some way to calculate whether adding more RAM (and so having bigger caches) is worth the cost, heat, power, etc. Gray and Putzolu's classic "The 5 minute rule for trading memory for disc accesses" (https://dl.acm.org/doi/pdf/10.1145/38713.38755 https://dl.acm.org/doi/pdf/10.1145/38713.38755) from 1987 is probably one of the most important CS systems papers of all time. In it, they lay out a way of thinking about memory and cache sizing by comparing the cost of holding cache to the cost of access (this isn't the first use of that line of thinking, but is a very influential statement of it). Back then, they found that storing 4kB in RAM for 5 minutes costs about the same as reading it back from storage. So if you're going to access something again within 5 minutes you should keep it around. The constants have change a lot (RAM is way cheaper, IOs are way cheaper, block sizes are typically bigger) since then, but the logic and way of thinking are largely timeless. The 5 minute rule is a quantitative way of thinking about the size of the working set, an idea that dates back at least to 1968 and Denning's "The working set model for program behavior" (https://dl.acm.org/doi/10.1145/363095.363141 https://dl.acm.org/doi/10.1145/363095.363141). Back to marginal costs - the marginal cost of converting empty RAM to cache is zero in the minute, but only because the full cost has been borne up front when the machine is purchased. It's not zero, just pre-paid.

GTP 3y ago
> Back to marginal costs - the marginal cost of converting empty RAM to cache is zero in the minute, but only because the full cost has been borne up front when the machine is purchased. It's not zero, just pre-paid. . Or, in other words, you get to fully use what you paid for.
- teruakohatu 3y ago
  I think the OPs point was that people tend to buy more RAM than they actually need because they have no idea how much RAM they actually need, because it's always used, and so err on the side of caution
  - Arrath 3y ago
    Well you don't have to call me out for running 64gb of ram in my home desktop like that.
LeifCarrotson 3y ago
Huh, never heard of that before. An interesting paper! Running the numbers - assuming 4k record size instead of 1k, ignoring data size changes, ignoring cache, ignoring electricity and rack costs, selecting a $60 Samsung 980 with 4xPCIe and a $95 set of 2x16GB DDR5-6400 DIMMs...I get $0.003/disk access/second/year and $0.0000113 for 4k of RAM, a ratio of 264. That is remarkably close to the original paper's ratio of 400, even though their disks only got 15 random reads per second, not 20,000, and cost $15,000, and their memory cost $1000/MB not $0.002/MB. I'm not sure the "Spend 10 bytes of memory to save 1 instruction per second" works equally well, especially given that processors are now multi-core pipelined complex beasts, but working naively, you could multiply price, frequency, and core count to calculate ~$0.01/MIP (instead of $50k). $0.01 is about the cost of 3 MB of RAM. Dividing both by a million you should spend 3 bytes, not 10 bytes, to save 1 instruction per second.
- justsomehnguy 3y ago
  > $60 Samsung 980 If this is a Hetzner machine then yes, but enterprise SSDs costs more, especially from enterprise vendors. But this only drives the storage cost up. More so, if you tend to send some big amount of data every 5 minutes and you are somewhat constrained by memory (32 / 1000 x 100 = 3.2%) then it would be easier to just read it from the storage again. If you are not constrained by storage bandwidth, of course. And by the way, the latest gaming consoles (at least PlayStation?) is designed around this concept - they trade having big amount of RAM (which in case of PS5 is shared between GPU and the OS) to just loading assets from the storage extremely fast 'just in time'. Which works fine for games.

dist-epoch 3y ago

> but each individual guest kernel doesn't know the full allocation I was under the impression that at least in some virtual machine types the guest kernel is collaborating with the host kernel through vm drivers to avoid this problem.

mjb 3y ago
Well, yeah. But (DAMON and friends aside), Linux doesn't handle that non-zero marginal memory cost well today.