4 ms·

Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ? Here is an idea: write an algorithm that keeps the most re

by jmulho 16y ago

Why on earth would you want to keep 236 million check-in documents (66 gig / 300 bytes) in memory ? Here is an idea: write an algorithm that keeps the most recently used 100 million check-in documents in memory. That'll save 38 gig of RAM. Or how about this idea from the 1970s: write an algorithm that keeps the most recently used 4 gig of check-in documents in memory. That will save 62 gig of RAM, and the most recently active 14 million users (4 gig / 300 bytes) will still enjoy instantaneous response. Can someone help me out with the math here?

ww520 16y ago
Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.
- silentbicycle 16y ago
  Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.
  - ww520 16y ago
    The problem isn't the page granularity, which is just a convenient size for the general case and works pretty well in most cases. The problem is to not keep excessive data in memory; in this case, they are keeping ALL data in memory due to some architectural decisions. OS caching and paging have work pretty well in keeping the frequently used data in memory and leave the rest on disk, and thus utilize the system resource more efficiently.
- jmulho 16y ago
  I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.
  - brown9-2 16y ago
    I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally? harryh's answer above about querying the user's entire history for each checkin answers this question though: http://news.ycombinator.com/item?id=1769909 http://news.ycombinator.com/item?id=1769909
    - dedward 16y ago
      IF your system is getting pounded with queries, and you suddenly have to page to serve those queries, the ability of your system to respond to queries and other operations goes WAY down, and the queries are just going to pile up, causing further swapping and load - it's a vicious cycle.

by jmulho 16y ago

ww520 16y ago

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

silentbicycle 16y ago
Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.
- ww520 16y ago
  The problem isn't the page granularity, which is just a convenient size for the general case and works pretty well in most cases. The problem is to not keep excessive data in memory; in this case, they are keeping ALL data in memory due to some architectural decisions. OS caching and paging have work pretty well in keeping the frequently used data in memory and leave the rest on disk, and thus utilize the system resource more efficiently.
jmulho 16y ago
I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.
- brown9-2 16y ago
  I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally? harryh's answer above about querying the user's entire history for each checkin answers this question though: http://news.ycombinator.com/item?id=1769909 http://news.ycombinator.com/item?id=1769909
  - dedward 16y ago
    IF your system is getting pounded with queries, and you suddenly have to page to serve those queries, the ability of your system to respond to queries and other operations goes WAY down, and the queries are just going to pile up, causing further swapping and load - it's a vicious cycle.