4 ms·

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly

by ww520 16y ago

Isn't that what the on-demanding disk cache (and paging) from the OS give you? Only the data being accessed and used are in memory. Unless they're constantly doing data churning over the whole dataset, there is no need to keep everything in memory.

silentbicycle 16y ago
Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.
- ww520 16y ago
  The problem isn't the page granularity, which is just a convenient size for the general case and works pretty well in most cases. The problem is to not keep excessive data in memory; in this case, they are keeping ALL data in memory due to some architectural decisions. OS caching and paging have work pretty well in keeping the frequently used data in memory and leave the rest on disk, and thus utilize the system resource more efficiently.
jmulho 16y ago
I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.
- brown9-2 16y ago
  I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally? harryh's answer above about querying the user's entire history for each checkin answers this question though: http://news.ycombinator.com/item?id=1769909 http://news.ycombinator.com/item?id=1769909
  - dedward 16y ago
    IF your system is getting pounded with queries, and you suddenly have to page to serve those queries, the ability of your system to respond to queries and other operations goes WAY down, and the queries are just going to pile up, causing further swapping and load - it's a vicious cycle.

by ww520 16y ago

silentbicycle 16y ago

Paging doesn't happen in 300 byte chunks, though. It's usually 4 KB, so if a record or one of twelve neighbors has been touched, it stays in. The granularity isn't right.

ww520 16y ago
The problem isn't the page granularity, which is just a convenient size for the general case and works pretty well in most cases. The problem is to not keep excessive data in memory; in this case, they are keeping ALL data in memory due to some architectural decisions. OS caching and paging have work pretty well in keeping the frequently used data in memory and leave the rest on disk, and thus utilize the system resource more efficiently.

jmulho 16y ago

I am talking about the database looking in memory for what it needs to answer a request; if it doesn't find what it needs, it reads from disk. Since it has been allowed limited memory, when it put something new in memory, it has to kick something old out. It uses a simple algorithm to identify what to keep and what to kick out. One of the benefits of this (obvious) design is that when too many users are active at the same time to keep all of them in memory, a few of them experience a small lag (as opposed to all of them experiencing a complete crash). If you have a few terabytes of disk then you have to really be asleep at the wheel (like for a decade) to totally crash.

brown9-2 16y ago
I was curious about a similar question to yours: if they had 66GB of RAM, why did going just slightly over the threshold cause such drastic paging for them - surely queries aren't touching all parts of their dataset equally? harryh's answer above about querying the user's entire history for each checkin answers this question though: http://news.ycombinator.com/item?id=1769909 http://news.ycombinator.com/item?id=1769909
- dedward 16y ago
  IF your system is getting pounded with queries, and you suddenly have to page to serve those queries, the ability of your system to respond to queries and other operations goes WAY down, and the queries are just going to pile up, causing further swapping and load - it's a vicious cycle.