3 ms·
"if the data set becomes too large to fit in memory on a single machine?" You just use mmap. If it become bigger than the disk capacity of single machine, you
by jdmoreira 11y ago
"if the data set becomes too large to fit in memory on a single machine?"
You just use mmap. If it become bigger than the disk capacity of single machine, you can still mount a distributed filesystem and use mmap.
- voltagex_ 11y agoSlightly off topic: http://yourdatafitsinram.com/ http://yourdatafitsinram.com/ and more seriously https://www.youtube.com/watch?v=SiSY1b0am5w https://www.youtube.com/watch?v=SiSY1b0am5w which is a really good talk about the challenges of sysadmining for scientists. What do you do when 2TB of RAM isn't enough?
- pdkl95 11y agoYou add an index and smaller-block loader. The size and structure of a "smaller-block" depends on the project and how you are using it. In more extreme situations, you to learn all about materialized views, cache invalidation, etc.