3 ms·
> you'll still need to keep track of which data is on the disk and which data isn't I'm not following this. As I see it, all you have to do is keep insure th
by nkurz 16y ago
> you'll still need to keep track of which data is on the
disk and which data isn't
I'm not following this. As I see it, all you have to do is keep insure that no more than 3GB are mapped for your process at a given time. You still let the kernel decide what's faulted in and when things are written out.
Maybe my perspective is skewed: I'm viewing it from the perspective of real-time search, with large posting lists being frequently read and updated, and lazily written out to disk. Is there a different situation that is harder?
> In short, i386 is not dead. And suggesting that it cannot handle "gigabytes of data" is absurd
I didn't mean to imply it couldn't, or that it wasn't in common use. Rather I was wondering why I as a open source software author (http://wiki.apache.org/incubator/LucyProposal http://wiki.apache.org/incubator/LucyProposal) would choose to make my life more complicated by targeting new software at a 32-bit platform at the expense of code simplicity.
Our current plan is to optimize for 64-bit systems, and allow 32-bit compatibility for smaller data sets (tens of GB) by sprinkling in mmap()/munmap() calls as necessary:
http://www.mail-archive.com/lucy-dev@lucene.apache.org/msg00888.html http://www.mail-archive.com/lucy-dev@lucene.apache.org/msg00...
Edit to add: I'm not trying to claim I'm definitively right. Rather, my point is that I care about the answers to these questions in practice as well as in theory. I worry about taking the wrong approach.