maybe you have to partition the file but I'm guessing random access will allow you to access it fine, I haven't ever poked a 35GB text file either. first thing

maybe you have to partition the file but I'm guessing random access will allow you to access it fine, I haven't ever poked a 35GB text file either. first thing I'd try is probably pseudo-this: while((entry=readDataPoint())!=null){ sha1e = sha1(entry); //database has log N index on sha1e if(database.contains(sha1e)) continue; //database object figures out insert syntax database.insert({sha1:sha1e,data:entry}); }