3 ms·
Any suggestions on how I can go about getting unique records from this type of file? If that's what you're after, you don't need to sort at all, you can pick o
by pvg 16y ago
Any suggestions on how I can go about getting unique records from this type of file?
If that's what you're after, you don't need to sort at all, you can pick out your uniques with a 3 line script in your favourite language, split up the file first, if necessary.
- aristus 16y agoThe file has 2+ billion lines. A set or hash table won't work very if if there are, say 500 million unique strings.
- pvg 16y agoHence the split. It's still likely to beat the crap out of sorting and it might not be needed, depending on the data. I do wonder what the dataset is that makes for such tiny, short lines.
- keefe 16y agoan in-memory set won't work. indexes don't have to be in memory all the time to provide log N lookup, I mean just think about how b-trees work. ...and this is a problem most databases have addressed.
- pvg 16y agoIt might well work, it depends a great deal on the data. In any event, I was talking about in-memory hashing, the responder's assumption that I was was correct.