2 ms·
> After a month I had most newsgroups - excepting binaries - and it came to 800GB. > Trying to index THAT lot was impossible Stupid question, but why would in
by zokier 9y ago
> After a month I had most newsgroups - excepting binaries - and it came to 800GB.
> Trying to index THAT lot was impossible
Stupid question, but why would indexing 800GB of newsgroup postings be impossible?
- mailxplorer 9y agoClucene was way too slow for body text, more than 1GB. I had my own header parser in C++ (though you can do that in Python easily). I'm trying again on that 800GB with KISS DB (append-only hashtable), and Elasticsearch. Doesn't matter if GPL because it's a website.
- GuacheSuedeHN 9y agoDo you mind sharing the code ? I think that is an interesting thing to see