3 ms·
I'm the author of this. I think I can index a lot more source code and still keep the speed. The important part of the index, the ngrams table, is only 400M or
by eklitzke 14y ago
I'm the author of this. I think I can index a lot more source code and still keep the speed. The important part of the index, the ngrams table, is only 400M or so of space and compresses pretty well. I had a test index at one point that was also indexing all of the C code in these projects:
clang-3.1.src
gcc-4.7.1
ghc-7.4.1
linux-3.4.4
llvm-3.1.src
lua-5.2.0
openjdk
perl-5.16.0
php-5.4.4
Python-2.7.3
Python-3.2.3
ruby-1.9.3-p194
And it only increased the size of the index by about 50%.