4 ms·
I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would
by rafaelmn 2y ago
I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.
- zahlman 2y agoThis seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?
- arccy 2y agowould it really change that quickly? you might get significant savings from just having keywords, common variable names, standard library functions
- rafaelmn 2y agoBut is it really moving that fast ? I suspect most fundamental terms in programming and the variations do not change often. You will always have keywords, built ins and the most popular concepts from libs/frameworks. So it is basically downloading a few hundred kb dictionary every year ?
- MBCook 2y agoYou’d have language keywords and such, yeah. But past that the most common words in a React project are going to be very different from a Vue project right?