4 ms·
Large-scale text mining (Google/Wikipedia). There's boatloads of patterns and data latent in an enormous popular text corpus. For instance, if you have two te
by lsb 19y ago
Large-scale text mining (Google/Wikipedia). There's boatloads of patterns and data latent in an enormous popular text corpus.
For instance, if you have two terms, and you want to see how similar they are, that's P(A|B). So do a search query for A&B and for B, and P(A|B)=P(A&B)/P(B), and that's a first approximation, a Google Distance, for arbitrary terms.
The information silos are only a couple years old, and already they're so valuable. I can't wait until the Internet's a century old.
- pixcavator 19y agoP(A|B) != P(B|A), so not really a distance...
- lsb 19y agoIt's from Cilibrasi and Vitanyi. They have shimming factors, so you take the min of A|B and B|A, then you throw in some logs to not penalize low probabilities.
- aswanson 19y agoHe's onto something here. What tools/languages are you using for this.
- lsb 19y agoWhen I'd stopped independent work on it, and brought it to Songkick, it was coded entirely in Javascript, with a reimplementation in Ruby (to check work). Email me (leebutterman/gmail) if you're interested.