4 ms·

It sounds like they released the tokenizer that powers bing, which one of the largest search engines in the world. That's pretty significant.

• dx034 8y ago
  From the Readme it appears that it will deliver the same results as NLTK, just much faster. Therefore publishing it won't give away any of the secret sauce that make up Bing's search results.
  • jiguang98 8y ago
    The tokenization results are not the same, you can see some examples in readme.
    • ma2rten 8y ago
      Just to clarify, I didn't mean to imply that this is Bing's secrete sauce. But if it runs at this scale, but it's probably extremely robust and optimized software.