5 ms·
I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that
by huahaiy 5y ago
I would appreciate people who claims that "the author does not know Lucene enough" to send a Pull Request, where Lucene is configured to do the same thing that T-Wand does, that is to guarantee documents containing more query terms to rank higher than those containing less query terms.
To those who says "I like Lucene because it does Tf-idf", please be assured that T-Wand also does tf-idf and uses vector space model.
To those who say "T-Wand is just Wand", please be reminded that you are just repeating what the article was saying. Of course T-Wand is Wand, but you miss the technical contribution of T-Wand, which is clearly laid out in "What's new in T-Wand" section of the article. Of course, it is understandable if you are not able to judge the merit of that contribution, for that's the job of the peer review panel, which I do not expect Hacker News readers to be. This work will be submitted to a peer reviewed conference with more experiments added and all trade-offs discussed.
This may come as a disappointment to some, but the paper will not have a comparison with Lucene, for it would be scientifically unsound.
However, it should be easy for Lucene to add T-Wand as an option, and I do hope so.
- Cavallium 5y agoI opened a PR to use the more precise timers. This would not improve the performance of any of the tests, but it will improve their time accuracy. I highlighted some other problems related to the Lucene benchmark, but since I can't program in Clojure I can't fix them with a PR: I opened 3 issues to describe what can be done to address them.
- huahaiy 5y agoMerged your PR. Thank you! If you do not know Clojure, that's fine, just outline what you want to do in Java, I can translate that into Clojure. Thanks. To be honest, all your proposed changes would not make material difference, for algorithmic differences cannot be made up by implementation details. As has already been demonstrated, T-Wand is implemented in a much slower language, Clojure. Clojure is implemented on top of JVM, it cannot possibly beat the same algorithm written in optimized Java. The only way to make a difference is to change the algorithm. So all your suggestions would not make a difference.
- gnabgib 5y agoWhy are you referring to yourself (the author) in the third person? It seems.. disingenuous
- huahaiy 5y agoCorrected. Thanks. I am not a native English speaker, so please do not read too much into it.
- deleted 5y ago[deleted]