4 ms·
Why is the text inside the document hard to index? Isn't that what Lucene/Solr is for? Not sure what you mean...
by shafqat 16y ago
Why is the text inside the document hard to index? Isn't that what Lucene/Solr is for? Not sure what you mean...
- byoung2 16y agoWhy is the text inside the document hard to index? Isn't that what Lucene/Solr is for? Not sure what you mean... You just proved my point...if I have to install a standalone Java full-text engine running on Tomcat, I've already gone beyond what most forum owners can do on shared hosting. Imagine searching through all posts on HN for the phrase "proved my point" anywhere in the post. There is no way other than iterating through all of the millions of posts. For commonly-searched phrases such as "P vs NP", "scalability", or "rate my startup" you can create an index for faster searching, but it is impossible to anticipate every possible search, and in particular, long tail searches which can be the most useful will be the least likely to be indexed.
- nkurz 16y agoImagine searching through all posts on HN for the phrase "proved my point" anywhere in the post. There is no way other than iterating through all of the millions of posts. This is not true. Rather, in addition to the document ID, the inverted index stores the position information for each term occurrence, and can very quickly find any phrase of your choosing. While it's possible that this isn't always the default behaviour, all modern general purpose search engines support this.
- danielnicollet 16y agoI concur with nkurz here. Inverted index are the basis for most full-text search engines and they can handle this quite well. Another issue with searching a large forum like HN though can be keeping the index up-to-date through crawling new posts constantly to add the terms they contain to the inverted index. However, you can usually build a simple script that adds news posts URLs to a list of priority crawl targets. Free HTDig (http://www.htdig.org/ http://www.htdig.org/) for instance could do the task pretty well and run on a budget hosting plan described above. It can index up to 100ks of documents easily and may require disk space upgrades for large sites but not much more. There are many other inverted index search engines available and some now easily index without crawling through XML feeds from your CMS or forum software.
- byoung2 16y agoSee my original post...I acknowledge that an inverted index can do it, but when you have to make it compatible with a wide range of hosting and technical ability as in the case of vBulletin, you are more limited. The typical forum owner will not install a standalone search engine, especially if his $5 hosting doesn't support it
- colinsidoti 16y agoYes, but perhaps larger forums with dedicated servers can handle it? If this is true, selling a Search Replacement for forum software could really be a viable business.
- danielnicollet 16y agoyes but you could do it with a hosted search solution which, can index without crawling if it can be fed xml feeds of old and new posts...