4 ms·

Thank you for trying it out. Re: performance - It's mining through ~40gb of data on server with 8gb of ram. - Also, we're not using caching of search results

by eserorg 18y ago

Thank you for trying it out. Re: performance - It's mining through ~40gb of data on server with 8gb of ram. - Also, we're not using caching of search results -- it computes on-the-fly for each query. - If we can get a hold of more servers, we should be able to bring down the query time below 1 second. Re: query "Test" - You have to search for something you're interested in.

okeumeni 18y ago
From my own experience 40GB and 8GB RAM is very good it should be enough for better performance. I don’t think you need more servers at this time. Think about it in order for you guys to have a meaningful search engine you will need data in the TB range how many servers will you need then? Spend more time fine tuning your search algorithm and processing you should get better performance out of what you have now. Then your repository will grow proportionally to your resources and you should be fine. When I said the search for 'test' did not return good result I meant you should do more work on relevancy.
- eserorg 18y ago
  It's a very tough problem. Queries such as "square", "blue", "fast", etc... will yield very poor results. PLSI tends to perform very well on more specific queries, such as "Paul Graham", "silicon graphics", etc... The problem with PLSI is that it is extremely computationally expensive -- which is why most internet-scale search engines don't use it. Our innovation was figuring out some tricks that have allowed us to improve performance dramatically. However, there is obviously still room for improvement. Our goal is to satisfy 80% of the queries with decent results -- and to leave the other 20% (square, etc...) to someone else. The interesting thing about PLSI is that it's able to rank documents from the text alone -- ignoring the link structure and other metadata. Therefore, we're thinking our algorithm will make the most sense in situations where there is lots of textual data without web-like link metadata. The two scenarios that come to mind where people need to text-mine documents outside the metadata-rich web are: (1) windows file shares on corporate intranets (2) large volumes of legal documents inside law firms Text-mining wikipedia is a proof-of-concept at this point

by eserorg 18y ago

okeumeni 18y ago

From my own experience 40GB and 8GB RAM is very good it should be enough for better performance. I don’t think you need more servers at this time. Think about it in order for you guys to have a meaningful search engine you will need data in the TB range how many servers will you need then? Spend more time fine tuning your search algorithm and processing you should get better performance out of what you have now. Then your repository will grow proportionally to your resources and you should be fine. When I said the search for 'test' did not return good result I meant you should do more work on relevancy.

eserorg 18y ago
It's a very tough problem. Queries such as "square", "blue", "fast", etc... will yield very poor results. PLSI tends to perform very well on more specific queries, such as "Paul Graham", "silicon graphics", etc... The problem with PLSI is that it is extremely computationally expensive -- which is why most internet-scale search engines don't use it. Our innovation was figuring out some tricks that have allowed us to improve performance dramatically. However, there is obviously still room for improvement. Our goal is to satisfy 80% of the queries with decent results -- and to leave the other 20% (square, etc...) to someone else. The interesting thing about PLSI is that it's able to rank documents from the text alone -- ignoring the link structure and other metadata. Therefore, we're thinking our algorithm will make the most sense in situations where there is lots of textual data without web-like link metadata. The two scenarios that come to mind where people need to text-mine documents outside the metadata-rich web are: (1) windows file shares on corporate intranets (2) large volumes of legal documents inside law firms Text-mining wikipedia is a proof-of-concept at this point