Good to know, thanks. Crawling certainly is easier, made especially so using tools like JSoup/BeautifulSoup, but areas like gist extraction, learned removal of

Good to know, thanks. Crawling certainly is easier, made especially so using tools like JSoup/BeautifulSoup, but areas like gist extraction, learned removal of navigational elements, the benefits of a truly massive IDF for terms, phrase extraction, date extraction etc all seem beyond the realm of a startup. I have written these pieces and know it can be a real time sync. I'm curious about your drag/drop ranking tool. Does this create a static or query dependent ranking boost? In my experience, the former is easy, but not very effective, the latter is much more difficult because of stemming, synonyms, partial match etc. I had to look up IMNSHO :)