3 ms·
What is being used as a crawler and is it integrated with Seekstorm? The same article referenced here https://deephn.org/?q=how+to+be+productive&filter=%7B%22h
by freediver 6y ago
What is being used as a crawler and is it integrated with Seekstorm?
The same article referenced here https://deephn.org/?q=how+to+be+productive&filter=%7B%22hashtags%22%3A%5B%22CPU%22%5D%7D https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... contains the phrase 'well-defined'
Any idea why doesn't the article surface when searching for this ?
- wolfgarbe 6y agoThe crawler is a part of SeekStorm. "Well-defined": Just a guess: We are doing key text extraction, i.e. we try not to index boilerplate stuff an and menu items. As "well-defined" is within a short list item, it might be accidentally skipped. So, that is not yet perfect, and considering the diversity in web page structure it probably never will. But we will try to improve.
- freediver 6y agoYes, if the preview is the representation of what you have indexed, then half of that article is missing. You may have identified the weakest link in your stack - crawler/extractor (which is notoriously hard to do, would be good if you provided more detail eg. do you use headless browser or simple GET request, do you crawl PDFs etc). Little use of the advanced stack on top of it, if the data does not end in the index in the first place. Hope you provide an update on this in the future. I'd probably sign up for a plan.
- wolfgarbe 6y agoNo, the preview is NOT the representation of what we have indexed. The preview ist limited to about 200 words to be compliant with fair use legislation. Indexing is limited to 1 MB per document.