4 ms·
Can you talk more about the specific? What kind of parsers did you guys use? How about storage? How often did you update pages?
by doh 6y ago
Can you talk more about the specific? What kind of parsers did you guys use? How about storage? How often did you update pages?
- jd20 6y agoYou should check out Manning's "Introduction to Information Retrieval", it has far more detail about web crawler architecture than I can write in a post, and served as a blueprint for much of Applebot's early design decisions.
- giu 6y agoNice, thanks for the recommendation! The book is freely available online at https://nlp.stanford.edu/IR-book/information-retrieval-book.html https://nlp.stanford.edu/IR-book/information-retrieval-book....