4 ms·

Ask HN: How did you learn about search engines/text processing ?

46 points by yr 16y ago

Any good videos/books/code ?

probably 16y ago
Text Processing in Python: http://gnosis.cx/TPiP/ http://gnosis.cx/TPiP/
tdmackey 16y ago
Introduction to Information Retrieval by Manning et al is a great text on the subject: http://nlp.stanford.edu/IR-book/information-retrieval-book.html http://nlp.stanford.edu/IR-book/information-retrieval-book.h...
deleted 16y ago
[deleted]
dejv 16y ago
You can take a look on Managing Gigabytes (http://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia-Information/dp/1558605703/ref=sr_1_1?ie=UTF8&s=books&qid=1272964874&sr=8-1 http://www.amazon.com/Managing-Gigabytes-Compressing-Multime...) It is nice book, but might be little bit outdated.
dhotson 16y ago
The basic data structure behind most text databases and search engines is an inverted index. Basically it's a map of words to a list of documents containing that word. eg {"hello": [1, 2], "world": [1,3,4], ...} (the numbers are document id's) So for example, the word 'hello' occurs in documents 1 and 2. 'world' occurs in documents 1,3 and 4. Doing boolean queries is also really easy with an inverted index. You basically get the document set for each word and then do a union on the sets for an OR query.. or an intersection to do an AND query. Pretty cool right?
- _delirium 16y ago
  The convention of calling this the "inverted" index never ceases to trip me up whenever I see it, since when it comes to IR, I always think of a normal index as being a word->locations mapping, like the index in the back of a book.
  - ctd 16y ago
    Heh, just happened to me, too. Have to remember we're in a technical context here. :)
grrrr 16y ago
The book by Manning (freely available online) has already been recommended. I would start with this. In addition there are a wealth of online video lectures that may inspire you: http://www.datawrangling.com/hidden-video-courses-in-math-science-and-engineering http://www.datawrangling.com/hidden-video-courses-in-math-sc... and http://videolectures.net/mlss04_hofmann_irtm/ http://videolectures.net/mlss04_hofmann_irtm/ and http://videolectures.net/Top/Computer_Science/ http://videolectures.net/Top/Computer_Science/ In so far as search engines go it's certainly worth playing around with Lucene. It's well implemented and you'll learn a lot of what really matters when it comes to indexing and retrieval. For the text processing (classification, data extraction) side It may also be worth brushing up on your stats (a good excuse to learn R) and checking out Mahout http://lucene.apache.org/mahout/ http://lucene.apache.org/mahout/
- arethuza 16y ago
  It is pretty nice that the different implementations of Lucene all use the same index file formats. There are some pretty nice tools to go with Lucene - I've used Luke quite a bit: http://code.google.com/p/luke/ http://code.google.com/p/luke/
DrJokepu 16y ago
Search and Text Processing course at university. Unfortunately I can't find anything related amongst MIT's online course materials.
uggedal 16y ago
For a high level overview I'd recommend Tim Bray's On Search series: http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTOC http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTO...
- deleted 16y ago
  [deleted]
- pasbesoin 16y ago
  I mentioned this myself -- I deleted the comment when I noticed your prior comment. Tim was nice enough to reply to my email query some years ago and point me to this (already written). It's not comprehensive, but it was helpful. I guess I'm mostly adding this comment to attest to the generosity inherent in the sharing of such information. Aka, "thanks".
rmc00 16y ago
I would echo the recommendations for Introduction to Information Retrieval. If you want something with the same concepts but a little less math, I liked Search Engine: Information Retrieval in Practice by Bruce Croft, et al as well. If you happen to be a python programmer, Natural Language Processing with Python by Steven Bird has some great examples of text processing.
gtani 16y ago
Whatever web app framework you favor, there should be plugins for SOLR and sphinx that make fulltext indexing with reasonable defaults pretty easy. i.e. for rails thinking sphinx. I used to use acts_as_solr (I think a lot of people use sunspot now, and Xapian). Play with a database or docs in a filesystem, do deltas of SOLR and sphinx, changing parameters like stopwords, token separators, stemmers, UTF-8 and ISO-Latin to ASCII mappings. See if you can get decent precision/recall metrics. There's quite a few degrees of freedom, depending on the database. http://www.computationalmedicine.org/challenge/cmcChallengeDetails.pdf http://www.computationalmedicine.org/challenge/cmcChallengeD... http://stackoverflow.com/questions/tagged/sphinx http://stackoverflow.com/questions/tagged/sphinx
vlad 16y ago
I'm literally in the library right now, working on my last homework for a Search Engines course taught by Distinguished Professor Bruce Croft. We're using his new book, Search Engines: Information Retrieval in Practice. You can find the slides that accompany the book here: http://www.search-engines-book.com http://www.search-engines-book.com - Slides, Data Sets http://www.pearsonhighered.com/croft1epreview/toc.html http://www.pearsonhighered.com/croft1epreview/toc.html - Book Table of Contents The book expands on the slides, as well as includes homework problems, some requiring the use or modifications of the open-source Galago Search Toolkit. http://www.galagosearch.org/quick-start.html http://www.galagosearch.org/quick-start.html
gregschlom 16y ago
I must cite Programming Collective Intelligence from Toby Segaram (http://oreilly.com/catalog/9780596529321 http://oreilly.com/catalog/9780596529321). Altough not entirely focused on search engines, it's an awesome book for anyone who wants to get their hands on some of the most useful algorithms for web apps, without having to deal with the math. I downloaded a torrent version, then bought the paperback version straight after.
- imp 16y ago
  I agree, PCI is a really cool book. All you need is some basic Python knowledge and it takes you through so many actual data mining examples. I used some of that code to analyze data using Amazon Elastic MapReduce, and got results that I thought would have taken an actual CS degree to get.
ghotli 16y ago
I've been deep into building a geocoder the past month. While we may get rid of Solr eventually, it was a great foot in the door to information retrieval. It helps that I have a problem to solve and a deadline, so I'm motivated to read and work through these books. These three texts have been very helpful. The last book is an excellent overview of text processing and some real world problems you may encounter writing your search engine. Solr 1.4 Enterprise Search Server http://www.amazon.com/Solr-1-4-Enterprise-Search-Server/dp/1847195881 http://www.amazon.com/Solr-1-4-Enterprise-Search-Server/dp/1... Programming Collective Intelligence http://www.amazon.com/Programming-Collective-Intelligence-Building-Applications/dp/0596529325 http://www.amazon.com/Programming-Collective-Intelligence-Bu... Building Search Applications: Lucene, LingPipe, and Gate http://www.amazon.com/Building-Search-Applications-Lucene-LingPipe/dp/0615204252 http://www.amazon.com/Building-Search-Applications-Lucene-Li...
deleted 16y ago
[deleted]
keefe 16y ago
downloading lucene is a nice place to start if you're a java person
jacquesm 16y ago
By building a small search engine. It took about 4 months and it was definitely worth it. Some of the problems that seem simple at first glance were terribly hard (such as reliably separating out the body text of a web page), and some that I thought would be hard turned out to be relatively easy (the actual index). It was a lot of fun, even if when I started out I was already fairly sure that I would not have the stamina nor the funds to commercialize it but as a learning experience it was great.

46 points by yr 16y ago

Any good videos/books/code ?

probably 16y ago

Text Processing in Python: http://gnosis.cx/TPiP/ http://gnosis.cx/TPiP/

tdmackey 16y ago

Introduction to Information Retrieval by Manning et al is a great text on the subject: http://nlp.stanford.edu/IR-book/information-retrieval-book.html http://nlp.stanford.edu/IR-book/information-retrieval-book.h...

deleted 16y ago

[deleted]

dejv 16y ago

You can take a look on Managing Gigabytes (http://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia-Information/dp/1558605703/ref=sr_1_1?ie=UTF8&s=books&qid=1272964874&sr=8-1 http://www.amazon.com/Managing-Gigabytes-Compressing-Multime...) It is nice book, but might be little bit outdated.

dhotson 16y ago

The basic data structure behind most text databases and search engines is an inverted index. Basically it's a map of words to a list of documents containing that word. eg {"hello": [1, 2], "world": [1,3,4], ...} (the numbers are document id's) So for example, the word 'hello' occurs in documents 1 and 2. 'world' occurs in documents 1,3 and 4. Doing boolean queries is also really easy with an inverted index. You basically get the document set for each word and then do a union on the sets for an OR query.. or an intersection to do an AND query. Pretty cool right?

_delirium 16y ago
The convention of calling this the "inverted" index never ceases to trip me up whenever I see it, since when it comes to IR, I always think of a normal index as being a word->locations mapping, like the index in the back of a book.
- ctd 16y ago
  Heh, just happened to me, too. Have to remember we're in a technical context here. :)

grrrr 16y ago

The book by Manning (freely available online) has already been recommended. I would start with this. In addition there are a wealth of online video lectures that may inspire you: http://www.datawrangling.com/hidden-video-courses-in-math-science-and-engineering http://www.datawrangling.com/hidden-video-courses-in-math-sc... and http://videolectures.net/mlss04_hofmann_irtm/ http://videolectures.net/mlss04_hofmann_irtm/ and http://videolectures.net/Top/Computer_Science/ http://videolectures.net/Top/Computer_Science/ In so far as search engines go it's certainly worth playing around with Lucene. It's well implemented and you'll learn a lot of what really matters when it comes to indexing and retrieval. For the text processing (classification, data extraction) side It may also be worth brushing up on your stats (a good excuse to learn R) and checking out Mahout http://lucene.apache.org/mahout/ http://lucene.apache.org/mahout/

arethuza 16y ago
It is pretty nice that the different implementations of Lucene all use the same index file formats. There are some pretty nice tools to go with Lucene - I've used Luke quite a bit: http://code.google.com/p/luke/ http://code.google.com/p/luke/

DrJokepu 16y ago

Search and Text Processing course at university. Unfortunately I can't find anything related amongst MIT's online course materials.

uggedal 16y ago

For a high level overview I'd recommend Tim Bray's On Search series: http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTOC http://www.tbray.org/ongoing/When/200x/2003/07/30/OnSearchTO...

deleted 16y ago
[deleted]
pasbesoin 16y ago
I mentioned this myself -- I deleted the comment when I noticed your prior comment. Tim was nice enough to reply to my email query some years ago and point me to this (already written). It's not comprehensive, but it was helpful. I guess I'm mostly adding this comment to attest to the generosity inherent in the sharing of such information. Aka, "thanks".

rmc00 16y ago

I would echo the recommendations for Introduction to Information Retrieval. If you want something with the same concepts but a little less math, I liked Search Engine: Information Retrieval in Practice by Bruce Croft, et al as well. If you happen to be a python programmer, Natural Language Processing with Python by Steven Bird has some great examples of text processing.

gtani 16y ago

Whatever web app framework you favor, there should be plugins for SOLR and sphinx that make fulltext indexing with reasonable defaults pretty easy. i.e. for rails thinking sphinx. I used to use acts_as_solr (I think a lot of people use sunspot now, and Xapian). Play with a database or docs in a filesystem, do deltas of SOLR and sphinx, changing parameters like stopwords, token separators, stemmers, UTF-8 and ISO-Latin to ASCII mappings. See if you can get decent precision/recall metrics. There's quite a few degrees of freedom, depending on the database. http://www.computationalmedicine.org/challenge/cmcChallengeDetails.pdf http://www.computationalmedicine.org/challenge/cmcChallengeD... http://stackoverflow.com/questions/tagged/sphinx http://stackoverflow.com/questions/tagged/sphinx

vlad 16y ago

I'm literally in the library right now, working on my last homework for a Search Engines course taught by Distinguished Professor Bruce Croft. We're using his new book, Search Engines: Information Retrieval in Practice. You can find the slides that accompany the book here: http://www.search-engines-book.com http://www.search-engines-book.com - Slides, Data Sets http://www.pearsonhighered.com/croft1epreview/toc.html http://www.pearsonhighered.com/croft1epreview/toc.html - Book Table of Contents The book expands on the slides, as well as includes homework problems, some requiring the use or modifications of the open-source Galago Search Toolkit. http://www.galagosearch.org/quick-start.html http://www.galagosearch.org/quick-start.html

gregschlom 16y ago

I must cite Programming Collective Intelligence from Toby Segaram (http://oreilly.com/catalog/9780596529321 http://oreilly.com/catalog/9780596529321). Altough not entirely focused on search engines, it's an awesome book for anyone who wants to get their hands on some of the most useful algorithms for web apps, without having to deal with the math. I downloaded a torrent version, then bought the paperback version straight after.

imp 16y ago
I agree, PCI is a really cool book. All you need is some basic Python knowledge and it takes you through so many actual data mining examples. I used some of that code to analyze data using Amazon Elastic MapReduce, and got results that I thought would have taken an actual CS degree to get.

ghotli 16y ago

I've been deep into building a geocoder the past month. While we may get rid of Solr eventually, it was a great foot in the door to information retrieval. It helps that I have a problem to solve and a deadline, so I'm motivated to read and work through these books. These three texts have been very helpful. The last book is an excellent overview of text processing and some real world problems you may encounter writing your search engine. Solr 1.4 Enterprise Search Server http://www.amazon.com/Solr-1-4-Enterprise-Search-Server/dp/1847195881 http://www.amazon.com/Solr-1-4-Enterprise-Search-Server/dp/1... Programming Collective Intelligence http://www.amazon.com/Programming-Collective-Intelligence-Building-Applications/dp/0596529325 http://www.amazon.com/Programming-Collective-Intelligence-Bu... Building Search Applications: Lucene, LingPipe, and Gate http://www.amazon.com/Building-Search-Applications-Lucene-LingPipe/dp/0615204252 http://www.amazon.com/Building-Search-Applications-Lucene-Li...

deleted 16y ago

[deleted]

keefe 16y ago

downloading lucene is a nice place to start if you're a java person

jacquesm 16y ago

By building a small search engine. It took about 4 months and it was definitely worth it. Some of the problems that seem simple at first glance were terribly hard (such as reliably separating out the body text of a web page), and some that I thought would be hard turned out to be relatively easy (the actual index). It was a lot of fun, even if when I started out I was already fairly sure that I would not have the stamina nor the funds to commercialize it but as a learning experience it was great.