3 ms·

Hey, thanks for the article. I'm still going through it now. Would it work for my scenario? I'm full text indexing large legal documents - 50 to even 300 pages.

by NDizzle 12y ago

Hey, thanks for the article. I'm still going through it now. Would it work for my scenario? I'm full text indexing large legal documents - 50 to even 300 pages. What I've been doing is just throwing the entire document (OCR) into a big mysql (Mariadb 10) LONGTEXT columns. Then I'm using a JDBC river to shoot those over to elasticsearch. I don't fully understand elasticsearch, because like you said it's insane. The query that I build to perform a simple search (think a google-like search box with a go button) is about 100 lines. Should I try out postgres full text search for my purposes? I guess I could change to indexing page by page, and then I could group on the document hash to stamp out duplicates. That would also get me a list of page numbers where the results were. I'm not sure if I could get the snippets though. (What I call a snippet is the word surrounding the match - I'm returning that to datatables so that the user can search within their search to further refine the search.) Here's what my search (and results) looks like now: http://imgur.com/Fjckva6 http://imgur.com/Fjckva6 As you can tell, I'm not too familiar with elasticsearch (A shameful offset input!) and I'm more comfortable in a RDBMS.

rachbelaid 12y ago
Which languages? English only? Anykind of fuzzy search required? I don't have experience with big data set but you may to read about GIST/GIN to pick the right index in your case (probably GIST) But from what I saw when I prepared this post, some people are getting some decent performance with full-text search on dataset like the size wikipedia. https://wiki.postgresql.org/images/2/25/Full-text_search_in_PostgreSQL_in_milliseconds-extended-version.pdf https://wiki.postgresql.org/images/2/25/Full-text_search_in_... I think that doesn't take long to try Postgres FTS if you are already using it but it may require more investment if you need to move from MySQL to PG.
- NDizzle 12y ago
  Thanks for the reply. English only. The only fuzzyness I've told my users about is "phrase here"~20 which would search for the given phrase within 20 words of each other. That helps for the imperfect OCR'd documents. A lot of the stuff we're indexing is best-guess OCR so it's nowhere near perfect. They are boring agreement documents that people have scribbled all over (setting up the redactions) so fuzzy is somewhat interesting to have. But it's not fuzzy in the sense of a typo, it's fuzzy in the sense that it's OCR'd garbage.

by NDizzle 12y ago

rachbelaid 12y ago

Which languages? English only? Anykind of fuzzy search required? I don't have experience with big data set but you may to read about GIST/GIN to pick the right index in your case (probably GIST) But from what I saw when I prepared this post, some people are getting some decent performance with full-text search on dataset like the size wikipedia. https://wiki.postgresql.org/images/2/25/Full-text_search_in_PostgreSQL_in_milliseconds-extended-version.pdf https://wiki.postgresql.org/images/2/25/Full-text_search_in_... I think that doesn't take long to try Postgres FTS if you are already using it but it may require more investment if you need to move from MySQL to PG.

NDizzle 12y ago
Thanks for the reply. English only. The only fuzzyness I've told my users about is "phrase here"~20 which would search for the given phrase within 20 words of each other. That helps for the imperfect OCR'd documents. A lot of the stuff we're indexing is best-guess OCR so it's nowhere near perfect. They are boring agreement documents that people have scribbled all over (setting up the redactions) so fuzzy is somewhat interesting to have. But it's not fuzzy in the sense of a typo, it's fuzzy in the sense that it's OCR'd garbage.