4 ms·

Quick context: I built this from the Open Library (Internet Archive) books dataset (https://openlibrary.org/ https://openlibrary.org/), as a follow up to this c

by jabo 6y ago

Quick context: I built this from the Open Library (Internet Archive) books dataset (https://openlibrary.org/ https://openlibrary.org/), as a follow up to this comment on the GoodReads post earlier today: https://news.ycombinator.com/item?id=25408186 https://news.ycombinator.com/item?id=25408186 Thank you @mekarpeles for helping me get access to the data quickly and giving me pointers about the schema. I should add - I built this in about 12 hours as a weekend project, so there might be some lurking issues. > Details about the Tech Stack: The dataset has ~28.6 million books and is indexed on Typesense [1], an open source alternative to Algolia/ElasticSearch that a friend and I are working on. The UI was built using the Typesense adapter for InstantSearch.js [2] and is a static site bundled using ParcelJS. The app is hosted on S3, with CloudFront for a CDN. The search backend is powered by a geo-distributed 3-node Typesense cluster running on Typesense Cloud [3], with nodes in Oregon, Frankfurt and Mumbai. Here's the source code: https://github.com/typesense/showcase-books-search https://github.com/typesense/showcase-books-search [1] https://github.com/typesense/typesense https://github.com/typesense/typesense [2] https://github.com/typesense/typesense-instantsearch-adapter https://github.com/typesense/typesense-instantsearch-adapter [3] https://cloud.typesense.org https://cloud.typesense.org

tmikaeld 6y ago
The typesense performance is really impressive, seems at par with the (also) impressive meilisearch. I'm going to try the free tier of typesense right now, perfect for my current use-case of site-search. How large is the dataset for the books? How large nodes are needed?
- jabo 6y ago
  Thank you! The OpenLibrary dataset has ~28M records, takes up 6.8GB on disk and 14.3GB in RAM when indexed in Typesense. Each node has 4vCPUs. Took me ~3 hours to index these 28M records. Could have probably been done in ~1.5hrs - most of the indexing time was because of cross-region latency between the 3 geo-distributed nodes in Oregon, Mumbai and Frankfurt.
  - tmikaeld 6y ago
    That's a lot of text to index, impressive! Is the indexed size 6.8GB or 14.3GB?
    - jabo 6y ago
      Thank you! The indexed size is 14.3GB.
- binarymax 6y ago
  I'm curious, what do you feel meilisearch offers that is better than elasticsearch or solr?
forgot-pw 6y ago
Someone really delivers after saying, "I could do that in a weekend." :) Well done!
- jabo 6y ago
  Thank you!
jillesvangurp 6y ago
I did a few quick tests. There are some issues with ranking and duplicates that I see. I typed "Neal Stephenson" and got lots of duplicates for his latest novels. Cleaning up the data might be necessary to fix this. It seems biased towards newer/recent titles. The ranking is also a bit tricky. If you type "Orwell" you get top results about George Orwell, rather than the book that made him famous (1984). I suspect this will be an issue with many literature works that will have a lot of books talking about these books. Spelling correction is kind of working but surfaces a lot of noise as well. E.g. Neal Stevenson just surfaces a lot of titles and authors with either Neal or Stevenson in them. But impressively it did find one Neal Stephenson title. Building good search and ranking is hard. So, not bad for a 12 hour effort.
- jabo 6y ago
  I’m currently ranking by text match score and publish date and some of the records had publish dates that were hard to parse by Date.parse(), so I just set those to 1970: https://github.com/typesense/showcase-books-search/blob/2d38ef8a49927b15d65d91adb035f166f664c801/scripts/indexer/transform_dataset.rb#L15 https://github.com/typesense/showcase-books-search/blob/2d38... I didn’t seem to find a popularity metric in the dataset, which would have solved the issue you pointed out about literary work having books written about them. I’ve left the typo correction settings to be a little loose, so need to reduce its aggressiveness. I’ll fix that.
allenleein 6y ago
Pretty good for a 12 hours hacking!
mekarpeles 6y ago
Jabo, I love to see this, more access for the book community -- especially how quickly it came together. Nice work Jabo and excited to see Typesense in action. Open Library gladly & thankfully tweeted its support of your hard work: https://twitter.com/openlibrary/status/1338531822536310789 https://twitter.com/openlibrary/status/1338531822536310789 In every intention of being additive to Jabo's work, thought I'd share a few resources folks may not be aware of: 1. Open Library's Full-text Search Major hat tip to the Internet Archive's Giovanni Damiola for this one: Folks may also appreciate the ability to full-text search across 4M of the Internet Archive's books on Open Library: https://blog.openlibrary.org/2018/07/14/search-full-text-within-4m-books https://blog.openlibrary.org/2018/07/14/search-full-text-wit.... You can try it directly here: http://openlibrary.org/search/inside?q=thanks%20for%20all%20the%20fish http://openlibrary.org/search/inside?q=thanks%20for%20all%20... As per usual, nearly all Open Library urls are themselves APIs, e.g.: http://openlibrary.org/search/inside.json?q=thanks%20for%20all%20the%20fish http://openlibrary.org/search/inside.json?q=thanks%20for%20a... 2. Open Library's search API By adding .json to our search endpoint, you can use Open Library's json search endpoint (which also contains bookcover links). You can also use this to fetch books by ISBN: http://openlibrary.org/search.json?q=isbn:9781400067824 http://openlibrary.org/search.json?q=isbn:9781400067824 3. Book Cover Service (has issues) We need to improve our cover service, it's definitely been neglected, but the cover IDs from our data dump, solr search index, and book APIs can be plugged in to our cover store to point to or fetch book covers. We don't yet have a current data bulk dump of covers available (it's on the roadmap) but if you need "all" the covers, please email us openlibrary@archive.org instead of hitting covers 1-by-1. https://openlibrary.org/dev/docs/api/covers https://openlibrary.org/dev/docs/api/covers 4. The new (unannounced) Open Library Explorer I'm honestly kind of hoping no one reads this far into the comments because we haven't announced this one yet and I'm sure there are major performance issues (it's beta), but cdrini recently added dewey + library of congress classifications to our search index... Making this masterpiece possible: https://openlibrary.org/explore https://openlibrary.org/explore The ability to browse Open Library digitally, with all the glory of a physical library.
- jabo 6y ago
  Thanks @mekarpeles!
jabo 6y ago
Also, forgot to mention this earlier: Please do contribute to OpenLibrary in any way you can. Here are some ways how: https://news.ycombinator.com/item?id=25408744 https://news.ycombinator.com/item?id=25408744

by jabo 6y ago

tmikaeld 6y ago

The typesense performance is really impressive, seems at par with the (also) impressive meilisearch. I'm going to try the free tier of typesense right now, perfect for my current use-case of site-search. How large is the dataset for the books? How large nodes are needed?

jabo 6y ago
Thank you! The OpenLibrary dataset has ~28M records, takes up 6.8GB on disk and 14.3GB in RAM when indexed in Typesense. Each node has 4vCPUs. Took me ~3 hours to index these 28M records. Could have probably been done in ~1.5hrs - most of the indexing time was because of cross-region latency between the 3 geo-distributed nodes in Oregon, Mumbai and Frankfurt.
- tmikaeld 6y ago
  That's a lot of text to index, impressive! Is the indexed size 6.8GB or 14.3GB?
  - jabo 6y ago
    Thank you! The indexed size is 14.3GB.
binarymax 6y ago
I'm curious, what do you feel meilisearch offers that is better than elasticsearch or solr?

forgot-pw 6y ago

Someone really delivers after saying, "I could do that in a weekend." :) Well done!

jabo 6y ago
Thank you!

jillesvangurp 6y ago

I did a few quick tests. There are some issues with ranking and duplicates that I see. I typed "Neal Stephenson" and got lots of duplicates for his latest novels. Cleaning up the data might be necessary to fix this. It seems biased towards newer/recent titles. The ranking is also a bit tricky. If you type "Orwell" you get top results about George Orwell, rather than the book that made him famous (1984). I suspect this will be an issue with many literature works that will have a lot of books talking about these books. Spelling correction is kind of working but surfaces a lot of noise as well. E.g. Neal Stevenson just surfaces a lot of titles and authors with either Neal or Stevenson in them. But impressively it did find one Neal Stephenson title. Building good search and ranking is hard. So, not bad for a 12 hour effort.

jabo 6y ago
I’m currently ranking by text match score and publish date and some of the records had publish dates that were hard to parse by Date.parse(), so I just set those to 1970: https://github.com/typesense/showcase-books-search/blob/2d38ef8a49927b15d65d91adb035f166f664c801/scripts/indexer/transform_dataset.rb#L15 https://github.com/typesense/showcase-books-search/blob/2d38... I didn’t seem to find a popularity metric in the dataset, which would have solved the issue you pointed out about literary work having books written about them. I’ve left the typo correction settings to be a little loose, so need to reduce its aggressiveness. I’ll fix that.

allenleein 6y ago

Pretty good for a 12 hours hacking!

mekarpeles 6y ago

Jabo, I love to see this, more access for the book community -- especially how quickly it came together. Nice work Jabo and excited to see Typesense in action. Open Library gladly & thankfully tweeted its support of your hard work: https://twitter.com/openlibrary/status/1338531822536310789 https://twitter.com/openlibrary/status/1338531822536310789 In every intention of being additive to Jabo's work, thought I'd share a few resources folks may not be aware of: 1. Open Library's Full-text Search Major hat tip to the Internet Archive's Giovanni Damiola for this one: Folks may also appreciate the ability to full-text search across 4M of the Internet Archive's books on Open Library: https://blog.openlibrary.org/2018/07/14/search-full-text-within-4m-books https://blog.openlibrary.org/2018/07/14/search-full-text-wit.... You can try it directly here: http://openlibrary.org/search/inside?q=thanks%20for%20all%20the%20fish http://openlibrary.org/search/inside?q=thanks%20for%20all%20... As per usual, nearly all Open Library urls are themselves APIs, e.g.: http://openlibrary.org/search/inside.json?q=thanks%20for%20all%20the%20fish http://openlibrary.org/search/inside.json?q=thanks%20for%20a... 2. Open Library's search API By adding .json to our search endpoint, you can use Open Library's json search endpoint (which also contains bookcover links). You can also use this to fetch books by ISBN: http://openlibrary.org/search.json?q=isbn:9781400067824 http://openlibrary.org/search.json?q=isbn:9781400067824 3. Book Cover Service (has issues) We need to improve our cover service, it's definitely been neglected, but the cover IDs from our data dump, solr search index, and book APIs can be plugged in to our cover store to point to or fetch book covers. We don't yet have a current data bulk dump of covers available (it's on the roadmap) but if you need "all" the covers, please email us openlibrary@archive.org instead of hitting covers 1-by-1. https://openlibrary.org/dev/docs/api/covers https://openlibrary.org/dev/docs/api/covers 4. The new (unannounced) Open Library Explorer I'm honestly kind of hoping no one reads this far into the comments because we haven't announced this one yet and I'm sure there are major performance issues (it's beta), but cdrini recently added dewey + library of congress classifications to our search index... Making this masterpiece possible: https://openlibrary.org/explore https://openlibrary.org/explore The ability to browse Open Library digitally, with all the glory of a physical library.

jabo 6y ago
Thanks @mekarpeles!

jabo 6y ago

Also, forgot to mention this earlier: Please do contribute to OpenLibrary in any way you can. Here are some ways how: https://news.ycombinator.com/item?id=25408744 https://news.ycombinator.com/item?id=25408744