3 ms·

There's 13 search engines in a dozen if you only want book title or author. What's lacking is a search index of the content of e-books. Something that will soon

There's 13 search engines in a dozen if you only want book title or author. What's lacking is a search index of the content of e-books. Something that will soon be incredibly important in the face of generative AI. Somebody here on HN told me it only takes a laptop to index the content of millions of books, while other people say the scope is almost impossible. Is there any project working on this?

bt1a 3y ago
Perhaps the initial creation of the index is indeed something that an average laptop could accomplish, but I'd imagine that frequently updating the index and serving requests against it would be compute-intensive. I have nothing to back this up but speculation. Would love to learn more!
CWuestefeld 3y ago
I believe that Calibre, the popular and free ebook management tool, now supports indexing the content all books in your library.
myco_logic 3y ago
Depends on how beefy that laptop is... I've been doing some local LLM stuff at work recently, and even with the amazing advances in quantization lately, doing that kind of stuff on a ThinkPad is feasible, but still strongly inferior to just renting out a VPS with a couple 4090/H100s for several hours. The biggest thing with summarizing stuff is that most local LLM models often don't have very big context-windows, so they have trouble with larger texts like even a short Vonnegut novel (I was just testing em' with summarizing GitHub issues, and even with a 16k token context window they still sometimes struggle if there are a lot of comments). There are probably smarter people than I who could get this working on a Raspberry Pi though... ;)
dmotz 3y ago
I have a side project that aims to organize your ebook highlight collections with on-device semantic search. [1] Right now it only indexes your own content but I'd like to add a mode that allows you to share your collection and let others find relevant ideas via semantic search -- a discovery platform for ideas found in books. It's open source if you want a sense of how it works now. [2] [1] https://emdash.ai/ https://emdash.ai/ [2] https://github.com/dmotz/emdash https://github.com/dmotz/emdash
dredmorbius 3y ago
The size of the index is far more dependent on the search text than the number of searched items. I believe Google documented some of this in its early days, noting that a search index returns the relevant metadata matching a specific query. The query space itself is largely based on both raw keywords and tuples (2- or 3-word ngrams if memory serves, though I'm hazy on this), the latter meeting some minimum frequency requirement. Longer search terms can be constructed from shorter ngrams. A typical advanced native-tongue English vocabulary is about 40,000 words. An expansive dictionary might contain fewer than 250,000 words, including obsolete ones. Mapping a vocabulary to works citing those words is relatively straightforward. Ngrams experience combinatorial expansion, but are still a reasonably constrained space. And we now have well over a quarter-century's experience indexing written content at Web scale. A laptop could probably make a decent cut at providing a useful index of many millions of books, though you'd probably want a somewhat larger system for a more comprehensive index, in particular to rank-index the search space, which is probably the more considerable challenge.
- carlosjobim 3y ago
  That was a great reply, and made me understand how these things work a lot better. Thank you!
  - dredmorbius 3y ago
    The key concept here is the inverted index: <https://en.wikipedia.org/wiki/Inverted_index https://en.wikipedia.org/wiki/Inverted_index>

bt1a 3y ago

Perhaps the initial creation of the index is indeed something that an average laptop could accomplish, but I'd imagine that frequently updating the index and serving requests against it would be compute-intensive. I have nothing to back this up but speculation. Would love to learn more!

CWuestefeld 3y ago

I believe that Calibre, the popular and free ebook management tool, now supports indexing the content all books in your library.

myco_logic 3y ago

Depends on how beefy that laptop is... I've been doing some local LLM stuff at work recently, and even with the amazing advances in quantization lately, doing that kind of stuff on a ThinkPad is feasible, but still strongly inferior to just renting out a VPS with a couple 4090/H100s for several hours. The biggest thing with summarizing stuff is that most local LLM models often don't have very big context-windows, so they have trouble with larger texts like even a short Vonnegut novel (I was just testing em' with summarizing GitHub issues, and even with a 16k token context window they still sometimes struggle if there are a lot of comments). There are probably smarter people than I who could get this working on a Raspberry Pi though... ;)

dmotz 3y ago

I have a side project that aims to organize your ebook highlight collections with on-device semantic search. [1] Right now it only indexes your own content but I'd like to add a mode that allows you to share your collection and let others find relevant ideas via semantic search -- a discovery platform for ideas found in books. It's open source if you want a sense of how it works now. [2] [1] https://emdash.ai/ https://emdash.ai/ [2] https://github.com/dmotz/emdash https://github.com/dmotz/emdash

dredmorbius 3y ago

The size of the index is far more dependent on the search text than the number of searched items. I believe Google documented some of this in its early days, noting that a search index returns the relevant metadata matching a specific query. The query space itself is largely based on both raw keywords and tuples (2- or 3-word ngrams if memory serves, though I'm hazy on this), the latter meeting some minimum frequency requirement. Longer search terms can be constructed from shorter ngrams. A typical advanced native-tongue English vocabulary is about 40,000 words. An expansive dictionary might contain fewer than 250,000 words, including obsolete ones. Mapping a vocabulary to works citing those words is relatively straightforward. Ngrams experience combinatorial expansion, but are still a reasonably constrained space. And we now have well over a quarter-century's experience indexing written content at Web scale. A laptop could probably make a decent cut at providing a useful index of many millions of books, though you'd probably want a somewhat larger system for a more comprehensive index, in particular to rank-index the search space, which is probably the more considerable challenge.

carlosjobim 3y ago
That was a great reply, and made me understand how these things work a lot better. Thank you!
- dredmorbius 3y ago
  The key concept here is the inverted index: <https://en.wikipedia.org/wiki/Inverted_index https://en.wikipedia.org/wiki/Inverted_index>