15 ms·
Ask HN: Has anybody built search on top of Anna's Archive?
Wouldn't this basically give us Google Books and searchable Scihub at the same time?
What would it cost?
- ggm 1y agoYou must mean free text search and page level return, because it already has full metadata indexing. The thing is AA doesn't hold the texts. They're disputable IPR and even a derived work would be a legal target.
- carlosjobim 1y ago> a derived work would be a legal target. Why would it? Google isn't prosecuted for indexing the web.
- 1970-01-01 1y agoOh it certainly is. https://www.reuters.com/sustainability/boards-policy-regulation/google-says-it-will-appeal-online-search-antitrust-decision-2025-05-31/ https://www.reuters.com/sustainability/boards-policy-regulat...
- trollbridge 1y agoThat's not prosecution for indexing the web. Google is being treated the same way AT&T was for telephones.
- 1970-01-01 1y agohttps://harvardlawreview.org/print/vol-138/united-states-v-google-llc/#footnote-ref-15 https://harvardlawreview.org/print/vol-138/united-states-v-g...
- bhaney 1y agoHonestly I don't think it would be that costly, but it would take a pretty long time to put together. I have a (few years old) copy of Library Genesis converted to plaintext and it's around 1TB. I think libgen proper was 50-100TB at the time, so we can probably assume that AA (~1PB) would be around 10-20TB when converted to plaintext. You'd probably spend several weeks torrenting a chunk of the archive, converting everything in it to plaintext, deleting the originals, then repeating with a new chunk until you have plaintext versions of everything in the archive. Then indexing all that for full text search would take even more storage and even more time, but still perfectly doable on commodity hardware. The main barriers are going to be reliably extracting plaintext from the myriad of formats in the archive, cleaning up the data, and selecting a decent full text search database (god help you if you pick wrong and decide you want to switch and re-index everything later).
- notpushkin 1y agoI think there’s a couple ways to improve it: 1. There’s a lot of variants of the same book. We only need one for the index. Perhaps for each ISBN, select the format easiest to parse. 2. We can download, convert and index top 100K books first, launch with these, and then continue indexing and adding other books.
- palmfacehn 1y agoThere should be a way to leverage compression when storing multiple editions of the same book.
- bawolff 1y agoFrom a good search perspective though you probably dont want 500 different versions of the same book popping up for a query
- palmfacehn 1y agoAgreed. I would prefer to see a single result for a single title. The option of pursuing different editions should follow from there.
- qingcharles 1y agoAnd without some sort of weighting system, it wouldn't even know which one is the best one to show the user.
- notpushkin 1y agoWe’ll also need to consider that some versions might be easier to index even though the user would prefer another version. E.g. if we have a TXT and EPub, we might want to index TXT (if it’s clean enough), but present user with EPub (with formatting and stuff). But it’s not a huge problem actually: just link to the search page instead and let the user decide what they want to download.
- nextos 1y agoAFAIK, Z-Library already does this, to some extent. Basic full-text queries do search inside the body of books and articles. It's a bit smaller than Anna's Archive, as they do host their own collections. From some locations, it's only easy to access through Tor.
- aaron695 1y ago[dead]
- bravesoul2 1y agoThis works in various search engines site:annas-archive.org avacado
- qingcharles 1y agoIt's not exactly clear, but OP is asking about indexing the content of all the documents, not the metadata (e.g. titles etc)
- imdavidsantiago 1y agoAs far as I know, no one has fully implemented full-text search directly over Anna's Archive. Technically it’s feasible with tools like Meilisearch, Elasticsearch, or Lucene, but the main challenges are: Converting all documents (PDFs, EPUBs, etc.) to clean plaintext. Indexing at scale efficiently. Managing potential legal issues. Z-Library does something similar, but it’s smaller in scope and doesn't integrate AA’s full catalog.
- bendangelo 1y agoI’ve done something like this before. Meilisearch will not be viable, because it indexes very slow and it takes up a lot of space. In my experience only Tantivy can index this much data. Check out Lnx.
- sam_lowry_ 1y agoLucene would fo fine as well, I guess. As much as I like the author of Tantivy, it is a toy compared to Lucene.
- _ache_ 1y agoTo manage the legal issues, you just have to put AI on the search. "AI search".
- samuel_shadrach 1y ago[dead]
- DaSexiestAlive 1y agoMebbe easier to just search Amazon or Goodreads. Like site:amazon.ca <query words> as someone has mentioned below. Every book has an ISBN 10 or 13 digit ISBN number to identify them. Unless it's some self-pub/amateur-hour situation by some paranoid prepper living in a faraday-cage-protected cage in Arkansas or Florida it's likely a publication with a title, an author and an ISBN number.
- pigeons 1y agoWhat about pre-1970 books?
- trollbridge 1y agoA self-pub amateur-hour book printed by a paranoid prepper living in a faraday cage is exactly the type of book I'd probably enjoy reading, but I doubt these exist anymore.
- pigeons 1y agoI know, remember Loopanics?
- renegat0x0 1y agoI have found some searche engines, but I do not think they're for Anna's. https://searchthearxiv.com/ https://searchthearxiv.com/ https://refseek.com/ https://refseek.com/ https://arxivxplorer.com/ https://arxivxplorer.com/
- simgt 1y agoRelated question, has Anna's archive been thoroughly filtered for non-copyright-related illegal material? Pedo, terrorism, etc. I've considered downloading a few chunks of it but I'm worried of ending up with content I really don't want to be anywhere near from.
- niux 1y agoHow might you inadvertently download illegal content while searching for legal content?
- lukan 1y agoHe said he wants to download lots of it in general, not specifical. Legit question, if you end up with dark material. I would assume pedo stuff is not really there, but the anarchist cookbook and alike likely will be.
- oguz-ismail 1y ago>I would assume pedo stuff is not really there Search for "lolicon"
- lukan 1y agoWell, I won't. But does it contain just text or real pictures? That would make a big legal difference I assume.
- qingcharles 1y agoIt makes very little legal difference in a lot of jurisdictions. They are considered the same.
- jxjnskkzxxhx 1y agoI thought that was anime pictures...?
- allenleee 1y agoHas anyone explored a different angle — like mapping out the 1,000 most frequently mentioned or cited books (across HN, Substack, Twitter, etc.), then turning their raw content into clean, structured data optimized for LLMs? Imagine curating these into thematic shelves — say, “Bill Gates’ Bookshelf” or “HN Canon” — and building an indie portal where anyone can semantically search across these high-signal texts. Kind of like an AI-searchable personal library of the internet’s favorite books.
- DocTomoe 1y agoWell, there's this: https://hacker-recommended-books.vercel.app/category/0/all-time/page/0/0 https://hacker-recommended-books.vercel.app/category/0/all-t...
- laserstrahl 1y agoThere’s an android app called OpenLip. [1] Description: Openlib is an open source app to download and read books from shadow library (Anna’s Archive). The App Has Built In Reader to Read Books. As Anna’s Archive doesn't have an API, the app works by sending requests to Anna’s Archive and parses the response to objects. The app extracts the mirrors from the responses, downloads the book and stores it in the application's document directory. Note : The app requires VPN to function properly . Without VPN the might show the captcha required page even after completing the captcha Main Features: Trending Books Download And Read Books With In-Built Viewer Supports Epub And Pdf Formats Open Books With Your Favourite Ebooks Reader Filter Books Sort Books [1]: https://f-droid.org/de/packages/com.app.openlib/ https://f-droid.org/de/packages/com.app.openlib/
- petra 1y agoZ-Library has a keyword search. Personally i didn't find it too useful, especially given Google Books exists. It's not easy to create a quality book search engine.
- podgorniy 1y agoThere is a search solution for zipped fb2 files. Not exactly what you need, but it has potential. The project has similar story to Anna's archive. There is 0.5 TB of archived books, and the project creates index of all the books with text, title and aruthor search capabilities, gives html UI for search and reading. On weak machine it takes about 2 hours to build that index. So if you have zipped archives of fb2, you can use the project to create web UI with search for those files. Without need of enough space to unpack all the files. You'll have to translate some russian though to get instructions on how to set it up. https://gitlab.com/opennota/fb2index/-/blob/master/README.ru.md https://gitlab.com/opennota/fb2index/-/blob/master/README.ru...
- Quin-tus 1y agohttps://book-finder.tiiny.site/ https://book-finder.tiiny.site/ More: https://rentry.co/StellaOctangulaIsCool https://rentry.co/StellaOctangulaIsCool
- HelloUsername 1y ago> https://book-finder.tiiny.site/ https://book-finder.tiiny.site/ That just redirects to https://yandex.com/search https://yandex.com/search
- Quin-tus 1y agoWell yeah but with a specific query with which you can search multiple libraries
- xbmcuser 1y agoFacebook did it's ai is trained on it so you can use that.
- teekert 1y agoProbably this was already done at Google, Meta, X and OpenAI, before training their LLMs.
- maartin0 1y agoThere's actually section in the Wikipedia page that explicitly says DeepSeek was trained on it
- net01 1y agoThey did! They conducted a competition https://annas-archive.org/blog/all-isbns-winners.html https://annas-archive.org/blog/all-isbns-winners.html , in which a few submissions exceeded the minimum requirements and implemented a good search tool & visualiser.
- carlosjobim 1y agoHow is this a text search of the books?
- outside1234 1y agoThe original question the poster made was not clear, so this is also an answer to it. It depends on what they meant by "search"
- perdomon 1y agoI think OP was more interested in the ability to text search through the contents. This competition was great and some of the entries were really informative, but none of them included a full text search of the contents of all books.
- carlosjobim 1y agoA functional full text search of the shadow libraries would be massive. It would have a comparable impact on humanity to the impact AI will have. And it's probably not difficult technically. Let's start a project to get this done! Edit: I have had this exact project as my dream for a couple of years, and even experimented a little bit. But I'm not a programmer, so I can only understand theoretically what would be needed for this to work. Anybody with the same dream, send me an e-mail to booksearch@fastmail.com and let's see what we can do to get the ball rolling!
- 1970-01-01 1y agoDon't do it. Just because you can, doesn't mean you should. Do you know if they have anywhere near the legal muscle to push back the flood of legal notices if you did this? Assume it survives because it doesn't have a wide open barn door to the public.
- bethekidyouwant 1y agoIt wouldn’t be called full text search of AA, It would be called full tech search of every book in the world.
- 1970-01-01 1y agoYou are asking a judge to consider that a book is ok to scrape because it's part of a much larger collection of books, perhaps the biggest and best collection, and therefore it's all OK because at scale means good.
- calibas 1y agoGoogle already successfully argued in court that creating an online search index of books constitutes fair use: https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,_Inc https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,....
- 1970-01-01 1y agoThe index isn't the full content. The OP search is about indexing the entire contents of the book for verbatim information retrieval.
- hbartab 1y agoSeeing as OpenAI & Co were trained on torrented books from similar places, I'm sure that ChatGPT provides an adequate search layer on top of Anna's Archive, though it is not as free from confabulations as one might hope for in a search engine. Edit: grammar
- underlines 1y agoyes, every major llm company did it: illegally using annas archive, the pile, common crawl, their own crawl, books2, libgen etc. and embed it into high dimensional space and do next token prediction on it.
- rkoots 1y ago[dead]
- whimsicalism 1y agosmall number of people willing to put in significant engineering hours for something that would be illegal and non-monetizable
- coolThingsFirst 1y agoNo, because you can't avert the legal issues of doing that.
- jimmydoe 1y agoFacebook said they leeched it, and Anna once mentioned a few companies most of them from China paid for it, so I assume the answer is yes someone has the data and very likely built the search, but no one will open it given the legal and reputation risk.
- fazlerocks 1y agoThe indexing costs would be nuts - Anna's Archive is like 200TB+ and growing fast. Even with decent search infra you're looking at serious compute/storage costs. Plus there's the obvious legal stuff that would make this a no-go for most companies with anything to lose. The decentralized thing they're doing probably makes way more sense.
- carlosjobim 1y agoHow serious compute/storage cost? $10 000 per month? $100 000 per month?
- samuel_shadrach 1y ago[dead]
- samuel_shadrach 1y ago[dead]
- happyzappy 1y ago[dead]