6 ms·

I wonder if there are any search engines dedicated to indexing these kinds of libraries. I know there's a decent one just for scihub, but it would be awesome if

I wonder if there are any search engines dedicated to indexing these kinds of libraries. I know there's a decent one just for scihub, but it would be awesome if I could do a Google-style search that returned the contents of books, magazines and journal articles instead of just websites.

delusional 4y ago
Wasn't that what google books was supposed to be?
- voisin 4y ago
  I assume Google abandoned this along with all their earlier mission statements in favour of building another chat app.
  - emanuensis 4y ago
    Supposedly all at Library of Congress (LoC) anyone have experience using it there?
  - b112 4y ago
    I can't imagine a life at google. So much promise, all turned to ash.
    - UkrainianJew 4y ago
      More like an ad company driven by dark patterns that needs to pretend being good and nice and innovative as otherwise it will lose goodwill just like Facebook did.
      - klipt 4y ago
        I think the Google founders were inherently nicer people than the Facebook founder, but their companies ended up converging on advertising because that's really the only way their products could be profitable.
        discordance 4y ago
        Is that really the only way to be profitable though? I believe going public inevitably leads to needing to follow growth, which then eventually corrupts virtuous missions.
  - lupire 4y ago
    Could you take a moment to check it Google Books search still exists? I'll give you a hint: https://books.google.com/?hl=en https://books.google.com/?hl=en > Search the world's most comprehensive index of full-text books.
    - voisin 4y ago
      I know it exists, but it has appeared to languish for years. They rely on third parties now for inclusion of books, whereas in the early days they innovated on their own with specialized scanning technology. They seemed quite proud of it a decade ago. When was the last time Google has touted their books project? Have they even integrated searching books into their main search (which was supposed to catalogue And make searchable all the world’s information)? It hasn’t been killed but it is clearly a zombie.
      - londons_explore 4y ago
        It was paralyzed by legal disputes with book publishers. In the years the lawsuits were going on, nearly everyone left the project. And then the lawyers have put in so many red lines that it's nearly impossible to make any changes to it.
        voisin 4y ago
        Yup, I read about that on Wikipedia, but I can't help but not care. If a company touts massive initiatives and then gets bogged down in lawsuits, it seems like they didn't do the basic due diligence to avoid that. (Uber, AirBNB, and others seem to also have these headwinds, though not to the extent that it led to permanent paralysis, so maybe Google made a bet they thought they'd win and then didn't, whereas these other companies did.). I can't help but wonder why, with Google's resources vs. Uber or AirBNB, they couldn't keep moving forward if they wanted to. Strike deals, pay people, whatever. If it matters (i.e. if it involved ads) they would have done it. Given Google's behaviour since the end of their period of true innovative excellence, I don't cut them much slack.
        4y ago
- natrys 4y ago
  Does anyone know of a good FOSS alternative for google books that could be self-hosted (for personal library)?
  - MegaDeKay 4y ago
    Does Calibre fit the bill? The program itself is great, but it supports a plugin system that really puts it over the top. One of them automagically strips off Adobe DRM for any book loaded in to it. https://calibre-ebook.com/ https://calibre-ebook.com/
    - natrys 4y ago
      Seems that with calibre-web[1] we can have nice web front-end. But it doesn't have full-text search inside the content of the books though, unlike google books. [1] https://github.com/janeczku/calibre-web https://github.com/janeczku/calibre-web
      - MegaDeKay 4y ago
        Calibre 6 adds full text search, though not sure if that works from the web interface. https://calibre-ebook.com/new-in/fifteen https://calibre-ebook.com/new-in/fifteen
  - mdp2021 4y ago
    One product could be "Ambar: Document Search Engine" https://github.com/RD17/ambar https://github.com/RD17/ambar > Ambar is an open-source document search engine with automated crawling, OCR, tagging and instant full-text search > *Easily deploy Ambar with a single docker-compose file; *Perform Google-like search through your documents and contents of your images; *Tag your documents; *Use a simple REST API to integrate Ambar into your workflow
- daniel_reetz 4y ago
  Google Books, like so many Google projects, had a dual purpose. Making books accessible is noble and on-mission. But more importantly natural language models can be trained on the scanned corpus. The same was true of the original GOOG 411, which provided a free service, but was really put in place to train up their voice recognition projects. This is a long running strategy of Google, and it's a shrewd one. The main thing is not to mistake it for a public good. It is an act of privatization.
  - visarga 4y ago
    That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. I think Google initially wanted to augment the web results with a large book collection to get "all the world information and make it searchable", same with Google News.
    - philsnow 4y ago
      > That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. ... pull the other one. Okay, your statement could be true depending on what you mean by "large". But what makes you think that companies like Google haven't been working on language models without releasing them and/or without discussing them publicly? There's an advantage to be had by keeping corporate secrets.
      - visarga 4y ago
        The gap between research and publication is real but its length is less than one year. I don't think even Google has the resources to do it secretly. Who would work on it and how could they have kept the secret so tight? AI researchers want to publish, especially the best ones, it's essential for their careers. Someone else could plant the flag on their discovery and claim the fame.
        philsnow 4y ago
- jrm4 4y ago
  But Google could never do it right. It strikes me as obvious that anything that's going try to be a genuinely modern library can't sidestep, or even "work with" the present capitalist+copyright regime. It will just have to be a fight.
zozbot234 4y ago
Book metadata is widely available via sites like e.g. Open Library. With good metadata, full text search is not as relevant.
- aaron695 4y ago
- scotty79 4y ago
  That's false. I often search for specific citation.
lupire 4y ago
books.google.com
sacrosanct 4y ago
There is the Imperial Library of Trantor: https://trantor.is/ https://trantor.is/ They offer a clearnet and a hidden service .onion incase you don’t want ISPs blocking access to it.
- generalizations 4y ago
  Unfortunately that doesn't do full text search, which is what I was trying to get at with the Google comparison. I don't search for websites based on their titles...

delusional 4y ago

Wasn't that what google books was supposed to be?

voisin 4y ago
I assume Google abandoned this along with all their earlier mission statements in favour of building another chat app.
- emanuensis 4y ago
  Supposedly all at Library of Congress (LoC) anyone have experience using it there?
- b112 4y ago
  I can't imagine a life at google. So much promise, all turned to ash.
  - UkrainianJew 4y ago
    More like an ad company driven by dark patterns that needs to pretend being good and nice and innovative as otherwise it will lose goodwill just like Facebook did.
    - klipt 4y ago
      I think the Google founders were inherently nicer people than the Facebook founder, but their companies ended up converging on advertising because that's really the only way their products could be profitable.
      - discordance 4y ago
        Is that really the only way to be profitable though? I believe going public inevitably leads to needing to follow growth, which then eventually corrupts virtuous missions.
- lupire 4y ago
  Could you take a moment to check it Google Books search still exists? I'll give you a hint: https://books.google.com/?hl=en https://books.google.com/?hl=en > Search the world's most comprehensive index of full-text books.
  - voisin 4y ago
    I know it exists, but it has appeared to languish for years. They rely on third parties now for inclusion of books, whereas in the early days they innovated on their own with specialized scanning technology. They seemed quite proud of it a decade ago. When was the last time Google has touted their books project? Have they even integrated searching books into their main search (which was supposed to catalogue And make searchable all the world’s information)? It hasn’t been killed but it is clearly a zombie.
    - londons_explore 4y ago
      It was paralyzed by legal disputes with book publishers. In the years the lawsuits were going on, nearly everyone left the project. And then the lawyers have put in so many red lines that it's nearly impossible to make any changes to it.
      - voisin 4y ago
        Yup, I read about that on Wikipedia, but I can't help but not care. If a company touts massive initiatives and then gets bogged down in lawsuits, it seems like they didn't do the basic due diligence to avoid that. (Uber, AirBNB, and others seem to also have these headwinds, though not to the extent that it led to permanent paralysis, so maybe Google made a bet they thought they'd win and then didn't, whereas these other companies did.). I can't help but wonder why, with Google's resources vs. Uber or AirBNB, they couldn't keep moving forward if they wanted to. Strike deals, pay people, whatever. If it matters (i.e. if it involved ads) they would have done it. Given Google's behaviour since the end of their period of true innovative excellence, I don't cut them much slack.
        4y ago
natrys 4y ago
Does anyone know of a good FOSS alternative for google books that could be self-hosted (for personal library)?
- MegaDeKay 4y ago
  Does Calibre fit the bill? The program itself is great, but it supports a plugin system that really puts it over the top. One of them automagically strips off Adobe DRM for any book loaded in to it. https://calibre-ebook.com/ https://calibre-ebook.com/
  - natrys 4y ago
    Seems that with calibre-web[1] we can have nice web front-end. But it doesn't have full-text search inside the content of the books though, unlike google books. [1] https://github.com/janeczku/calibre-web https://github.com/janeczku/calibre-web
    - MegaDeKay 4y ago
      Calibre 6 adds full text search, though not sure if that works from the web interface. https://calibre-ebook.com/new-in/fifteen https://calibre-ebook.com/new-in/fifteen
- mdp2021 4y ago
  One product could be "Ambar: Document Search Engine" https://github.com/RD17/ambar https://github.com/RD17/ambar > Ambar is an open-source document search engine with automated crawling, OCR, tagging and instant full-text search > *Easily deploy Ambar with a single docker-compose file; *Perform Google-like search through your documents and contents of your images; *Tag your documents; *Use a simple REST API to integrate Ambar into your workflow
daniel_reetz 4y ago
Google Books, like so many Google projects, had a dual purpose. Making books accessible is noble and on-mission. But more importantly natural language models can be trained on the scanned corpus. The same was true of the original GOOG 411, which provided a free service, but was really put in place to train up their voice recognition projects. This is a long running strategy of Google, and it's a shrewd one. The main thing is not to mistake it for a public good. It is an act of privatization.
- visarga 4y ago
  That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. I think Google initially wanted to augment the web results with a large book collection to get "all the world information and make it searchable", same with Google News.
  - philsnow 4y ago
    > That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection. ... pull the other one. Okay, your statement could be true depending on what you mean by "large". But what makes you think that companies like Google haven't been working on language models without releasing them and/or without discussing them publicly? There's an advantage to be had by keeping corporate secrets.
    - visarga 4y ago
      The gap between research and publication is real but its length is less than one year. I don't think even Google has the resources to do it secretly. Who would work on it and how could they have kept the secret so tight? AI researchers want to publish, especially the best ones, it's essential for their careers. Someone else could plant the flag on their discovery and claim the fame.
      - philsnow 4y ago
jrm4 4y ago
But Google could never do it right. It strikes me as obvious that anything that's going try to be a genuinely modern library can't sidestep, or even "work with" the present capitalist+copyright regime. It will just have to be a fight.

zozbot234 4y ago

Book metadata is widely available via sites like e.g. Open Library. With good metadata, full text search is not as relevant.

aaron695 4y ago
scotty79 4y ago
That's false. I often search for specific citation.

lupire 4y ago

books.google.com

sacrosanct 4y ago

There is the Imperial Library of Trantor: https://trantor.is/ https://trantor.is/ They offer a clearnet and a hidden service .onion incase you don’t want ISPs blocking access to it.

generalizations 4y ago
Unfortunately that doesn't do full text search, which is what I was trying to get at with the Google comparison. I don't search for websites based on their titles...