3 ms·

Very cool! Do you have a recommended way for an agent to see an index of the books and epub links? (I can’t quite tell if that’s an egregious abuse of the site

Very cool! Do you have a recommended way for an agent to see an index of the books and epub links? (I can’t quite tell if that’s an egregious abuse of the site or you’re perfectly fine to share without human eye balls hitting your www?)

e0d075b569cd 5mo ago
[flagged]
kay_o 5mo ago
Check out https://www.gutenberg.org/ebooks/offline_catalogs.html https://www.gutenberg.org/ebooks/offline_catalogs.html Don't hit the site with agent. The section furtherst bottom machine readable.
JSeiko 5mo ago
not yet, but that's not a bad idea imo. Dealing with Ai crawler traffic is definitely a challenge if that's what you were referring to.
ancientcatz 5mo ago
OPDS?
- gluejar 5mo ago
  OPDS 2.0 coming RSN. email us if you want to test. OPDS 0.x is currently available (not recommended) by adding .opds to the end of a url
jzs 5mo ago
Now i'm not associated with gutenberg in any form, but they do have a page for offline consumption: https://www.gutenberg.org/ebooks/offline_catalogs.html https://www.gutenberg.org/ebooks/offline_catalogs.html Perhaps you can find the information you are looking for there. However if you plan on scraping or otherwise hitting them with a ton of traffic, consider at least to donate a good amount for the traffic you cause them. It ain't free after all.
- JSeiko 5mo ago
  Donations are always appreciated ;)
- jimnotgym 5mo ago
  Presumably if you paid them enough money they would give you the books without you having to pay to scrape at all?
samcollins 5mo ago
Thanks for the answers! Found it: > All Project Gutenberg metadata are available digitally in the XML/RDF format. This is updated daily (other than the legacy format mentioned below). Please use one of these files as input to a database or other tools you may be developing, instead of crawling or roboting the website. And strongly consider a donation! (My addition) https://www.gutenberg.org/ebooks/offline_catalogs.html#the-project-gutenberg-catalog-metadata-in-machine-readable-format https://www.gutenberg.org/ebooks/offline_catalogs.html#the-p...
gluejar 5mo ago
if what you want is all the text, please use the tarball or data files at https://www.gutenberg.org/cache/epub/feeds https://www.gutenberg.org/cache/epub/feeds
dredmorbius 5mo ago
Possibly ZIMs is of interest: <https://ebookfoundation.org/openzim.html https://ebookfoundation.org/openzim.html> (via: <https://news.ycombinator.com/item?id=48152200 https://news.ycombinator.com/item?id=48152200>).

e0d075b569cd 5mo ago

[flagged]

kay_o 5mo ago

Check out https://www.gutenberg.org/ebooks/offline_catalogs.html https://www.gutenberg.org/ebooks/offline_catalogs.html Don't hit the site with agent. The section furtherst bottom machine readable.

JSeiko 5mo ago

not yet, but that's not a bad idea imo. Dealing with Ai crawler traffic is definitely a challenge if that's what you were referring to.

ancientcatz 5mo ago

OPDS?

gluejar 5mo ago
OPDS 2.0 coming RSN. email us if you want to test. OPDS 0.x is currently available (not recommended) by adding .opds to the end of a url

jzs 5mo ago

Now i'm not associated with gutenberg in any form, but they do have a page for offline consumption: https://www.gutenberg.org/ebooks/offline_catalogs.html https://www.gutenberg.org/ebooks/offline_catalogs.html Perhaps you can find the information you are looking for there. However if you plan on scraping or otherwise hitting them with a ton of traffic, consider at least to donate a good amount for the traffic you cause them. It ain't free after all.

JSeiko 5mo ago
Donations are always appreciated ;)
jimnotgym 5mo ago
Presumably if you paid them enough money they would give you the books without you having to pay to scrape at all?

samcollins 5mo ago

Thanks for the answers! Found it: > All Project Gutenberg metadata are available digitally in the XML/RDF format. This is updated daily (other than the legacy format mentioned below). Please use one of these files as input to a database or other tools you may be developing, instead of crawling or roboting the website. And strongly consider a donation! (My addition) https://www.gutenberg.org/ebooks/offline_catalogs.html#the-project-gutenberg-catalog-metadata-in-machine-readable-format https://www.gutenberg.org/ebooks/offline_catalogs.html#the-p...

gluejar 5mo ago

if what you want is all the text, please use the tarball or data files at https://www.gutenberg.org/cache/epub/feeds https://www.gutenberg.org/cache/epub/feeds

dredmorbius 5mo ago

Possibly ZIMs is of interest: <https://ebookfoundation.org/openzim.html https://ebookfoundation.org/openzim.html> (via: <https://news.ycombinator.com/item?id=48152200 https://news.ycombinator.com/item?id=48152200>).