5 ms·

Show HN: @smoores/epub, a JavaScript library for working with EPUB publications

156 points by smoores 2y ago

Howdy! I've just written a blog post about this, and I figured I would share it here: https://smoores.dev/post/announcing_smoores_epub/ https://smoores.dev/post/announcing_smoores_epub/. As I've been working on Storyteller[1], I've been developing a library for working with EPUB files, since that's a large amount of the work that Storyteller does. After a friend asked for advice on creating EPUB books in Node.js, I decided to publish Storyteller's EPUB library as a standalone NPM package. I really love the EPUB spec, and I think the Node.js developer community deserves an actively maintained library for working with it! [1]: https://smoores.gitlab.io/storyteller https://smoores.gitlab.io/storyteller

hombre_fatal 2y ago
Extracting a library from a real world project is one of my favorite parts of software. I'm sure the march of LLMs will continue eating into this pie, and that's a good thing (most of it is a distraction from the real work), but I love polishing a library on my laptop in a cafe. It's like working on a painting or something.
- deleted 2y ago
  [deleted]
- smoores 2y ago
  It was, actually, very enjoyable! When we pulled React ProseMirror[1] out of the NYT text editor, it was a pretty laborious process that we had to careful plan and execute for months, and we still ended up with an internal fork for a while. By contrast, this was mostly just moving a file around and then writing documentation and cleaning up the public API. I rather enjoy thinking about and modeling library APIs in general, so I actually had a lot of fun with it! [1]: https://www.npmjs.com/package/@nytimes/react-prosemirror https://www.npmjs.com/package/@nytimes/react-prosemirror
gavmor 2y ago
Cool! I totally could have used this earlier this year... can't remember what for... Interesting choice to publish from the storyteller "monorepo." Is that because it evolved in situ, and you've no impetus to incur the overhead of extraction?
- smoores 2y ago
  Hahaha, well if it comes to you, the library will still be there for you :) Right, this was actually just a file within the Storyteller web package to start. It was fairly well defined, and so pretty easy to pull out into another package in the monorepo, but Storyteller is the primary consumer at the moment, and I want to be able to develop them in sync. Plus, it provides a great test bed for development of the library! edit: I forgot to mention that the eventual goal is to (hopefully!) publish this package as @storyteller/epub, along with any other packages that end up split out of Storyteller. That will probably include at least a @storyteller/synchronize and a @storyteller/cli. Unfortunately, someone seems to have snagged the @storyteller org on NPM several years ago and left it to languish without really using it, so I'm waiting to see whether GitHub will consider this squatting and transfer the org to me. I've also tried reaching out to the developer that owns the org, but they don't seem to have been active on GitHub or NPM for the past 5 years or so, and my only real strategy for reaching out to them was to open an issue on one of their other GitHub projects!
curtis3389 2y ago
Something I've been wondering: why do ebooks take so long to render? My kindle seems good at it, but opening an ebook in calibre/fbreader/etc can take minutes or even fail in some readers depending on the ebook.
- smoores 2y ago
  I would guess there are multiple potential pitfalls here. Firstly, not all ebook formats are created equal -- Storyteller only operates on EPUB files, because EPUB is an open source format and it supports Media Overlays (read-aloud) natively. I can only really speak to that format, but there are others (MOBI, PDF, etc). An EPUB is just a ZIP archive of XML and XHTML files (plus other assets, like images). Partly, I suspect, because of the dearth of actively maintained open source projects in the space, and partly because of the nature of tech in the book publishing industry, EPUB generation software used by authors and publishers often messes up this spec, which means that EPUB readers sometimes need to have fairly complex fallback logic for trying to figure out how to render a book. Also, because EPUBs are ZIP archives, some readers may either unzip the entire book into memory or "explode" it into an unzipped directory on disk, both of which may result in some slowness, especially if the book has lots of large resources. The newest Brandon Sanderson novel, for example, is ~300MB _zipped_. Additionally, and perhaps more importantly, EPUBs (and I believe MOBIs as well) represent content as XHTML and CSS, which means that readers very often need to use a browser or webview to actually render the book. Precisely how they deliver this content into the webview can have a huge impact on performance; most browser don't love to be told to format entire novels worth of content into text columns, for example.
  - curtis3389 2y ago
    Thank you so much! That's incredibly enlightening!
    - smoores 2y ago
      Of course! I'm hoping to have a web reader with Media Overlay support built in to Storyteller available in the next few months, along with some much needed library management tooling, so maybe that will be useful for you! I'll try to make it snappy :)
seanwessmith 2y ago
sometimes life just delivers. was looking for this 2 days ago
- smoores 2y ago
  Outstanding. Let me know if you run into any issues!
revskill 2y ago
Genius. Thanks.
herrvogel- 2y ago
Storyteller seems pretty cool in general. Can it be used to host books for other people?
- smoores 2y ago
  Thanks! Absolutely. You can invite users to your Storyteller server and give them whatever permissions are appropriate (e.g. you can choose whether they can only download books, or can also manage uploading and syncing books and/or managing users). It has SMTP support for emailing invites, or it can just generate invite links for you to share yourself. More info here: https://smoores.gitlab.io/storyteller/docs/administering#inviting-users https://smoores.gitlab.io/storyteller/docs/administering#inv...
  - 9dev 2y ago
    Oh my! This looks very neat, and I’ve been working on something similar to Storyteller (i think): https://github.com/project-kiosk/kiosk https://github.com/project-kiosk/kiosk I don’t get around working on it right now, but maybe there’s something useful there for you.
deleted 2y ago
[deleted]
cyberax 2y ago
BTW, I've had this idea in my mind for a while, after slowly working through my e-library infrastructure. Do you think it might be a good idea to set up a site to share the aligned overlays from Storyteller? This way, people won't have to waste CPU/GPU time re-aligning the same files over and over again. It should be OK from a copyright perspective, as it won't be distributing any copyrighted material, only the media overlay information.
- smoores 2y ago
  That’s a really interesting idea! The more I think about it, the more I like it. A challenge I foresee is that the media overlays are only reusable if you have the exact same input EPUB file, and have processed it with Storyteller to mark up the sentence boundaries. EPUBs have unique identifiers, though, so maybe this would be fine! We’d need to add a new processing flow to Storyteller, but it should be doable. Feel free to hit me up in the Storyteller chat if you want to discuss more! Thanks for sharing this idea!
  - NoahKAndrews 2y ago
    You could require that the input files have the same sha256 hash, that would presumably be more robust than trusting an ID from the file itself
    - smoores 2y ago
      Yeah I was toying around with that, too… but folks often mess around with metadata in tools like Calibre and Audiobookshelf in ways that wouldn’t have an impact on Storyteller’s sync, but would change their hash. On the other hand, I don’t know how various publishers handle EPUB dc:identifiers and that may not be robust enough, either. We could try doing something like hashing only the contents of spine items (including their file names, since that’s how media overlays refer to content)
      - cyberax 2y ago
        I was going to suggest to use the same approach as the old CD tagging systems. Count the number of words in each chapter to create a "book fingerprint". It's highly likely to be globally unique, and it can also help with the missing forewords/afterwords/bonus content sections. In addition, you can also add fuzzy matching for the title.
antonyt 2y ago
I feel stupid. I read “automatically syncing ebooks and audiobooks” and thought StoryTeller was a file synchronization service (like SyncThing) that for some reason only supported certain file types. Maybe “syncing ebooks with audiobooks” would be clearer? Also entirely possible this is just a me problem, not a general one. Really cool project!
- trevor-e 2y ago
  I thought the exact same thing, especially since one of the main selling points is around self-hosting which I thought was for the file syncing.
- smoores 2y ago
  This is like... comically hard to express succinctly haha. We've been having several conversations about how to explain it quickly to new folks. I like the phrase that the Readium folks use, "guided narration", but I don't know how useful that is for folks that aren't already familiar with what Storyteller does
  - chrismorgan 2y ago
    “Aligning” seems a perfectly good word to me. It’s the technical term that gets used, and it’s not particularly overloaded so people aren’t going to expect it to mean something else, and it’s easy to demonstrate even pictorially. (I would also use the term “synchronising” rather than “syncing”, because “syncing” has kinda developed a meaning of its own, even though it came from the same place. And use the word “with”, and try to use the words “text”. “Synchronising ebook text with audiobooks” is way better than “syncing ebooks and audiobooks”.)
  - rahimnathwani 2y ago
    Maybe you can use examples of the 'why'? - Switch between reading and listening whenever you want - Follow the text while listening to audiobooks - Enjoy audiobooks while reading along with the text Or even simpler: - Read or listen - switch anytime - See the words as you listen - Listen while you read along
tills13 2y ago
I'm going to be annoying and simply laugh at the fact it's a 2k line, single file in source. It's TypeScript! You HAVE to compile it to ship it as an npm package (_technically_ you don't but). Split it up into smaller modules with `import` statements!
- chrismorgan 2y ago
  Quite seriously: why? What actual advantage do you get? People tend to split things up into separate files far too eagerly. A single two-thousand-line file is nothing. Especially when most of that is in one class, and needs to be, so you can’t really split things up, or at the very least there would be runtime overhead to it.
  - smoores 2y ago
    Thanks, yeah I agree. The vast majority of the code in this library lives in a single class definition. Is it possible to move the implementations into separate files? Totally. Would that make the codebase more legible? I think at the moment, I would argue no, it would actually hurt legibility. If the class needs to grow dramatically, then maybe we’ll need a different approach, but I think this is actually the right thing to do for now!
Daiz 2y ago
As someone who has built custom EPUB authoring scripts in Node.js for work and also quite likes the EPUB spec: cool! I expect I'll still keep using said scripts for work, but I have some side project ideas which would involve actually reading EPUB files too - I'll probably give this a spin for those.
- smoores 2y ago
  Thanks! Looking forward to your feedback!
jspizziri 2y ago
You mentioned readium in one of your comments. I’m curious why you didn’t use the readium spec and corresponding ecosystem. Coming in cold and just glanced at your blog post, and I see you’re using RN. There is a RN package that leverages readium under the hood for ebook reading (I know because I’m the author), don’t know if it’s a weird actually has the api you need. EDIT: storyteller is super interesting. Dug around a bit in the code and see that you do seem to be using readium for some things, so I must just be missing some nuance.
- smoores 2y ago
  Readium is fantastic, and Storyteller uses it basically whenever possible. But Readium is exclusively for reading EPUB contents, and doesn’t have any support for modifying or creating them, which is the primary purpose of this library!

156 points by smoores 2y ago

hombre_fatal 2y ago

Extracting a library from a real world project is one of my favorite parts of software. I'm sure the march of LLMs will continue eating into this pie, and that's a good thing (most of it is a distraction from the real work), but I love polishing a library on my laptop in a cafe. It's like working on a painting or something.

deleted 2y ago
[deleted]
smoores 2y ago
It was, actually, very enjoyable! When we pulled React ProseMirror[1] out of the NYT text editor, it was a pretty laborious process that we had to careful plan and execute for months, and we still ended up with an internal fork for a while. By contrast, this was mostly just moving a file around and then writing documentation and cleaning up the public API. I rather enjoy thinking about and modeling library APIs in general, so I actually had a lot of fun with it! [1]: https://www.npmjs.com/package/@nytimes/react-prosemirror https://www.npmjs.com/package/@nytimes/react-prosemirror

gavmor 2y ago

Cool! I totally could have used this earlier this year... can't remember what for... Interesting choice to publish from the storyteller "monorepo." Is that because it evolved in situ, and you've no impetus to incur the overhead of extraction?

smoores 2y ago
Hahaha, well if it comes to you, the library will still be there for you :) Right, this was actually just a file within the Storyteller web package to start. It was fairly well defined, and so pretty easy to pull out into another package in the monorepo, but Storyteller is the primary consumer at the moment, and I want to be able to develop them in sync. Plus, it provides a great test bed for development of the library! edit: I forgot to mention that the eventual goal is to (hopefully!) publish this package as @storyteller/epub, along with any other packages that end up split out of Storyteller. That will probably include at least a @storyteller/synchronize and a @storyteller/cli. Unfortunately, someone seems to have snagged the @storyteller org on NPM several years ago and left it to languish without really using it, so I'm waiting to see whether GitHub will consider this squatting and transfer the org to me. I've also tried reaching out to the developer that owns the org, but they don't seem to have been active on GitHub or NPM for the past 5 years or so, and my only real strategy for reaching out to them was to open an issue on one of their other GitHub projects!

curtis3389 2y ago

Something I've been wondering: why do ebooks take so long to render? My kindle seems good at it, but opening an ebook in calibre/fbreader/etc can take minutes or even fail in some readers depending on the ebook.

smoores 2y ago
I would guess there are multiple potential pitfalls here. Firstly, not all ebook formats are created equal -- Storyteller only operates on EPUB files, because EPUB is an open source format and it supports Media Overlays (read-aloud) natively. I can only really speak to that format, but there are others (MOBI, PDF, etc). An EPUB is just a ZIP archive of XML and XHTML files (plus other assets, like images). Partly, I suspect, because of the dearth of actively maintained open source projects in the space, and partly because of the nature of tech in the book publishing industry, EPUB generation software used by authors and publishers often messes up this spec, which means that EPUB readers sometimes need to have fairly complex fallback logic for trying to figure out how to render a book. Also, because EPUBs are ZIP archives, some readers may either unzip the entire book into memory or "explode" it into an unzipped directory on disk, both of which may result in some slowness, especially if the book has lots of large resources. The newest Brandon Sanderson novel, for example, is ~300MB _zipped_. Additionally, and perhaps more importantly, EPUBs (and I believe MOBIs as well) represent content as XHTML and CSS, which means that readers very often need to use a browser or webview to actually render the book. Precisely how they deliver this content into the webview can have a huge impact on performance; most browser don't love to be told to format entire novels worth of content into text columns, for example.
- curtis3389 2y ago
  Thank you so much! That's incredibly enlightening!
  - smoores 2y ago
    Of course! I'm hoping to have a web reader with Media Overlay support built in to Storyteller available in the next few months, along with some much needed library management tooling, so maybe that will be useful for you! I'll try to make it snappy :)

seanwessmith 2y ago

sometimes life just delivers. was looking for this 2 days ago

smoores 2y ago
Outstanding. Let me know if you run into any issues!

revskill 2y ago

Genius. Thanks.

herrvogel- 2y ago

Storyteller seems pretty cool in general. Can it be used to host books for other people?

smoores 2y ago
Thanks! Absolutely. You can invite users to your Storyteller server and give them whatever permissions are appropriate (e.g. you can choose whether they can only download books, or can also manage uploading and syncing books and/or managing users). It has SMTP support for emailing invites, or it can just generate invite links for you to share yourself. More info here: https://smoores.gitlab.io/storyteller/docs/administering#inviting-users https://smoores.gitlab.io/storyteller/docs/administering#inv...
- 9dev 2y ago
  Oh my! This looks very neat, and I’ve been working on something similar to Storyteller (i think): https://github.com/project-kiosk/kiosk https://github.com/project-kiosk/kiosk I don’t get around working on it right now, but maybe there’s something useful there for you.

deleted 2y ago

[deleted]

cyberax 2y ago

BTW, I've had this idea in my mind for a while, after slowly working through my e-library infrastructure. Do you think it might be a good idea to set up a site to share the aligned overlays from Storyteller? This way, people won't have to waste CPU/GPU time re-aligning the same files over and over again. It should be OK from a copyright perspective, as it won't be distributing any copyrighted material, only the media overlay information.

smoores 2y ago
That’s a really interesting idea! The more I think about it, the more I like it. A challenge I foresee is that the media overlays are only reusable if you have the exact same input EPUB file, and have processed it with Storyteller to mark up the sentence boundaries. EPUBs have unique identifiers, though, so maybe this would be fine! We’d need to add a new processing flow to Storyteller, but it should be doable. Feel free to hit me up in the Storyteller chat if you want to discuss more! Thanks for sharing this idea!
- NoahKAndrews 2y ago
  You could require that the input files have the same sha256 hash, that would presumably be more robust than trusting an ID from the file itself
  - smoores 2y ago
    Yeah I was toying around with that, too… but folks often mess around with metadata in tools like Calibre and Audiobookshelf in ways that wouldn’t have an impact on Storyteller’s sync, but would change their hash. On the other hand, I don’t know how various publishers handle EPUB dc:identifiers and that may not be robust enough, either. We could try doing something like hashing only the contents of spine items (including their file names, since that’s how media overlays refer to content)
    - cyberax 2y ago
      I was going to suggest to use the same approach as the old CD tagging systems. Count the number of words in each chapter to create a "book fingerprint". It's highly likely to be globally unique, and it can also help with the missing forewords/afterwords/bonus content sections. In addition, you can also add fuzzy matching for the title.

antonyt 2y ago

I feel stupid. I read “automatically syncing ebooks and audiobooks” and thought StoryTeller was a file synchronization service (like SyncThing) that for some reason only supported certain file types. Maybe “syncing ebooks with audiobooks” would be clearer? Also entirely possible this is just a me problem, not a general one. Really cool project!

trevor-e 2y ago
I thought the exact same thing, especially since one of the main selling points is around self-hosting which I thought was for the file syncing.
smoores 2y ago
This is like... comically hard to express succinctly haha. We've been having several conversations about how to explain it quickly to new folks. I like the phrase that the Readium folks use, "guided narration", but I don't know how useful that is for folks that aren't already familiar with what Storyteller does
- chrismorgan 2y ago
  “Aligning” seems a perfectly good word to me. It’s the technical term that gets used, and it’s not particularly overloaded so people aren’t going to expect it to mean something else, and it’s easy to demonstrate even pictorially. (I would also use the term “synchronising” rather than “syncing”, because “syncing” has kinda developed a meaning of its own, even though it came from the same place. And use the word “with”, and try to use the words “text”. “Synchronising ebook text with audiobooks” is way better than “syncing ebooks and audiobooks”.)
- rahimnathwani 2y ago
  Maybe you can use examples of the 'why'? - Switch between reading and listening whenever you want - Follow the text while listening to audiobooks - Enjoy audiobooks while reading along with the text Or even simpler: - Read or listen - switch anytime - See the words as you listen - Listen while you read along

tills13 2y ago

I'm going to be annoying and simply laugh at the fact it's a 2k line, single file in source. It's TypeScript! You HAVE to compile it to ship it as an npm package (_technically_ you don't but). Split it up into smaller modules with `import` statements!

chrismorgan 2y ago
Quite seriously: why? What actual advantage do you get? People tend to split things up into separate files far too eagerly. A single two-thousand-line file is nothing. Especially when most of that is in one class, and needs to be, so you can’t really split things up, or at the very least there would be runtime overhead to it.
- smoores 2y ago
  Thanks, yeah I agree. The vast majority of the code in this library lives in a single class definition. Is it possible to move the implementations into separate files? Totally. Would that make the codebase more legible? I think at the moment, I would argue no, it would actually hurt legibility. If the class needs to grow dramatically, then maybe we’ll need a different approach, but I think this is actually the right thing to do for now!

Daiz 2y ago

As someone who has built custom EPUB authoring scripts in Node.js for work and also quite likes the EPUB spec: cool! I expect I'll still keep using said scripts for work, but I have some side project ideas which would involve actually reading EPUB files too - I'll probably give this a spin for those.

smoores 2y ago
Thanks! Looking forward to your feedback!

jspizziri 2y ago

You mentioned readium in one of your comments. I’m curious why you didn’t use the readium spec and corresponding ecosystem. Coming in cold and just glanced at your blog post, and I see you’re using RN. There is a RN package that leverages readium under the hood for ebook reading (I know because I’m the author), don’t know if it’s a weird actually has the api you need. EDIT: storyteller is super interesting. Dug around a bit in the code and see that you do seem to be using readium for some things, so I must just be missing some nuance.

smoores 2y ago
Readium is fantastic, and Storyteller uses it basically whenever possible. But Readium is exclusively for reading EPUB contents, and doesn’t have any support for modifying or creating them, which is the primary purpose of this library!