5 ms·

"text mining" Hmmm. It's not so easy to do this with ebooks viewed in a graphical ebook reader. Note he didn't say "text search". He said mining. Does this

"text mining" Hmmm. It's not so easy to do this with ebooks viewed in a graphical ebook reader. Note he didn't say "text search". He said mining. Does this suggest noncommercial library books will offer research capabilities that commercial ebooks will not? I want my ebooks in ASCII format. And that certainly goes for textbooks. less(1) is my "ebook reader".

pyre 14y ago
Are you suggesting that ebooks should be formatted in groff and devoid of page-breaks?
- bookworm_ 14y ago
  Should be a format that can be grep'd, made into PostScript, DejaVu, etc. Versatile. ASCII works well for that. Open, non-proprietary format. Not only that, working with raw text is so much faster than anything else. less(1) is the most responsive "ebook reader" I've ever used. It never chokes on huge files. less -n Ever try reading 100 or more PDF's in one sitting? But with 100 ASCII docs, it's possible to skim them fast using less, or mine some text with some UNIX utility. I like what this judge said. Digitizing books not just about putting the book on a "paper-like" screen with beautiful fonts, it's about enabling new utility. Working with the text.
  - chris_mahan 14y ago
    epub is xhtml+ wrapper
    - pyre 14y ago
      If you're targeting grep as a tool to operate on your data, then HTML isn't the best 'raw' format as grep isn't HTML-aware. It can't ignore markup, or translate encoded values (e.g. & => &).
      - lucian1900 14y ago
        But ePub as a format is open and well understood, and there are XHTML parsers in every language you might care about.
      - bduerst 14y ago
        I thought we were talking about viewing ebooks.
  - pyre 14y ago
    > It never chokes on huge files You've never tried to reach the end of a 200MB log file with a cold cache... :P
    - bookworm_ 14y ago
      It can handle files that big no problem. Use the -n switch to supress page numbering. But with a 200MB file, chances are you will only want to view part of it. Unless you have superhuman reading powers. ;) Dissect the part you want with some filter (sed, awk, tail, whatever) and feed the result to less.
  - TeMPOraL 14y ago
    > Should be a format that can be grep'd, made into PostScript, DejaVu, etc. Versatile. ASCII works well for that. Open, non-proprietary format. Org Mode. http://orgmode.org/ http://orgmode.org/
  - hellrich 14y ago
    I'm in for plaintext, but ASCII is ill suited for most languages.
- lucian1900 14y ago
  Yes, I don't want page-breaks in my ebooks. My reader is unlikely to be precisely the size the publisher put breaks on.
  - lallysingh 14y ago
    Technology doesn't make this an either-or situation, and some time reading RFCs really makes the pain of ascii obvious. Just ask for reflowable pdf. Figures, equations, cross-references (with hyperlinks), actual typography and design, full-text search, and the ability to adapt to your viewer. If you don't like that implementation, then may I suggest looking into improvements to the algorithm, instead of brutally stripping away content at books to fit an unnecessarily crude model?
    - lucian1900 14y ago
      I don't understand how PDF could be reflowable. I'd like a nice ePub, but again, no page-breaks. They'll be wrong for most devices anyway.
drivebyacct2 14y ago
What [major] eBook format keep their content in something other than ASCII (or Unicode or text of some kind)?
magicalist 14y ago
> Does this suggest noncommercial library books will offer research capabilities that commercial ebooks will not? If they are scanned noncommercial library books, I guess. I'm not sure how this is really relevant though. Scanning for this use may be fair use (hooray! some sanity), but the publishers aren't providing an ASCII file to the libraries either, the libraries are making a copy...hence the need for a fair use decision.
- deleted 14y ago
  [deleted]
- bookworm_ 14y ago
  I guess the question is whether libraries will use OCR to try to produce text that can be mined, or will they just make images.
kalleboo 14y ago
Textbooks in pure ASCII sounds like a pain. No hyperlinks? No index?
- TheGateKeeper 14y ago
  As plane-Jane as can be sounds absolutely great for text piping etc.

pyre 14y ago

Are you suggesting that ebooks should be formatted in groff and devoid of page-breaks?

bookworm_ 14y ago
Should be a format that can be grep'd, made into PostScript, DejaVu, etc. Versatile. ASCII works well for that. Open, non-proprietary format. Not only that, working with raw text is so much faster than anything else. less(1) is the most responsive "ebook reader" I've ever used. It never chokes on huge files. less -n Ever try reading 100 or more PDF's in one sitting? But with 100 ASCII docs, it's possible to skim them fast using less, or mine some text with some UNIX utility. I like what this judge said. Digitizing books not just about putting the book on a "paper-like" screen with beautiful fonts, it's about enabling new utility. Working with the text.
- chris_mahan 14y ago
  epub is xhtml+ wrapper
  - pyre 14y ago
    If you're targeting grep as a tool to operate on your data, then HTML isn't the best 'raw' format as grep isn't HTML-aware. It can't ignore markup, or translate encoded values (e.g. & => &).
    - lucian1900 14y ago
      But ePub as a format is open and well understood, and there are XHTML parsers in every language you might care about.
    - bduerst 14y ago
      I thought we were talking about viewing ebooks.
- pyre 14y ago
  > It never chokes on huge files You've never tried to reach the end of a 200MB log file with a cold cache... :P
  - bookworm_ 14y ago
    It can handle files that big no problem. Use the -n switch to supress page numbering. But with a 200MB file, chances are you will only want to view part of it. Unless you have superhuman reading powers. ;) Dissect the part you want with some filter (sed, awk, tail, whatever) and feed the result to less.
- TeMPOraL 14y ago
  > Should be a format that can be grep'd, made into PostScript, DejaVu, etc. Versatile. ASCII works well for that. Open, non-proprietary format. Org Mode. http://orgmode.org/ http://orgmode.org/
- hellrich 14y ago
  I'm in for plaintext, but ASCII is ill suited for most languages.
lucian1900 14y ago
Yes, I don't want page-breaks in my ebooks. My reader is unlikely to be precisely the size the publisher put breaks on.
- lallysingh 14y ago
  Technology doesn't make this an either-or situation, and some time reading RFCs really makes the pain of ascii obvious. Just ask for reflowable pdf. Figures, equations, cross-references (with hyperlinks), actual typography and design, full-text search, and the ability to adapt to your viewer. If you don't like that implementation, then may I suggest looking into improvements to the algorithm, instead of brutally stripping away content at books to fit an unnecessarily crude model?
  - lucian1900 14y ago
    I don't understand how PDF could be reflowable. I'd like a nice ePub, but again, no page-breaks. They'll be wrong for most devices anyway.

drivebyacct2 14y ago

What [major] eBook format keep their content in something other than ASCII (or Unicode or text of some kind)?

magicalist 14y ago

> Does this suggest noncommercial library books will offer research capabilities that commercial ebooks will not? If they are scanned noncommercial library books, I guess. I'm not sure how this is really relevant though. Scanning for this use may be fair use (hooray! some sanity), but the publishers aren't providing an ASCII file to the libraries either, the libraries are making a copy...hence the need for a fair use decision.

deleted 14y ago
[deleted]
bookworm_ 14y ago
I guess the question is whether libraries will use OCR to try to produce text that can be mined, or will they just make images.

kalleboo 14y ago

Textbooks in pure ASCII sounds like a pain. No hyperlinks? No index?

TheGateKeeper 14y ago
As plane-Jane as can be sounds absolutely great for text piping etc.