9 ms·

How does Firefox's Reader View work? (2020)

244 points by pmoriarty 5y ago

deleted 5y ago
[deleted]
pipeline_peak 5y ago
Does anyone know how difficult it would be to develop a reader view only browser?
- actually_a_dog 5y ago
  You don't need to. All you'd have to do is hack up one of the reader view extensions so that it's triggered when loading a page rather than by clicking a button.
  - tannhaeuser 5y ago
    I guess OP eyes a reader mode-only browser as an achievable milestone on the way to develop a browser (something sorely missing in "web standards" only serving to pull up the ladder), rather than as an app from a user PoV.
    - pipeline_peak 5y ago
      This, I could've been more specific. I basically just want to design a simple HTML4 (or doable parts of HTML 5 ) layout engine to render static documents and avoid any dynamic components.
- 878654Tom 5y ago
  There are a couple of (not very popular) plugins that can set Reader view as default. One of them is Automatic Reader View[1] which has an option to to show all pages in reader-view. 1. https://addons.mozilla.org/en-US/firefox/addon/automatic-reader-view/ https://addons.mozilla.org/en-US/firefox/addon/automatic-rea...
- btdmaster 5y ago
  Try about:reader?url=YOUR_URL_HERE. If that does not work, it means the URL does not support it.
  - classichasclass 5y ago
    Sadly, no longer works on Firefox Android. I used to use this trick a lot.
- betwixthewires 5y ago
  Well, there's a few options out there for something like this, it depends on what you're going for specifically. There's Gemini and gopher, protocols designed around transferring and rendering plain documents. There are terminal based browsers like Lynx that render pages in a terminal, of course it's all text on your end. If you're talking about a GUI program to do what reader view does and only that, the reader view code is open source and available from Mozilla, I'm sure it wouldn't be much to build a webview app for mobile or a simple GUI that sits on top of curl or wget or something like that, fetches the page, processes a URL and renders the text. You're probably going to be manually entering URLs though, I'm not sure in the modern web how you even click from one site to another, how you even wind up at a URL for a written article using only reader view.
- hansel_der 5y ago
  https://news.ycombinator.com/item?id=30864714 https://news.ycombinator.com/item?id=30864714
- phgn 5y ago
  I think the problem here is not on the technical side. The places where we find stuff to read (aggregators, homepages, social networks) just don't work well if you're only extracting the text and links. There's also a huge long tail of parsing issues because web pages are not static documents. You'd want a fallback to the original HTML -- so why not use your main browser? There are extensions that activate reader mode automatically on pages where it's supported [0]. [0] I'm working on one of them: https://github.com/lindylearn/unclutter https://github.com/lindylearn/unclutter
Arnavion 5y ago
>Getting author information a.k.a. byline relies primarily on either on the correct appearance two attribute=value pairs rel=author or itemprop=author, or a regexp soup matching potential byline matches. The heuristics are indeed brittle, which is why I had to add: <span class="work-around-firefox-reader-mode-bug" rel="author">‌</span> ... to my website. Without that, Reader Mode would look for the first HTML element that contained the string "author", assume its content was the name of the author of the article, and make it a byline right below the title. Of course the element it picked was a section header, with the content "Completing an authorization". So not only did I end up with a blog post written by "Completing an authorization", but also the section that was supposed to have that header no longer had a header. So the hack is to add another element before the website content that it picks instead. ‌ is a ZWNJ, which I used because Reader Mode is too smart for its own good and will ignore an element that has empty text content or just an NBSP. So a ZWNJ is the next-best thing. But it's great when it works, and I'm glad it exists. I have JS disabled by default and have lost count of the number of websites that use CSS to hide their content and JS to make it visible. Reader Mode defeats them easily.
- deleted 5y ago
  [deleted]
- emsixteen 5y ago
  Why use a ZWNJ and not just, well, an actual author name? I'm probably missing something here.
  - Arnavion 5y ago
    Because it's my website and I get to decide whether to have an author name or not. It's bad enough that browsers assume my website, that's made up entirely of free-flowing text inside p tags, won't fit on phone screens unless I add a meta viewport tag. I'm not going to also make changes to the content to satisfy them.
    - account42 5y ago
      > It's bad enough that browsers assume my website, that's made up entirely of free-flowing text inside p tags, won't fit on phone screens unless I add a meta viewport tag. This anoys me to no end as well. It's especially ridiculous now that mobile browsers provide a "Desktop" mode that can be used as a fallback for those sites that do need a larger page width.
hans1729 5y ago
Ha, I just worked through this a week ago: https://searchfox.org/mozilla-release/source/toolkit/components/reader/Readability.js https://searchfox.org/mozilla-release/source/toolkit/compone...
mediumdeviation 5y ago
> for instance, any node with class skyscraper is considered hidden (maybe the logic is that those elements are shrouded by clouds). Ha, you wish. Skyscraper is the name of a standard Internet advertising unit. It’s fairly common for adblockers to block it. https://www.marketingterms.com/dictionary/skyscraper_ad/ https://www.marketingterms.com/dictionary/skyscraper_ad/
- pantulis 5y ago
  Is it still being used?
  - zapt02 5y ago
    Yes, it's a standard industry term and still used on many sites.
busymom0 5y ago
My Hacker News client HACK for iOS and Android has reader mode ability browser. It can be set to be the default for hacker news sites in the app too. While on iOS, I was able to use the reader mode feature provided by SFSafariViewController, that wasn't available on android. So I had to read a ton about this. I ended up using a heavily modified Kotlin version of Readability: https://github.com/dankito/Readability4J https://github.com/dankito/Readability4J https://play.google.com/store/apps/details?id=com.pranapps.hack https://play.google.com/store/apps/details?id=com.pranapps.h... https://apps.apple.com/us/app/id1464477788 https://apps.apple.com/us/app/id1464477788
- dewey 5y ago
  Thanks for Hack, it's what I'm currently using!
tannhaeuser 5y ago
Wondering if this (plus Safari's reader mode heuristics if those are any different) could form the basis of a much-needed reduced HTML subset. Like, for the HTML we're actually searching, as opposed to the trackfest and seo'd articles search engines are giving us. Edit: so much for the illusion of "semantic HTML" ie where you need heuristics and are entering an arm's race vs publishers to make your HTML even readable
uniqueuid 5y ago
It's really nice that browsers offer reader modes, but they are frustratingly incomplete. Really, let's switch to a user perspective once and consider - what if I always want reader mode? This is technologically complete impossible and all the solutions are a band-aid. Firefox and others' attempts rely on the page authors' goodwill. But some pages will always attempt to frustrate reader modes. Alternative approaches for content extraction use machine learning such as [1], but they of course need to be updated for culture- language- and technology-specific changes. It's a mess and will remain so for the foreseeable future. [1] https://github.com/dragnet-org/dragnet https://github.com/dragnet-org/dragnet
- classichasclass 5y ago
  I specifically wrote always-available, "sticky" (default: stays Reader View even if you go to another page) and automatic (goes auto on subpages or entire domains, as you specify) Reader View support built-in to TenFourFox to help out on slower systems.
- paxys 5y ago
  It's a mess by design. In an ideal world no browser would need a reader mode, because pages themselves would be reader-friendly by design. Page authors, of course, are instead more interested in optimizing for advertising and tracking. Reader mode, much like adblock extensions, works directly against their interests. In fact we wouldn't even need any new web standards to implement something like this. Any website following the latest Web Content Accessibility Guidelines and using the correct page structure and aria- tags should be fully semantically parsable and displayable in any format or medium.
  - Moru 5y ago
    I'm sure they can live without my views. If I can't read the page I can always press the back button. I'm not going to buy anything from any ads anyway so no loss on either side except a few seconds of time lost for me. Just got a link to a documentation for an API that I can't read because of too low contrast. Dark Reader plugin nicely fixed it so didn't even have to try reader mode. It sucks to get old anyway, you young people don't have to make it worse. You will be in the same seat in 20-odd years so plan for it :-)
productceo 5y ago
To HN users reading this comment, where did you find the best reader view implementation? Seems like it's a really difficult task.
- miket 5y ago
  https://www.diffbot.com/products/extract/ https://www.diffbot.com/products/extract/ works pretty well.
gumby 5y ago
I use safari and have it set to use reader view by default. Much more pleasant!
intsunny 5y ago
I'm always amazed at how Firefox's reader view circumvents so many javascript paywalls. I'm surprised at how incredibly mono-cultured web developers have become in testing browsers other than Chrome. OR .... they know, and they don't care ....
- porker 5y ago
  Often they don't care because this makes it easy for Google to still index their content and blocks most humans.
yason 5y ago
If only we had some descriptive language to mark up the actual, readable content of a web page in the simplest form and then have all the auxiliary content, table-of-contents, dynamic functionality, and styling described separately so that the browser would know what content is essential and what is optional...
stavros 5y ago
Thinking about this and the Gemini protocol, it seems to me that HTML has become too user-hostile, or at least it gives too much rope for site developers to hang users. The way I'd solve this problem would be to develop a new file format that was similar to Markdown but contained semantic elements (eg "this image is relevant to this bit of text", or "this text explains that text". This would be rendered into HTML by a browser extension so the browser could display it, which would help keep authors honest (because if the server renders it, we're just using HTML again). The format would contain no styling information at all, apart from the semantic tags, so styling would be entirely on the client. You could pick the way you wanted your articles to look, which would probably be different per device. This sounds a bit like epub and would probably be great for book reader devices as well, while being extremely light to load because there isn't much you can do with it.
- superasn 5y ago
  I think somebody came up with something similar to this called Aftertext(1). I remember it as I found the concept kinda interesting. Unfortunately the easy part is designing the protocol, the hard part is universal rendering and adoption. (1) https://news.ycombinator.com/item?id=29643313 https://news.ycombinator.com/item?id=29643313
  - stavros 5y ago
    Hmm, that's an interesting idea, though I think the format looks too cumbersome. Still, that's mostly what I meant, thanks for the link!
- jraph 5y ago
  Unfortunately, it seems like a technical solution to a society-wide human problem to me.
  - solarkraft 5y ago
    Tech people love throwing technology at social problems. Plain HTML is great for documents. It’s just the browsers‘ failure to render it nicely that makes authors feel like it’s necessary to style to make it presentable.
rcarmo 5y ago
I implemented a variation of the Readability algorithm some 9 years ago, in case anyone needs a server-side Python version and is interested in dragging it (kicking and screaming) into the 2020s: https://github.com/rcarmo/soup-strainer https://github.com/rcarmo/soup-strainer
- leodriesch 5y ago
  Mozilla’s readability library is also usable on the server with Node.js and JSDOM. That’s if you are fine with using JavaScript on the backend.
solarkraft 5y ago
I haven’t directly compared them, but I have also found mercury parser (https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser) to be pretty reliable. Its advantage is that you can directly pass it a page instead of having to give it a DOM. Since it turns a website into very plain (X)HTML it‘s fairly easy to use it to make a browsing proxy or automatically produce epub files for e-readers, which is what I do. Edit: Here’s the proof of concept type code I use: https://gist.github.com/solarkraft/d6306f17a761fcb5ce47f2be73327213 https://gist.github.com/solarkraft/d6306f17a761fcb5ce47f2be7... It’s a bit crappy, but it works for me :-)
simonw 5y ago
If you're interested in playing with Mozilla Readability.js from the command line I came up with this recipe using my shot-scraper tool the other day: https://til.simonwillison.net/shot-scraper/readability https://til.simonwillison.net/shot-scraper/readability pip install shot-scraper shot-scraper install shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ " async () => { const readability = await import('https://cdn.skypack.dev/@mozilla/readability'); return (new readability.Readability(document)).parse(); }"
- dotancohen 5y ago
  shot-scraper is terrific, thank you! Right now I'm using a bash script instead of YAML file for screenshotting multiple sites I maintain, as there is no option to add a timestamp to the filenames, something like this (simplified): declare -A capture capture[www.foo.com]=https://www.foo.com/ capture[www.foo.com-bar]=https://www.foo.com/bar for key in "${!capture[@]}" ; do shot-scraper ${capture[$key]} -o - > $key_$(date +"%Y-%m-%d_%H_%M") done Otherwise it is a terrific tool, able to screenshot websites that are otherwise very difficult due to CDNs, javascript, etc. Thank you very much!
longnguyen 5y ago
Readability is amazing. I find it works most of the time. Currently I'm using it to parse a web article, package into a .mobi file and send it directly to my Kindle Paperwhite. If you want to read your favorite content on Kindle, give it a try[0] [0]: https://ktool.io https://ktool.io
- zulln 5y ago
  For websites, do you have examples where it outperform Amazon's own solution? https://www.amazon.com/sendtokindle/ https://www.amazon.com/sendtokindle/
  - longnguyen 5y ago
    KTool[0] is fairly new so I can't claim it "outperform" Amazon's solution. However if you read the recent reviews from their Chrome Extension page[1], the majority of complaints was that the article never arrived. I just tried sending this article[2] and 10 minutes later it still hasn't delivered yet. This article contains a lot of images, so I understand it's not going to be quick, but with KTool, it does deliver within 2 minutes. But that is not the only reason why I build KTool. I don't want the articles to include links when reading on my Kindle because it's bad UX (easy to mis-tap), distracting and reduce comprehension (see here[3] and here[4]). Instead, I push all these links to the bottom of the article, which make it still accessible yet improve the reading experience. [0]: https://ktool.io https://ktool.io [1]: https://chrome.google.com/webstore/detail/send-to-kindle-for-google/cgdjpilhipecahhcilnafpblkieebhea https://chrome.google.com/webstore/detail/send-to-kindle-for... [2]: https://waitbutwhy.com/2017/04/neuralink.html https://waitbutwhy.com/2017/04/neuralink.html [3]: https://www.wired.com/2010/05/ff-nicholas-carr/ https://www.wired.com/2010/05/ff-nicholas-carr/ [4]: https://www.roughtype.com/?p=1378 https://www.roughtype.com/?p=1378
    - longnguyen 5y ago
      Update: 5 hours later, no sign of the article being delivered. It think it's safe to assume the Amazon solution doesn't work ¯\_(ツ)_/¯
- 5y ago
adamgordonbell 5y ago
I found readability pretty useful for building a plain text version of articles. From there, I'd love to take a RSS feed and transform it to ebook. https://earthly-tools.com/text-mode https://earthly-tools.com/text-mode https://news.ycombinator.com/item?id=30829568 https://news.ycombinator.com/item?id=30829568
- criddell 5y ago
  Evernote (remember them?) had a pretty good tool called Clipper that would save a reader-version of the page to Evernote. It was pretty great, but I haven't used it lately.
- cxr 5y ago
  I'm surprised that more effort hasn't gone into hacking on "line mode" (terminal-based) Web browsers—esp. for reading developer documentation—and absence of the resulting effect that this should have among hackers on their own sites. I find most people who consider their projects to be "no-frills" have quite a few frills indeed. This includes personal blogs, but Sourcehut and the constant praise for its "clean" UI is one other example of this sort of thing. Load it up in w3m or Lynx and it very much feels like a site intended to be read in a conventional desktop browser but filtered and constrained to a text-only medium. Actually clean design would start with, "How would I expect this to feel if I made a custom, terminal-based (but not command-line driven) Sourcehut client?" and then figuring out the appropriate markup you'd need to spit out to achieve that in a line mode browser that understands the basics—forms, content ordering, etc. (PS: Now transport that same artifact to a conventional browser. Is the experience better or worse that what's currently available?)
BuckRogers 5y ago
I used Firefox for 19 years without interruption until last summer when I switched off of it. Its reader view was one of its best features. It works on more pages than competing browser's reading modes. While they removed most of the features that I had grown to love over the past two decades, another standout for me was the ability to have multiple picture in picture instances running.
bni 5y ago
I use this in a proxy to be able to browse the web on my Amiga 1200
jokoon 5y ago
I think it won't work on techcrunch, since it somehow blurs the page. But generally it's awesome.
JangoSteve 5y ago
This seems to be a prime example of the Pareto Principle, or the 80/20 rule. About 12 years ago, I had built something to do something similar and published how I did it [1]. It was a very simple process, which my younger self called "the meat algorithm", as in, how to get the meat of an article. It was far less code and worked perfectly 95% of the time (though, the average web-page was also a little simpler 12 years ago). But that code would have quickly ballooned out if our use-case had called for addressing the other 5% of webpages, as the Firefox Reader View must do. [1] https://www.alfajango.com/blog/create-a-printable-format-for-any-webpage-with-ruby-and-nokogiri https://www.alfajango.com/blog/create-a-printable-format-for...
acrophobic 5y ago
I've been working on several web extractors project, so I think I could share some of my findings while working on them. Granted it's been several months since I worked on it so I might be forgetting some things. There are several open source projects for extracting web contents. However, there are three extractors that I've worked with and give us good result: - readability.js[1], web extractor by Mozilla that used in Firefox. - dom-distiller[2], web extractor by Chromium team, written in Java. - trafilatura[3], Python package by Adrien Barbaresi from BBAW[4]. First, readability.js, as expected is the most famous extractor. It's a single file Javascript library with modest 2,000+ lines of code, released under Apache license. Since it's in JS, you can use it wherever you want, either in web page using `script` tag or by using it in Node project. Next, DomDistiller is extractor that used in Chromium. It uses Java language with whopping 14,000+ lines of code and can only be used as part of Chromium browser, so you can't exactly use it as standalone library or CLI. Finally, Trafilatura is a Python package released under GPLv3 license. Created in order to build a text databases[5] for NLP research, it mainly intended for German web pages. However, as development continues, it works really great with other languages. It's a bit slow though compared to Readability.js. All of those three work in similar way: extract metadata, remove unneeded contents, and finally returns the cleaned up content. Their differences (that I remembered) are: - In Readability, they insist to make no special rules for any website, while DomDistiller and Trafilatura give a small exception for popular sites like Wikipedia. Thanks to this, if you use Readability.js in Wikipedia pages, it will shows `[edit]` button thorough the extracted content. - Readability has a small function to detect whether a web page can be converted to reader mode. While it's not really accurate, it's quite convenient to have. - In DomDistiller, the metadata extraction is more thorough than the others. It supports OpenGraph, Schema.org, and even the old IE Reading View mark up tags. - Since DomDistiller is only usable within Chromium, it has the advantage to be able to use CSS styling to determine if an element is important or not. If an element is styled to be invisible (e.g. `display: none`) then it will be deemed unimportant. However, according to a research[6] this step is actually doesn't really affect the extraction result. - DomDistiller also has an experimental feature to find and extract next page in sites that separated its article to several partial pages. - For Trafilatura, since it was created for collecting web corpus, it main ability is extracting text and the publication date of a web page. For the latter, they've created a Python package named htmldate[7] whose only purpose is to extract the publication or modification date for a web page. - Trafilatura also has an experimental feature to remove elements that repeated too often. The idea is if the element occured too often, then it's not important to the reader. I've found benchmark[8] that compare the performance between the extractors, and it said that Trafilatura has the best accuracy compared to the others. However, before you start rushing to use Trafilatura, you should remember that Trafilatura is intended for gathering web corpus, so it's really great for extracting text content, but IIRC is not as good as Readability.js and DomDistiller for extracting a proper article with images and embedded iframes (depending on how you look, it could be a feature though). By the way, if you are using Go and need to use a web extractor, I already ported all three of them to Go[9][10][11] including their dependencies[12][13], so have fun with it. [1]: https://github.com/mozilla/readability https://github.com/mozilla/readability [2]: https://github.com/chromium/dom-distiller https://github.com/chromium/dom-distiller [3]: https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura [4]: https://www.bbaw.de/en/ https://www.bbaw.de/en/ [5]: https://www.dwds.de/d/k-web https://www.dwds.de/d/k-web [6]: https://arxiv.org/abs/1811.03661 https://arxiv.org/abs/1811.03661 [7]: https://github.com/adbar/htmldate https://github.com/adbar/htmldate [8]: https://github.com/scrapinghub/article-extraction-benchmark https://github.com/scrapinghub/article-extraction-benchmark [9]: https://github.com/go-shiori/go-readability https://github.com/go-shiori/go-readability [10]: https://github.com/markusmobius/go-domdistiller https://github.com/markusmobius/go-domdistiller [11]: https://github.com/markusmobius/go-trafilatura https://github.com/markusmobius/go-trafilatura [12]: https://github.com/markusmobius/go-htmldate https://github.com/markusmobius/go-htmldate [13]: https://github.com/markusmobius/go-dateparser https://github.com/markusmobius/go-dateparser
Stampo00 5y ago
Thank you for this! I've always wondered where this code lives because I love reader view and I'd like to use it as a base style for my blog.
zerop 5y ago
Can this engine be used to build something like pocket? Does pocket use the same tech to fetch content of pages ?
masukomi 5y ago
For those wondering if there's a redability lib in their favorite language. Here's a list of them all (as far as i know) plus the original arc-90 implementation https://github.com/masukomi/arc90-readability/#readability https://github.com/masukomi/arc90-readability/#readability Please submit a PR if there's something i don't have listed there.

244 points by pmoriarty 5y ago

deleted 5y ago

[deleted]

pipeline_peak 5y ago

Does anyone know how difficult it would be to develop a reader view only browser?

actually_a_dog 5y ago
You don't need to. All you'd have to do is hack up one of the reader view extensions so that it's triggered when loading a page rather than by clicking a button.
- tannhaeuser 5y ago
  I guess OP eyes a reader mode-only browser as an achievable milestone on the way to develop a browser (something sorely missing in "web standards" only serving to pull up the ladder), rather than as an app from a user PoV.
  - pipeline_peak 5y ago
    This, I could've been more specific. I basically just want to design a simple HTML4 (or doable parts of HTML 5 ) layout engine to render static documents and avoid any dynamic components.
878654Tom 5y ago
There are a couple of (not very popular) plugins that can set Reader view as default. One of them is Automatic Reader View[1] which has an option to to show all pages in reader-view. 1. https://addons.mozilla.org/en-US/firefox/addon/automatic-reader-view/ https://addons.mozilla.org/en-US/firefox/addon/automatic-rea...
btdmaster 5y ago
Try about:reader?url=YOUR_URL_HERE. If that does not work, it means the URL does not support it.
- classichasclass 5y ago
  Sadly, no longer works on Firefox Android. I used to use this trick a lot.
betwixthewires 5y ago
Well, there's a few options out there for something like this, it depends on what you're going for specifically. There's Gemini and gopher, protocols designed around transferring and rendering plain documents. There are terminal based browsers like Lynx that render pages in a terminal, of course it's all text on your end. If you're talking about a GUI program to do what reader view does and only that, the reader view code is open source and available from Mozilla, I'm sure it wouldn't be much to build a webview app for mobile or a simple GUI that sits on top of curl or wget or something like that, fetches the page, processes a URL and renders the text. You're probably going to be manually entering URLs though, I'm not sure in the modern web how you even click from one site to another, how you even wind up at a URL for a written article using only reader view.
hansel_der 5y ago
https://news.ycombinator.com/item?id=30864714 https://news.ycombinator.com/item?id=30864714
phgn 5y ago
I think the problem here is not on the technical side. The places where we find stuff to read (aggregators, homepages, social networks) just don't work well if you're only extracting the text and links. There's also a huge long tail of parsing issues because web pages are not static documents. You'd want a fallback to the original HTML -- so why not use your main browser? There are extensions that activate reader mode automatically on pages where it's supported [0]. [0] I'm working on one of them: https://github.com/lindylearn/unclutter https://github.com/lindylearn/unclutter

Arnavion 5y ago

>Getting author information a.k.a. byline relies primarily on either on the correct appearance two attribute=value pairs rel=author or itemprop=author, or a regexp soup matching potential byline matches. The heuristics are indeed brittle, which is why I had to add: <span class="work-around-firefox-reader-mode-bug" rel="author">‌</span> ... to my website. Without that, Reader Mode would look for the first HTML element that contained the string "author", assume its content was the name of the author of the article, and make it a byline right below the title. Of course the element it picked was a section header, with the content "Completing an authorization". So not only did I end up with a blog post written by "Completing an authorization", but also the section that was supposed to have that header no longer had a header. So the hack is to add another element before the website content that it picks instead. ‌ is a ZWNJ, which I used because Reader Mode is too smart for its own good and will ignore an element that has empty text content or just an NBSP. So a ZWNJ is the next-best thing. But it's great when it works, and I'm glad it exists. I have JS disabled by default and have lost count of the number of websites that use CSS to hide their content and JS to make it visible. Reader Mode defeats them easily.

deleted 5y ago
[deleted]
emsixteen 5y ago
Why use a ZWNJ and not just, well, an actual author name? I'm probably missing something here.
- Arnavion 5y ago
  Because it's my website and I get to decide whether to have an author name or not. It's bad enough that browsers assume my website, that's made up entirely of free-flowing text inside p tags, won't fit on phone screens unless I add a meta viewport tag. I'm not going to also make changes to the content to satisfy them.
  - account42 5y ago
    > It's bad enough that browsers assume my website, that's made up entirely of free-flowing text inside p tags, won't fit on phone screens unless I add a meta viewport tag. This anoys me to no end as well. It's especially ridiculous now that mobile browsers provide a "Desktop" mode that can be used as a fallback for those sites that do need a larger page width.

hans1729 5y ago

Ha, I just worked through this a week ago: https://searchfox.org/mozilla-release/source/toolkit/components/reader/Readability.js https://searchfox.org/mozilla-release/source/toolkit/compone...

mediumdeviation 5y ago

> for instance, any node with class skyscraper is considered hidden (maybe the logic is that those elements are shrouded by clouds). Ha, you wish. Skyscraper is the name of a standard Internet advertising unit. It’s fairly common for adblockers to block it. https://www.marketingterms.com/dictionary/skyscraper_ad/ https://www.marketingterms.com/dictionary/skyscraper_ad/

pantulis 5y ago
Is it still being used?
- zapt02 5y ago
  Yes, it's a standard industry term and still used on many sites.

busymom0 5y ago

My Hacker News client HACK for iOS and Android has reader mode ability browser. It can be set to be the default for hacker news sites in the app too. While on iOS, I was able to use the reader mode feature provided by SFSafariViewController, that wasn't available on android. So I had to read a ton about this. I ended up using a heavily modified Kotlin version of Readability: https://github.com/dankito/Readability4J https://github.com/dankito/Readability4J https://play.google.com/store/apps/details?id=com.pranapps.hack https://play.google.com/store/apps/details?id=com.pranapps.h... https://apps.apple.com/us/app/id1464477788 https://apps.apple.com/us/app/id1464477788

dewey 5y ago
Thanks for Hack, it's what I'm currently using!

tannhaeuser 5y ago

Wondering if this (plus Safari's reader mode heuristics if those are any different) could form the basis of a much-needed reduced HTML subset. Like, for the HTML we're actually searching, as opposed to the trackfest and seo'd articles search engines are giving us. Edit: so much for the illusion of "semantic HTML" ie where you need heuristics and are entering an arm's race vs publishers to make your HTML even readable

uniqueuid 5y ago

It's really nice that browsers offer reader modes, but they are frustratingly incomplete. Really, let's switch to a user perspective once and consider - what if I always want reader mode? This is technologically complete impossible and all the solutions are a band-aid. Firefox and others' attempts rely on the page authors' goodwill. But some pages will always attempt to frustrate reader modes. Alternative approaches for content extraction use machine learning such as [1], but they of course need to be updated for culture- language- and technology-specific changes. It's a mess and will remain so for the foreseeable future. [1] https://github.com/dragnet-org/dragnet https://github.com/dragnet-org/dragnet

classichasclass 5y ago
I specifically wrote always-available, "sticky" (default: stays Reader View even if you go to another page) and automatic (goes auto on subpages or entire domains, as you specify) Reader View support built-in to TenFourFox to help out on slower systems.
paxys 5y ago
It's a mess by design. In an ideal world no browser would need a reader mode, because pages themselves would be reader-friendly by design. Page authors, of course, are instead more interested in optimizing for advertising and tracking. Reader mode, much like adblock extensions, works directly against their interests. In fact we wouldn't even need any new web standards to implement something like this. Any website following the latest Web Content Accessibility Guidelines and using the correct page structure and aria- tags should be fully semantically parsable and displayable in any format or medium.
- Moru 5y ago
  I'm sure they can live without my views. If I can't read the page I can always press the back button. I'm not going to buy anything from any ads anyway so no loss on either side except a few seconds of time lost for me. Just got a link to a documentation for an API that I can't read because of too low contrast. Dark Reader plugin nicely fixed it so didn't even have to try reader mode. It sucks to get old anyway, you young people don't have to make it worse. You will be in the same seat in 20-odd years so plan for it :-)

productceo 5y ago

To HN users reading this comment, where did you find the best reader view implementation? Seems like it's a really difficult task.

miket 5y ago
https://www.diffbot.com/products/extract/ https://www.diffbot.com/products/extract/ works pretty well.

gumby 5y ago

I use safari and have it set to use reader view by default. Much more pleasant!

intsunny 5y ago

I'm always amazed at how Firefox's reader view circumvents so many javascript paywalls. I'm surprised at how incredibly mono-cultured web developers have become in testing browsers other than Chrome. OR .... they know, and they don't care ....

porker 5y ago
Often they don't care because this makes it easy for Google to still index their content and blocks most humans.

yason 5y ago

If only we had some descriptive language to mark up the actual, readable content of a web page in the simplest form and then have all the auxiliary content, table-of-contents, dynamic functionality, and styling described separately so that the browser would know what content is essential and what is optional...

stavros 5y ago

Thinking about this and the Gemini protocol, it seems to me that HTML has become too user-hostile, or at least it gives too much rope for site developers to hang users. The way I'd solve this problem would be to develop a new file format that was similar to Markdown but contained semantic elements (eg "this image is relevant to this bit of text", or "this text explains that text". This would be rendered into HTML by a browser extension so the browser could display it, which would help keep authors honest (because if the server renders it, we're just using HTML again). The format would contain no styling information at all, apart from the semantic tags, so styling would be entirely on the client. You could pick the way you wanted your articles to look, which would probably be different per device. This sounds a bit like epub and would probably be great for book reader devices as well, while being extremely light to load because there isn't much you can do with it.

superasn 5y ago
I think somebody came up with something similar to this called Aftertext(1). I remember it as I found the concept kinda interesting. Unfortunately the easy part is designing the protocol, the hard part is universal rendering and adoption. (1) https://news.ycombinator.com/item?id=29643313 https://news.ycombinator.com/item?id=29643313
- stavros 5y ago
  Hmm, that's an interesting idea, though I think the format looks too cumbersome. Still, that's mostly what I meant, thanks for the link!
jraph 5y ago
Unfortunately, it seems like a technical solution to a society-wide human problem to me.
- solarkraft 5y ago
  Tech people love throwing technology at social problems. Plain HTML is great for documents. It’s just the browsers‘ failure to render it nicely that makes authors feel like it’s necessary to style to make it presentable.

rcarmo 5y ago

I implemented a variation of the Readability algorithm some 9 years ago, in case anyone needs a server-side Python version and is interested in dragging it (kicking and screaming) into the 2020s: https://github.com/rcarmo/soup-strainer https://github.com/rcarmo/soup-strainer

leodriesch 5y ago
Mozilla’s readability library is also usable on the server with Node.js and JSDOM. That’s if you are fine with using JavaScript on the backend.

solarkraft 5y ago

I haven’t directly compared them, but I have also found mercury parser (https://github.com/postlight/mercury-parser https://github.com/postlight/mercury-parser) to be pretty reliable. Its advantage is that you can directly pass it a page instead of having to give it a DOM. Since it turns a website into very plain (X)HTML it‘s fairly easy to use it to make a browsing proxy or automatically produce epub files for e-readers, which is what I do. Edit: Here’s the proof of concept type code I use: https://gist.github.com/solarkraft/d6306f17a761fcb5ce47f2be73327213 https://gist.github.com/solarkraft/d6306f17a761fcb5ce47f2be7... It’s a bit crappy, but it works for me :-)

simonw 5y ago

If you're interested in playing with Mozilla Readability.js from the command line I came up with this recipe using my shot-scraper tool the other day: https://til.simonwillison.net/shot-scraper/readability https://til.simonwillison.net/shot-scraper/readability pip install shot-scraper shot-scraper install shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ " async () => { const readability = await import('https://cdn.skypack.dev/@mozilla/readability'); return (new readability.Readability(document)).parse(); }"

dotancohen 5y ago
shot-scraper is terrific, thank you! Right now I'm using a bash script instead of YAML file for screenshotting multiple sites I maintain, as there is no option to add a timestamp to the filenames, something like this (simplified): declare -A capture capture[www.foo.com]=https://www.foo.com/ capture[www.foo.com-bar]=https://www.foo.com/bar for key in "${!capture[@]}" ; do shot-scraper ${capture[$key]} -o - > $key_$(date +"%Y-%m-%d_%H_%M") done Otherwise it is a terrific tool, able to screenshot websites that are otherwise very difficult due to CDNs, javascript, etc. Thank you very much!

longnguyen 5y ago

Readability is amazing. I find it works most of the time. Currently I'm using it to parse a web article, package into a .mobi file and send it directly to my Kindle Paperwhite. If you want to read your favorite content on Kindle, give it a try[0] [0]: https://ktool.io https://ktool.io

zulln 5y ago
For websites, do you have examples where it outperform Amazon's own solution? https://www.amazon.com/sendtokindle/ https://www.amazon.com/sendtokindle/
- longnguyen 5y ago
  KTool[0] is fairly new so I can't claim it "outperform" Amazon's solution. However if you read the recent reviews from their Chrome Extension page[1], the majority of complaints was that the article never arrived. I just tried sending this article[2] and 10 minutes later it still hasn't delivered yet. This article contains a lot of images, so I understand it's not going to be quick, but with KTool, it does deliver within 2 minutes. But that is not the only reason why I build KTool. I don't want the articles to include links when reading on my Kindle because it's bad UX (easy to mis-tap), distracting and reduce comprehension (see here[3] and here[4]). Instead, I push all these links to the bottom of the article, which make it still accessible yet improve the reading experience. [0]: https://ktool.io https://ktool.io [1]: https://chrome.google.com/webstore/detail/send-to-kindle-for-google/cgdjpilhipecahhcilnafpblkieebhea https://chrome.google.com/webstore/detail/send-to-kindle-for... [2]: https://waitbutwhy.com/2017/04/neuralink.html https://waitbutwhy.com/2017/04/neuralink.html [3]: https://www.wired.com/2010/05/ff-nicholas-carr/ https://www.wired.com/2010/05/ff-nicholas-carr/ [4]: https://www.roughtype.com/?p=1378 https://www.roughtype.com/?p=1378
  - longnguyen 5y ago
    Update: 5 hours later, no sign of the article being delivered. It think it's safe to assume the Amazon solution doesn't work ¯\_(ツ)_/¯
5y ago

adamgordonbell 5y ago

I found readability pretty useful for building a plain text version of articles. From there, I'd love to take a RSS feed and transform it to ebook. https://earthly-tools.com/text-mode https://earthly-tools.com/text-mode https://news.ycombinator.com/item?id=30829568 https://news.ycombinator.com/item?id=30829568

criddell 5y ago
Evernote (remember them?) had a pretty good tool called Clipper that would save a reader-version of the page to Evernote. It was pretty great, but I haven't used it lately.
cxr 5y ago
I'm surprised that more effort hasn't gone into hacking on "line mode" (terminal-based) Web browsers—esp. for reading developer documentation—and absence of the resulting effect that this should have among hackers on their own sites. I find most people who consider their projects to be "no-frills" have quite a few frills indeed. This includes personal blogs, but Sourcehut and the constant praise for its "clean" UI is one other example of this sort of thing. Load it up in w3m or Lynx and it very much feels like a site intended to be read in a conventional desktop browser but filtered and constrained to a text-only medium. Actually clean design would start with, "How would I expect this to feel if I made a custom, terminal-based (but not command-line driven) Sourcehut client?" and then figuring out the appropriate markup you'd need to spit out to achieve that in a line mode browser that understands the basics—forms, content ordering, etc. (PS: Now transport that same artifact to a conventional browser. Is the experience better or worse that what's currently available?)

BuckRogers 5y ago

I used Firefox for 19 years without interruption until last summer when I switched off of it. Its reader view was one of its best features. It works on more pages than competing browser's reading modes. While they removed most of the features that I had grown to love over the past two decades, another standout for me was the ability to have multiple picture in picture instances running.

bni 5y ago

I use this in a proxy to be able to browse the web on my Amiga 1200

jokoon 5y ago

I think it won't work on techcrunch, since it somehow blurs the page. But generally it's awesome.

JangoSteve 5y ago

This seems to be a prime example of the Pareto Principle, or the 80/20 rule. About 12 years ago, I had built something to do something similar and published how I did it [1]. It was a very simple process, which my younger self called "the meat algorithm", as in, how to get the meat of an article. It was far less code and worked perfectly 95% of the time (though, the average web-page was also a little simpler 12 years ago). But that code would have quickly ballooned out if our use-case had called for addressing the other 5% of webpages, as the Firefox Reader View must do. [1] https://www.alfajango.com/blog/create-a-printable-format-for-any-webpage-with-ruby-and-nokogiri https://www.alfajango.com/blog/create-a-printable-format-for...

acrophobic 5y ago

I've been working on several web extractors project, so I think I could share some of my findings while working on them. Granted it's been several months since I worked on it so I might be forgetting some things. There are several open source projects for extracting web contents. However, there are three extractors that I've worked with and give us good result: - readability.js[1], web extractor by Mozilla that used in Firefox. - dom-distiller[2], web extractor by Chromium team, written in Java. - trafilatura[3], Python package by Adrien Barbaresi from BBAW[4]. First, readability.js, as expected is the most famous extractor. It's a single file Javascript library with modest 2,000+ lines of code, released under Apache license. Since it's in JS, you can use it wherever you want, either in web page using `script` tag or by using it in Node project. Next, DomDistiller is extractor that used in Chromium. It uses Java language with whopping 14,000+ lines of code and can only be used as part of Chromium browser, so you can't exactly use it as standalone library or CLI. Finally, Trafilatura is a Python package released under GPLv3 license. Created in order to build a text databases[5] for NLP research, it mainly intended for German web pages. However, as development continues, it works really great with other languages. It's a bit slow though compared to Readability.js. All of those three work in similar way: extract metadata, remove unneeded contents, and finally returns the cleaned up content. Their differences (that I remembered) are: - In Readability, they insist to make no special rules for any website, while DomDistiller and Trafilatura give a small exception for popular sites like Wikipedia. Thanks to this, if you use Readability.js in Wikipedia pages, it will shows `[edit]` button thorough the extracted content. - Readability has a small function to detect whether a web page can be converted to reader mode. While it's not really accurate, it's quite convenient to have. - In DomDistiller, the metadata extraction is more thorough than the others. It supports OpenGraph, Schema.org, and even the old IE Reading View mark up tags. - Since DomDistiller is only usable within Chromium, it has the advantage to be able to use CSS styling to determine if an element is important or not. If an element is styled to be invisible (e.g. `display: none`) then it will be deemed unimportant. However, according to a research[6] this step is actually doesn't really affect the extraction result. - DomDistiller also has an experimental feature to find and extract next page in sites that separated its article to several partial pages. - For Trafilatura, since it was created for collecting web corpus, it main ability is extracting text and the publication date of a web page. For the latter, they've created a Python package named htmldate[7] whose only purpose is to extract the publication or modification date for a web page. - Trafilatura also has an experimental feature to remove elements that repeated too often. The idea is if the element occured too often, then it's not important to the reader. I've found benchmark[8] that compare the performance between the extractors, and it said that Trafilatura has the best accuracy compared to the others. However, before you start rushing to use Trafilatura, you should remember that Trafilatura is intended for gathering web corpus, so it's really great for extracting text content, but IIRC is not as good as Readability.js and DomDistiller for extracting a proper article with images and embedded iframes (depending on how you look, it could be a feature though). By the way, if you are using Go and need to use a web extractor, I already ported all three of them to Go[9][10][11] including their dependencies[12][13], so have fun with it. [1]: https://github.com/mozilla/readability https://github.com/mozilla/readability [2]: https://github.com/chromium/dom-distiller https://github.com/chromium/dom-distiller [3]: https://github.com/adbar/trafilatura https://github.com/adbar/trafilatura [4]: https://www.bbaw.de/en/ https://www.bbaw.de/en/ [5]: https://www.dwds.de/d/k-web https://www.dwds.de/d/k-web [6]: https://arxiv.org/abs/1811.03661 https://arxiv.org/abs/1811.03661 [7]: https://github.com/adbar/htmldate https://github.com/adbar/htmldate [8]: https://github.com/scrapinghub/article-extraction-benchmark https://github.com/scrapinghub/article-extraction-benchmark [9]: https://github.com/go-shiori/go-readability https://github.com/go-shiori/go-readability [10]: https://github.com/markusmobius/go-domdistiller https://github.com/markusmobius/go-domdistiller [11]: https://github.com/markusmobius/go-trafilatura https://github.com/markusmobius/go-trafilatura [12]: https://github.com/markusmobius/go-htmldate https://github.com/markusmobius/go-htmldate [13]: https://github.com/markusmobius/go-dateparser https://github.com/markusmobius/go-dateparser

Stampo00 5y ago

Thank you for this! I've always wondered where this code lives because I love reader view and I'd like to use it as a base style for my blog.

zerop 5y ago

Can this engine be used to build something like pocket? Does pocket use the same tech to fetch content of pages ?

masukomi 5y ago

For those wondering if there's a redability lib in their favorite language. Here's a list of them all (as far as i know) plus the original arc-90 implementation https://github.com/masukomi/arc90-readability/#readability https://github.com/masukomi/arc90-readability/#readability Please submit a PR if there's something i don't have listed there.