5 ms·

Fathom: a framework for understanding web pages

146 points by nachtigall 9y ago

gavinpc 9y ago
We've been seeing more Datalog-inspired DSL's around here, and that's a good thing. Fathom surely has uses beyond those the OP mentioned. But as for those use cases... well, it just makes me sad. Obviously people have perverse incentives to make the kind of noise that Mozilla is bemoaning, and those people will find a way to game any system --- especially a highly readable one!
- grincho 9y ago
  > Fathom surely has uses beyond those the OP mentioned. (Author here.) In fact, Fathom isn't particularly coupled to the DOM, apart from the dom() call that acts as the initial source of data and some of its optional utility procedures. With a few tweaks, you could use it for any score-and-rank problem.
throwaway2016a 9y ago
I like this post but I disagree with the premise... the whole "webpages don't implement microformats and RDF so we're going to take away all their control" doesn't settle right with me. If stores using microformats and semantic markup is important then give value added to the places that support it and people will start using. If readability is important start penalizing sites with bad readability indexes. But please don't take over my UX. I can't think of a faster way to stifle innovation.[1] Try the documentation, much less political https://mozilla.github.io/fathom/intro.html#why https://mozilla.github.io/fathom/intro.html#why [1] That's an exaggeration. I can think of lots of better ways.
- r3bl 9y ago
  I agree with you on one hand that a browser shouldn't mess with the layout, but it could provide an alternative layout of the content (kind of like the reader mode does) in the click of a button, and I'll be more than happy to use it (the same way I got used to reading articles in a single, clean layout. But, how do you think that a browser could theoretically penalize a website?
  - throwaway2016a 9y ago
    > But, how do you think that a browser could theoretically penalize a website? Easier to do on a search engine. But for a browser they already do: many browsers are or plan to call out sites that don't use HTTPs. Or as another example: the warning that happens when you try to access a page that uses a self-signed certificate. Granted the penalty needs to be proportional to the crime. So maybe the penalty and value add go together. If the value add is a useful feature that everyone wants and demands then not having it becomes a penalty to the sites that don't offer it.
- CapacitorSet 9y ago
  >If readability is important start penalizing sites with bad readability indexes. That's relatively easy to do when you're Google and you develop both a browser and the most used search engine; not so much when you're Mozilla and have no apparent mean to apply pressure to websites.
- TeMPOraL 9y ago
  > But please don't take over my UX. The thing is, it's not your UX, it's my UX. The more power I have to easily clean out all the "design" crap websites throw at me, the better my experience and convenience in using the web.
  - throwaway2016a 9y ago
    > The thing is, it's not your UX, it's my UX. The more power I have to easily clean out all the "design" crap websites throw at me, the better my experience and convenience in using the web. In my opinion, absolutely not true. If a UX annoys you so much that you don't want to use a site... stop using that site. That is the power you have. Vote with your wallet. You have the power to deprive the site of your business. That's it. Not to remove ads, not to change the UX, just to stop using it.
    - TeMPOraL 9y ago
      But why? You're sending bytes to my computer, I get to do with them whatever I want. If you want to restrict me from doing that, DRM-ed protocols are the way to go.
      - throwaway2016a 9y ago
        You are receiving bytes that I pretty clearly own the copyright to. You have an implied limited license to the content. You are not buying the content. But to be clear. I actually lean towards agreeing with you up to a point. And I think the topic shifted slightly from my original intend. But it is my opinion that the point you are pushing it on other people (by bundling it with your browser) is where I draw the line. I'm okay if you, TeMPOraL do some post processing of the data but if Mozilla (a browser) or Comcast (an ISP) does it on your behalf without your say and without the content creators say. Especially considering that I am responsible for customer service on my own site.
        TeMPOraL 9y ago
        > but if Mozilla (a browser) or Comcast (an ISP) does it on your behalf without your say and without the content creators say. > Especially considering that I am responsible for customer service on my own site. That's a reasonable position I haven't considered. Thanks.
vinceguidry 9y ago
Would using something like this run you afoul of the CFAA for rules prohibiting web scraping?
- throwaway2016a 9y ago
  I haven't read of CFAA and scraping so I had to do some research and it is fascinating. I just read through http://www.sociallyawareblog.com/2014/07/21/data-for-the-taking-using-the-cfaa-to-combat-web-scraping/ http://www.sociallyawareblog.com/2014/07/21/data-for-the-tak... It seems that even though people keep trying to use CFAA for scraping it almost always gets thrown out as long as the source is publicly accessible. Even if the ToS prohibits scraping explicitly. Seems people have had more luck with just plain old copyright. But back to this tool in particular... depends entirely on what you are using the data for. Don't use it to gain access to data you wouldn't normally have or give other people access to data they wouldn't normally have and I can't see why CFAA applies.
- derefr 9y ago
  Is it scraping if you're doing it on a client device, to present to said client?
andy_ppp 9y ago
This looks incredible for starting to understand web content in useful ways. Will definitely give this a try for a product I'm building.
unabst 9y ago
> The browser could recognize a Log In link, follow it in the background, and log you in, As a web developer, this is exactly what I don't want. Just the other day I got bit by font boosting on Mobile Chrome. Couldn't figure out for the life of me why my h1 was bigger than I had explicitly specified. No traces of anything going on on the desktop either. Turns out, my page was being tampered with because someone at Chrome had a "brilliant" idea for mobile. The fix is to specify max-height to something inconsequential like 1000000px. My page just had a header, but font boosting destroys navigation menus and tiles too. Thank you but no thank you. There cannot be any conflict between browser and web developer. If we have to fight and hack one another, we're both failing. Font boosting needs to be explicit. And nothing should be done without the explicit intent of the author of the page. If the browser wants to provide automatic login, then great. Please outline exactly how to enable it, make it as easy and automated as possible, and help decrease my workload. Thank you and thank you.
- flukus 9y ago
  > There cannot be any conflict between browser and web developer. If we have to fight and hack one another, we're both failing. It sounds like you're understanding is wrong, You're css is a polite suggestion to the browser, nothing more. If you want pixel perfect rendering then publish PDF's.
  - unabst 9y ago
    Code should be explicit. There should be no "in between the lines" and the browser should not be guessing what the developer or consumer wants, let alone override what was explicitly declared. Then having to "politely suggest" max-height: 1000000px is not the right conversation.
    - braveo 9y ago
      I agree with you with the caveat that the customer themselves can, and should, be changing things to suit their own needs. But at that point if the customer screws something up, they understand that's on them.
acdha 9y ago
Previous discussion: https://github.com/mozilla/fathom https://github.com/mozilla/fathom
untangle 9y ago
Prev HN Comments: https://news.ycombinator.com/item?id=12060787 https://news.ycombinator.com/item?id=12060787
hnruss 9y ago
I looked at mozilla/activity-stream a bit to try to find some examples of fathom usage, but didn't find any. Then I figured out that it doesn't depend directly on fathom-web, it depends on page-metadata-parser (which then depends on fathom-web). Here's the code in that project which uses Fathom: https://github.com/mozilla/page-metadata-parser/blob/master/parser.js https://github.com/mozilla/page-metadata-parser/blob/master/...
nebabyte 9y ago
> That scores within 7% of Readability’s output on a selection of its own test cases "on a selection of" its own test cases = lol > Fathom is a data-flow language like Prolog, so data conveniently “turns up” when there are applicable rules that haven’t yet seen it Why are you trying to explain the concept of declarative programming without just telling people about declarative programming? Just because it's not common in a scene doesn't make it some proprietary concept you've just introduced there. > The best part is that Fathom rulesets are data That's actually kinda nice > In 70 lines, Insert that comic about 'just a few lines' masking the function calls that actually don't 'replace' a system, just change where work is being done when flexibility is not needed
- TeMPOraL 9y ago
  > Insert that comic about 'just a few lines' masking the function calls that actually don't 'replace' a system, just change where work is being done when flexibility is not needed Do you have link for that comic? I think haven't seen it. The concept you describe is very important though - abstractions like that are usually about moving complexity around (compare Turing tarpit - all Turing-complete languages are in principle equivalent; the difference is in their distribution of complexity along the axis of programs you want to write).
- grincho 9y ago
  You might prefer the more technical introduction at https://mozilla.github.io/fathom/intro.html#specific-areas-we-address https://mozilla.github.io/fathom/intro.html#specific-areas-w.... It talks about Fathom in terms of declarativeness. > "on a selection of" One has to start somewhere! At this early stage, my aim is to demonstrate that Fathom has value for simplifying the implementation of recognizers, not to claim a polished, production-ready Readability alternative. Though, frankly, getting to the latter would be a fun project for someone: just write more features (in the ML sense), and add more tuning data. There are lots of low-hanging TODOs in the code around https://github.com/mozilla/fathom/blob/master/examples/readability.js#L52 https://github.com/mozilla/fathom/blob/master/examples/reada....
dlwdlw 9y ago
Often times, trying to reduce illegibility kills the ecosystem. Centralized economies for example, though easier to understand and control, often destroyed economies. Certain choices cause divergence, they increase your optionality in the future. Other choices cause convergence to one thing. Convergence is only desired if and only if the one true thing is the one true thing. So neither divergent nor convergent thinking is good in itself unless you apply another filter on how you see the future. You either see messiness as indicative of the future having possibility, or you see the present as a broken world in need of fixing. (Or isolated areas of perfection needing protection and isolation)

146 points by nachtigall 9y ago

We've been seeing more Datalog-inspired DSL's around here, and that's a good thing. Fathom surely has uses beyond those the OP mentioned. But as for those use cases... well, it just makes me sad. Obviously people have perverse incentives to make the kind of noise that Mozilla is bemoaning, and those people will find a way to game any system --- especially a highly readable one!

grincho 9y ago
> Fathom surely has uses beyond those the OP mentioned. (Author here.) In fact, Fathom isn't particularly coupled to the DOM, apart from the dom() call that acts as the initial source of data and some of its optional utility procedures. With a few tweaks, you could use it for any score-and-rank problem.

throwaway2016a 9y ago

I like this post but I disagree with the premise... the whole "webpages don't implement microformats and RDF so we're going to take away all their control" doesn't settle right with me. If stores using microformats and semantic markup is important then give value added to the places that support it and people will start using. If readability is important start penalizing sites with bad readability indexes. But please don't take over my UX. I can't think of a faster way to stifle innovation.[1] Try the documentation, much less political https://mozilla.github.io/fathom/intro.html#why https://mozilla.github.io/fathom/intro.html#why [1] That's an exaggeration. I can think of lots of better ways.

r3bl 9y ago
I agree with you on one hand that a browser shouldn't mess with the layout, but it could provide an alternative layout of the content (kind of like the reader mode does) in the click of a button, and I'll be more than happy to use it (the same way I got used to reading articles in a single, clean layout. But, how do you think that a browser could theoretically penalize a website?
- throwaway2016a 9y ago
  > But, how do you think that a browser could theoretically penalize a website? Easier to do on a search engine. But for a browser they already do: many browsers are or plan to call out sites that don't use HTTPs. Or as another example: the warning that happens when you try to access a page that uses a self-signed certificate. Granted the penalty needs to be proportional to the crime. So maybe the penalty and value add go together. If the value add is a useful feature that everyone wants and demands then not having it becomes a penalty to the sites that don't offer it.
CapacitorSet 9y ago
>If readability is important start penalizing sites with bad readability indexes. That's relatively easy to do when you're Google and you develop both a browser and the most used search engine; not so much when you're Mozilla and have no apparent mean to apply pressure to websites.
TeMPOraL 9y ago
> But please don't take over my UX. The thing is, it's not your UX, it's my UX. The more power I have to easily clean out all the "design" crap websites throw at me, the better my experience and convenience in using the web.
- throwaway2016a 9y ago
  > The thing is, it's not your UX, it's my UX. The more power I have to easily clean out all the "design" crap websites throw at me, the better my experience and convenience in using the web. In my opinion, absolutely not true. If a UX annoys you so much that you don't want to use a site... stop using that site. That is the power you have. Vote with your wallet. You have the power to deprive the site of your business. That's it. Not to remove ads, not to change the UX, just to stop using it.
  - TeMPOraL 9y ago
    But why? You're sending bytes to my computer, I get to do with them whatever I want. If you want to restrict me from doing that, DRM-ed protocols are the way to go.
    - throwaway2016a 9y ago
      You are receiving bytes that I pretty clearly own the copyright to. You have an implied limited license to the content. You are not buying the content. But to be clear. I actually lean towards agreeing with you up to a point. And I think the topic shifted slightly from my original intend. But it is my opinion that the point you are pushing it on other people (by bundling it with your browser) is where I draw the line. I'm okay if you, TeMPOraL do some post processing of the data but if Mozilla (a browser) or Comcast (an ISP) does it on your behalf without your say and without the content creators say. Especially considering that I am responsible for customer service on my own site.
      - TeMPOraL 9y ago
        > but if Mozilla (a browser) or Comcast (an ISP) does it on your behalf without your say and without the content creators say. > Especially considering that I am responsible for customer service on my own site. That's a reasonable position I haven't considered. Thanks.

vinceguidry 9y ago

Would using something like this run you afoul of the CFAA for rules prohibiting web scraping?

throwaway2016a 9y ago
I haven't read of CFAA and scraping so I had to do some research and it is fascinating. I just read through http://www.sociallyawareblog.com/2014/07/21/data-for-the-taking-using-the-cfaa-to-combat-web-scraping/ http://www.sociallyawareblog.com/2014/07/21/data-for-the-tak... It seems that even though people keep trying to use CFAA for scraping it almost always gets thrown out as long as the source is publicly accessible. Even if the ToS prohibits scraping explicitly. Seems people have had more luck with just plain old copyright. But back to this tool in particular... depends entirely on what you are using the data for. Don't use it to gain access to data you wouldn't normally have or give other people access to data they wouldn't normally have and I can't see why CFAA applies.
derefr 9y ago
Is it scraping if you're doing it on a client device, to present to said client?

andy_ppp 9y ago

This looks incredible for starting to understand web content in useful ways. Will definitely give this a try for a product I'm building.

unabst 9y ago

> The browser could recognize a Log In link, follow it in the background, and log you in, As a web developer, this is exactly what I don't want. Just the other day I got bit by font boosting on Mobile Chrome. Couldn't figure out for the life of me why my h1 was bigger than I had explicitly specified. No traces of anything going on on the desktop either. Turns out, my page was being tampered with because someone at Chrome had a "brilliant" idea for mobile. The fix is to specify max-height to something inconsequential like 1000000px. My page just had a header, but font boosting destroys navigation menus and tiles too. Thank you but no thank you. There cannot be any conflict between browser and web developer. If we have to fight and hack one another, we're both failing. Font boosting needs to be explicit. And nothing should be done without the explicit intent of the author of the page. If the browser wants to provide automatic login, then great. Please outline exactly how to enable it, make it as easy and automated as possible, and help decrease my workload. Thank you and thank you.

flukus 9y ago
> There cannot be any conflict between browser and web developer. If we have to fight and hack one another, we're both failing. It sounds like you're understanding is wrong, You're css is a polite suggestion to the browser, nothing more. If you want pixel perfect rendering then publish PDF's.
- unabst 9y ago
  Code should be explicit. There should be no "in between the lines" and the browser should not be guessing what the developer or consumer wants, let alone override what was explicitly declared. Then having to "politely suggest" max-height: 1000000px is not the right conversation.
  - braveo 9y ago
    I agree with you with the caveat that the customer themselves can, and should, be changing things to suit their own needs. But at that point if the customer screws something up, they understand that's on them.

acdha 9y ago

Previous discussion: https://github.com/mozilla/fathom https://github.com/mozilla/fathom

untangle 9y ago

Prev HN Comments: https://news.ycombinator.com/item?id=12060787 https://news.ycombinator.com/item?id=12060787

hnruss 9y ago

I looked at mozilla/activity-stream a bit to try to find some examples of fathom usage, but didn't find any. Then I figured out that it doesn't depend directly on fathom-web, it depends on page-metadata-parser (which then depends on fathom-web). Here's the code in that project which uses Fathom: https://github.com/mozilla/page-metadata-parser/blob/master/parser.js https://github.com/mozilla/page-metadata-parser/blob/master/...

nebabyte 9y ago

> That scores within 7% of Readability’s output on a selection of its own test cases "on a selection of" its own test cases = lol > Fathom is a data-flow language like Prolog, so data conveniently “turns up” when there are applicable rules that haven’t yet seen it Why are you trying to explain the concept of declarative programming without just telling people about declarative programming? Just because it's not common in a scene doesn't make it some proprietary concept you've just introduced there. > The best part is that Fathom rulesets are data That's actually kinda nice > In 70 lines, Insert that comic about 'just a few lines' masking the function calls that actually don't 'replace' a system, just change where work is being done when flexibility is not needed

TeMPOraL 9y ago
> Insert that comic about 'just a few lines' masking the function calls that actually don't 'replace' a system, just change where work is being done when flexibility is not needed Do you have link for that comic? I think haven't seen it. The concept you describe is very important though - abstractions like that are usually about moving complexity around (compare Turing tarpit - all Turing-complete languages are in principle equivalent; the difference is in their distribution of complexity along the axis of programs you want to write).
grincho 9y ago
You might prefer the more technical introduction at https://mozilla.github.io/fathom/intro.html#specific-areas-we-address https://mozilla.github.io/fathom/intro.html#specific-areas-w.... It talks about Fathom in terms of declarativeness. > "on a selection of" One has to start somewhere! At this early stage, my aim is to demonstrate that Fathom has value for simplifying the implementation of recognizers, not to claim a polished, production-ready Readability alternative. Though, frankly, getting to the latter would be a fun project for someone: just write more features (in the ML sense), and add more tuning data. There are lots of low-hanging TODOs in the code around https://github.com/mozilla/fathom/blob/master/examples/readability.js#L52 https://github.com/mozilla/fathom/blob/master/examples/reada....

dlwdlw 9y ago

Often times, trying to reduce illegibility kills the ecosystem. Centralized economies for example, though easier to understand and control, often destroyed economies. Certain choices cause divergence, they increase your optionality in the future. Other choices cause convergence to one thing. Convergence is only desired if and only if the one true thing is the one true thing. So neither divergent nor convergent thinking is good in itself unless you apply another filter on how you see the future. You either see messiness as indicative of the future having possibility, or you see the present as a broken world in need of fixing. (Or isolated areas of perfection needing protection and isolation)