15 ms·

Years ago I made a news aggregator site which involved dynamically digesting rss feeds and then generating an iframe display of the target story alongside relat

Years ago I made a news aggregator site which involved dynamically digesting rss feeds and then generating an iframe display of the target story alongside related stories from other sites. The site caught on in certain circles and pageviews skyrocketed within a few months. I was not the only person pursuing this strategy at the time. Before a year had gone by, almost all the major news sites had updated their pages with JavaScript intended to either block iframing altogether, or to “break out” of the iframe, redirecting the viewer to their actual page. We even got a few nasty letters and emails about it with terms like “litigation” casually thrown in. The project became untenable because of this and I shut it down. Fast forward a few years and imagine my dismay when I see Google Amp doing essentially the same thing - but on a much larger scale....

chrismorgan 6y ago
Mind you, the problems with iframe embedding are technical and security-related, and it’s quite possible that news sites broke iframing at that time not because of you or any one else specifically, but because that was about the time people started making noise about clickjacking attacks, and so they protected themselves against that form of technical abuse by blocking such embedding. AMP doesn’t suffer from that technical problem, instead saying “please implement your page in such a way that it’s safe to embed and can’t possibly cause trouble for either of us if we happen to host it instead of you”.
notjustanymike 6y ago
Years ago I worked as a frontend dev for Newsweek & The Daily Beast and saw the other side of this. It boiled down to money, brand, advertising and engagement. Wrapping the content in an iframe allowed the host to monetize our content by decorating it with ads. This would dillute the CTR on our ads, resulting in less money. Big sites enter into contracts with major brands for full page advertisements, which could include contractual requirements that theirs is the only ad on the page. Serving and decorating a site would create headaches for us to explain why their ads weren't alone. The iframe also reduced engagement as reader would only read the embedded article and move on. An actual visitor of the site had a far higher likelihood to navigate to other pages. Higher engagement resulted in more money and some brand loyalty. This also dramatically hurt sharing as the browser URL wasn't ours, but the hosts (which I also hate about AMP). Your site messed with a lot of the ways their business generated revenue, in an industry that was already struggling to adapt to "the internet".
- mathattack 6y ago
  It would seem like the content creator has the right to make their site as easy or hard to aggregate as they want. If you’re in the aggregation business, you have to convince them of the benefits to make it easier. If you can’t, you’re not adding value (to them) for what you’re taking.
  - Chris2048 6y ago
    There was a time when corps pushed the idea that you'd need permission to link to their content, where does this begin/end? seems to me there should be some control on behalf of the user wrt to how data is displayed: It maybe your content, but it's running on my hardware; you say "3rd party not allowed to run ads, only us" I say "my hardware, you aren't allowed to run ads either".
    - mpclark 6y ago
      ...then the publisher says "OK, you can't show my content then"
      - martimarkov 6y ago
        Then I scrape the page using JS from the browser side and render it dynamically. It’s all client side. Where would this sit on that line? It’s just an example but for the 1000 ppl trying to create laws there are 10000x trying to bypass them. I don’t think restrictions are the way forward but as of now I haven’t come up with a viable alternative.
        derivagral 6y ago
        If a brick and mortar store refuses services and kicks you out for disruption... breaking in at night, taking what you want, and leaving money is still illegal. I love web scraping (it and tampermonkey basically got me into CS) but lets not act as if this behavior has no moral precedent.
thrower123 6y ago
I realized today that it is nearly impossible to load anything in an iframe anymore... Which is really fun if you want to load content in tabs in the Microsoft Teams client, because that is how that works...
- 9HZZRfNlpR 6y ago
  You still can if you're the server owner.
oefrha 6y ago
> Google Amp doing essentially the same thing Iframe is a security trainwreck. You could clickjack Google's login widget iframe to steal account details as late as 2018, maybe even later.[1] Whatever legitimate gripes you may have with AMP, it's not the same as iframes at all. [1] https://news.ycombinator.com/item?id=17044518 https://news.ycombinator.com/item?id=17044518
- Lammy 6y ago
  So it's a privacy trainwreck now instead?
  - oefrha 6y ago
    AMP allows Google Analytics which all these sites are using anyway, and disallows the dozens of other third party trackers typically seen on these sites, so no, it's a net improvement, or at least not worse.
- musicale 6y ago
  > Iframe is a security trainwreck That seems like it should be relatively fixable. For example, iframes could be visible, clearly marked (including the URL), and sandboxed like any separate page. One could go further by insisting that they are display-only and can only be displayed rather than interacted with. Then an iframe would basically be a glorified jpg with a URL.
  - oefrha 6y ago
    Then you need to kiss embedded video players and login/auth widgets goodbye, which are like 95%[1] of what iframes are used for outside ads. But if you just want to embed completely static article, I suppose it could work, but then there’s no incentive for anyone to allow them. [1] Yes, I pulled that number out of my ____.
    - hyperdimension 6y ago
      "Deep Rectal Extraction"
paxys 6y ago
> Fast forward a few years and imagine my dismay when I see Google Amp doing essentially the same thing - but on a much larger scale.... Amp isn't even close to the same thing as iFrames. And the difference is they did it with the publishers' consent. Why is it okay to put their content on your own site?
- sbarre 6y ago
  While I agree with your objection to the comparison of AMP and iframes, I think "with the publisers' consent" is a tough phrasing to swallow in this context, given that many publishers were put in a position of having to do AMP or risk seeing their content pushed down the page and lose a ton of traffic.
  - Blahah 6y ago
    That's because the publishers are dependent on Google, because they have no discoverability of their own. It's a systemic problem but it's not Google's problem that they are the only discovery platform for most media outlets. Those brands don't have the inherent value or reputation they think they do, or it wouldn't matter where they were on google results.
wolco2 6y ago
The missing piece is you had no leverage. As PG says leverage is everything.
rkeene2 6y ago
Why were you dismayed if your idea was implemented, but better (larger scale, no iframe) ?
amoorthy 6y ago
So cool that you and notjustanymike worked on this idea. My startup, The Factual, does something similar, grouping related articles across the political spectrum and rating them for credibility in a transparent (if imperfect) manner. We don't use iframes so your clicks open a new page as with most aggregators. As for a business model we've found that a freemium model with a modest subscription fee is a good way to monetize while making this accessible to a wide audience.
deleted 6y ago
[deleted]
velomash 6y ago
These days it's a security concern to allow any random site to iFrame your property due to clickjacking. This is why you see X-Frame-Options headers blocking / whitelisting who can iFrame sites.
dheera 6y ago
Quarter-baked idea: I wish we had a better way to build these kinds of sites anonymously and in a decentralized fashion, like Bitcoin/Ethereum. No matter how much they want ligitate and threaten they wouldn't actually know who to send the threat to, and it wouldn't be possible to take it down.
- a1369209993 6y ago
  It's called TOR. Pressure your web browser maintainers to support it (not that that will help without antitrust action, but the problem isn't lack of a technical solution).
musicale 6y ago
> Before a year had gone by, almost all the major news sites had updated their pages with JavaScript intended to either block iframing altogether, or to “break out” of the iframe, redirecting the viewer to their actual page. We even got a few nasty letters and emails about it with terms like “litigation” casually thrown in. Web feature: works as intended, allowing aggregation and republishing Businesses/lawyers: we hate that, shut it down! > Fast forward a few years and imagine my dismay when I see Google Amp doing essentially the same thing Google has the deep pockets to defend itself against spurious lawsuits.
- underwater 6y ago
  AMP is opt in, so no drama there. What Google have is a giant big carrot in the form of the news carousel.
niutech 6y ago
You could use X-Frame-Bypass (https://github.com/niutech/x-frame-bypass https://github.com/niutech/x-frame-bypass) to circumvent frame busting.

chrismorgan 6y ago

Mind you, the problems with iframe embedding are technical and security-related, and it’s quite possible that news sites broke iframing at that time not because of you or any one else specifically, but because that was about the time people started making noise about clickjacking attacks, and so they protected themselves against that form of technical abuse by blocking such embedding. AMP doesn’t suffer from that technical problem, instead saying “please implement your page in such a way that it’s safe to embed and can’t possibly cause trouble for either of us if we happen to host it instead of you”.

notjustanymike 6y ago

Years ago I worked as a frontend dev for Newsweek & The Daily Beast and saw the other side of this. It boiled down to money, brand, advertising and engagement. Wrapping the content in an iframe allowed the host to monetize our content by decorating it with ads. This would dillute the CTR on our ads, resulting in less money. Big sites enter into contracts with major brands for full page advertisements, which could include contractual requirements that theirs is the only ad on the page. Serving and decorating a site would create headaches for us to explain why their ads weren't alone. The iframe also reduced engagement as reader would only read the embedded article and move on. An actual visitor of the site had a far higher likelihood to navigate to other pages. Higher engagement resulted in more money and some brand loyalty. This also dramatically hurt sharing as the browser URL wasn't ours, but the hosts (which I also hate about AMP). Your site messed with a lot of the ways their business generated revenue, in an industry that was already struggling to adapt to "the internet".

mathattack 6y ago
It would seem like the content creator has the right to make their site as easy or hard to aggregate as they want. If you’re in the aggregation business, you have to convince them of the benefits to make it easier. If you can’t, you’re not adding value (to them) for what you’re taking.
- Chris2048 6y ago
  There was a time when corps pushed the idea that you'd need permission to link to their content, where does this begin/end? seems to me there should be some control on behalf of the user wrt to how data is displayed: It maybe your content, but it's running on my hardware; you say "3rd party not allowed to run ads, only us" I say "my hardware, you aren't allowed to run ads either".
  - mpclark 6y ago
    ...then the publisher says "OK, you can't show my content then"
    - martimarkov 6y ago
      Then I scrape the page using JS from the browser side and render it dynamically. It’s all client side. Where would this sit on that line? It’s just an example but for the 1000 ppl trying to create laws there are 10000x trying to bypass them. I don’t think restrictions are the way forward but as of now I haven’t come up with a viable alternative.
      - derivagral 6y ago
        If a brick and mortar store refuses services and kicks you out for disruption... breaking in at night, taking what you want, and leaving money is still illegal. I love web scraping (it and tampermonkey basically got me into CS) but lets not act as if this behavior has no moral precedent.

thrower123 6y ago

I realized today that it is nearly impossible to load anything in an iframe anymore... Which is really fun if you want to load content in tabs in the Microsoft Teams client, because that is how that works...

9HZZRfNlpR 6y ago
You still can if you're the server owner.

oefrha 6y ago

> Google Amp doing essentially the same thing Iframe is a security trainwreck. You could clickjack Google's login widget iframe to steal account details as late as 2018, maybe even later.[1] Whatever legitimate gripes you may have with AMP, it's not the same as iframes at all. [1] https://news.ycombinator.com/item?id=17044518 https://news.ycombinator.com/item?id=17044518

Lammy 6y ago
So it's a privacy trainwreck now instead?
- oefrha 6y ago
  AMP allows Google Analytics which all these sites are using anyway, and disallows the dozens of other third party trackers typically seen on these sites, so no, it's a net improvement, or at least not worse.
musicale 6y ago
> Iframe is a security trainwreck That seems like it should be relatively fixable. For example, iframes could be visible, clearly marked (including the URL), and sandboxed like any separate page. One could go further by insisting that they are display-only and can only be displayed rather than interacted with. Then an iframe would basically be a glorified jpg with a URL.
- oefrha 6y ago
  Then you need to kiss embedded video players and login/auth widgets goodbye, which are like 95%[1] of what iframes are used for outside ads. But if you just want to embed completely static article, I suppose it could work, but then there’s no incentive for anyone to allow them. [1] Yes, I pulled that number out of my ____.
  - hyperdimension 6y ago
    "Deep Rectal Extraction"

paxys 6y ago

> Fast forward a few years and imagine my dismay when I see Google Amp doing essentially the same thing - but on a much larger scale.... Amp isn't even close to the same thing as iFrames. And the difference is they did it with the publishers' consent. Why is it okay to put their content on your own site?

sbarre 6y ago
While I agree with your objection to the comparison of AMP and iframes, I think "with the publisers' consent" is a tough phrasing to swallow in this context, given that many publishers were put in a position of having to do AMP or risk seeing their content pushed down the page and lose a ton of traffic.
- Blahah 6y ago
  That's because the publishers are dependent on Google, because they have no discoverability of their own. It's a systemic problem but it's not Google's problem that they are the only discovery platform for most media outlets. Those brands don't have the inherent value or reputation they think they do, or it wouldn't matter where they were on google results.

wolco2 6y ago

The missing piece is you had no leverage. As PG says leverage is everything.

rkeene2 6y ago

Why were you dismayed if your idea was implemented, but better (larger scale, no iframe) ?

amoorthy 6y ago

So cool that you and notjustanymike worked on this idea. My startup, The Factual, does something similar, grouping related articles across the political spectrum and rating them for credibility in a transparent (if imperfect) manner. We don't use iframes so your clicks open a new page as with most aggregators. As for a business model we've found that a freemium model with a modest subscription fee is a good way to monetize while making this accessible to a wide audience.

deleted 6y ago

[deleted]

velomash 6y ago

These days it's a security concern to allow any random site to iFrame your property due to clickjacking. This is why you see X-Frame-Options headers blocking / whitelisting who can iFrame sites.

dheera 6y ago

Quarter-baked idea: I wish we had a better way to build these kinds of sites anonymously and in a decentralized fashion, like Bitcoin/Ethereum. No matter how much they want ligitate and threaten they wouldn't actually know who to send the threat to, and it wouldn't be possible to take it down.

a1369209993 6y ago
It's called TOR. Pressure your web browser maintainers to support it (not that that will help without antitrust action, but the problem isn't lack of a technical solution).

musicale 6y ago

> Before a year had gone by, almost all the major news sites had updated their pages with JavaScript intended to either block iframing altogether, or to “break out” of the iframe, redirecting the viewer to their actual page. We even got a few nasty letters and emails about it with terms like “litigation” casually thrown in. Web feature: works as intended, allowing aggregation and republishing Businesses/lawyers: we hate that, shut it down! > Fast forward a few years and imagine my dismay when I see Google Amp doing essentially the same thing Google has the deep pockets to defend itself against spurious lawsuits.

underwater 6y ago
AMP is opt in, so no drama there. What Google have is a giant big carrot in the form of the news carousel.

niutech 6y ago

You could use X-Frame-Bypass (https://github.com/niutech/x-frame-bypass https://github.com/niutech/x-frame-bypass) to circumvent frame busting.