3 ms·

I've been thinking about this for a few days but just realized that this is a complete end run around all web scraping in general. All 'adversarial compatibili

I've been thinking about this for a few days but just realized that this is a complete end run around all web scraping in general. All 'adversarial compatibility' from projects like Nitter, Teddit, Invidious, and youtube-dl go out the window. Any archive site (archive.org, archive.ph, etc.) can be blocked by sites requiring attestation. And just like the book industry was terrified of piracy and were 'rescued' by Kindle, so too will journalism outlets that can't find a business model flock to Google to save them. This is going to be rough.

deleted 3y ago
[deleted]
userbinator 3y ago
Any archive site (archive.org, archive.ph, etc.) can be blocked by sites requiring attestation. What will happen if such a thing actually happens is that the underground market for "trusted device" farms grows, not too different from what's currently already happening but possibly at a far larger scale. Of course, that means the financially motivated scraping services still keep going while the honest individuals wanting user-agent freedom get screwed, just like with many other forms of DRM...
- wraptile 3y ago
  This has been happening already. The market is trying really hard to price out web scraping through scraper detection technologies and it's kinda working - scraping is becoming non-existent in user-space apps. It's also extremely discriminatory. Try running a single scrape with a developing country's IP and Linux, you'll be blocked at TLS step lol
  - CalRobert 3y ago
    But of course search engines are fine
    - wraptile 3y ago
      Having your cake and eating it too is a natural goal of every business and honestly it was just a matter of time till web pages figured out they can have the benefits of public data and avoid the costs. Web scraping and botting is basically a solved problem too - just put a login gate for the data which allows you to legally litigate against scrapers and bots. Done. However, nobody wants to lose the benefits of public data so here we are.
      - CalRobert 3y ago
        I used to care about respecting robots.txt until it was clear that established search engines are fine but any newcomers can go right to hell.
  - altfredd 3y ago
    > The market is trying really hard to price out web scraping... scraping is becoming non-existent in user-space apps Uhh... Those two matters are pretty much unrelated to each other. Scraping is becoming non-existing because the era of static web pages has ended. No need to "scrap" when you have a nice, performant JSON REST API provided for you.
    - flagrant_taco 3y ago
      SSG vs SSR really has nothing to do with whether an API exists to provide the data you would otherwise need to scrape. When was the last time you saw a site with a JSON API providing metadata, like the json-ld for a product on an e-commerce site? Or an API just for the open graph data? How would you even discover these APIs for sites that you don't own? It's also worth noting that very, very few JSON APIs today are actually REST. They rarely include all the context needed, and in general JSON is much less useful than XML when you're talking to other APIs that you don't own since JSON can't easily describe the shape and datatypes of the content.
    - wraptile 3y ago
      > No need to "scrap" when you have a nice, performant JSON REST API provided for you. There are no performant json rest APIs provided these days though. The days of public APIs are long gone.
      - altfredd 3y ago
        HTML "APIs" weren't meant for public either. In practice, if there is a mobile app, there is an API. Whether it's creators object to your usage is mostly their own problem.
- jaflo 3y ago
  Basically the captcha solving industry.
dceddia 3y ago
Yeah, exactly this, and on top of that, it also conveniently for Google makes it impossible or wildly expensive to build an index of the web if most of it is behind this attestation stuff.
vxNsr 3y ago
the thought of this being used is making me much more strongly consider moving to firefox, there are still things I don't love about it, like many of the extensions I use are still chromium only, but now I really feel like I don't have a choice.

deleted 3y ago

[deleted]

userbinator 3y ago

Any archive site (archive.org, archive.ph, etc.) can be blocked by sites requiring attestation. What will happen if such a thing actually happens is that the underground market for "trusted device" farms grows, not too different from what's currently already happening but possibly at a far larger scale. Of course, that means the financially motivated scraping services still keep going while the honest individuals wanting user-agent freedom get screwed, just like with many other forms of DRM...

wraptile 3y ago
This has been happening already. The market is trying really hard to price out web scraping through scraper detection technologies and it's kinda working - scraping is becoming non-existent in user-space apps. It's also extremely discriminatory. Try running a single scrape with a developing country's IP and Linux, you'll be blocked at TLS step lol
- CalRobert 3y ago
  But of course search engines are fine
  - wraptile 3y ago
    Having your cake and eating it too is a natural goal of every business and honestly it was just a matter of time till web pages figured out they can have the benefits of public data and avoid the costs. Web scraping and botting is basically a solved problem too - just put a login gate for the data which allows you to legally litigate against scrapers and bots. Done. However, nobody wants to lose the benefits of public data so here we are.
    - CalRobert 3y ago
      I used to care about respecting robots.txt until it was clear that established search engines are fine but any newcomers can go right to hell.
- altfredd 3y ago
  > The market is trying really hard to price out web scraping... scraping is becoming non-existent in user-space apps Uhh... Those two matters are pretty much unrelated to each other. Scraping is becoming non-existing because the era of static web pages has ended. No need to "scrap" when you have a nice, performant JSON REST API provided for you.
  - flagrant_taco 3y ago
    SSG vs SSR really has nothing to do with whether an API exists to provide the data you would otherwise need to scrape. When was the last time you saw a site with a JSON API providing metadata, like the json-ld for a product on an e-commerce site? Or an API just for the open graph data? How would you even discover these APIs for sites that you don't own? It's also worth noting that very, very few JSON APIs today are actually REST. They rarely include all the context needed, and in general JSON is much less useful than XML when you're talking to other APIs that you don't own since JSON can't easily describe the shape and datatypes of the content.
  - wraptile 3y ago
    > No need to "scrap" when you have a nice, performant JSON REST API provided for you. There are no performant json rest APIs provided these days though. The days of public APIs are long gone.
    - altfredd 3y ago
      HTML "APIs" weren't meant for public either. In practice, if there is a mobile app, there is an API. Whether it's creators object to your usage is mostly their own problem.
jaflo 3y ago
Basically the captcha solving industry.

dceddia 3y ago

Yeah, exactly this, and on top of that, it also conveniently for Google makes it impossible or wildly expensive to build an index of the web if most of it is behind this attestation stuff.

vxNsr 3y ago

the thought of this being used is making me much more strongly consider moving to firefox, there are still things I don't love about it, like many of the extensions I use are still chromium only, but now I really feel like I don't have a choice.