4 ms·

I can't help but feel a little conflicted about things like this in the current climate. On one hand, as a developer trying to just get some updated stuff into

I can't help but feel a little conflicted about things like this in the current climate. On one hand, as a developer trying to just get some updated stuff into a spreadsheet, this looks extremely useful. On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.

imathrowaway 3y ago
Agreed, we've seen a lot of governing bodies _now_ start scraping to fight scam sites and "fake" sites, so there's a lot of legitimate use-cases popping up. It's not necessarily just about gathering e-commerce data or backfilling lack of an API anymore. Source: Founder@browserless.io
- Matheus28 3y ago
  So how will you ensure your service is only used for "good"?
  - 67j67j67n 3y ago
    no services remain "good" so why ask the question. it gets ignored or answered but then a decade or two down the line the philosophy gets scrapped, a story as old as time.
    - chankstein38 3y ago
      I still don't understand why Google publicly removed "Do No Evil" from their company. It's one thing to not follow it but to actively remove it feels weird. Like they wanted to say "HEY Y'ALL GET READY FOR EVIL!"
      - ziddoap 3y ago
        You can go to https://abc.xyz/investor/google-code-of-conduct/ https://abc.xyz/investor/google-code-of-conduct/ and ctrl+f "evil", and you'll see it's still there.
        chankstein38 3y ago
        Interesting I remember a bunch of articles and stuff talking about them removing the do no evil bit. Guess I didn't do enough research. Appreciate the correction!
        qingcharles 3y ago
        Can confirm this Google document contains the word "evil."
        deleted 3y ago
        [deleted]
  - mattigames 3y ago
    Same way the cooking-knives company ensures those are only used for cooking.
- scarface_74 3y ago
  I fail to see any legitimate use case for getting around protections that web dm developers put in to help prevent this.
- mmcclure 3y ago
  I know this is slightly off topic, but I have to ask: Why not create a new account rather than use the same self-described throwaway account you used to ask about raising prices (for browserless?)[1]? For what it's worth, I initially assumed you likely weren't legitimate and didn't engage, so if you want to keep contributing to threads as yourself you might want to consider using a new/different username. [1] https://news.ycombinator.com/item?id=21271224 https://news.ycombinator.com/item?id=21271224
- hatsix 3y ago
  the problem isn't scraping, the problem is the "content" that is being "created". A site I worked on changed our captcha, and in the 24 hours after, we had a 90% reduction in new accounts. We had a full API including webhooks, the only private data was on the account page, there was no reason to scrape, but even if you did, there was no reason to be logged in... these accounts were used to harass, intimidate and scam legitimate users.
deleted 3y ago
[deleted]
zexodus 3y ago
Every single time I see these scraping discussions I get the same thoughts: Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app? At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.
- cyanydeez 3y ago
  the growing IP centipede is an interesting GIGO dilemma
- arp242 3y ago
  If the data is public or semi-public then chances are I consented to display that data there. I consented to it being used on that site, for the purpose of that site. Not for random other companies. And most scraping isn't done by users. It's done by companies. For profit. Often for less than enlightened reasons. LinkedIn is a good example: I want my data displayed to people on that job site. I don't want it harvested by every recruiter under the sun who will then spam me. I certainly don't want that data sold between those recruiters long after I deleted my account on LinkedIn. Tinder and sites like that are also an obvious example: yes it's (semi-)public, but I also wouldn't want it to be scraped and harvested by some company – I just want it to be shown temporarily to a limited set of people.
  - diego_sandoval 3y ago
    In general, I don't think people should have a moral right to decide where and how the data that they made public is used, or to decide if it can get scraped or not. And, in general, I take the fact that you published something on the Internet as a tacit moral consentment for the rest of the world to use it how they want. This comes with a couple of big asterisks, because (1) Copyright law exists, and I generally try to not break the law, even if I don't agree with it. But the discussion in this thread is mostly separate from copyright: for instance, I don't think a court would see someone scraping and redistributing data from someone's LinkedIn profile as a copyright infringement case. And (2) because I think that in some specific cases, using published data can be morally wrong, but not as a general rule.
parentheses 3y ago
It's community hostile. The problem is we don't consider the internet a community.
- throw10920 3y ago
  That's incorrect. The fact that the internet is not a community is not a problem - it's objectively correct and morally desirable. It's neither possible, nor desirable, nor good, to treat the internet as a community. It isn't, and we wouldn't want it to be in the first place, just like we wouldn't want all the nations of the world to be fused together.
  - imp0cat 3y ago
    It may not be now, but some people remember the time when the internet felt more like a community and it was glorious.
    - throw10920 3y ago
      That was only possible because the internet was far smaller. There are now billions of people with internet access - the cat is out of the bag. The best you can do is try to form your own little community in a corner of the internet - which is actually a great idea. Mass social media has so many problems - trying to make a little nook for yourself and a small group of enthusiasts/friends seems like the best way to go.
contravariant 3y ago
On the other hand, I'm not sure if the whole idea of people visiting a website where they ought to behave and play by the house-rules is doing the world much good. And I'm not just saying this because one of my pet projects was crippled by an overzealous bot-prevention system (yes I was using a program I wrote myself to access your system, no that doesn't make me a bot). Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable. Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.
dheera 3y ago
> keep stuff accessible to humans and not have it scraped for models I suspect that over the next 100 years, as AI and biotech come together, the boundaries between human and bot will start to blur. At some point it will be unethical to differentiate between human, bot, and everything in-between. I argue that that time is now, if at the very least for the sake of AI-based assistive technologies to be developed to turn a disabled {100% human} into a highly functional {80% human + 20% bot}. Bot detection should, for this reason, be doomed as an ADA violation.
1vuio0pswjnm7 3y ago
Perhaps the "conflict" lies in the notion that a "developer trying to just get some stuff updated into a spreadsheet" is somehow different from "people right now that want to keep stuff acceessible to humans and not have it scraped for models [for use by developers]." What if all users of the www were treated equally. There is in practice a conflict of interest between www users calling themselves "developers" and the rest of www users. Yet these "developers" purport to act 100% on behalf of the rest of www users, not themselves. The failure to acknowledge any self-interest is dishonest.
3abiton 3y ago
I agree, being a dev myself hosting few website, and a data enthusiast, it is very hard to balance both, except if all data was open source and released for all it's owners. All online platforms, especially social media relay on users input (reddit, HN, Twitter, etc ...) yet the trend seems to be heading the other way (closing down apis, etc ...), imo this is quite unfair. Not to mention awful practices by shopping giants like Amazon and Aliexpress, Airlines, Booking.com (et al.) to profile people and "scam them". The data ethic question is lost on both sides, only the few decent ones that remain are the true victims, you mom'n'pop website.
throw10920 3y ago
The problem of scraping content for models should be solved on a legal level. Trying to do it on a technical level is mostly futile, and wrong, because it actively makes legitimate uses worse. If you really really really want a technical mitigation in addition to a legal one, then paywalls are vastly better than incredibly messy and problematic heuristic bot-detection. There's a huge payment problem for content anyway, so paywalls would mitigate or solve many other problems in addition to the AI-training-data one.

imathrowaway 3y ago

Agreed, we've seen a lot of governing bodies _now_ start scraping to fight scam sites and "fake" sites, so there's a lot of legitimate use-cases popping up. It's not necessarily just about gathering e-commerce data or backfilling lack of an API anymore. Source: Founder@browserless.io

Matheus28 3y ago
So how will you ensure your service is only used for "good"?
- 67j67j67n 3y ago
  no services remain "good" so why ask the question. it gets ignored or answered but then a decade or two down the line the philosophy gets scrapped, a story as old as time.
  - chankstein38 3y ago
    I still don't understand why Google publicly removed "Do No Evil" from their company. It's one thing to not follow it but to actively remove it feels weird. Like they wanted to say "HEY Y'ALL GET READY FOR EVIL!"
    - ziddoap 3y ago
      You can go to https://abc.xyz/investor/google-code-of-conduct/ https://abc.xyz/investor/google-code-of-conduct/ and ctrl+f "evil", and you'll see it's still there.
      - chankstein38 3y ago
        Interesting I remember a bunch of articles and stuff talking about them removing the do no evil bit. Guess I didn't do enough research. Appreciate the correction!
      - qingcharles 3y ago
        Can confirm this Google document contains the word "evil."
      - deleted 3y ago
        [deleted]
- mattigames 3y ago
  Same way the cooking-knives company ensures those are only used for cooking.
scarface_74 3y ago
I fail to see any legitimate use case for getting around protections that web dm developers put in to help prevent this.
mmcclure 3y ago
I know this is slightly off topic, but I have to ask: Why not create a new account rather than use the same self-described throwaway account you used to ask about raising prices (for browserless?)[1]? For what it's worth, I initially assumed you likely weren't legitimate and didn't engage, so if you want to keep contributing to threads as yourself you might want to consider using a new/different username. [1] https://news.ycombinator.com/item?id=21271224 https://news.ycombinator.com/item?id=21271224
hatsix 3y ago
the problem isn't scraping, the problem is the "content" that is being "created". A site I worked on changed our captcha, and in the 24 hours after, we had a 90% reduction in new accounts. We had a full API including webhooks, the only private data was on the account page, there was no reason to scrape, but even if you did, there was no reason to be logged in... these accounts were used to harass, intimidate and scam legitimate users.

deleted 3y ago

[deleted]

zexodus 3y ago

Every single time I see these scraping discussions I get the same thoughts: Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app? At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.

cyanydeez 3y ago
the growing IP centipede is an interesting GIGO dilemma
arp242 3y ago
If the data is public or semi-public then chances are I consented to display that data there. I consented to it being used on that site, for the purpose of that site. Not for random other companies. And most scraping isn't done by users. It's done by companies. For profit. Often for less than enlightened reasons. LinkedIn is a good example: I want my data displayed to people on that job site. I don't want it harvested by every recruiter under the sun who will then spam me. I certainly don't want that data sold between those recruiters long after I deleted my account on LinkedIn. Tinder and sites like that are also an obvious example: yes it's (semi-)public, but I also wouldn't want it to be scraped and harvested by some company – I just want it to be shown temporarily to a limited set of people.
- diego_sandoval 3y ago
  In general, I don't think people should have a moral right to decide where and how the data that they made public is used, or to decide if it can get scraped or not. And, in general, I take the fact that you published something on the Internet as a tacit moral consentment for the rest of the world to use it how they want. This comes with a couple of big asterisks, because (1) Copyright law exists, and I generally try to not break the law, even if I don't agree with it. But the discussion in this thread is mostly separate from copyright: for instance, I don't think a court would see someone scraping and redistributing data from someone's LinkedIn profile as a copyright infringement case. And (2) because I think that in some specific cases, using published data can be morally wrong, but not as a general rule.

parentheses 3y ago

It's community hostile. The problem is we don't consider the internet a community.

throw10920 3y ago
That's incorrect. The fact that the internet is not a community is not a problem - it's objectively correct and morally desirable. It's neither possible, nor desirable, nor good, to treat the internet as a community. It isn't, and we wouldn't want it to be in the first place, just like we wouldn't want all the nations of the world to be fused together.
- imp0cat 3y ago
  It may not be now, but some people remember the time when the internet felt more like a community and it was glorious.
  - throw10920 3y ago
    That was only possible because the internet was far smaller. There are now billions of people with internet access - the cat is out of the bag. The best you can do is try to form your own little community in a corner of the internet - which is actually a great idea. Mass social media has so many problems - trying to make a little nook for yourself and a small group of enthusiasts/friends seems like the best way to go.

contravariant 3y ago

On the other hand, I'm not sure if the whole idea of people visiting a website where they ought to behave and play by the house-rules is doing the world much good. And I'm not just saying this because one of my pet projects was crippled by an overzealous bot-prevention system (yes I was using a program I wrote myself to access your system, no that doesn't make me a bot). Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable. Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.

dheera 3y ago

> keep stuff accessible to humans and not have it scraped for models I suspect that over the next 100 years, as AI and biotech come together, the boundaries between human and bot will start to blur. At some point it will be unethical to differentiate between human, bot, and everything in-between. I argue that that time is now, if at the very least for the sake of AI-based assistive technologies to be developed to turn a disabled {100% human} into a highly functional {80% human + 20% bot}. Bot detection should, for this reason, be doomed as an ADA violation.

1vuio0pswjnm7 3y ago

Perhaps the "conflict" lies in the notion that a "developer trying to just get some stuff updated into a spreadsheet" is somehow different from "people right now that want to keep stuff acceessible to humans and not have it scraped for models [for use by developers]." What if all users of the www were treated equally. There is in practice a conflict of interest between www users calling themselves "developers" and the rest of www users. Yet these "developers" purport to act 100% on behalf of the rest of www users, not themselves. The failure to acknowledge any self-interest is dishonest.

3abiton 3y ago

I agree, being a dev myself hosting few website, and a data enthusiast, it is very hard to balance both, except if all data was open source and released for all it's owners. All online platforms, especially social media relay on users input (reddit, HN, Twitter, etc ...) yet the trend seems to be heading the other way (closing down apis, etc ...), imo this is quite unfair. Not to mention awful practices by shopping giants like Amazon and Aliexpress, Airlines, Booking.com (et al.) to profile people and "scam them". The data ethic question is lost on both sides, only the few decent ones that remain are the true victims, you mom'n'pop website.

throw10920 3y ago

The problem of scraping content for models should be solved on a legal level. Trying to do it on a technical level is mostly futile, and wrong, because it actively makes legitimate uses worse. If you really really really want a technical mitigation in addition to a legal one, then paywalls are vastly better than incredibly messy and problematic heuristic bot-detection. There's a huge payment problem for content anyway, so paywalls would mitigate or solve many other problems in addition to the AI-training-data one.