4 ms·

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there. Unethi

> Seems very unethical, no? Who uses service providers like this? The whole point of anti-bot measures is to get rid of bots - you are not wanted there. Unethical just because it does something someone else doesn't want? I guess it depends on why and what the intention is. I don't have time to sit 24/7 in front of a computer to get a ticket to some events, does that mean it's unethical for me to use my own bot so I can purchase a ticket to bands I'm a fan of? Probably not. But if I did so for scalping purposes? Then yeah, I'd agree it's unethical. The whole point of anti-anti-bot measures is to be able to do things even if others don't think that thing should be automated, so from the hacker news audience, I think quite a lot of us have at one point or another engaged in stuff like that. Doing so merely for profits of course stinks, but for you to be able to have a fighting chance against scalpers? Probably OK.

joatmon-snoo 4mo ago
An example I ran into recently: I wanted to scrape pricing data for used cars, to better inform a friend's decision about what to purchase. I know there's a relationship between mileage and depreciation, but wanted to have a better sense of what that relationship is to know whether a given car was over or underpriced. Similarly, if I was pulling that data to build a service of my own to offer to users... is that unethical?
- sroussey 4mo ago
  All of these questions are easily answered by the question: can I run the bot on the same PC I use regularly? If so, then do it there. If not, then don’t do it at all.
  - fc417fc802 4mo ago
    Why should technical capability to evade countermeasures dictate whether or not something is ethical? My view is that scraping remains ethical as long as your actions aren't causing technical problems for the operator. If anything, a retailer attempting to hide pricing data is what's unethical in my view.
  - pocksuppet 4mo ago
    This is often really good for your bot, because anti-bot providers are loathe to block what, as far as they know, is a residential CGNAT address. Sometimes you get more success scraping from home with Firefox or Chrome, than with an army of proxy networks.
- adolph 4mo ago
  > scrape pricing data for used cars Time was you could get lovely json feeds from every site by iterating the inspector curl statement. Now-a-days you can't even use Selenium without Cloudflare getting grouchy. Last fall had to make my spreadsheet like a cave-person control c, control v. It wouldn't be so bad if the dealer aggregators' coverage was xor, but you have to dedupe listings. Then there is the whole online salespeople who don't show up at the dealership.
  - inigyou 4mo ago
    There's a JavaScript property called navigator.webdriver that returns true if selenium is in use. Obviously, every antibot system checks it. Obviously, you can patch it to always say false.
mystifyingpoi 4mo ago
> even if others don't think that thing should be automated It's an interesting thought that can be further explored. Could anything that's considered "unwanted" by a third party considered unethical, if I do it anyway? If the hotel self-service restaurant has a sign "don't take the food out" and I take 1 apple in my pocket for a snack, is it unethical? Or maybe the sign is just for people that would otherwise take $100 of watermelons out of the cantina daily and try to resell it on the beach.
- Nathanba 4mo ago
  We ("society") do things to people that they don't want all the time, namely punishing them via the legal system. Then there are things that other people are doing that are immoral and should be punished for even though they are not illegal. And the whole class of inactions where we don't e.g buy something because it's overpriced and that's certainly something the seller doesn't like.
skybrian 4mo ago
What do you think of Anubis and Cloudflare? If they block your bot, is that unethical? Seems like doing business with other people should normally be based on mutual consent, not whatever you can get away with technically.
- inigyou 4mo ago
  Mutual consent to what? I didn't agree for rich people to own all the housing around here, is it mutual consent if I agree to pay one to get some, or is it extortion?
  - skybrian 4mo ago
    Yes, when you buy a house it’s usually because the buyer and seller agreed to it. It seems better than, say, a foreclosure or an eviction.
turtlebits 4mo ago
Its unethical because you're intentionally bypassing restrictions. Just because others do it doesn't mean its okay. If you saw a sign in a store that said "1 per person" or "for registered guests only", would you ignore it?
- embedding-shape 4mo ago
  > Its unethical because you're intentionally bypassing restrictions I'd still consider why the restriction is there and why I'm thinking of breaking it, before deciding if it's unethical or not. It depends, basically. Generally I follow the rules and restrictions, but maybe see them more as guidelines or suggestions.
- orf 4mo ago
  Was Rosa Parks unethical for sitting down on a bus? The point is that the context matters: both the users context and the context of the restriction. It’s not as clear cut as “ignoring restrictions = bad”. The restriction itself can be unethical, in the same way that bypassing a restriction can be unethical.
  - BoorishBears 4mo ago
    Woah now, I'm for headless browsers but let's not start comparing any of this to Rosa Parks lol. The reality is a lot of interesting, trivially harmful to non harmful things are illegal and we still do them anyways.
  - saintfire 4mo ago
    [flagged]
    - Dylan16807 4mo ago
      You're taking an analogy way too seriously so: you too.
  - Natfan 4mo ago
    we need a new version of Godwin's Law after this comment. orf's law: > As an online discussion grows longer, the probability of a comparison with Rosa Parks approaches one."
choilive 4mo ago
We use headless browser providers because the companies we interact with don't and won't create a proper API for us to use. Lots of legacy web apps/portals. Saves thousands of man hours.
fc417fc802 4mo ago
> But if I did so for scalping purposes? Then yeah, I'd agree it's unethical. Is scalping actually unethical though? Sure it's unpopular, but I'd argue that's just your average person not properly grasping supply and demand and thinking through the consequences. If you want to sell something below market then you should raffle it off and take extensive measures to prevent transfer of ownership. The current practice is trying to pretend it's an open market while fixing the price, then getting angry when the obvious consequences materialize. Scalpers are merely the agents that correct a market inefficiency introduced by a dysfunctional status quo.
- saltmate 4mo ago
  Noone thinks concert ticket sales are an open market tho. They have one seller that sets the price, no competition between different sellers. It only becomes a "market" after scalpers buy all contingent to resell.
  - darkwater 4mo ago
    Unbelievable how somebody could defend scalping. There is no ethical or moral value in that practice outside of "I can earn money with that".
    - fc417fc802 4mo ago
      I'm not defending the overall arrangement, simply pointing out that the blame is misplaced. If you want to sell below market value in a capitalist system then you must take appropriate measures to prevent a market from forming. More or less by design if one can form then it will. Similarly I do not attempt to blame the rank and file employees of the ad tech industry for the actions of their employers, nor of the defense industry for the actions of the government. As an individual living under such a system either you move to make money when the opportunity arises or someone else will instead and you will lose out.
      - inigyou 4mo ago

joatmon-snoo 4mo ago

An example I ran into recently: I wanted to scrape pricing data for used cars, to better inform a friend's decision about what to purchase. I know there's a relationship between mileage and depreciation, but wanted to have a better sense of what that relationship is to know whether a given car was over or underpriced. Similarly, if I was pulling that data to build a service of my own to offer to users... is that unethical?

sroussey 4mo ago
All of these questions are easily answered by the question: can I run the bot on the same PC I use regularly? If so, then do it there. If not, then don’t do it at all.
- fc417fc802 4mo ago
  Why should technical capability to evade countermeasures dictate whether or not something is ethical? My view is that scraping remains ethical as long as your actions aren't causing technical problems for the operator. If anything, a retailer attempting to hide pricing data is what's unethical in my view.
- pocksuppet 4mo ago
  This is often really good for your bot, because anti-bot providers are loathe to block what, as far as they know, is a residential CGNAT address. Sometimes you get more success scraping from home with Firefox or Chrome, than with an army of proxy networks.
adolph 4mo ago
> scrape pricing data for used cars Time was you could get lovely json feeds from every site by iterating the inspector curl statement. Now-a-days you can't even use Selenium without Cloudflare getting grouchy. Last fall had to make my spreadsheet like a cave-person control c, control v. It wouldn't be so bad if the dealer aggregators' coverage was xor, but you have to dedupe listings. Then there is the whole online salespeople who don't show up at the dealership.
- inigyou 4mo ago
  There's a JavaScript property called navigator.webdriver that returns true if selenium is in use. Obviously, every antibot system checks it. Obviously, you can patch it to always say false.

mystifyingpoi 4mo ago

> even if others don't think that thing should be automated It's an interesting thought that can be further explored. Could anything that's considered "unwanted" by a third party considered unethical, if I do it anyway? If the hotel self-service restaurant has a sign "don't take the food out" and I take 1 apple in my pocket for a snack, is it unethical? Or maybe the sign is just for people that would otherwise take $100 of watermelons out of the cantina daily and try to resell it on the beach.

Nathanba 4mo ago
We ("society") do things to people that they don't want all the time, namely punishing them via the legal system. Then there are things that other people are doing that are immoral and should be punished for even though they are not illegal. And the whole class of inactions where we don't e.g buy something because it's overpriced and that's certainly something the seller doesn't like.

skybrian 4mo ago

What do you think of Anubis and Cloudflare? If they block your bot, is that unethical? Seems like doing business with other people should normally be based on mutual consent, not whatever you can get away with technically.

inigyou 4mo ago
Mutual consent to what? I didn't agree for rich people to own all the housing around here, is it mutual consent if I agree to pay one to get some, or is it extortion?
- skybrian 4mo ago
  Yes, when you buy a house it’s usually because the buyer and seller agreed to it. It seems better than, say, a foreclosure or an eviction.

turtlebits 4mo ago

Its unethical because you're intentionally bypassing restrictions. Just because others do it doesn't mean its okay. If you saw a sign in a store that said "1 per person" or "for registered guests only", would you ignore it?

embedding-shape 4mo ago
> Its unethical because you're intentionally bypassing restrictions I'd still consider why the restriction is there and why I'm thinking of breaking it, before deciding if it's unethical or not. It depends, basically. Generally I follow the rules and restrictions, but maybe see them more as guidelines or suggestions.
orf 4mo ago
Was Rosa Parks unethical for sitting down on a bus? The point is that the context matters: both the users context and the context of the restriction. It’s not as clear cut as “ignoring restrictions = bad”. The restriction itself can be unethical, in the same way that bypassing a restriction can be unethical.
- BoorishBears 4mo ago
  Woah now, I'm for headless browsers but let's not start comparing any of this to Rosa Parks lol. The reality is a lot of interesting, trivially harmful to non harmful things are illegal and we still do them anyways.
- saintfire 4mo ago
  [flagged]
  - Dylan16807 4mo ago
    You're taking an analogy way too seriously so: you too.
- Natfan 4mo ago
  we need a new version of Godwin's Law after this comment. orf's law: > As an online discussion grows longer, the probability of a comparison with Rosa Parks approaches one."

choilive 4mo ago

We use headless browser providers because the companies we interact with don't and won't create a proper API for us to use. Lots of legacy web apps/portals. Saves thousands of man hours.

fc417fc802 4mo ago

> But if I did so for scalping purposes? Then yeah, I'd agree it's unethical. Is scalping actually unethical though? Sure it's unpopular, but I'd argue that's just your average person not properly grasping supply and demand and thinking through the consequences. If you want to sell something below market then you should raffle it off and take extensive measures to prevent transfer of ownership. The current practice is trying to pretend it's an open market while fixing the price, then getting angry when the obvious consequences materialize. Scalpers are merely the agents that correct a market inefficiency introduced by a dysfunctional status quo.

saltmate 4mo ago
Noone thinks concert ticket sales are an open market tho. They have one seller that sets the price, no competition between different sellers. It only becomes a "market" after scalpers buy all contingent to resell.
- darkwater 4mo ago
  Unbelievable how somebody could defend scalping. There is no ethical or moral value in that practice outside of "I can earn money with that".
  - fc417fc802 4mo ago
    I'm not defending the overall arrangement, simply pointing out that the blame is misplaced. If you want to sell below market value in a capitalist system then you must take appropriate measures to prevent a market from forming. More or less by design if one can form then it will. Similarly I do not attempt to blame the rank and file employees of the ad tech industry for the actions of their employers, nor of the defense industry for the actions of the government. As an individual living under such a system either you move to make money when the opportunity arises or someone else will instead and you will lose out.
    - inigyou 4mo ago