12 ms·

So you want to scrape like the big boys (2021)

321 points by aragonite 2y ago

blantonl 2y ago
This tends to be a very unpopular opinion around here, but in almost all cases I find Internet scraping to be unethical and downright malicious. I'm not saying all cases, but I'm saying almost. A lot of the actors involved tend to be hustle culture types who think they are OWED your data, regardless of the ethics, laws, being a good citizen, whatever. They will blatantly disregard terms of service and hide behind massive setups such as these to circumvent protection etc. And the problem is, if you run any sort of business or service that is data oriented, there will be thousands of people that will do this, which will cause you to devote enormous amounts of time, effort, money, and infrastructure just to mitigate the issues involved with data scraping. That's before you are even addressing whether or not these people are "stealing" your data. People who feel they are entitled to the crux of your business aren't bothered by being nice in the way they take it - they'll launch services that will cripple infrastructure. Whenever I deal with a scraping process that decides it wants my entire business, and it wants all of it RIGHT NOW, or in 5 minutes, I want to find the person and sit them down in a room and tell them "hey, develop your own ideas and business. Ok? Thanks" And if you think this was a problem before, it's exponentially worse over the past few months with every Tom, Susan, and Harry deciding they must have all your data to train their new LLM AI model. By the thousands.
- vouaobrasil 2y ago
  I absolutely agree. In fact, I think the problem is that like everything, there is an optimal point for efficiency, and crossing that line by making things "too easy" when it comes to data means too much power for one person to handle ethically. Absolute power may corrupt absolutely, but near absolutely power also corrupts quite nicely, too. In short, we should have limits to amount of scraping possible, simply because humans can never be trusted past a certain point to remain ethical. After all, ethics at its first approximation is only a mechanism to improve societal cohesiveness, and it only works as long as the person doesn't have enough power to "do away" with society.
  - jumby 2y ago
    Would you make the same argument of the inverse: data gathering?
    - vouaobrasil 2y ago
      Yes, I would. There is a law of diminishing returns for all technical and scientific inquiry.
- deleted 2y ago
  [deleted]
- flir 2y ago
  There's a lot of local history locked up in facebook's nostalgia groups. I want to archive it in an open format. I want to grab new rental listings and put them in an RSS feed, so I only look at each one once. That's my uses for data scraping right now. If that destroys someone's business, I don't actually care. Maybe it's selfish, but my right to re-format data for my own convenience outweighs their right to make a profit.
  - throwaway11460 2y ago
    Not that I think you shouldn't do it or you're doing something wrong, but describing it as a right irks me the wrong way. You don't have any right to expect someone else's computers to work for you.
    - flir 2y ago
      I'm not sure how to phrase it except in terms of competing rights, but I take your point. At the point where I'm scraping, the data's on my computer though.
      - solarkraft 2y ago
        You could call them interests . It's often in a business's interest to format data in a specific way to make money, for example interlacing it with ads.
        flir 2y ago
        Nice.
  - blantonl 2y ago
    If that destroys someone's business, I don't actually care. Maybe it's selfish, but my right to re-format data for my own convenience outweighs their right to make a profit. Exhibit A
    - 2y ago
- brigadier132 2y ago
  > hustle culture types It seems like you have this imaginary strawman that you hate and it seems like that's the foundation of why you dislike this.
  - blantonl 2y ago
    No. The foundation of why I dislike it is simple. If I own some data, then I get to dictate the terms of how that data is used. Period. “Hustle culture types” is simply a little anecdote about the types that would look you in the eye and tell you they are entitled to disregard what I said above. They’ll usually wrap it in some altruistic bs to justify as well.
    - some1else 2y ago
      Serving HTML will get you scraped. Your terms don't overrule fair use.
    - throwaway11460 2y ago
      Why do you put it on the open internet if you don't want machines to find and read it? ToS is nice but you can't expect that it applies - the user (of the machine doing the scraping) might be a child which makes the potential contract automatically void, for example. Also, there are people under jurisdictions where such things have no power, or that don't recognize your rights to the data. And the whole thing of putting data out publicly and then just expecting machines to see the pile of data and go "oh so where do I sign the ToS?" is weird... Just put it behind a rate limited API key...
      - blantonl 2y ago
        What makes you think putting data on the Internet all the sudden means I unilaterally surrender the rights to my intellectual property? If I choose to make my data available to some businesses to make discovery of it easier, and I choose to decline to allow others to unilaterally copy my data to develop a different business, that's my right. And it is unethical and unreasonable for any other person to assume otherwise that they are entitled to the same rights I granted someone else. If I own some data, I get to the be arbitrator of the who/what/when/where on the use of the data. Period.
- deleted 2y ago
  [deleted]
- malwrar 2y ago
  If your business is just that you have a bundle of information and expose it over an open website, I’m not really sure how you’re able to maintain a mentality that you are somehow entitled to ownership of that information. You already put it out there, it’s now public, any illusion to exclusivity is now gone because anyone could come along at any time and make a copy without your knowledge. A moral position on this issue is even more confusing to me. Do you think that you e.g. own the knowledge on which radio frequencies are used where? Do you think you have a moral claim on ownership of (presumably unpaid) user-submitted information? I think the only legitimate moral grievance you have is high traffic volumes from inconsiderate scrapers.
  - blantonl 2y ago
    Do you think you have a moral claim on ownership of (presumably unpaid) user-submitted information? You damn right I do. I own, develop, and maintain the entire system that enabled the body of works to exist in the first place. Do you think that you have a claim on ownership of the data because you drove by, saw what you liked, and decided that now you'll just rip the baton out of my hand?
    - malwrar 2y ago
      > You damn right I do. I own, develop, and maintain the entire system that enabled the body of works to exist in the first place. I don’t think that meets the bar. Running a website is absolutely not equivalent to the collective effort people put in to populate that website with the information that actually gives the overall artifact its value. There is a large history of outrage when similar information repository websites with user-generated content violate expectations of openness. Nevermind the fact that the actual information itself isn’t even private or proprietary, just obscure and distributed. > Do you think that you have a claim on ownership of the data because you drove by, saw what you liked, and decided that now you'll just rip the baton out of my hand? I wouldn’t claim ownership nor want to, when I scrape stuff I usually just want information in a different format. I’m confused as to how you think you can even “own” data to begin with. Suppose that your users uploaded songs instead of RF info, do you believe you own their music solely because they chose to share it on your site? Do you think your users would believe that?
- greenbandit 2y ago
  I use web scraping to identify and monitor fraud. Exhibit A: https://archive.ph/0ZUA8 https://archive.ph/0ZUA8 This website is used to recruit people to set up "lead generation" Google Business Profiles and leave paid reviews. Exhibit B: https://archive.ph/WWZuw https://archive.ph/WWZuw This is an example of the Craigslist ad used to initially attract people to the website above. Exhibit C: https://archive.ph/wip/7Xig4 https://archive.ph/wip/7Xig4 This is one of the Google Maps contributors which left paid reviews. If you start with the reviews on that profile, you'll find a network of Google Business Profiles for fake service-area businesses connected through paid reviews. Web scraping allows me to collect this type of data at scale. I also use scraping to monitor the status of fake listings. If they are removed, the actor behind them will often get them reinstated. This allows me to report them again.
  - blantonl 2y ago
    I don't care if you use Web scraping to solve the Israeli / Palestinian conflict. You're not entitled to anyone's data, computers, services, etc because you've decided for altruistic reasons that it is appropriate. Cool use case. Love it. Fascinating stuff. But if Google told you to stop, would you? Or would you instead decide to build a 5 server cluster of 200 4G modems spread across continents to continue your work? Because if you did I would assume that you've decided to move on from a cute little altruistic process into a commercial use of someone else's data to make a profit.
    - greenbandit 2y ago
      > cute little altruistic process Maybe it is not the opinion which is unpopular, but the way it is being presented.
    - ansc 2y ago
      >I don't care if you use Web scraping to solve the Israeli / Palestinian conflict. Maybe you should though. It's always worth it to think about which giant's shoulder you're standing on. It's giants all the way down.
    - dmkii 2y ago
- tengbretson 2y ago
  Is it unethical for a mouse to eat the cheese without triggering the trap?
- hipadev23 2y ago
  I find it aptly hilarious that your own business model at broadcastify.com is recording publicly accessible radio broadcasts and then selling access to those recordings for commercial gain.
  - blantonl 2y ago
    Why is that hilarious? We developed an entire community, infrastructure, system, architecture, everything, from scratch, and provide access to something that never existed in the first place on the Internet. That's a significant key difference here. This would be analogous to you thinking ancestory.com is "aptly hilarious" for arguing against someone just scraping their site for content. What makes you think you should be entitled to drive by the very unique house that we built, and pointing right at that house and saying "I think I'll take that all of that for myself!"
    - hipadev23 2y ago
      Because you fail to see the very obvious parallels to scraping. I’m not criticizing your business (I think you provide a valuable service) but your hypocritical stance on what forms of publicly available information are allowed to be gathered and repackaged. Google’s original (and OpenAI’s) business model was also building a scraping infrastructure, system, and architecture, from scratch — and providing access to something that never existed in the first place.
      - blantonl 2y ago
        It's completely perpendicular, not parallel. Public safety communications are radio waves that are broadcasted and the ability to passively monitor them is enshrined in United States law. That is a massively key difference. If I was sending data into your home from my infrastructure without any action from you whatsoever, and you were reaching up into the air and gathering it and repackaging it, AND the law said that I have no intellectual property rights to said data, then that's a whole different story.
        2y ago
- dale_glass 2y ago
  > Whenever I deal with a scraping process that decides it wants my entire business, and it wants all of it RIGHT NOW, or in 5 minutes, I want to find the person and sit them down in a room and tell them "hey, develop your own ideas and business. Ok? Thanks" That's a lot of righteous anger for somebody building a business on top of other people's data. "Broadcastify is the worlds largest source of public safety, aircraft, rail, and marine radio live audio streams." I have no sympathy whatsoever. You're just complaining about the very thing you're doing. If it's fair for you to do that, it's fair for others to do it to you.
  - blantonl 2y ago
    They volunteer to provide the data to us. Every single last one of them. Nowhere in our business model did we make the conscious decision to say "hey, look at that business, they have something, and I'm going to take it."
    - bsuvc 2y ago
      Reading public website data is not "taking it". It is still there. Observing publicly available information is not theft, nor is it illegal. Of course copyright rules apply, but that is for if you reproduce something.
      - blantonl 2y ago
        reproduce something No one is developing a 5 server cluster with 200+ 4g modems to observe publicly available information. They are using said cluster to deliberately work around blocks, rate limits, and restrictions on scrapers who are scraping content solely to reproduce the data and use it for commercial purposes (make money)
    - schlipity 2y ago
      Aren't you also volunteering your data? Don't browsers just talk to your webserver and say "Hey, what do you have?" and your site responds in kind.
- juunpp 2y ago
  > but in almost all cases I find Internet scraping to be unethical and downright malicious. The Web (you said the "Internet", but you meant the Web) was not envisioned to be a commercial space. Your statement is antithetical to the original idea of the open Web. It's when the MBAs joined the party circa 2k and decided to profit out of it that all of these confused and wrong opinions about what the Web should be arose and that lead to the situation today. Your statement is a vast display of zero historical context. MBAs are obviously not very concerned with history. They just want to protect their own little turd for their own little profit and vanity, which is why they now put it behind a paywall, JS, and anti-bot proxies.
deleted 2y ago
[deleted]
mellosouls 2y ago
A curious title. "So you want to scrape like the unethical boys?" I guess doesn't scan so well. Bad boys maybe? I'm pretty sure Internet Archive, etc don't in fact misrepresent what they are to crawl websites...
- echelon 2y ago
  > unethical Using and transforming information in useful ways is unethical if it results in a profit? That's what our brains do, too.
  - llamaimperative 2y ago
    No, destroying incentives to produce and share information is unethical (and more importantly, self-defeating). Brains that consume information don’t destroy that incentive, they produce it. Intermediating that and capturing all of the value for yourself is the unethical part, just like all forms of rent-seeking.
    - echelon 2y ago
      > destroying incentives Internet usage and content creation are increasing, not decreasing. I continue to publish comments, code, and images that presumably get used to train models. My incentive hasn't been destroyed. > rent-seeking Supply and demand set the prices. Subscription services provide value and continue to invest in their product, catalog, and/or service. Property owners handle asset ownership and upkeep problems at scale. Inefficiencies will be met with competition, and businesses not providing value will be out-competed. Data under-availability is an inefficiency holding us back from bigger and better things.
- vasco 2y ago
  Tell that to the most used website in the world, which is basically a scrapping-and-sorting machine.
  - blantonl 2y ago
gwittel 2y ago
I’m really mixed on this. Anti bot stuff is increasingly a pain point for security research. Working in this space, I have to work against these systems. Threat actors use Cloudflare and other services to gate their payloads. That’s a problem for our customers who are trying to find/detect things like brand impersonation and credential phish. Cloudflare has been completely unhelpful. They just don’t care.
- heipei 2y ago
  Seconding this. Evading detection has become a real cake-walk since threat actors are able to sign up for a free Cloudflare account and then put their phishing site on their 2-hours old domain behind a level of protection backed by a $20B company. Funny that you almost never see phishing on Akamai ;) Disclaimer: We operate in this space so we obviously have an interest in being able to detect these threats going forward.
  - rashkov 2y ago
    Why not Akamai?
    - nkozyra 2y ago
      Cost.
  - throwaway48476 2y ago
    Cloudflare is the ultimate example of creating the problem and selling the solution.
    - zinglersen 2y ago
      I was under the (naive?) impression that Cloudflare a SaaS startup poster child. Do you mind expanding on your comment?
      - throwaway48476 2y ago
deleted 2y ago
[deleted]
anyfactor 2y ago
I was a professional web scraper. I still keep up to date with the industry. These days, you do not make money by doing web scraping; you make money selling services to web scrapers. There are tons of web scraping SAAS and services out there, as well as dozens of residential proxy providers. Most anti-bot mechanisms evolve so quickly that you can make a decent income just by working in a traditional software engineering role dedicated entirely to engineering anti-anti-bot solutions. As these mechanisms evolve rapidly, working for a web scraping company is more stable than pursuing web scraping as a profession. Web scrapers get paid by projects, making it an unstable job in the long run. High-level web scraping requires operational investments in residential proxies and renting out servers. Additionally, low-end jobs pay very little. Brightdata hosting a conference on web scraping, which should indicate the profitability of selling services in large-scale web scraping.
- RockRobotRock 2y ago
  I've been writing scrapers on Upwork for many years. I'm sick of doing project based work and want to work at/start a scraping SaaS. Any advice?
  - anyfactor 2y ago
    I would recommend checking Google to see if you can find any job openings. Please remember that it is a niche industry, so there may not be many companies currently hiring. But honestly, if you are looking to make a full-time living, consider choosing another niche as web scraping jobs require you to consistently stay on top of your game. Most full-time jobs involve scraping data from big tech companies, and you are on your own to find solutions in bypassing anti-bot measures.
- jimz 2y ago
  The irony is that before I realized it was so easy I would just open source the code - not on Github, mind you, since the likes of Akamai would DMCA pretty quickly, but playing a little bit of jurisdictional arbitrage I put it on Gitee - the Chinese copycat of Github. I don't have a background in any of this, but companies like the brag and it's not hard to put two and two together. It also was a practical way to enable me to place wagers on sports automatically - which was more or less my actual day job - and was pretty good for learning programming quickly in your late 20s. Instead almost immediately I got inundated by sneaker botters in China and in English from somewhere that doesn't use it as a native language, judging from the idiosyncratic use. I kept the code up for a bit but took it down not because of any legal threats (good luck with DMCA-ing a platform endorsed by the CCP, even though I have no love for the party, I also find the American attitude that places intellectual property over real property in practice - from my experience as a defense attorney - to be just as screwed up in terms of priorities, just a matter of degrees. What made me take it down was the fact that I did not want to work in a customer service job or really for anyone, and judging by the requests, it was mostly consisted of "you do the work but we'll split the profits", which I can't believe anyone would fall for. But since the internet is forever, some parts of code that specifically worked to emulate Cyberfed-Akamai from 0.8 to 2.3 are probably still floating around. My bad. I don't wear shoes normally - flip flops or nothing after having to wear a suit to work for a decade - and have no idea beyond what happens in NBA2K. Although cybersecurity firms making products that someone who learned how to program in their mid 20s and put online within 3 years and had it work should be pretty ashamed of how much they charge, considering that I haven't even taken a math course since 11th grade and had too much of an ADHD problem to watch videos or even read more than blog posts or documentation. Everything I learned, I learned by copying from Github and similar services until it worked. There must be a lot of snake oil being sold out there, maybe most of it, since the insidiousness of the whole thing is that selling bunk solutions seldom gets you in trouble anyway, while actual crime - rape, murder, robbery and the like - are largely lagging because the police simply prefer to complain about culture war bs instead of actually, you know, do their jobs. Who knew Judith Butler was THIS spot on.
sublinear 2y ago
> Those companies employ ill-adjusted individuals that do nothing else than look for the most recent techniques to fingerprint browsers ... When normal people are out drinking beers in the pub on Friday night, these individuals invent increasingly bizarre ways to fingerprint browsers and detect bots ;) Why not both on a friday night?
KieranMac 2y ago
I'm a lawyer that works in the web-scraping space, and I always chuckle when I read threads like this. Almost every company that we now consider a monopolist (or their affiliates) in the tech space used scraping a part of their process to build their business, and almost every one of those same monopolists now prohibits startups and competitors from scraping their data (which, invariably, is not actually "their" data in any sort of legally cognizable sense). And so perhaps the ethics of web scraping are not so straightforward. And neither are the legal issues associated with it. I wrote an article about that last fall that got some attention here. https://news.ycombinator.com/item?id=37264676 https://news.ycombinator.com/item?id=37264676
- jMyles 2y ago
  > And so perhaps the ethics of web scraping are not so straightforward. It strikes me that the _ethics_ of web scraping are extremely straightforward and cognizable with a terse analysis: * You can respond however you like to my HTTP request, and I can parse your response however I like. Simple, traditional, common. This is the way that conversations have occurred since the dawn of human communication, no? > the legal issues associated with it. But aren't these, without exception, fabrics spun out of the cloth that shields established players with the threat of state violence? This is not particularly new, and seems to fit in the pathetic-and-predictable file. Moreover, the broader cheap attempt to cast this in "intellectual" property terms, and to attach that to protection of artists and creators, warrants a very particular eye-roll for its illogic.
  - elicksaur 2y ago
    If I say, “Hey, please don’t text me anymore. I’m going to block this number,” and you respond by buying 500 phones in five cities and text me nonstop, is that ethical?
    - andai 2y ago
      Not sure the metaphor works here. For example most sites let Google scrape them as much as it likes, but go out of their way to block other robots. By doing so they are effectively forcing the whole world to use (or support, since smaller search engines have to piggyback on the big ones wih special status, and pay them) proprietary spyware. In your analogy, most websites block everyone except the biggest pervert known to man.
graemep 2y ago
Anti bot stuff also seems to be a security threat and privacy threat: preventing users from accessing your site if using VMs, port scanning, various froms of fingerprinting
- Terr_ 2y ago
  I prefer the approach of an algorithmic challenge that forces the "new visitor" to spend some CPU cycles. It's a clear process, doesn't involve privacy risks or strange sneaky games, and tends to fail in ways that a human can at least see and report, as opposed to mysterious outages.
  - theamk 2y ago
    .. and also annoys people with slow hardware while costing very little to serious scrapers? How much CPU time can you burn so people on 3 year old phones can see it, and how much will it cost scrapers?
    - graemep 2y ago
      Even a very slight challenge is a problem for scrapers: they have to do it far more frequently. Its better than captchas and whatever Cloudflare does in terms of overall nuisance.
dang 2y ago
Discussed at the time: Scrape like the big boys - https://news.ycombinator.com/item?id=29117022 https://news.ycombinator.com/item?id=29117022 - Nov 2021 (189 comments)
lyu07282 2y ago
> Every website can access rotation and velocity data from Android data without asking for permission. What????!!! That's nuts
qweqwe14 2y ago
> Those companies employ ill-adjusted individuals that do nothing else than look for the most recent techniques to fingerprint browsers [...] When normal people are out drinking beers in the pub on Friday night, these individuals invent increasingly bizarre ways to fingerprint browsers and detect bots ;) What's the deal with "ill-adjusted" and "normal people"? I'm gonna say it right now, the reason why these individuals do this is because it's way more interesting and fun than building some bullshit React website for some boring business for the 20th time (this is just an example, not attacking React here, no need to freak out) It's fun because you get to solve an actual real-world challenge and find new ways to do something. Same with things like developing exploits. Those who do this are not "ill-adjusted", they are in fact normal people that do what they are passionate about. The whole mentality of "anyone who does something I don't like is ill-adjusted" is just absolutely insane.
- snowwrestler 2y ago
  That entire paragraph is a joke. That’s why there is a little wink at the end.
  - qweqwe14 2y ago
    It's not clear if it's a joke specifically because of the addition of "ill-adjusted"
    - snowwrestler 2y ago
      That is the joke. It’s hyperbole.
Havoc 2y ago
Interesting. Busy building a project that requires scraping (pretty low rate) Have been puzzling what to do about the rejection cases. A single cheap android might just fill the gap.

321 points by aragonite 2y ago

This tends to be a very unpopular opinion around here, but in almost all cases I find Internet scraping to be unethical and downright malicious. I'm not saying all cases, but I'm saying almost. A lot of the actors involved tend to be hustle culture types who think they are OWED your data, regardless of the ethics, laws, being a good citizen, whatever. They will blatantly disregard terms of service and hide behind massive setups such as these to circumvent protection etc. And the problem is, if you run any sort of business or service that is data oriented, there will be thousands of people that will do this, which will cause you to devote enormous amounts of time, effort, money, and infrastructure just to mitigate the issues involved with data scraping. That's before you are even addressing whether or not these people are "stealing" your data. People who feel they are entitled to the crux of your business aren't bothered by being nice in the way they take it - they'll launch services that will cripple infrastructure. Whenever I deal with a scraping process that decides it wants my entire business, and it wants all of it RIGHT NOW, or in 5 minutes, I want to find the person and sit them down in a room and tell them "hey, develop your own ideas and business. Ok? Thanks" And if you think this was a problem before, it's exponentially worse over the past few months with every Tom, Susan, and Harry deciding they must have all your data to train their new LLM AI model. By the thousands.

vouaobrasil 2y ago
I absolutely agree. In fact, I think the problem is that like everything, there is an optimal point for efficiency, and crossing that line by making things "too easy" when it comes to data means too much power for one person to handle ethically. Absolute power may corrupt absolutely, but near absolutely power also corrupts quite nicely, too. In short, we should have limits to amount of scraping possible, simply because humans can never be trusted past a certain point to remain ethical. After all, ethics at its first approximation is only a mechanism to improve societal cohesiveness, and it only works as long as the person doesn't have enough power to "do away" with society.
- jumby 2y ago
  Would you make the same argument of the inverse: data gathering?
  - vouaobrasil 2y ago
    Yes, I would. There is a law of diminishing returns for all technical and scientific inquiry.
deleted 2y ago
[deleted]
flir 2y ago
There's a lot of local history locked up in facebook's nostalgia groups. I want to archive it in an open format. I want to grab new rental listings and put them in an RSS feed, so I only look at each one once. That's my uses for data scraping right now. If that destroys someone's business, I don't actually care. Maybe it's selfish, but my right to re-format data for my own convenience outweighs their right to make a profit.
- throwaway11460 2y ago
  Not that I think you shouldn't do it or you're doing something wrong, but describing it as a right irks me the wrong way. You don't have any right to expect someone else's computers to work for you.
  - flir 2y ago
    I'm not sure how to phrase it except in terms of competing rights, but I take your point. At the point where I'm scraping, the data's on my computer though.
    - solarkraft 2y ago
      You could call them interests . It's often in a business's interest to format data in a specific way to make money, for example interlacing it with ads.
      - flir 2y ago
        Nice.
- blantonl 2y ago
  If that destroys someone's business, I don't actually care. Maybe it's selfish, but my right to re-format data for my own convenience outweighs their right to make a profit. Exhibit A
  - 2y ago
brigadier132 2y ago
> hustle culture types It seems like you have this imaginary strawman that you hate and it seems like that's the foundation of why you dislike this.
- blantonl 2y ago
  No. The foundation of why I dislike it is simple. If I own some data, then I get to dictate the terms of how that data is used. Period. “Hustle culture types” is simply a little anecdote about the types that would look you in the eye and tell you they are entitled to disregard what I said above. They’ll usually wrap it in some altruistic bs to justify as well.
  - some1else 2y ago
    Serving HTML will get you scraped. Your terms don't overrule fair use.
  - throwaway11460 2y ago
    Why do you put it on the open internet if you don't want machines to find and read it? ToS is nice but you can't expect that it applies - the user (of the machine doing the scraping) might be a child which makes the potential contract automatically void, for example. Also, there are people under jurisdictions where such things have no power, or that don't recognize your rights to the data. And the whole thing of putting data out publicly and then just expecting machines to see the pile of data and go "oh so where do I sign the ToS?" is weird... Just put it behind a rate limited API key...
    - blantonl 2y ago
      What makes you think putting data on the Internet all the sudden means I unilaterally surrender the rights to my intellectual property? If I choose to make my data available to some businesses to make discovery of it easier, and I choose to decline to allow others to unilaterally copy my data to develop a different business, that's my right. And it is unethical and unreasonable for any other person to assume otherwise that they are entitled to the same rights I granted someone else. If I own some data, I get to the be arbitrator of the who/what/when/where on the use of the data. Period.
deleted 2y ago
[deleted]
malwrar 2y ago
If your business is just that you have a bundle of information and expose it over an open website, I’m not really sure how you’re able to maintain a mentality that you are somehow entitled to ownership of that information. You already put it out there, it’s now public, any illusion to exclusivity is now gone because anyone could come along at any time and make a copy without your knowledge. A moral position on this issue is even more confusing to me. Do you think that you e.g. own the knowledge on which radio frequencies are used where? Do you think you have a moral claim on ownership of (presumably unpaid) user-submitted information? I think the only legitimate moral grievance you have is high traffic volumes from inconsiderate scrapers.
- blantonl 2y ago
  Do you think you have a moral claim on ownership of (presumably unpaid) user-submitted information? You damn right I do. I own, develop, and maintain the entire system that enabled the body of works to exist in the first place. Do you think that you have a claim on ownership of the data because you drove by, saw what you liked, and decided that now you'll just rip the baton out of my hand?
  - malwrar 2y ago
    > You damn right I do. I own, develop, and maintain the entire system that enabled the body of works to exist in the first place. I don’t think that meets the bar. Running a website is absolutely not equivalent to the collective effort people put in to populate that website with the information that actually gives the overall artifact its value. There is a large history of outrage when similar information repository websites with user-generated content violate expectations of openness. Nevermind the fact that the actual information itself isn’t even private or proprietary, just obscure and distributed. > Do you think that you have a claim on ownership of the data because you drove by, saw what you liked, and decided that now you'll just rip the baton out of my hand? I wouldn’t claim ownership nor want to, when I scrape stuff I usually just want information in a different format. I’m confused as to how you think you can even “own” data to begin with. Suppose that your users uploaded songs instead of RF info, do you believe you own their music solely because they chose to share it on your site? Do you think your users would believe that?
greenbandit 2y ago
I use web scraping to identify and monitor fraud. Exhibit A: https://archive.ph/0ZUA8 https://archive.ph/0ZUA8 This website is used to recruit people to set up "lead generation" Google Business Profiles and leave paid reviews. Exhibit B: https://archive.ph/WWZuw https://archive.ph/WWZuw This is an example of the Craigslist ad used to initially attract people to the website above. Exhibit C: https://archive.ph/wip/7Xig4 https://archive.ph/wip/7Xig4 This is one of the Google Maps contributors which left paid reviews. If you start with the reviews on that profile, you'll find a network of Google Business Profiles for fake service-area businesses connected through paid reviews. Web scraping allows me to collect this type of data at scale. I also use scraping to monitor the status of fake listings. If they are removed, the actor behind them will often get them reinstated. This allows me to report them again.
- blantonl 2y ago
  I don't care if you use Web scraping to solve the Israeli / Palestinian conflict. You're not entitled to anyone's data, computers, services, etc because you've decided for altruistic reasons that it is appropriate. Cool use case. Love it. Fascinating stuff. But if Google told you to stop, would you? Or would you instead decide to build a 5 server cluster of 200 4G modems spread across continents to continue your work? Because if you did I would assume that you've decided to move on from a cute little altruistic process into a commercial use of someone else's data to make a profit.
  - greenbandit 2y ago
    > cute little altruistic process Maybe it is not the opinion which is unpopular, but the way it is being presented.
  - ansc 2y ago
    >I don't care if you use Web scraping to solve the Israeli / Palestinian conflict. Maybe you should though. It's always worth it to think about which giant's shoulder you're standing on. It's giants all the way down.
  - dmkii 2y ago
tengbretson 2y ago
Is it unethical for a mouse to eat the cheese without triggering the trap?
hipadev23 2y ago
I find it aptly hilarious that your own business model at broadcastify.com is recording publicly accessible radio broadcasts and then selling access to those recordings for commercial gain.
- blantonl 2y ago
  Why is that hilarious? We developed an entire community, infrastructure, system, architecture, everything, from scratch, and provide access to something that never existed in the first place on the Internet. That's a significant key difference here. This would be analogous to you thinking ancestory.com is "aptly hilarious" for arguing against someone just scraping their site for content. What makes you think you should be entitled to drive by the very unique house that we built, and pointing right at that house and saying "I think I'll take that all of that for myself!"
  - hipadev23 2y ago
    Because you fail to see the very obvious parallels to scraping. I’m not criticizing your business (I think you provide a valuable service) but your hypocritical stance on what forms of publicly available information are allowed to be gathered and repackaged. Google’s original (and OpenAI’s) business model was also building a scraping infrastructure, system, and architecture, from scratch — and providing access to something that never existed in the first place.
    - blantonl 2y ago
      It's completely perpendicular, not parallel. Public safety communications are radio waves that are broadcasted and the ability to passively monitor them is enshrined in United States law. That is a massively key difference. If I was sending data into your home from my infrastructure without any action from you whatsoever, and you were reaching up into the air and gathering it and repackaging it, AND the law said that I have no intellectual property rights to said data, then that's a whole different story.
      - 2y ago
dale_glass 2y ago
> Whenever I deal with a scraping process that decides it wants my entire business, and it wants all of it RIGHT NOW, or in 5 minutes, I want to find the person and sit them down in a room and tell them "hey, develop your own ideas and business. Ok? Thanks" That's a lot of righteous anger for somebody building a business on top of other people's data. "Broadcastify is the worlds largest source of public safety, aircraft, rail, and marine radio live audio streams." I have no sympathy whatsoever. You're just complaining about the very thing you're doing. If it's fair for you to do that, it's fair for others to do it to you.
- blantonl 2y ago
  They volunteer to provide the data to us. Every single last one of them. Nowhere in our business model did we make the conscious decision to say "hey, look at that business, they have something, and I'm going to take it."
  - bsuvc 2y ago
    Reading public website data is not "taking it". It is still there. Observing publicly available information is not theft, nor is it illegal. Of course copyright rules apply, but that is for if you reproduce something.
    - blantonl 2y ago
      reproduce something No one is developing a 5 server cluster with 200+ 4g modems to observe publicly available information. They are using said cluster to deliberately work around blocks, rate limits, and restrictions on scrapers who are scraping content solely to reproduce the data and use it for commercial purposes (make money)
  - schlipity 2y ago
    Aren't you also volunteering your data? Don't browsers just talk to your webserver and say "Hey, what do you have?" and your site responds in kind.
juunpp 2y ago
> but in almost all cases I find Internet scraping to be unethical and downright malicious. The Web (you said the "Internet", but you meant the Web) was not envisioned to be a commercial space. Your statement is antithetical to the original idea of the open Web. It's when the MBAs joined the party circa 2k and decided to profit out of it that all of these confused and wrong opinions about what the Web should be arose and that lead to the situation today. Your statement is a vast display of zero historical context. MBAs are obviously not very concerned with history. They just want to protect their own little turd for their own little profit and vanity, which is why they now put it behind a paywall, JS, and anti-bot proxies.

deleted 2y ago

[deleted]

mellosouls 2y ago

A curious title. "So you want to scrape like the unethical boys?" I guess doesn't scan so well. Bad boys maybe? I'm pretty sure Internet Archive, etc don't in fact misrepresent what they are to crawl websites...

echelon 2y ago
> unethical Using and transforming information in useful ways is unethical if it results in a profit? That's what our brains do, too.
- llamaimperative 2y ago
  No, destroying incentives to produce and share information is unethical (and more importantly, self-defeating). Brains that consume information don’t destroy that incentive, they produce it. Intermediating that and capturing all of the value for yourself is the unethical part, just like all forms of rent-seeking.
  - echelon 2y ago
    > destroying incentives Internet usage and content creation are increasing, not decreasing. I continue to publish comments, code, and images that presumably get used to train models. My incentive hasn't been destroyed. > rent-seeking Supply and demand set the prices. Subscription services provide value and continue to invest in their product, catalog, and/or service. Property owners handle asset ownership and upkeep problems at scale. Inefficiencies will be met with competition, and businesses not providing value will be out-competed. Data under-availability is an inefficiency holding us back from bigger and better things.
vasco 2y ago
Tell that to the most used website in the world, which is basically a scrapping-and-sorting machine.
- blantonl 2y ago

gwittel 2y ago

I’m really mixed on this. Anti bot stuff is increasingly a pain point for security research. Working in this space, I have to work against these systems. Threat actors use Cloudflare and other services to gate their payloads. That’s a problem for our customers who are trying to find/detect things like brand impersonation and credential phish. Cloudflare has been completely unhelpful. They just don’t care.

heipei 2y ago
Seconding this. Evading detection has become a real cake-walk since threat actors are able to sign up for a free Cloudflare account and then put their phishing site on their 2-hours old domain behind a level of protection backed by a $20B company. Funny that you almost never see phishing on Akamai ;) Disclaimer: We operate in this space so we obviously have an interest in being able to detect these threats going forward.
- rashkov 2y ago
  Why not Akamai?
  - nkozyra 2y ago
    Cost.
- throwaway48476 2y ago
  Cloudflare is the ultimate example of creating the problem and selling the solution.
  - zinglersen 2y ago
    I was under the (naive?) impression that Cloudflare a SaaS startup poster child. Do you mind expanding on your comment?
    - throwaway48476 2y ago

deleted 2y ago

[deleted]

anyfactor 2y ago

I was a professional web scraper. I still keep up to date with the industry. These days, you do not make money by doing web scraping; you make money selling services to web scrapers. There are tons of web scraping SAAS and services out there, as well as dozens of residential proxy providers. Most anti-bot mechanisms evolve so quickly that you can make a decent income just by working in a traditional software engineering role dedicated entirely to engineering anti-anti-bot solutions. As these mechanisms evolve rapidly, working for a web scraping company is more stable than pursuing web scraping as a profession. Web scrapers get paid by projects, making it an unstable job in the long run. High-level web scraping requires operational investments in residential proxies and renting out servers. Additionally, low-end jobs pay very little. Brightdata hosting a conference on web scraping, which should indicate the profitability of selling services in large-scale web scraping.

RockRobotRock 2y ago
I've been writing scrapers on Upwork for many years. I'm sick of doing project based work and want to work at/start a scraping SaaS. Any advice?
- anyfactor 2y ago
  I would recommend checking Google to see if you can find any job openings. Please remember that it is a niche industry, so there may not be many companies currently hiring. But honestly, if you are looking to make a full-time living, consider choosing another niche as web scraping jobs require you to consistently stay on top of your game. Most full-time jobs involve scraping data from big tech companies, and you are on your own to find solutions in bypassing anti-bot measures.
jimz 2y ago
The irony is that before I realized it was so easy I would just open source the code - not on Github, mind you, since the likes of Akamai would DMCA pretty quickly, but playing a little bit of jurisdictional arbitrage I put it on Gitee - the Chinese copycat of Github. I don't have a background in any of this, but companies like the brag and it's not hard to put two and two together. It also was a practical way to enable me to place wagers on sports automatically - which was more or less my actual day job - and was pretty good for learning programming quickly in your late 20s. Instead almost immediately I got inundated by sneaker botters in China and in English from somewhere that doesn't use it as a native language, judging from the idiosyncratic use. I kept the code up for a bit but took it down not because of any legal threats (good luck with DMCA-ing a platform endorsed by the CCP, even though I have no love for the party, I also find the American attitude that places intellectual property over real property in practice - from my experience as a defense attorney - to be just as screwed up in terms of priorities, just a matter of degrees. What made me take it down was the fact that I did not want to work in a customer service job or really for anyone, and judging by the requests, it was mostly consisted of "you do the work but we'll split the profits", which I can't believe anyone would fall for. But since the internet is forever, some parts of code that specifically worked to emulate Cyberfed-Akamai from 0.8 to 2.3 are probably still floating around. My bad. I don't wear shoes normally - flip flops or nothing after having to wear a suit to work for a decade - and have no idea beyond what happens in NBA2K. Although cybersecurity firms making products that someone who learned how to program in their mid 20s and put online within 3 years and had it work should be pretty ashamed of how much they charge, considering that I haven't even taken a math course since 11th grade and had too much of an ADHD problem to watch videos or even read more than blog posts or documentation. Everything I learned, I learned by copying from Github and similar services until it worked. There must be a lot of snake oil being sold out there, maybe most of it, since the insidiousness of the whole thing is that selling bunk solutions seldom gets you in trouble anyway, while actual crime - rape, murder, robbery and the like - are largely lagging because the police simply prefer to complain about culture war bs instead of actually, you know, do their jobs. Who knew Judith Butler was THIS spot on.

sublinear 2y ago

> Those companies employ ill-adjusted individuals that do nothing else than look for the most recent techniques to fingerprint browsers ... When normal people are out drinking beers in the pub on Friday night, these individuals invent increasingly bizarre ways to fingerprint browsers and detect bots ;) Why not both on a friday night?

KieranMac 2y ago

I'm a lawyer that works in the web-scraping space, and I always chuckle when I read threads like this. Almost every company that we now consider a monopolist (or their affiliates) in the tech space used scraping a part of their process to build their business, and almost every one of those same monopolists now prohibits startups and competitors from scraping their data (which, invariably, is not actually "their" data in any sort of legally cognizable sense). And so perhaps the ethics of web scraping are not so straightforward. And neither are the legal issues associated with it. I wrote an article about that last fall that got some attention here. https://news.ycombinator.com/item?id=37264676 https://news.ycombinator.com/item?id=37264676

jMyles 2y ago
> And so perhaps the ethics of web scraping are not so straightforward. It strikes me that the _ethics_ of web scraping are extremely straightforward and cognizable with a terse analysis: * You can respond however you like to my HTTP request, and I can parse your response however I like. Simple, traditional, common. This is the way that conversations have occurred since the dawn of human communication, no? > the legal issues associated with it. But aren't these, without exception, fabrics spun out of the cloth that shields established players with the threat of state violence? This is not particularly new, and seems to fit in the pathetic-and-predictable file. Moreover, the broader cheap attempt to cast this in "intellectual" property terms, and to attach that to protection of artists and creators, warrants a very particular eye-roll for its illogic.
- elicksaur 2y ago
  If I say, “Hey, please don’t text me anymore. I’m going to block this number,” and you respond by buying 500 phones in five cities and text me nonstop, is that ethical?
  - andai 2y ago
    Not sure the metaphor works here. For example most sites let Google scrape them as much as it likes, but go out of their way to block other robots. By doing so they are effectively forcing the whole world to use (or support, since smaller search engines have to piggyback on the big ones wih special status, and pay them) proprietary spyware. In your analogy, most websites block everyone except the biggest pervert known to man.

graemep 2y ago

Anti bot stuff also seems to be a security threat and privacy threat: preventing users from accessing your site if using VMs, port scanning, various froms of fingerprinting

Terr_ 2y ago
I prefer the approach of an algorithmic challenge that forces the "new visitor" to spend some CPU cycles. It's a clear process, doesn't involve privacy risks or strange sneaky games, and tends to fail in ways that a human can at least see and report, as opposed to mysterious outages.
- theamk 2y ago
  .. and also annoys people with slow hardware while costing very little to serious scrapers? How much CPU time can you burn so people on 3 year old phones can see it, and how much will it cost scrapers?
  - graemep 2y ago
    Even a very slight challenge is a problem for scrapers: they have to do it far more frequently. Its better than captchas and whatever Cloudflare does in terms of overall nuisance.

dang 2y ago

Discussed at the time: Scrape like the big boys - https://news.ycombinator.com/item?id=29117022 https://news.ycombinator.com/item?id=29117022 - Nov 2021 (189 comments)

lyu07282 2y ago

> Every website can access rotation and velocity data from Android data without asking for permission. What????!!! That's nuts

qweqwe14 2y ago

> Those companies employ ill-adjusted individuals that do nothing else than look for the most recent techniques to fingerprint browsers [...] When normal people are out drinking beers in the pub on Friday night, these individuals invent increasingly bizarre ways to fingerprint browsers and detect bots ;) What's the deal with "ill-adjusted" and "normal people"? I'm gonna say it right now, the reason why these individuals do this is because it's way more interesting and fun than building some bullshit React website for some boring business for the 20th time (this is just an example, not attacking React here, no need to freak out) It's fun because you get to solve an actual real-world challenge and find new ways to do something. Same with things like developing exploits. Those who do this are not "ill-adjusted", they are in fact normal people that do what they are passionate about. The whole mentality of "anyone who does something I don't like is ill-adjusted" is just absolutely insane.

snowwrestler 2y ago
That entire paragraph is a joke. That’s why there is a little wink at the end.
- qweqwe14 2y ago
  It's not clear if it's a joke specifically because of the addition of "ill-adjusted"
  - snowwrestler 2y ago
    That is the joke. It’s hyperbole.

Havoc 2y ago

Interesting. Busy building a project that requires scraping (pretty low rate) Have been puzzling what to do about the rejection cases. A single cheap android might just fill the gap.