5 ms·

And why is there need for scraping proxies? Because greedy capitalists do not allow to scrape their sites.

kortilla 3y ago
It has nothing to do with greedy capitalists. I don’t want anyone scraping my site at all. I don’t charge anything for it.
- codedokode 3y ago
  Let's take as an example a website which compares prices in different stores and shows which has the cheapest price. You can do it manually, but using automation it is faster and more convenient. Is doesn't make sense to read websites manually when you can use a script or a language model. Obviously for consumer it is better to be able to scrape sites. It is only those store owners (greedy capitalists) who do not want consumer to know that their prices are inflated. Another thing is looking for some information, it is better just to have a language model go around the web and summarize the data for you rather than read someone's site with white letters on black background and weird font.
dewey 3y ago
It's not uncommon that companies ask other companies to scrape their site as they don't have the tech resources to build an API / integration for whatever they want to have.
sim7c00 3y ago
if u run a small site and ppl scrape it aggresively that can rack up ur bills.depending on where u host ur site. ofcourse not an issue for billion dollar companies, but a line needs to be drawn somewhere. also, whats the purpose of the scraping? usually its greedy capatalist purposes, so.then ur point is a bit moot, dont u think?
- codedokode 3y ago
  Let's say someone makes a site that compares prices in different stores. It needs a lot of scraping, but is useful for consumers. Obviously, the only ones who are against it will be the store owners.
  - noAnswer 3y ago
    Unless you go full blown DoS you will not be able to scrap 100.000 of articles multiple times a day. Geizhals for example compares prices every 10 minutes. It does this by working with the stores (they provide a price list) not against them. If a store owner doesn't want the reach, it's their loss. IMHO no need for a DoS attack.
  - zztop44 3y ago
    And then someone else makes another site that also compares prices in different stores. Instead of scraping all the stores they can just scrape the first comparison site and invest the time saved into SEO :)
opportune 3y ago
Sites have a right to try to block scrapers. Scraping can incur significant costs and users may not want it - as a LinkedIn user, I want my profile to be indexed and able to viewed without logging in from the Web, but I also don’t want my entire profile scraped and resold/rehosted for marketing. It’s also often the case that the website put in significant working acquiring/curating/cleaning their data, and they are being scraped by other commercial entities trying to just resell it (blocking that isn’t being greedy, the data is basically being stolen for other commercial entities). So it’s not just being greedy. Individuals also, IMO, have a right to sell access to their network for scraping-via-proxy. But they should be prepared to deal with the consequences, like a potential IP ban. Most people using VPNs that resell their residential network for scraping probably don’t know that’s happening, and many scrapers are indeed doing something bad, which is why there is a disdain for the practice.
- withinboredom 3y ago
  I guess secret shoppers are illegal too.
  - pests 3y ago
    Secret shoppers are paid by the company in question. How is that in any way relevant?
  - opportune 3y ago
    Not only did I not say anything about legality, I did not even say scraping is “bad”, just that it often is.
- ipaddr 3y ago
  Not to worry linkedin already packages and sells your data no scrapers needed.
  - scarface_74 3y ago
    The difference is why else would I use LinkedIn except for my job data to be available for recruiters and employers? Yes, I realize that LinkedIn is a self aggrandizing cesspool. I never post anything there.

It has nothing to do with greedy capitalists. I don’t want anyone scraping my site at all. I don’t charge anything for it.

codedokode 3y ago
Let's take as an example a website which compares prices in different stores and shows which has the cheapest price. You can do it manually, but using automation it is faster and more convenient. Is doesn't make sense to read websites manually when you can use a script or a language model. Obviously for consumer it is better to be able to scrape sites. It is only those store owners (greedy capitalists) who do not want consumer to know that their prices are inflated. Another thing is looking for some information, it is better just to have a language model go around the web and summarize the data for you rather than read someone's site with white letters on black background and weird font.

dewey 3y ago

It's not uncommon that companies ask other companies to scrape their site as they don't have the tech resources to build an API / integration for whatever they want to have.

sim7c00 3y ago

if u run a small site and ppl scrape it aggresively that can rack up ur bills.depending on where u host ur site. ofcourse not an issue for billion dollar companies, but a line needs to be drawn somewhere. also, whats the purpose of the scraping? usually its greedy capatalist purposes, so.then ur point is a bit moot, dont u think?

codedokode 3y ago
Let's say someone makes a site that compares prices in different stores. It needs a lot of scraping, but is useful for consumers. Obviously, the only ones who are against it will be the store owners.
- noAnswer 3y ago
  Unless you go full blown DoS you will not be able to scrap 100.000 of articles multiple times a day. Geizhals for example compares prices every 10 minutes. It does this by working with the stores (they provide a price list) not against them. If a store owner doesn't want the reach, it's their loss. IMHO no need for a DoS attack.
- zztop44 3y ago
  And then someone else makes another site that also compares prices in different stores. Instead of scraping all the stores they can just scrape the first comparison site and invest the time saved into SEO :)

opportune 3y ago

Sites have a right to try to block scrapers. Scraping can incur significant costs and users may not want it - as a LinkedIn user, I want my profile to be indexed and able to viewed without logging in from the Web, but I also don’t want my entire profile scraped and resold/rehosted for marketing. It’s also often the case that the website put in significant working acquiring/curating/cleaning their data, and they are being scraped by other commercial entities trying to just resell it (blocking that isn’t being greedy, the data is basically being stolen for other commercial entities). So it’s not just being greedy. Individuals also, IMO, have a right to sell access to their network for scraping-via-proxy. But they should be prepared to deal with the consequences, like a potential IP ban. Most people using VPNs that resell their residential network for scraping probably don’t know that’s happening, and many scrapers are indeed doing something bad, which is why there is a disdain for the practice.

withinboredom 3y ago
I guess secret shoppers are illegal too.
- pests 3y ago
  Secret shoppers are paid by the company in question. How is that in any way relevant?
- opportune 3y ago
  Not only did I not say anything about legality, I did not even say scraping is “bad”, just that it often is.
ipaddr 3y ago
Not to worry linkedin already packages and sells your data no scrapers needed.
- scarface_74 3y ago
  The difference is why else would I use LinkedIn except for my job data to be available for recruiters and employers? Yes, I realize that LinkedIn is a self aggrandizing cesspool. I never post anything there.