5 ms·

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of l

I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.

ohyeshedid 5y ago
Seemingly, most of those people don't have a realistic concept of scale.
Aperocky 5y ago
Anyone who has a minor website knows that majority of the traffic are bot. Imagine if the goal is images and videos, now you've got yourself some heavy duty scraper that could cost the website owner lots of data fees.
kodah 5y ago
When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.
- rstupek 5y ago
  Or bots using ecommerce sites to test credit card validity for use elsewhere
  - herbst 5y ago
    For whatever reason in years of running a SaaS this only happened maybe 3 times and never with my online shop. I guess using stripe and a few basic security settings keeps them away mostly
- nocturnial 5y ago
  > I'm sure there are folks who use bots and scrapers for home automation I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.
  - maxgashkov 5y ago
    Some people have smart mirrors or any other kind of 'news display', so it may be useful for them to scrape the data they may think relevant (this may be weather data, stocks, or even the new Nintendo Switch availability at their nearby retailer).
  - 3np 5y ago
    I don't think anyone mentioned crawling (in my mind, "crawling" refers to a long-running process over several pages or sites and "scraping" could be a single fetch of a single URI) I'm sure there are others, just from the top of my head: * Electricity prices (as OP mentioned). Especially for people with solar panels or multiple options for heating. * Watching for availability/prices of products or new homes one might be on the lookout for. Notifications at price drops/availability * Public transport: next bus/trains from closest station, delays and interruptions * IMDB/tvdb/etc for monitored shows and movies. Common with sonarr. * Air quality, covid outbreaks, whatnot
- tyingq 5y ago
  There's also a fair amount of scraping for things like... - Reselling aggregated data - Competitive pricing and inventory data - "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve - Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it. - And, sort of in the space you mentioned, searching for existing vulnerabilities by various signatures, or trying to brute-force guess things like passwords.
  - kodah 5y ago
    Most of these I threw under "disrupting business processes". My company considered most of these to be threats to the integrity of their business.
matheusmoreira 5y ago
What could my scraper which makes 1 HTTP request per day possibly cost the webmaster?
- ChuckMcM 5y ago
  Nothing of course. But follow that string a little further. So you do some thing which once a day scrapes a site and pulls off some data that you use in your thing. Maybe you talk about it to friends, or you have this thing as one of your github repositories. Some of your friends download the repository and also start using your thing. They talk to people about how cool your thing is, or what it does and the nice convenience of automating something that you used to have to do manually. There are 86,400 seconds in a 24 hour period, probably folks won't change your code at all at first, and as it diffuses into the community some webmaster starts seeing this weird spike of queries that happen once a day at some time. Different addresses but always the same kind of request. It's not a problem when its like 10 or 20 qps burst but when it starts getting up to 100 - 200 or worse 1000 - 2000, it causes the system to perhaps spin up additional instances that it isn't going to need after the burst and waste money. So the webmaster starts denying those requests with a 404. Now sometimes your code works and sometimes it doesn't but you don't know that the webmaster is fighting you yet. Maybe eventually you start randomly varying the request time, or the people who have copied your thing are in more varied time zones so you it starts getting spread over the day. now the webmaster is seeing bursts of traffic nearly every hour on the hour and that is weird so a more aggressive mitigation strategy is enacted. People using your thing complain that it keeps breaking so you look into it and realize that the site is trying to block your requests. Perhaps you don't understand why this is, or perhaps you do and don't care, either way you come up with some strategies that avoid the block (maybe your rotate the user-agent or something). Now the query traffic is spiking again and the webmaster is getting complaints that this 'bot traffic' is resulting in useless AWS fees because it isn't part of the revenue traffic and it is forcing the service to add more resources for their customers. Not all scrapers are malicious, but my experience is that it is rare that a non-malicious scraper application isn't talked about and shared (amongst people who have a similar itch that the thing is scratching) and because its all open source it spreads around.
  - 5y ago

ohyeshedid 5y ago

Seemingly, most of those people don't have a realistic concept of scale.

Aperocky 5y ago

Anyone who has a minor website knows that majority of the traffic are bot. Imagine if the goal is images and videos, now you've got yourself some heavy duty scraper that could cost the website owner lots of data fees.

kodah 5y ago

When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs that become overhead. Bots are usually coded with retry mechanisms and ways to change connection criteria on subsequent retries.

rstupek 5y ago
Or bots using ecommerce sites to test credit card validity for use elsewhere
- herbst 5y ago
  For whatever reason in years of running a SaaS this only happened maybe 3 times and never with my online shop. I guess using stripe and a few basic security settings keeps them away mostly
nocturnial 5y ago
> I'm sure there are folks who use bots and scrapers for home automation I know this is off-topic, but I'm really curious. How does scraping the web help with home automation? Maybe downloading weather data could help, but crawling the web? I think I'm missing something about home automation.
- maxgashkov 5y ago
  Some people have smart mirrors or any other kind of 'news display', so it may be useful for them to scrape the data they may think relevant (this may be weather data, stocks, or even the new Nintendo Switch availability at their nearby retailer).
- 3np 5y ago
  I don't think anyone mentioned crawling (in my mind, "crawling" refers to a long-running process over several pages or sites and "scraping" could be a single fetch of a single URI) I'm sure there are others, just from the top of my head: * Electricity prices (as OP mentioned). Especially for people with solar panels or multiple options for heating. * Watching for availability/prices of products or new homes one might be on the lookout for. Notifications at price drops/availability * Public transport: next bus/trains from closest station, delays and interruptions * IMDB/tvdb/etc for monitored shows and movies. Common with sonarr. * Air quality, covid outbreaks, whatnot
tyingq 5y ago
There's also a fair amount of scraping for things like... - Reselling aggregated data - Competitive pricing and inventory data - "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve - Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it. - And, sort of in the space you mentioned, searching for existing vulnerabilities by various signatures, or trying to brute-force guess things like passwords.
- kodah 5y ago
  Most of these I threw under "disrupting business processes". My company considered most of these to be threats to the integrity of their business.

matheusmoreira 5y ago

What could my scraper which makes 1 HTTP request per day possibly cost the webmaster?

ChuckMcM 5y ago
Nothing of course. But follow that string a little further. So you do some thing which once a day scrapes a site and pulls off some data that you use in your thing. Maybe you talk about it to friends, or you have this thing as one of your github repositories. Some of your friends download the repository and also start using your thing. They talk to people about how cool your thing is, or what it does and the nice convenience of automating something that you used to have to do manually. There are 86,400 seconds in a 24 hour period, probably folks won't change your code at all at first, and as it diffuses into the community some webmaster starts seeing this weird spike of queries that happen once a day at some time. Different addresses but always the same kind of request. It's not a problem when its like 10 or 20 qps burst but when it starts getting up to 100 - 200 or worse 1000 - 2000, it causes the system to perhaps spin up additional instances that it isn't going to need after the burst and waste money. So the webmaster starts denying those requests with a 404. Now sometimes your code works and sometimes it doesn't but you don't know that the webmaster is fighting you yet. Maybe eventually you start randomly varying the request time, or the people who have copied your thing are in more varied time zones so you it starts getting spread over the day. now the webmaster is seeing bursts of traffic nearly every hour on the hour and that is weird so a more aggressive mitigation strategy is enacted. People using your thing complain that it keeps breaking so you look into it and realize that the site is trying to block your requests. Perhaps you don't understand why this is, or perhaps you do and don't care, either way you come up with some strategies that avoid the block (maybe your rotate the user-agent or something). Now the query traffic is spiking again and the webmaster is getting complaints that this 'bot traffic' is resulting in useless AWS fees because it isn't part of the revenue traffic and it is forcing the service to add more resources for their customers. Not all scrapers are malicious, but my experience is that it is rare that a non-malicious scraper application isn't talked about and shared (amongst people who have a similar itch that the thing is scratching) and because its all open source it spreads around.
- 5y ago