9 ms·

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff,

by hobs 10y ago

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

madamelic 10y ago
>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them. In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine. tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.
- muglug 10y ago
  > If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
  - cookiecaper 10y ago
    Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too. Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser. Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect. Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser. It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.
    - DashRattlesnake 10y ago
      > I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was saved too. Bandwidth is certainly part of it, but there's also also database and app-server load (which may be the actual bottleneck) that a scraper isn't necessarily bypassing.
      - cookiecaper 10y ago
        Yeah, I just have a hard time buying that a scraper that does less than a conventional desktop browser is going to accidentally stumble across something that causes the server-side to flip out. I'm not really sure in what case your hypothetical is plausible. Scrapers are usually used to get publicly-available data more efficiently. What you're describing would basically require the scraper to hammer an invisible endpoint somewhere, but there's no reason the scraper would do that -- it just wants to get the data displayed by the site in a more efficient manner. I suppose the browser could enforce a cooldown on an expensive callback via JavaScript, which a scraper would circumvent, but IMO that's not a fair reason to say scrapers are disallowed; cooldowns should be enforced server-side. There's no way to ensure that a user is going to execute your script. That's just part of the deal. Everything about scrapers means less server load; no images, no wandering around the site trying to find the right place, no heavy JavaScript callbacks that invoke server-side application load, etc. Scrapers are just highly-optimized browsing devices targeting specific pieces of data; it's logical that they would be cheaper to serve than a desktop user who's concerned about aesthetics and the like. In our specific case, those JavaScripts we didn't download included instructions to make over 100 AJAX requests on every page load. No wonder users were looking for something more efficient. So I agree that a scraper isn't necessarily bypassing some load-heavy operations, but I find it highly implausible that a non-malicious scraper would be invoking operations that cause extra load (beyond just hitting the site too often). Frankly, I'd be surprised if there was a functional scraper that regularly invoked more resource cost per-session than a typical desktop browsing session to get equivalent data.
  - chucksmash 10y ago
    Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so after N very reasonably spaced queries the site would reproducibly start throwing an uncaught MAX_DB_CONNECTIONS_EXCEEDED and just be down for everybody everywhere who might've wanted to use it.
    - baddox 10y ago
      It seems like you could easily hit those scaling issues by manually browsing the website. While I agree that it sucks to take down a site by scraping, in that specific case it sounds like the performance issues are their fault and not yours. That said, once I realized the effect my scraping had, I would (hopefully) cease my scraping.
      - chucksmash 10y ago
        So the thing is, I could totally believe they never saw this traffic pattern under normal load. I'd expect bar trivia scores in a certain mid-sized US city are one of those niche things where you have a very low number of uniques but each unique then pokes around on 9 or 10 pages while they're there. The fact that the site didn't crash during normal browsing was what originally led me to speculate they were maintaining an open DB connection per session. If that was indeed the issue, I could totally imagine they'd only rarely (never?) had 100+ "concurrent..ish" unique visitors.
        Grishnakh 10y ago
        Ok, then why couldn't you revise your scraper so that it did everything in a single session, to avoid this problem? To me, for private, personal use, a scraper should emulate a normal human browser as much as possible to avoid causing site problems and to avoid detection. If what you're doing can be done in the background, or by a cron process at some odd hour, it doesn't have to be fast at all, and you can set the timings to be similar to a normal human.
  - antisthenes 10y ago
    You're saying the site should pay him a consulting fee for the free load-testing service he provides?
  - wwweston 10y ago
    One fair baseline is whether or not the custom User Agent you're using to scrape has request timing that's on the order of what a fast human visitor might do. If the site can't handle that, it's certainly not the UA's fault.
  - bigiain 10y ago
    I mostly agree with the post's author on the "I'm just automating something I'd otherwise be doing manually". If the local weather service publishes, say, barometric charts on their site - but has a TOS that prohibits me from scraping, and my alternative was to just hit their site every day and right-click and save-as on the chart - I feel absolutely no compunction in automating that. You need to be careful of the slippery slope though, when it's easy to grab every day's local barometric chart, it become too easy to think "Hey, I just need to stick that in a loop and I can grab 1000 different charts every day!". I'd personally _not_ do that. If it's something I'm likely to do "by hand" but would occasionally miss a day or three, I'll automate it no matter what the TOS says.
- minimaxir 10y ago
  > tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business. Analyzing data that you're not allowed to access gives you/your company a competitive advantage, which is affecting their service/business even if it's not posted/distributed publically.
  - Nicksil 10y ago
    I don't follow your argument. How does one get their scraper access to data they would otherwise not be able to access through 'normal' browsing techniques?
    - Raphmedia 10y ago
      Example I know of: You can scrape your competitor's Facebook pages since their creation and output nice graphs of which posts generated what kind of likes and suscriptions. This data is usually limited to the owner of the page.
      - wongarsu 10y ago
        But I can get the same data with manual browsing. And I can just as well pay some chinese to collect the data and input it in my database. Automated scraping is just a way to drastically reduce labor costs for information collection. Sure, it's a competitive advantage, but I think disallowing it or calling it unethical is a pretty big can of worms. Why is it ok if something is done by humans but not ok if a computer does it by himself?
      - deleted 10y ago
        [deleted]
      - charlesdm 10y ago
        This is called hustling. I love it. What's not to like?
      - Trundle 10y ago
        If someone was to sit down and use paper + pencil + time to accomplish the same thing, would you still have issue with it? It's publicly available data. Should you also not watch your competitors television ads or walk in to their physical store and browse around?
- sjwright 10y ago
  > Scraping, in my opinion, isn't black hat unless you are ... stealing info. And as a webmaster, how can I tell the difference before it's too late?
pierrebai 10y ago
To know the TOS of a page, you need to read it. To know which links are part of a site and which are not, you need to follow the link. Having a TOS as part of the page content is akin to having a sign in a room, only readable by entering the room, that says "you are not allowed to enter this room". Yes, this defense is being petty abotu details, but I find businesses using post-hoc discoverable limitations to limit people rights annoying.
Fiahil 10y ago
Sometimes, scraping a website is the only way at your disposal to fetch relevant information, you've paid for (or not). It could be simply the opening schedule of your local administration or the status of different pieces of public infrastructure. If your administration don't have the resources (and it's often the case) to maintain a proper JSON API for you to fetch with a fancy python lib, then, it's not "super bad netizen stuff" to scrap a few HTML/PDF/XLS, parse them and display them for convenient public consumption on your personal website (and paying for the bandwidth). It's 2016. State-companies holding a third party responsible for their own outages and poor planning is _bad faith_[1]. ETL? Never heard of it? [1]: https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-data-a-paris https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-d... (french)
flukus 10y ago
> This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. How so? I send a web request, they send me the content in a response. If they aren't happy with that then they should refuse my request.
Mikushi 10y ago
Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out. Being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them (even though, all the scraping in the world won't have any impact).
- hobs 10y ago
  So its moral to continue bad behavior because someone else did it?
  - cookiecaper 10y ago
    It's not bad behavior. The companies that profit off this try to make you think it's bad behavior because they don't want to risk your taking any profit away, and they've installed laws that let them get away with this. They can violate to their heart's content, but unless someone else in the oligopoly sues over the matter (which they would never do, because the precedent may prevent their abuse of the law), the peons will be forced to comply. That's not how a competitive marketplace works, and it's why we have such a hard time breaking gridlock on web properties.
Mikushi 10y ago
Ah, Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out. Scrapping is the way of the web, that you like it or not. Really, being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them.
malz 10y ago
> misuse of other's resources against their will Nonsense, there is no implication that this activity is illicit. Many sites (I have worked with hundreds) are happy to be included in my service, but don't have the technical ability to provide a data feed. They were delighted when I told them I could aggregate their content without any extra work on their part. We respect TOS, we respect robots.txt and so on. Just because you study scraping techniques doesn't mean you intend to break the law. > Breaking captchas and the like is basically blackhat work Um, captchas only work if they work. If breaking them is trivial, they shouldn't exist. Don't shoot the messenger for pointing out the front door is unlocked.
- paulddraper 10y ago
  "Don't shoot the messenger for pointing out the lock on your door can be picked"
  - drabiega 10y ago
    "Don't shoot the messenger for pointing out that robots are capable of opening doors."
madshiva 10y ago
So you think that people should waste their time solving captcha are the solution? people are paid to solve captcha, there's always something like that and then the users suffer more. It's not a solution at all.
angry_octet 10y ago
'Bad netizen stuff'? Is this a comment from 1997? Breaking captchas is 'blackhat'? What cozy hippy internet alternative reality does this come from? These are the same websites and companies that are loading evercookies and doing browser fingerprinting, that break as much as possible the anonymity citizens should enjoy, with Real Name policies, using network analysis to find who your friends are and what your politics and buying habbits are, that routinely rip private information from you cell phone and share it with oppressive regimes. You're not in Kansas anymore Toto.
TeMPOraL 10y ago
I disagree. DOSing a site is bad behaviour, regardless of how you do this. But accessing it in an automated way instead of a browser? Not really. The deal on the Internet is like this: a website owner can provide whatever they want, and a visitor can read it however they want. Discriminating visitors based on whether or not they seem to be bots instead of people is going beyond what the site provider should do. So is detecting and blocking people using adblockers.
- cookiecaper 10y ago
  IMO all's fair as long as the solution is technical. Dragging it into the courts because you can't figure out how to stop them technically (especially if they're not actually disrupting anything) is inappropriate. We need updated legislation that covers malicious actors that issue DDoS attacks but leaves normal people that scrape consciously and carefully alone.
- Grishnakh 10y ago
  I agree with your assertion about "the deal on the Internet", but I disagree with your conclusions. IMO, site owners should be able to discriminate all they want. However, when someone's browser (or other software...) makes a request to that site, and the site serves them some data, the user should be able to do with that what they will: either honor or decline the requests for them to download various ads or JavaScript, for instance. It should be up to the site owner to craft their site to follow their policies and whims. What I'm completely against is the idea of using the government and law enforcement to enforce some site owner's policies. The only exception I can see for this is extreme cases where this general principle falls down: DOS attacks, for instance. If I can modify my web browser to view a site, but skip the ads, that should be my right. If the site owner codes their site to detect this and then blocks my request to see their site, that should be their right. If I modify my ad-blocker to get around their ad-blocker-block, that should be my right, and so on. As long as we don't get into something like DDOS territory where a reasonable web site has no good technological way of avoiding the problem caused by a user, this isn't something for government to get involved in.
  - 10y ago

by hobs 10y ago

madamelic 10y ago

>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them. In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine. tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.

muglug 10y ago
> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
- cookiecaper 10y ago
  Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too. Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser. Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect. Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser. It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.
  - DashRattlesnake 10y ago
    > I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was saved too. Bandwidth is certainly part of it, but there's also also database and app-server load (which may be the actual bottleneck) that a scraper isn't necessarily bypassing.
    - cookiecaper 10y ago
      Yeah, I just have a hard time buying that a scraper that does less than a conventional desktop browser is going to accidentally stumble across something that causes the server-side to flip out. I'm not really sure in what case your hypothetical is plausible. Scrapers are usually used to get publicly-available data more efficiently. What you're describing would basically require the scraper to hammer an invisible endpoint somewhere, but there's no reason the scraper would do that -- it just wants to get the data displayed by the site in a more efficient manner. I suppose the browser could enforce a cooldown on an expensive callback via JavaScript, which a scraper would circumvent, but IMO that's not a fair reason to say scrapers are disallowed; cooldowns should be enforced server-side. There's no way to ensure that a user is going to execute your script. That's just part of the deal. Everything about scrapers means less server load; no images, no wandering around the site trying to find the right place, no heavy JavaScript callbacks that invoke server-side application load, etc. Scrapers are just highly-optimized browsing devices targeting specific pieces of data; it's logical that they would be cheaper to serve than a desktop user who's concerned about aesthetics and the like. In our specific case, those JavaScripts we didn't download included instructions to make over 100 AJAX requests on every page load. No wonder users were looking for something more efficient. So I agree that a scraper isn't necessarily bypassing some load-heavy operations, but I find it highly implausible that a non-malicious scraper would be invoking operations that cause extra load (beyond just hitting the site too often). Frankly, I'd be surprised if there was a functional scraper that regularly invoked more resource cost per-session than a typical desktop browsing session to get equivalent data.
- chucksmash 10y ago
  Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so after N very reasonably spaced queries the site would reproducibly start throwing an uncaught MAX_DB_CONNECTIONS_EXCEEDED and just be down for everybody everywhere who might've wanted to use it.
  - baddox 10y ago
    It seems like you could easily hit those scaling issues by manually browsing the website. While I agree that it sucks to take down a site by scraping, in that specific case it sounds like the performance issues are their fault and not yours. That said, once I realized the effect my scraping had, I would (hopefully) cease my scraping.
    - chucksmash 10y ago
      So the thing is, I could totally believe they never saw this traffic pattern under normal load. I'd expect bar trivia scores in a certain mid-sized US city are one of those niche things where you have a very low number of uniques but each unique then pokes around on 9 or 10 pages while they're there. The fact that the site didn't crash during normal browsing was what originally led me to speculate they were maintaining an open DB connection per session. If that was indeed the issue, I could totally imagine they'd only rarely (never?) had 100+ "concurrent..ish" unique visitors.
      - Grishnakh 10y ago
        Ok, then why couldn't you revise your scraper so that it did everything in a single session, to avoid this problem? To me, for private, personal use, a scraper should emulate a normal human browser as much as possible to avoid causing site problems and to avoid detection. If what you're doing can be done in the background, or by a cron process at some odd hour, it doesn't have to be fast at all, and you can set the timings to be similar to a normal human.
- antisthenes 10y ago
  You're saying the site should pay him a consulting fee for the free load-testing service he provides?
- wwweston 10y ago
  One fair baseline is whether or not the custom User Agent you're using to scrape has request timing that's on the order of what a fast human visitor might do. If the site can't handle that, it's certainly not the UA's fault.
- bigiain 10y ago
  I mostly agree with the post's author on the "I'm just automating something I'd otherwise be doing manually". If the local weather service publishes, say, barometric charts on their site - but has a TOS that prohibits me from scraping, and my alternative was to just hit their site every day and right-click and save-as on the chart - I feel absolutely no compunction in automating that. You need to be careful of the slippery slope though, when it's easy to grab every day's local barometric chart, it become too easy to think "Hey, I just need to stick that in a loop and I can grab 1000 different charts every day!". I'd personally _not_ do that. If it's something I'm likely to do "by hand" but would occasionally miss a day or three, I'll automate it no matter what the TOS says.
minimaxir 10y ago
> tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business. Analyzing data that you're not allowed to access gives you/your company a competitive advantage, which is affecting their service/business even if it's not posted/distributed publically.
- Nicksil 10y ago
  I don't follow your argument. How does one get their scraper access to data they would otherwise not be able to access through 'normal' browsing techniques?
  - Raphmedia 10y ago
    Example I know of: You can scrape your competitor's Facebook pages since their creation and output nice graphs of which posts generated what kind of likes and suscriptions. This data is usually limited to the owner of the page.
    - wongarsu 10y ago
      But I can get the same data with manual browsing. And I can just as well pay some chinese to collect the data and input it in my database. Automated scraping is just a way to drastically reduce labor costs for information collection. Sure, it's a competitive advantage, but I think disallowing it or calling it unethical is a pretty big can of worms. Why is it ok if something is done by humans but not ok if a computer does it by himself?
    - deleted 10y ago
      [deleted]
    - charlesdm 10y ago
      This is called hustling. I love it. What's not to like?
    - Trundle 10y ago
      If someone was to sit down and use paper + pencil + time to accomplish the same thing, would you still have issue with it? It's publicly available data. Should you also not watch your competitors television ads or walk in to their physical store and browse around?
sjwright 10y ago
> Scraping, in my opinion, isn't black hat unless you are ... stealing info. And as a webmaster, how can I tell the difference before it's too late?

pierrebai 10y ago

To know the TOS of a page, you need to read it. To know which links are part of a site and which are not, you need to follow the link. Having a TOS as part of the page content is akin to having a sign in a room, only readable by entering the room, that says "you are not allowed to enter this room". Yes, this defense is being petty abotu details, but I find businesses using post-hoc discoverable limitations to limit people rights annoying.

Fiahil 10y ago

Sometimes, scraping a website is the only way at your disposal to fetch relevant information, you've paid for (or not). It could be simply the opening schedule of your local administration or the status of different pieces of public infrastructure. If your administration don't have the resources (and it's often the case) to maintain a proper JSON API for you to fetch with a fancy python lib, then, it's not "super bad netizen stuff" to scrap a few HTML/PDF/XLS, parse them and display them for convenient public consumption on your personal website (and paying for the bandwidth). It's 2016. State-companies holding a third party responsible for their own outages and poor planning is _bad faith_[1]. ETL? Never heard of it? [1]: https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-data-a-paris https://citymapper.com/i/1208/soutenez-citymapper-et-lopen-d... (french)

flukus 10y ago

> This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. How so? I send a web request, they send me the content in a response. If they aren't happy with that then they should refuse my request.

Mikushi 10y ago

Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out. Being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them (even though, all the scraping in the world won't have any impact).

hobs 10y ago
So its moral to continue bad behavior because someone else did it?
- cookiecaper 10y ago
  It's not bad behavior. The companies that profit off this try to make you think it's bad behavior because they don't want to risk your taking any profit away, and they've installed laws that let them get away with this. They can violate to their heart's content, but unless someone else in the oligopoly sues over the matter (which they would never do, because the precedent may prevent their abuse of the law), the peons will be forced to comply. That's not how a competitive marketplace works, and it's why we have such a hard time breaking gridlock on web properties.

Mikushi 10y ago

Ah, Instagram or Facebook, they thrive on stolen or relinked content and monetize it day in day out. Scrapping is the way of the web, that you like it or not. Really, being amazed at this kind of bad behaviour where the targets are some of the most despicable companies on the web is a bit ironic. Scrape away, these companies hurt the web, let's hurt them.

malz 10y ago

> misuse of other's resources against their will Nonsense, there is no implication that this activity is illicit. Many sites (I have worked with hundreds) are happy to be included in my service, but don't have the technical ability to provide a data feed. They were delighted when I told them I could aggregate their content without any extra work on their part. We respect TOS, we respect robots.txt and so on. Just because you study scraping techniques doesn't mean you intend to break the law. > Breaking captchas and the like is basically blackhat work Um, captchas only work if they work. If breaking them is trivial, they shouldn't exist. Don't shoot the messenger for pointing out the front door is unlocked.

paulddraper 10y ago
"Don't shoot the messenger for pointing out the lock on your door can be picked"
- drabiega 10y ago
  "Don't shoot the messenger for pointing out that robots are capable of opening doors."

madshiva 10y ago

So you think that people should waste their time solving captcha are the solution? people are paid to solve captcha, there's always something like that and then the users suffer more. It's not a solution at all.

angry_octet 10y ago

'Bad netizen stuff'? Is this a comment from 1997? Breaking captchas is 'blackhat'? What cozy hippy internet alternative reality does this come from? These are the same websites and companies that are loading evercookies and doing browser fingerprinting, that break as much as possible the anonymity citizens should enjoy, with Real Name policies, using network analysis to find who your friends are and what your politics and buying habbits are, that routinely rip private information from you cell phone and share it with oppressive regimes. You're not in Kansas anymore Toto.

TeMPOraL 10y ago

I disagree. DOSing a site is bad behaviour, regardless of how you do this. But accessing it in an automated way instead of a browser? Not really. The deal on the Internet is like this: a website owner can provide whatever they want, and a visitor can read it however they want. Discriminating visitors based on whether or not they seem to be bots instead of people is going beyond what the site provider should do. So is detecting and blocking people using adblockers.

cookiecaper 10y ago
IMO all's fair as long as the solution is technical. Dragging it into the courts because you can't figure out how to stop them technically (especially if they're not actually disrupting anything) is inappropriate. We need updated legislation that covers malicious actors that issue DDoS attacks but leaves normal people that scrape consciously and carefully alone.
Grishnakh 10y ago
I agree with your assertion about "the deal on the Internet", but I disagree with your conclusions. IMO, site owners should be able to discriminate all they want. However, when someone's browser (or other software...) makes a request to that site, and the site serves them some data, the user should be able to do with that what they will: either honor or decline the requests for them to download various ads or JavaScript, for instance. It should be up to the site owner to craft their site to follow their policies and whims. What I'm completely against is the idea of using the government and law enforcement to enforce some site owner's policies. The only exception I can see for this is extreme cases where this general principle falls down: DOS attacks, for instance. If I can modify my web browser to view a site, but skip the ads, that should be my right. If the site owner codes their site to detect this and then blocks my request to see their site, that should be their right. If I modify my ad-blocker to get around their ad-blocker-block, that should be my right, and so on. As long as we don't get into something like DDOS territory where a reasonable web site has no good technological way of avoiding the problem caused by a user, this isn't something for government to get involved in.
- 10y ago