24 ms·

The Scraping Problem and Ethics

241 points by codezero 12y ago

elwell 12y ago
Is it ethical to scrape from a scraper?
- programminggeek 12y ago
  Who watches the watchmen?
- drdeadringer 12y ago
  Is it ethical to scrape from a service you DO pay for, instead of using their API or other interface?
- andreasvc 12y ago
  Ethical at least requires respecting robots.txt.
  - elwell 12y ago
    If robots.txt said to jump off a bridge, would your spider do it?
duey 12y ago
The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.
- awor 12y ago
  Did you read the article? In both cases the offending parties were told to contact RBS to obtain a proper license/account.
  - kijin 12y ago
    That's exactly the problem that GP is talking about. McAfee were just being the usual assholes, but the first guy mentioned in the blog post could have been converted into a paying customer if the pricing scheme were clearly outlined on the web. Since "Contact us" account tiers are usually reserved for the very high end, he probably assumed that it would cost him an arm and a leg. It shouldn't be too difficult to come up with a handful of moderately priced account tiers, each targeting a different type of customer, as well as an open-ended tier at the end ("Contact us") for those with special needs and deep pockets.
    - brc 12y ago
      So many times I say this to people. Sometimes the reason people aren't buying is because they can't find the price, and aren't willing to pay the mental price of talking to a sales person to find out. I understand the sales psychology in making sure you enter into a proper discussion with people to make sure their needs are right, and extracting the maximum consumer surplus from them. But there is a non-trivial pricing point where this makes sense, and by not having any anonymous-sign up, you're cutting off your sales curve below this point. I think it's very easy to convince yourself it's easier just to sign up customers over the phone, but doing so without at least testing takeup of simpler tiers is an incomplete picture.
      - nemothekid 12y ago
        I get this line of thinking if your you can sell your offering for $100 bucks a month. But what if your minimum offering is $2500/mo? Correct me if I'm wrong, but I've yet to see a company - that only sales through enterprise - put up a sign saying where their plans start at $2,500. In most cases I don't believe these services are forcing customers to talk on the phone because they think they will convert them. Chances are that sign up isn't as easy as a 1-2-click, and involved set up and understanding of the customers needs is required before services can be rendered.
        kijin 12y ago
- lukejduncan 12y ago
  It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."
  - duey 12y ago
    Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.
    - danielweber 12y ago
      Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.
      - DanBC 12y ago
        And it sucks as a method of buying something.
      - thaumasiotes 12y ago
        Perhaps a method used exclusively by very large companies is that way for a reason? Of course purchasing by talking to someone on the phone is extremely difficult. The only reason people sell that way is to make sure they can hassle you as much as possible during the sale.
      - the_ancient 12y ago
        and I have a policy that if the company does not practice open pricing by publishing their prices online I will not do business with them.. Fuck that non-sense of pricing based on what ever they think they can scam me out of
        danielweber 12y ago
        That's fine. 'patio11 has lots of stories about that. However, the lack of up-front pricing data isn't being used to justify "I won't do business with you." It's justifying "I'm going to take all your stuff anyway."
lugg 12y ago
Pretty sad. McAfee I dont really expect much less from. They've always been a bunch of scam artists / con's in my mind. I dontbhave time to look but couple of things striked me as odd. Is there a reason you dont lock down your request limits? Also why dont you secure access? Allow free accounts to be made and issue apis keys for them, at least that way you can much more easily rate limit access to the apis and heavily rate limit front end web requests down to reasonable numbers.
DigitalSea 12y ago
If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder for people to obtain the information (to the best of your abilities) and paying will become more of an option because they won't go to the trouble of scraping it if it's impossible and will take a lot of time to do so. Think of your offering as a car. You currently have no car alarm or immobiliser, if you install a immobiliser and car alarm, you will make it very hard for a thief to steal your car.
- gadders 12y ago
  If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."
  - dspillett 12y ago
    You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage. You would no doubt eventually win any resulting legal argument, assuming you could afford to carry the argument on to that conclusion. Sending no data, or limited data, would be safe though. A better method would be to set "default" pricing (something high but not ridiculous, that could easily be negotiated downwards if they contact you) and make access beyond a few requests a click-through (or better: have them respond to an email before progressing further) where they agree to that pricing if they are using the information commercially.
    - andreasvc 12y ago
      You don't have to return bogus data, you can return an HTTP error code: 402 payment required.
      - 12y ago
jwcrux 12y ago
As a somewhat unrelated side note, I'm actually working on providing a free API written in Golang for CVE and CVSS entries. You can find it here: https://github.com/jordan-wright/cve-api https://github.com/jordan-wright/cve-api
kevcampb 12y ago
I see that the "Open Source" Vulnerability Database changed it's name to the "Open Sourced" Vulnerability Database in July 2013. https://web.archive.org/web/20130714002216/http://www.osvdb.org/osvdb_license https://web.archive.org/web/20130714002216/http://www.osvdb.... I guess they were hoping no one would notice the subtle but substantial change to their service
yaur 12y ago
Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.
- userbinator 12y ago
  Maybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...
  - yhager 12y ago
    Not completely by the specification, but I think this one works as expected. user-agent: * disallow: / user-agent: Googlebot allow: /
    - yaur 12y ago
      I think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.
      - thefreeman 12y ago
        By a single search engine which probably provides the vast majority of their traffic. While I don't necessarily agree with the concept of only allowing google to index your site, comparing a search engine which feeds you business to a company reselling your data with no attribution is not really fair in my opinion.
        yaur 12y ago
        Neither of the two parties mentioned is likely to resell the data directly, both are likely to to create a derivative work which they will exploit commercially as will Google. What part of the ToS make one OK while and the other forbidden? How does this work when a smart scraper can just pull from the Google or archive.org cache?
kylemaxwell 12y ago
Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.
- Afforess 12y ago
  Scraping isn't a crime, but by the same token, neither is rate limiting and banning ips.
- codezero 12y ago
  It can be a crime if it is a violation of the site's terms of service and the requests violate the host's robots.txt. I'm not sure whether there is a legal precedent though, in some cases you could call it a denial of service if the requests are not rate limited, and in other cases it might be considered an inappropriate access (see Weev, though he eventually won appeal).
  - deleted 12y ago
    [deleted]
    - acrefoot 12y ago
      Actually, Terms of Service violations fall under the Computer Fraud and Abuse Act, since ToS agreements can lay out under which circumstances that authorization for access to computer systems is given. That sort of obscene generality is the reason for proposals such as Aaron's Law, but to my knowledge there are no such protections today.
      - quesera 12y ago
        Totally correct. Parent comment deleted for lack of usefulness. Thank you for the correction.
      - aidenn0 12y ago
        While what you have stated is certainly the position held by the US AG, I don't believe it's been tested in court yet.
msie 12y ago
Why not put all information behind a login page and force people to sign up? It looks like this site will be used by only a few people anyways. You can then also track scrapers by login.
paulhauggis 12y ago
What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical". This is straight from the open security foundation website: "We believe that security information and services should be easily accessible for all who have the need for such information and services"
- codezero 12y ago
  While your sentiment is reasonable, I think the main difference here is that McAfee and others mentioned host their own private vuln databases and do not share them with anyone, so they were scraping to increase their own private resources for commercial use. Aaron was scraping private resources to share publicly.
  - jamra 12y ago
    More than that, he was scraping private resources that were freely populated. He was not robbing content creators of their money. He was circumventing a paywall to what should be free data.
    - web007 12y ago
      Is OSVDB any different? "Open Source" Vulnerability Database implies openness. The maintainers are trying to implement a paywall. I feel bad for OSVDB from a sysadmin perspective, but if Aaron's case was so polarizing for essentially the same thing, why isn't everyone jumping on the Hate Train here?
    - npizzolato 12y ago
      That's quite the mental leap. He was circumventing a paywall, but that's not robbing anyone of money because it should have been free in the first place? Well, it wasn't free, even if you think you it should be. Thus the paywall.
      - codezero 12y ago
cproctor 12y ago
I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served to identified scrapers with 5% bogus vulnerabilities. Or is that about as ethical as spiking trees to prevent illegal logging?
- lukejduncan 12y ago
  I've definitely seen this done in practice
- freshhawk 12y ago
  Heh, I've always called this the map makers trick (found an article about it here: https://theweek.com/article/index/241967/trap-streets-the-crafty-trick-mapmakers-use-to-fight-plagiarism https://theweek.com/article/index/241967/trap-streets-the-cr...) although I guess that is specific to putting a small amount of fake data in your dataset to prove someone else used it. The hellbanning metaphor does fit for return large amounts of poison results. It could be like spiking trees I guess, but that depends on the potential for harm. I just looked it up and was surprised to find that only one injury has ever been reported due to tree spiking. I guess it's a better talking point than actual tactic. And like lukejduncan said, this is definitely done in practice.
  - deleted 12y ago
    [deleted]
  - calroc 12y ago
    The point of spiking trees isn't to hurt people, in fact the tree-huggers tell the loggers that the trees have been spiked so that they won't attempt to cut the trees. The point of the tactic is to prevent damage to the forest, not to actually hurt anyone.
ChuckMcM 12y ago
This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical turk and paid them each a dollar to do one search, it probably would get them more information. Look at folks like 80legs or other 'distributed' scrapers. They exist almost solely to subvert these service terms. Are they evil? Creative? One of the things that stands out is called out in this article. The people involved really want this information, so much that they are willing to expend time and effort to construct scraping bots and what have you . Why not just buy it? How is it that someone gets a request from their boss to get some information, but their boss expects them to get it for free? Can you imagine if they said, "We need pens, pencils, notebooks, staplers, the works for the office here. Oh and you can't spend any money getting that stuff, just get it here." Would they construct some elaborate raid on a nearby Office supplies store using a mercenary army of criminals? Why do that with information? We did an experiment where we would 'grep the web' for you, basically run a regex over a multi-billion page crawl, give you the first 50 results for "free" and you could buy the complete set. I think we sold exactly one of those. It is a weird thing, the OP captured it perfectly. [1] Its a violation of the terms of service.
- bizark1342 12y ago
  Looks like 80legs that you mention rebranded themselves as "Datafiniti" at some point recently. http://blog.datafiniti.net/?p=230 http://blog.datafiniti.net/?p=230
  - aruggirello 12y ago
    which in Italian literally means 'end of Data'... kind of ironic isn't it?
- meowface 12y ago
  I'm assuming that the vast majority of the time, it's a case of an individual or small group that wants the information, not a real corporation. Usually they have no intention of paying for anything or playing by the rules. I have absolutely no explanation regarding McAfee though, considering the billions McAfee and its parent company, Intel, makes in revenue yearly.
diziet 12y ago
What is the price? I assume the price is in the 5-10k+/month range, which is a reasonable amount for data like this.
alexnking 12y ago
I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today. The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.
- ksk 12y ago
  >make it hard for a company like Google or Yahoo to be founded today. Is that a bad thing? >The internet isn't about "don't take my stuff", it's about spreading that stuff around. Try asking Google if they want to "share" their database of crawled data. >I'm confused by people who want to make their data public, but want to control exactly how people access it. Me too.
  - newaccountfool 12y ago
    >Is that a bad thing? Yes, no question about it. More players in the market means more competition and more competition means a better service.
    - ksk 12y ago
      >More players in the market means more competition and more competition means a better service I'm not sure if this is a joke. So, I'll refrain from replying.
      - parktheredcar 12y ago
        Could you explain your point of view? I'd like to understand both sides here and I think I understand why more competition would be good, but now why it wouldn't be.
        ksk 12y ago
        >I think I understand why more competition would be good, but now why it wouldn't be. That is not what I said at all. I don't believe that more competition necessarily leads to better product/service. In fact I believe that in most cases it does not. In capitalist economies, companies try their hardest to avoid competing. Competition forces companies to reduce costs, and not necessarily increase quality. The quality of a product is not some number that people can read and go "oh yeah this product is better". Marketing people try hard to invent such pointless numbers (e.g. Megapixels in cameras.. horsepower in cars , etc etc). Also many CEOs don't have the first clue on how the product is actually made, much less increase its quality. They rely on these same 'marketing numbers' that their underlings serve them with. So, they too can go "oh yeah this number is increasing so our product is getting better". It seems like a lot of people are brainwashed into believing this free market utopia where things just automatically get better because everyone is competing and the customer is this genius who can figure out which company is delivering a better product. Its sort of like thinking "Well if I'm nice to everyone, everyone will be nice to me.". And then you realize that the real world is a dark place filled with assholes, where slavery is still rampant and many of the goods and services we consume are dependent on the exploitation of natural resources or other fellow humans. Sorry if reading all that bummed you out. I'm really a quite a cheerful person :P
binarysolo 12y ago
Given these people's immediate reaction to resort to scraping, I'm honestly kinda surprised they didn't setup basic practices of at least going through proxies (or free ride off of Tor, ugh).
im3w1l 12y ago
>including an expansive watch list capability, I would avoid the word expansive. Too similar to expensive. Especially when emailing people who are not native English speakers.
im3w1l 12y ago
I was once developing a program for automated trading for my own personal use. In the beginning I made periodical requests to my broker, scraped out the price info, and made orders if the price was right. Once I had something sort of working I decided that I wanted to do this the right way and ask them for permission. They told me I would have to subscribe to a feed through some other program. After many hours of reading the manual and trying to figure out how to transfer the data to my application I again got it sort of working. But I had already started to lose interest and other things had come up in my life. How is this relevant? Well I would have much preferred paying for scraping, than trying to learn some new api. It increased the transaction cost. If you further have to negotiate with a partner company, that sounds like even more transaction cost, from having to send emails back and forth and the mental effort of negotiating.
- wslh 12y ago
  Beyond the moral discussion, I think a market for web scraping is a good thing because currently there are a lot of unconnected people trying to buy/sell this service. Freelancer sites has a lot of offerings for web scraping but this niche has its own issues.
- im3w1l 12y ago
  paying [the site in question] for scraping I mean
bitJericho 12y ago
I would hope they sent a bill.
NoMoreNicksLeft 12y ago
Maybe it's the supervillain in me, but I think I would try to come up with software to recognize the scraping attempts, and rather than ban them just have it generate fake data on the fly.
detroitcoder 12y ago
Amateurs. If you are going to crawl distribute it across unique IP addresses.
e28eta 12y ago
We had a former Google Maps PM give a talk about building APIs, and he addressed scraping. Something he pointed out is that the traffic pattern from scrapers differs from regular users. Regular users make requests for the same things often. Scrapers usually only request any given item once. The really obvious ones do it in order from the beginning. He had a hilarious map showing the requests of someone requesting business information starting from the top left corner of the map. They got really good information on all of the businesses between the North Pole and Greenland before being blocked.
notastartup 12y ago
Scraping is not an ethics discussion. It's what you do with the data that falls into the topic. Selling email lists scraped from websites to spammers would be one. Using email lists to prevent spam would not be one. The article is basically saying "I want to charge people for information I have made public online, they won't pay, so they are obviously thieves by refusing to do it manually by hand like they are supposed to." Gimme a fucking break here. If you don't want the information to disseminate, DO NOT PUBLISH IT ONLINE.
pbreit 12y ago
It's hard to get excited about this. The data owner supposedly has a mission to make the data available but then is concerned with such a thing. It doesn't strike me as that difficult or expensive to make that sized database available.
jmzbond 12y ago
This also goes way beyond scraping, in the sense of small start-ups doing all kinds of hacks or clever manips to stay within the bounds of a free trial service. As someone who sees both sides… I don't know what to say. I'm running test landing pages on Heroku with New Relic that pings the sites every minute to ensure the dynes keep spinning and my users don't experience downtime. While I'm careful to stay within fair use, this is at best obnoxious, because if everyone did this, Heroku would certainly need to redefine what's free. From my POV though, I am a bootstrapped entrepreneur and supporting 5 landing pages. I simply don't have resources to pay for a dyno and test everything I have in my head, especially not combined with the many other resources I'd need to start paying for as well. Or consider the kid in Florida who used Parse' free account for hundreds of thousands of users. [1] (The article was on HN a few weeks ago, this was not it's central point, just something I took away relevant to this comment.) Part of the cause I think is that we live in a world where we're so used to having things be free, it becomes an entitlement. Another is that all these examples of start-up hacks and hustle stories, we kind of laud, don't we? Everyone talks about how Airbnb scraped Craigslist and got a huge boon that way, but few in critical tones. Should we? Or is that how competition and new products get created (i.e., if the scraping hadn't happened, perhaps Airbnb and the whole sharing economy would be less successful today). These are philosophical questions, and I don't really have a solution, but they are things to think about. [1] http://pando.com/2014/04/30/how-a-florida-kids-stupid-app-saved-his-familys-home-and-landed-him-on-the-main-stage-of-facebooks-f8/ http://pando.com/2014/04/30/how-a-florida-kids-stupid-app-sa...
Gigablah 12y ago
Looks like they forgot to conceal the full name and email of the guy from s21sec in one of the quoted emails...

241 points by codezero 12y ago

elwell 12y ago

Is it ethical to scrape from a scraper?

programminggeek 12y ago
Who watches the watchmen?
drdeadringer 12y ago
Is it ethical to scrape from a service you DO pay for, instead of using their API or other interface?
andreasvc 12y ago
Ethical at least requires respecting robots.txt.
- elwell 12y ago
  If robots.txt said to jump off a bridge, would your spider do it?

duey 12y ago

The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.

awor 12y ago
Did you read the article? In both cases the offending parties were told to contact RBS to obtain a proper license/account.
- kijin 12y ago
  That's exactly the problem that GP is talking about. McAfee were just being the usual assholes, but the first guy mentioned in the blog post could have been converted into a paying customer if the pricing scheme were clearly outlined on the web. Since "Contact us" account tiers are usually reserved for the very high end, he probably assumed that it would cost him an arm and a leg. It shouldn't be too difficult to come up with a handful of moderately priced account tiers, each targeting a different type of customer, as well as an open-ended tier at the end ("Contact us") for those with special needs and deep pockets.
  - brc 12y ago
    So many times I say this to people. Sometimes the reason people aren't buying is because they can't find the price, and aren't willing to pay the mental price of talking to a sales person to find out. I understand the sales psychology in making sure you enter into a proper discussion with people to make sure their needs are right, and extracting the maximum consumer surplus from them. But there is a non-trivial pricing point where this makes sense, and by not having any anonymous-sign up, you're cutting off your sales curve below this point. I think it's very easy to convince yourself it's easier just to sign up customers over the phone, but doing so without at least testing takeup of simpler tiers is an incomplete picture.
    - nemothekid 12y ago
      I get this line of thinking if your you can sell your offering for $100 bucks a month. But what if your minimum offering is $2500/mo? Correct me if I'm wrong, but I've yet to see a company - that only sales through enterprise - put up a sign saying where their plans start at $2,500. In most cases I don't believe these services are forcing customers to talk on the phone because they think they will convert them. Chances are that sign up isn't as easy as a 1-2-click, and involved set up and understanding of the customers needs is required before services can be rendered.
      - kijin 12y ago
lukejduncan 12y ago
It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."
- duey 12y ago
  Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.
  - danielweber 12y ago
    Talking to someone on the phone is not "extremely difficult." Large companies buy stuff by talking to people on the phone all the time.
    - DanBC 12y ago
      And it sucks as a method of buying something.
    - thaumasiotes 12y ago
      Perhaps a method used exclusively by very large companies is that way for a reason? Of course purchasing by talking to someone on the phone is extremely difficult. The only reason people sell that way is to make sure they can hassle you as much as possible during the sale.
    - the_ancient 12y ago
      and I have a policy that if the company does not practice open pricing by publishing their prices online I will not do business with them.. Fuck that non-sense of pricing based on what ever they think they can scam me out of
      - danielweber 12y ago
        That's fine. 'patio11 has lots of stories about that. However, the lack of up-front pricing data isn't being used to justify "I won't do business with you." It's justifying "I'm going to take all your stuff anyway."

lugg 12y ago

Pretty sad. McAfee I dont really expect much less from. They've always been a bunch of scam artists / con's in my mind. I dontbhave time to look but couple of things striked me as odd. Is there a reason you dont lock down your request limits? Also why dont you secure access? Allow free accounts to be made and issue apis keys for them, at least that way you can much more easily rate limit access to the apis and heavily rate limit front end web requests down to reasonable numbers.

DigitalSea 12y ago

If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck. You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder for people to obtain the information (to the best of your abilities) and paying will become more of an option because they won't go to the trouble of scraping it if it's impossible and will take a lot of time to do so. Think of your offering as a car. You currently have no car alarm or immobiliser, if you install a immobiliser and car alarm, you will make it very hard for a thief to steal your car.

gadders 12y ago
If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."
- dspillett 12y ago
  You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage. You would no doubt eventually win any resulting legal argument, assuming you could afford to carry the argument on to that conclusion. Sending no data, or limited data, would be safe though. A better method would be to set "default" pricing (something high but not ridiculous, that could easily be negotiated downwards if they contact you) and make access beyond a few requests a click-through (or better: have them respond to an email before progressing further) where they agree to that pricing if they are using the information commercially.
  - andreasvc 12y ago
    You don't have to return bogus data, you can return an HTTP error code: 402 payment required.
    - 12y ago

jwcrux 12y ago

As a somewhat unrelated side note, I'm actually working on providing a free API written in Golang for CVE and CVSS entries. You can find it here: https://github.com/jordan-wright/cve-api https://github.com/jordan-wright/cve-api

kevcampb 12y ago

I see that the "Open Source" Vulnerability Database changed it's name to the "Open Sourced" Vulnerability Database in July 2013. https://web.archive.org/web/20130714002216/http://www.osvdb.org/osvdb_license https://web.archive.org/web/20130714002216/http://www.osvdb.... I guess they were hoping no one would notice the subtle but substantial change to their service

yaur 12y ago

Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.

userbinator 12y ago
Maybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...
- yhager 12y ago
  Not completely by the specification, but I think this one works as expected. user-agent: * disallow: / user-agent: Googlebot allow: /
  - yaur 12y ago
    I think that userbinator's point is that the DO conditionally allow scraping which makes their position even more tenuous IMO.
    - thefreeman 12y ago
      By a single search engine which probably provides the vast majority of their traffic. While I don't necessarily agree with the concept of only allowing google to index your site, comparing a search engine which feeds you business to a company reselling your data with no attribution is not really fair in my opinion.
      - yaur 12y ago
        Neither of the two parties mentioned is likely to resell the data directly, both are likely to to create a derivative work which they will exploit commercially as will Google. What part of the ToS make one OK while and the other forbidden? How does this work when a smart scraper can just pull from the Google or archive.org cache?

kylemaxwell 12y ago

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.) This doesn't even rise to the level of what Weev did.

Afforess 12y ago
Scraping isn't a crime, but by the same token, neither is rate limiting and banning ips.
codezero 12y ago
It can be a crime if it is a violation of the site's terms of service and the requests violate the host's robots.txt. I'm not sure whether there is a legal precedent though, in some cases you could call it a denial of service if the requests are not rate limited, and in other cases it might be considered an inappropriate access (see Weev, though he eventually won appeal).
- deleted 12y ago
  [deleted]
  - acrefoot 12y ago
    Actually, Terms of Service violations fall under the Computer Fraud and Abuse Act, since ToS agreements can lay out under which circumstances that authorization for access to computer systems is given. That sort of obscene generality is the reason for proposals such as Aaron's Law, but to my knowledge there are no such protections today.
    - quesera 12y ago
      Totally correct. Parent comment deleted for lack of usefulness. Thank you for the correction.
    - aidenn0 12y ago
      While what you have stated is certainly the position held by the US AG, I don't believe it's been tested in court yet.

msie 12y ago

Why not put all information behind a login page and force people to sign up? It looks like this site will be used by only a few people anyways. You can then also track scrapers by login.

paulhauggis 12y ago

What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical". This is straight from the open security foundation website: "We believe that security information and services should be easily accessible for all who have the need for such information and services"

codezero 12y ago
While your sentiment is reasonable, I think the main difference here is that McAfee and others mentioned host their own private vuln databases and do not share them with anyone, so they were scraping to increase their own private resources for commercial use. Aaron was scraping private resources to share publicly.
- jamra 12y ago
  More than that, he was scraping private resources that were freely populated. He was not robbing content creators of their money. He was circumventing a paywall to what should be free data.
  - web007 12y ago
    Is OSVDB any different? "Open Source" Vulnerability Database implies openness. The maintainers are trying to implement a paywall. I feel bad for OSVDB from a sysadmin perspective, but if Aaron's case was so polarizing for essentially the same thing, why isn't everyone jumping on the Hate Train here?
  - npizzolato 12y ago
    That's quite the mental leap. He was circumventing a paywall, but that's not robbing anyone of money because it should have been free in the first place? Well, it wasn't free, even if you think you it should be. Thus the paywall.
    - codezero 12y ago

cproctor 12y ago

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served to identified scrapers with 5% bogus vulnerabilities. Or is that about as ethical as spiking trees to prevent illegal logging?

lukejduncan 12y ago
I've definitely seen this done in practice
freshhawk 12y ago
Heh, I've always called this the map makers trick (found an article about it here: https://theweek.com/article/index/241967/trap-streets-the-crafty-trick-mapmakers-use-to-fight-plagiarism https://theweek.com/article/index/241967/trap-streets-the-cr...) although I guess that is specific to putting a small amount of fake data in your dataset to prove someone else used it. The hellbanning metaphor does fit for return large amounts of poison results. It could be like spiking trees I guess, but that depends on the potential for harm. I just looked it up and was surprised to find that only one injury has ever been reported due to tree spiking. I guess it's a better talking point than actual tactic. And like lukejduncan said, this is definitely done in practice.
- deleted 12y ago
  [deleted]
- calroc 12y ago
  The point of spiking trees isn't to hurt people, in fact the tree-huggers tell the loggers that the trees have been spiked so that they won't attempt to cut the trees. The point of the tactic is to prevent damage to the forest, not to actually hurt anyone.

ChuckMcM 12y ago

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical turk and paid them each a dollar to do one search, it probably would get them more information. Look at folks like 80legs or other 'distributed' scrapers. They exist almost solely to subvert these service terms. Are they evil? Creative? One of the things that stands out is called out in this article. The people involved really want this information, so much that they are willing to expend time and effort to construct scraping bots and what have you . Why not just buy it? How is it that someone gets a request from their boss to get some information, but their boss expects them to get it for free? Can you imagine if they said, "We need pens, pencils, notebooks, staplers, the works for the office here. Oh and you can't spend any money getting that stuff, just get it here." Would they construct some elaborate raid on a nearby Office supplies store using a mercenary army of criminals? Why do that with information? We did an experiment where we would 'grep the web' for you, basically run a regex over a multi-billion page crawl, give you the first 50 results for "free" and you could buy the complete set. I think we sold exactly one of those. It is a weird thing, the OP captured it perfectly. [1] Its a violation of the terms of service.

bizark1342 12y ago
Looks like 80legs that you mention rebranded themselves as "Datafiniti" at some point recently. http://blog.datafiniti.net/?p=230 http://blog.datafiniti.net/?p=230
- aruggirello 12y ago
  which in Italian literally means 'end of Data'... kind of ironic isn't it?
meowface 12y ago
I'm assuming that the vast majority of the time, it's a case of an individual or small group that wants the information, not a real corporation. Usually they have no intention of paying for anything or playing by the rules. I have absolutely no explanation regarding McAfee though, considering the billions McAfee and its parent company, Intel, makes in revenue yearly.

diziet 12y ago

What is the price? I assume the price is in the 5-10k+/month range, which is a reasonable amount for data like this.

alexnking 12y ago

I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today. The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.

ksk 12y ago
>make it hard for a company like Google or Yahoo to be founded today. Is that a bad thing? >The internet isn't about "don't take my stuff", it's about spreading that stuff around. Try asking Google if they want to "share" their database of crawled data. >I'm confused by people who want to make their data public, but want to control exactly how people access it. Me too.
- newaccountfool 12y ago
  >Is that a bad thing? Yes, no question about it. More players in the market means more competition and more competition means a better service.
  - ksk 12y ago
    >More players in the market means more competition and more competition means a better service I'm not sure if this is a joke. So, I'll refrain from replying.
    - parktheredcar 12y ago
      Could you explain your point of view? I'd like to understand both sides here and I think I understand why more competition would be good, but now why it wouldn't be.
      - ksk 12y ago
        >I think I understand why more competition would be good, but now why it wouldn't be. That is not what I said at all. I don't believe that more competition necessarily leads to better product/service. In fact I believe that in most cases it does not. In capitalist economies, companies try their hardest to avoid competing. Competition forces companies to reduce costs, and not necessarily increase quality. The quality of a product is not some number that people can read and go "oh yeah this product is better". Marketing people try hard to invent such pointless numbers (e.g. Megapixels in cameras.. horsepower in cars , etc etc). Also many CEOs don't have the first clue on how the product is actually made, much less increase its quality. They rely on these same 'marketing numbers' that their underlings serve them with. So, they too can go "oh yeah this number is increasing so our product is getting better". It seems like a lot of people are brainwashed into believing this free market utopia where things just automatically get better because everyone is competing and the customer is this genius who can figure out which company is delivering a better product. Its sort of like thinking "Well if I'm nice to everyone, everyone will be nice to me.". And then you realize that the real world is a dark place filled with assholes, where slavery is still rampant and many of the goods and services we consume are dependent on the exploitation of natural resources or other fellow humans. Sorry if reading all that bummed you out. I'm really a quite a cheerful person :P

binarysolo 12y ago

Given these people's immediate reaction to resort to scraping, I'm honestly kinda surprised they didn't setup basic practices of at least going through proxies (or free ride off of Tor, ugh).

im3w1l 12y ago

>including an expansive watch list capability, I would avoid the word expansive. Too similar to expensive. Especially when emailing people who are not native English speakers.

im3w1l 12y ago

I was once developing a program for automated trading for my own personal use. In the beginning I made periodical requests to my broker, scraped out the price info, and made orders if the price was right. Once I had something sort of working I decided that I wanted to do this the right way and ask them for permission. They told me I would have to subscribe to a feed through some other program. After many hours of reading the manual and trying to figure out how to transfer the data to my application I again got it sort of working. But I had already started to lose interest and other things had come up in my life. How is this relevant? Well I would have much preferred paying for scraping, than trying to learn some new api. It increased the transaction cost. If you further have to negotiate with a partner company, that sounds like even more transaction cost, from having to send emails back and forth and the mental effort of negotiating.

wslh 12y ago
Beyond the moral discussion, I think a market for web scraping is a good thing because currently there are a lot of unconnected people trying to buy/sell this service. Freelancer sites has a lot of offerings for web scraping but this niche has its own issues.
im3w1l 12y ago
paying [the site in question] for scraping I mean

bitJericho 12y ago

I would hope they sent a bill.

NoMoreNicksLeft 12y ago

Maybe it's the supervillain in me, but I think I would try to come up with software to recognize the scraping attempts, and rather than ban them just have it generate fake data on the fly.

detroitcoder 12y ago

Amateurs. If you are going to crawl distribute it across unique IP addresses.

e28eta 12y ago

We had a former Google Maps PM give a talk about building APIs, and he addressed scraping. Something he pointed out is that the traffic pattern from scrapers differs from regular users. Regular users make requests for the same things often. Scrapers usually only request any given item once. The really obvious ones do it in order from the beginning. He had a hilarious map showing the requests of someone requesting business information starting from the top left corner of the map. They got really good information on all of the businesses between the North Pole and Greenland before being blocked.

notastartup 12y ago

Scraping is not an ethics discussion. It's what you do with the data that falls into the topic. Selling email lists scraped from websites to spammers would be one. Using email lists to prevent spam would not be one. The article is basically saying "I want to charge people for information I have made public online, they won't pay, so they are obviously thieves by refusing to do it manually by hand like they are supposed to." Gimme a fucking break here. If you don't want the information to disseminate, DO NOT PUBLISH IT ONLINE.

pbreit 12y ago

It's hard to get excited about this. The data owner supposedly has a mission to make the data available but then is concerned with such a thing. It doesn't strike me as that difficult or expensive to make that sized database available.

jmzbond 12y ago

This also goes way beyond scraping, in the sense of small start-ups doing all kinds of hacks or clever manips to stay within the bounds of a free trial service. As someone who sees both sides… I don't know what to say. I'm running test landing pages on Heroku with New Relic that pings the sites every minute to ensure the dynes keep spinning and my users don't experience downtime. While I'm careful to stay within fair use, this is at best obnoxious, because if everyone did this, Heroku would certainly need to redefine what's free. From my POV though, I am a bootstrapped entrepreneur and supporting 5 landing pages. I simply don't have resources to pay for a dyno and test everything I have in my head, especially not combined with the many other resources I'd need to start paying for as well. Or consider the kid in Florida who used Parse' free account for hundreds of thousands of users. [1] (The article was on HN a few weeks ago, this was not it's central point, just something I took away relevant to this comment.) Part of the cause I think is that we live in a world where we're so used to having things be free, it becomes an entitlement. Another is that all these examples of start-up hacks and hustle stories, we kind of laud, don't we? Everyone talks about how Airbnb scraped Craigslist and got a huge boon that way, but few in critical tones. Should we? Or is that how competition and new products get created (i.e., if the scraping hadn't happened, perhaps Airbnb and the whole sharing economy would be less successful today). These are philosophical questions, and I don't really have a solution, but they are things to think about. [1] http://pando.com/2014/04/30/how-a-florida-kids-stupid-app-saved-his-familys-home-and-landed-him-on-the-main-stage-of-facebooks-f8/ http://pando.com/2014/04/30/how-a-florida-kids-stupid-app-sa...

Gigablah 12y ago

Looks like they forgot to conceal the full name and email of the guy from s21sec in one of the quoted emails...