27 ms·

Web Scraping in 2016

• minimaxir 10y ago
  Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues-after-scraping-of-user-data/d/d-id/1113362 http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupid-profiles-taken-down-dmca-claim/ https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API following rate limits or b) if there is no API, and the data has the explicit purpose of being shared publically (e.g blogs), I follow robots.txt. Of course, most companies have a do-not-scrape clause in their ToS anyways, to my personal frustration. (Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.)
  • yeowMeng 10y ago
    I am no expert, but I always thought you could scrape without consequence provided you never distribute your scrapings?
  • kuschku 10y ago
    Or you just move to a locale where scraping is legal, and any contractual terms saying otherwise are null and void. I’d assume a lot of HN users are from such locales. We don’t always have to assume US laws apply globally – they don’t.
    • madamelic 10y ago
      Scraping being illegal is as dumb as saying it's illegal to take photos in public. You aren't affecting anyone if you do it respectfully.
      • elmigranto 10y ago
        Being dumb doesn't prevent laws from existing (we all know those "funny US law stuff" in the realms of "no kissing toy camels on the cheek (but the mouths are okay)". Also, since this is somewhat untouched territory, don't be so sure that you'll get a judge who is as well-versed in web scarping and infrastructure as you, or shares your opinions on the subject. (And given that precedents are so important in US laws, you better hope someone else before you didn't get such a judge.)
    • cookiecaper 10y ago
      I was actually searching for such a jurisdiction as my startup was shut down by a company that invoked the CFAA late last year. What do you suggest? The EU is even worse than the US when it comes to data freedom and tech access. The law on the books in many former British colonies provides marginally more protection (the "Telecommunications Act"), but it'd probably still be disputable, and you'd be shut down anyway unless you had millions sitting around with your lawyers' name on it.
      • detaro 10y ago
        It depends on what you are doing. The CFAA is very far-reaching, but of course many aspects were the US answer is "CFAA" are covered by other laws. [EDIT: removed outdated information superseded by european decisions, which make the situation a lot less clear] anti-scraping: If somebody were to offer a telephone book database online and you created a copy of that to sell on your own, you'd almost certainly loose in the EU (since unlike in the US, databases as pure collections of facts have their own copyright protections) The legally safest locations probably are outside the western world if you are targeting western sites.
        • cookiecaper 10y ago
          >Pro-scraping: Last big case I remember here was a flight-search site that did flight search and booking(!) via a scraper and Ryanair lost when they tried to sue them for that, since they couldn't argue convincingly how that was damaging them. Every case I've seen wrt Ryanair (they sue a lot of people) has resulted in a win for Ryanair. Do you have details on the case you're describing? >anti-scraping: [...] Scraping purely factual data is one of my points of defense in the US. I don't want to give it away. >It's still risky though, the safest locations probably are outside the western world if you are targeting western sites. Yeah, this was ultimately the conclusion I had to come to. However, outside the West, the Western companies will just send someone with a briefcase full of $100 bills and pay them off. Corrupt government officials in these locations want the goodwill of a big American company a lot more than they care about any particular random guy. There is only one workable solution: run the service totally anonymously and maintain good opsec so that your cover isn't blown. All under the table. This has its own issues, like making it difficult to receive payment and putting one at much greater legal risk than a mere CFAA dispute, but it's the only option if you don't plan to get shut down.
  • hobs 10y ago
    This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
    • madamelic 10y ago
      >Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with requests because of your scraping, yeah you need to knock it off. If you throttle your scraper in proportion to the size of their site, you aren't really harming them. In regards to "stealing info", as long as you aren't taking info and selling it as your own (which it seems OP is indeed doing), that is just fine. tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business.
      • muglug 10y ago
        > If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.
        • cookiecaper 10y ago
          Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was conserved too. Because we were a specialized browser used by people looking for a very specific piece of data, we could employ caching mechanisms that meant that each person could get their request fulfilled without having to hit the data source's servers. We also had a regular pacing algorithm that meant our users were contacting the site way less than they would've been if they were using a conventional desktop browser. Our service saved the data source a large amount of resource cost. When we were shut down, their site struggled for about two weeks to return to stability. I think they had anticipated the opposite effect. Our service also saved our users a large amount of time. We were accessing publicly-available factual data that was not copyrightable (but only available from this one source's site). There's no reason that the user should be able to choose between Firefox and Chrome but not a task-specialized browser. It is true that some people will (usually accidentally) cause a DDoS with scrapers because the target site is not properly configured, but the same thing could be done with desktop browsers. It doesn't mean that scrapers should be disadvantaged.
  • cookiecaper 10y ago
    Yes, it's important to understand that in the United States, web scraping is usually an illegal activity under the CFAA. If you draw enough attention, your scrape target will notice and threaten you, and probably follow through with the suit. Since the CFAA prescribes both civil and criminal penalties, you may even find yourself in jail for accessing data without the company's approval. Aaron Swartz was being prosecuted under these provisions for scraping public domain data. The CFAA is a really bad law and creates the network effect lock-in that we all considered a natural part of the web. It doesn't have to be that way -- users should be free to use any browsing appliance they want, including so-called "scrapers". Big companies like Google not only got their start by flagrantly violating the CFAA, copyright, and privacy laws, but they continue to do so. The moral of the story is hurry up and get big before you get sued or arrested. There's a long history of ridiculous web scraping rulings based on technical misunderstandings by neophyte judges, including Ticketmaster v. RMG, where infringement was found because the company scraped data out of a page with the Ticketmaster logo on it. Facebook sued a company called Power Ventures which read out only the user's own data. The founder was found personally liable for $3 million in damages. Facebook did this because they don't want it to be easy for their users to move between social media services. If it's easy, Facebook has to compete on merit instead of just keeping switching costs high. Facebook doesn't like that, so they sue people who make it possible -- and the law says they should win. We badly need a revised law, but the powers-that-be will strongly oppose it because it would threaten their monopoly over web properties. They continue to flaunt their strategic ignorance of these laws and then take shelter behind them to stop risk from small innovators (i.e., having to compete fair and square). In the real world, we have a lot of laws that mostly prevent this kind of bad behavior. In cyberspace, the structure is such that most of those laws are not applicable. We need to update and port the pro-small-business logic we have for meatspace companies so that it counts online too. The state of affairs online is really bad. I want to get a law called the "Consumer Data Freedom Act" passed, which would allow users to access any web property with any non-disruptive browsing device, including custom scrapers that don't impose much more load than a typical user browsing session would.
    • 0xdeadbeefbabe 10y ago
      Yes laws are neat and a reason for attending law school I suppose. I'm of the simpleton opinion that TCP/IP and the other protocols are the law of the net, and you ought to start with those.
  • twa927 10y ago
    How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
    • minimaxir 10y ago
      That analogy is not equitable. If you take photographs of a building while on the building's property, they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.
      • mindslight 10y ago
        Sure, but they do not have the right to retroactively declare you as having been trespassing, nor even to preemptively put up a "no photography" sign and have you arrested for trespassing if you disobey it. The entire point of protocols is to precisely define the terms of communication. The status code is '200 OK', not '200 OK/Asterisk'. But of course if lawlers didn't force themselves into the situation, they'd be out of jobs. As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database.
        • cookiecaper 10y ago
          >As an aside, I'd really like to see a browser plugin that would scrape sites in the normal course of access, storing the proceeds in a distributed public database. This would be copyright infringement, since the content of the page is a substantive unique work that is automatically copyrighted by its author. A site that doesn't want you scraping its content is not going to want you posting dumps of its pages. Much like BitTorrent, they'd get into the protocol and send subpoenas to the ISPs behind the IPs that serve their pages, and use that info to sue the customer. When my company was shut down by a legal threat related to scraping, I did suggest to my lawyer that we create something like a browser extension that would grab the data we needed out of normal client-side browsing sessions. This wouldn't be as nice as controlling the flow of information ourselves but it would've worked OK. My lawyer strongly suggested avoiding that as it could've been construed as conspiratorial conduct that would've made criminal prosecution more likely.
  • jurgenwerk 10y ago
    There are hundreds of paid services that scrape Google heavily (search engine ranking trackers). How are they legal?
    • cookiecaper 10y ago
      They aren't, or at least, they won't be if Google decides it doesn't like them anymore and decides to bring the matter to court. The CFAA says it's a crime to exceed "authorized access". Authorized access is whatever the server's owner says it is. If they change their mind, you must cease and desist or risk both civil and criminal penalties. A contract defining the length and nature of your authorization from the server's owner would go a long way to establishing your rights to access, but no one is going to give that to a small player.
      • lossolo 10y ago
        You forgot to add: In USA.
        • cookiecaper 10y ago
          Unfortunately this is true in almost all of the developed world. While developing countries may not have specific legal prohibitions, we know that that doesn't stop big companies from having their way. Heck, even a first-world country like Sweden couldn't resist the pressure from Hollywood to prosecute and jail the operators of the Pirate Bay, which had long been recognized as totally legal in Sweden. Another issue is that on the internet, jurisdiction is a very messy affair. An American judge will likely determine that California and/or the federal government has jurisdiction over such a case because Google is based in California. Most developed countries have treaties with one another that allow them to enforce foreign civil judgments on behalf of the jurisdiction that entered them. Most developed countries also have mutual extradition treaties. The countries that don't can easily be paid off by an interested party.
    • lossolo 10y ago
      They probably doing it from country where it's legal. In most countries there is no law that would be applicable in this case.
  • franciskim 10y ago
    On the ethics side, I don't scrape large amounts of data - eg. giving clients lead gen (x leads for y dollars) - in fact, I have never done a scraping job and don't intend to do those jobs for profit. For me it's purely for personal use and my little side projects. I don't even like the word scraping because it comes loaded with so many negative connotations (which sparked this whole comment thread) - and for a good reason - it's reflective of how the the demand in the market. People want cheap leads to spam, and that's bad use of technology. Generally I tend to focus more on words and phases like 'automation' and 'scripting a bot'. I'm just automating my life, I'm writing a bot to replace what I would have to do on a daily basis - like looking on Facebook for some gifs and videos then manually posting them to my site. Would I spend an hour each and every day doing this? No, I'm much more lazier than that. Who is anyone to tell me what I can and can't automate in my life?
    • ccvannorman 10y ago
      This is exactly my response to "you can't legally scrape my site because of TOS." I don't think anyone has a legal right to tell me HOW I use their service. Making "browsing my website using a script you wrote yourself" illegal is akin to "You cannot use the tab key to tab between fields on my website, you must only use the touchpad to move the cursor over each field individually." It's baloney.
      • cookiecaper 10y ago
        Under the CFAA, they do have the right to determine what constitutes authorized access. If they say you're unauthorized for using the wrong buttons on your mouse, then you're unauthorized. It's treated very similarly to trespass on private land. You can try telling the judge it's baloney, but if he's going by current precedent, he probably won't agree with you.
        • ccvannorman 10y ago
          It is not the same flavor of violation as trespassing on land, more like riding a bicycle on a sidewalk. Fortunately, I'm not in the business of scraping sites, but I still find this legal precedent abhorable, and I hope it gets struck down in court when push comes to shove. I would certainly vote that way if given the chance.
  • sandGorgon 10y ago
    The code is pretty cool. Thanks for releasing that! May I ask why you built your own scraper infrastructure and not build it on top of a known framework like scrapy (which is in python as well).
  • siegecraft 10y ago
    I think the lessons learned from those lawsuits was to always have some sort of 3rd-party intermediary scraping consultancy firm you engage that is totally not just your business under another name.
  • headmelted 10y ago
    One thing I'm not quite clear on here. I understand the use of ToS clauses to prevent scraping but I do kind of wonder to what extent they have authority here. IANAL, but surely this would fall under copyright law? While re-publishing copyright-protected data without consent is probably unlawful in your region (like scraping an art site and re-posting the images), I wouldn't think just scraping data points for a different purpose (like scraping amazon for the purposes of price comparison) is nearly so clear cut (or enforceable), but maybe I'm just naive.
    • cookiecaper 10y ago
      The content falls under copyright law. The problem is that you have to enter the company's servers to obtain this data, and the CFAA says that the company can treat their public-facing web servers like private property, and if you're caught "trespassing", you can be sued and jailed. Scraping plaintiffs are usually granted an injunction based on "trespass to chattels" (among other rationales), i.e., trespass to an individual's property (as opposed to land). Companies like PriceZombie are forced to stop because the CFAA says that Amazon can prevent them from accessing their servers by decree alone. A ToS isn't even really necessary for this, but it helps them pin down their argument. PriceZombie could try to get the data from third-party caches, but it only solves part of the problem, because copyright and trademarks come back into the picture once you have a replica of the target page. In Ticketmaster v. RMG Technologies, the judge found RMG infringing on Ticketmaster's trademarks and copyrights because the page they were scraping included Ticketmaster's logo. The judge said the copy of the full page that existed momentarily in RAM while the scraper extracted the non-copyrightable data constituted a copy that infringed on Ticketmaster's rights, even though the logo was never used by the application in any way, it just happened to be on the page.
  • downandout 10y ago
    Obviously it's a good idea to follow TOS. But as a practical matter, they have to know that you're doing it before they can take action. You wouldn't want to put up a site announcing that you're selling scraped LinkedIn data, for example. But if that data is valuable to your business - collecting names of people that work in certain positions at certain companies so that you can do targeted snail mail campaigns for example - you could quietly scrape and use the data without issue. Use proxies and prosper.
    • cookiecaper 10y ago
      This only goes so far, and if you get found out, you're looking at willful infringement (which usually triples damages) and probably criminal charges under the CFAA. However, it should be acknowledged that there are many people making quiet livings off scrapes that are not legal. There are even a few companies making loud livings off such scrapes, like Google. If you're not going to run it totally anonymously, you should be prepared to jettison and repackage it when you get found it (so that you appear to be complying with the C&D). Scraping is a huge part of the web, and everyone does it. It sucks that it has to live underground because only big companies can duke it out in court.
      • downandout 10y ago
        TOS violations have been found to not be subject to the criminal provisions of the CFAA. The only circumstance under which it would become a criminal issue is if they successfully sued you for it, and obtained a judgment that included a provision ordering you to cease scraping. Ignoring such a court order would then potentially expose you to a criminal contempt of court action.
  • tangue 10y ago
    It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.
    • sjwright 10y ago
      As the owner of a large website, I don't care what you think. I block by default and whitelist when I decide it's in my interest. If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic. Unless you have a way for me to automatically identify "honourable" scrapers such as yourself as distinct from the thousands upon thousands of extremely dodgy scrapers from across the world, my policy shall remain.
      • bigtunacan 10y ago
        As the user of large websites I don't care. I'm not going to read the TOS and I will continue to scrape what I like since it makes my life more convenient. Like OP when blocked I'll just drive my scraping through a web browser which is the same as I've done for years on various sites that never provided APIs.
        • louprado 10y ago
          "As the user of large websites I don't care". Are you sure ? Do you want your OK Cupid or LinkedIn profile to be crossposted on another website without your knowledge.
          • lerpa 10y ago
            If you don't want your data public, then don't make it public in the first place. That's a good rule of thumb.
            • vidarh 10y ago
  • ChuckMcM 10y ago
    I was going to post something similar. When you go to all that trouble that the web site owner is pretty clearly trying to prevent, that is convincing evidence that you are breaking the terms of service. And breaking the terms of service for a web site has been held to be a civil violation (a number of times on Ebay and Amazon) and potentially a CFAA violation by the Justice department.
    • downandout 10y ago
      Actually it's been held that TOS violations are NOT subject to the criminal provisions of the CFAA.
      • ChuckMcM 10y ago
        Are you referring to the MySpace case? Or the July 2016 decision by the Ninth Circuit (https://cdn.ca9.uscourts.gov/datastore/opinions/2016/07/05/14-10037.pdf https://cdn.ca9.uscourts.gov/datastore/opinions/2016/07/05/1...). In US v Nosal it seems like they come down in favor of a CFAA violation if the user acts in an unauthorized way. The author of the piece talks about bypassing captcha's which are, in one interpretation, a demand for authorization (by proving that you are a human and not a program) and by circumventing that authorization they have stepped quite clearly into CFAA territory. If you were referring to a different decision I'd love to read it. I follow this stuff (and at one time explored what legal action our startup could take against scrapers). In our case we also offered a paid API so it was fairly easy to establish damages.
        • downandout 10y ago
          The case you're referring to is an entirely different set of circumstances. From the text: "The panel held that the defendant, a former employee whose computer access credentials were revoked, acted “without authorization” in violation of the CFAA when he or his former employee co-conspirators used the login credentials of a current employee to gain access to computer data owned by the former employer and to circumvent the revocation of access. " I think that case is unambiguous - this guy was using someone else's credentials to access secured systems after having been explicitly told that he could not. I was referring to the MySpace case. I don't think these two cases are in conflict; IMO they are very different. Additionally, for our purposes in this comment thread, we're talking about scraping of publicly available websites by outside parties, not by former employees whose access has been explicitly revoked. That is different than either of these cases.
  • jbmorgado 10y ago
    I actually wonder if that constraining by the ToS has any legal validity in EU. Since in here, typical that kind of stuff cannot be enforced legally.
  • kbenson 10y ago
    > (Disclosure: I have developed a Facebook Page Post Scraper [https://github.com/minimaxir/facebook-page-post-scraper https://github.com/minimaxir/facebook-page-post-scraper] which explicitly follows the permissions set by the Facebook API.) I played with the idea of creating some social aggregation type service with some friends (as a business). The more I read about FB's past behavior with regard to this, and how essential they are to any sort of service like, that, I canned the project. Regardless of what their TOS say, if you get on their radar and they send you a cease-and-desist, it's game over. Facebook is not in the business of subverting their revenue stream, so if you are making money off them and it's preventing them from capitalizing on their users, don't expect to last long if you exist by their grace. Really, there's an interesting space between so small nobody cares and large enough that getting shut down is a real problem. A lot of projects start small and end up (relatively) large, but without a good way to pay for the service itself. While not every service needs to be a business and make money, once you reach the level where you risk either being shut out of your data source or you need to somehow work out an understanding with that source, how do you approach that when being able to pay is off the table? Not to mention the problem approaching before you have to and forcing the situation, or waiting too long and risking the wrath of the source because you've abused their service as long as you have. Has anyone else been in this situation and found an approach that works?
• jgmmo 10y ago
  Good stuff. I do a good bit of scraping, and made RubyRetriever[1] to make my life easier but it seems like I'm getting roadblocked on occasion, probably due to some of the things you mention in your article. Is there any way for a site to verify that only their JS and CSS files are linked? Like preventing injection? [1]: https://github.com/joenorton/rubyretriever https://github.com/joenorton/rubyretriever
  • franciskim 10y ago
    I got the 100th star on the repo! What do you mean by the verifying part?
  • throwanem 10y ago
    You could inspect the src attributes of script tags, and the href attributes of link tags with rel="stylesheet", for acceptable domains. I doubt it would cover all cases, but it might be a start.
• dchuk 10y ago
  Scraping with Selenium in Docker is pretty great, especially because you can use the Docker API itself to spin up/shut down containers at will. So you can spin up a container to hit a specific URL in a second, scrape whatever you're looking for, then kill the container. This can be done via a job queue (sidekiq if you're using Ruby) to do all sorts of fun stuff. That aside, hitting Insta like this is playing with fire, because you're really dealing with Facebook and their legal team.
  • franciskim 10y ago
    True that, I hope Zuck sues me so I'll get extra famous
  • spikej 10y ago
    Serious question: What do you gain from having an extra layer like docker?
    • franciskim 10y ago
      Selenium grid runs in docker, so it's easy to have multiple instances running. Better control.
      • elsonrodriguez 10y ago
        Also, if use Kubernetes to manage the grid you can scale out to your credit card limit on GKE: https://github.com/kubernetes/kubernetes/tree/master/examples/selenium https://github.com/kubernetes/kubernetes/tree/master/example...
      • ramblenode 10y ago
        What are the advantages of this versus a thread pool of web drivers? I'm not really familiar with Selenium Grid.
        • paulryanrogers 10y ago
          Grid can dynamically dispatch based on the browser and capabilities you want when you create the session.
• writeslowly 10y ago
  Have you run into any issues from running all of your scrapers off of AWS, or just from sites detecting that you're accessing large numbers of pages in some sort of obvious pattern? I guess I was hoping there would be sites with more interesting ways to screw with web scrapers (rearranging certain page elements or something) than just throwing up a CAPTCHA.
  • madamelic 10y ago
    Most really don't. A lot of big sites don't seem to care, at least in my experience. The few that I've seen just 'ban' your IP for a few minutes. If you hit Wikipedia too much too quickly, they will essentially refuse to serve you for a while. It was a number of years ago I was doing it, but basically you would be scraping then you would just stop getting info (Maybe I wasn't reading response codes and could've realized quicker what was happening)
    • detaro 10y ago
      Wikipedia provides you with an API and guidelines on how to use it, so you really shouldn't be scraping it directly or so much you hit enforced limits.
      • mod 10y ago
        "It was a number of years ago..."
      • user5994461 10y ago
        Wikipedia provides archives of all its content. No need to scrap it when you can readily download a nicely formatted .xml.zip file containing all knowledge written by mankind.
  • franciskim 10y ago
    I'm not actually doing a lot of hits, so it's generally been ok. I can just rotate my IP or solve the CAPTCHA.
  • 10y ago
• ben_jones 10y ago
  Currently getting 502 Gateway. Guessing this post is also trending on reddit and we hugged it to death :(.
  • deleted 10y ago
    [deleted]
  • franciskim 10y ago
    Just upgraded my EC2 :)
  • franciskim 10y ago
    I'm on Reddit?
    • soared 10y ago
      Google Analytics > Acquisition > Source/Medium > type "reddit" in search bar. Add secondary dimension "referral path"
    • ben_jones 10y ago
      It's a total guess. HN rarely hugs sites to death compared to Reddit (IMO).
• Jake232 10y ago
  Not wanting to thread hijack, but just going to post an article I wrote a few years back as it covers a few other things that are still relevant; and often still gets referenced. May it'll help some people out in combination with OP's post. http://jakeaustwick.me/python-web-scraping-resource/ http://jakeaustwick.me/python-web-scraping-resource/
  • franciskim 10y ago
    Nice post Jake!
  • mdaniel 10y ago
    I was surprised to not see Scrapy listed, but then I saw there were some comments about it - but seriously, doing by hand what Scrapy has spent years perfecting is highly suboptimal. I guess the distinction is between whether one wants to just "toy around" or run the spider for-real.
• franciskim 10y ago
  Sorry guys, hit by traffic - just scaling my EC2 at the moment.
  • franciskim 10y ago
    ok on m4.4xlarge now :)
  • niftich 10y ago
    No worries, we had your page scraped just in case ;) Google Cache link: http://webcache.googleusercontent.com/search?q=cache:https://franciskim.co/2016/08/24/dont-need-no-stinking-api-web-scraping-2016-beyond/ http://webcache.googleusercontent.com/search?q=cache:https:/... Archive.is link: http://archive.is/DQccs http://archive.is/DQccs
    • franciskim 10y ago
      haha :)
  • fareesh 10y ago
    Is it common for developers in the eCommerce space to use scrapers as a means to aggressively push automated price-match algorithms? I've been asked to do this a number of times, was just curious as to how prevalent it is.
    • hk__2 10y ago
      Yes, everybody scrapes the prices of the others.
      • rckclmbr 10y ago
        Most companies will use resources not in their datacenter and not identifiable. Executives know it's sketchy, but they do it anyway.
      • fareesh 10y ago
        With stuff like Facebook opengraph (e.g. og:price) and other meta tags meant to help search engines and social networks get this sort of data to display inline, do you think it's inevitable that complex scraping will no longer be needed in a practical sense since everyone will be inadvertently optimizing their markup in a way that you could write a really simple parser to grab the data?
• prashnts 10y ago
  A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr: $.ajaxSetup({ dataFilter: function (data, type) { if (this.url === 'some url that you want to watch!') { // Do anything with the data here awesomeMethod(this.data) } return data } }) I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page down every 2 seconds, and the `awesomeMethod` just initiated the download. Pasted it all in dev-tools console, and the cheap "scraper" was ready!
  • franciskim 10y ago
    Yup I've been known to do this as well :) I'd have a Node.js + Mongo endpoint ready on the other side.
    • zappo2938 10y ago
      Why not user Nightmare with Node.js + Mongo? Here is an example of injecting a jQuery script into a page with jQuery loaded and getting nicely formatted information returned. [1] [1]https://github.com/adam-s/playboy-fm/blob/master/server/scraper/monthNames.js https://github.com/adam-s/playboy-fm/blob/master/server/scra...
  • macromaniac 10y ago
    Another trick: You can hover over elements in chrome with F12 and the inspect tool, then right click > copy > copy selector and chrome will generate one for you that way you don't have to actually do any work. With a selector it's easy to grab data, here's a linux command that gets every user that posted in this thread: lynx -base -source 'https://news.ycombinator.com/item?id=12345693' | hxnormalize -x | \ hxselect -c -s '\n' "td > table > tbody > tr > td.default > div:nth-child(1) > span > a.hnuser" Here are the most frequent commenters: 27 cookiecaper 22 franciskim 6 fake-name 4 niftich 4 flukus 4 elmigranto 4 downandout 3 tedunangst 3 siegecraft 3 muglug 3 minimaxir 3 madamelic
  • pault 10y ago
• slig 10y ago
  I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas. Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.
  • franciskim 10y ago
    I think I might have some trouble with some reCAPTCHA stuff, but there must be ways around it. I agree with you on your point about AWS.
  • greglindahl 10y ago
    There are a fair number of people in China etc running personal VPNs on AWS.
  • user5994461 10y ago
    There can be legitimate traffic coming from AWS, if not the site itself. It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that. On the other hand, Tor is likely to be 90% evil. When in doubt, just block it. (That makes me think, I should run some proper stats and maybe publish a blog post about that. )
    • slig 10y ago
      > There can be legitimate traffic coming from AWS, if not the site itself. The traffic from the site itself, if it's hosted there, would come from the intranet IP address, right? Not the public facing one. > It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that. Agreed, but it's fairly easy to block the AWS IP traffic on web endpoints and not on the API endpoints.
• mosburger 10y ago
  > AngelList even detects PhamtomJS (have not seen other sites do this). I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and will likely retire that bot soon. It kinda sucks that I'm just grabbing publicly-available content in a very low-bandwidth way, but I really can't convince myself that what I'm doing is very ethical. My to-do list includes making my crawler into a more well-behaved bot and that will have to go.
  • franciskim 10y ago
    I think PhantomJS has a bit of a giveaway in the headers where 2 lines are reversed compared to a normal Chrome, I always thought AngelList detected this flaw. Although I have heard there are builds of PhantomJS where this flaw does not exist.
  • ixtli 10y ago
    I think you may want to decouple your ethical analysis from which private company is making the most money. Remember that the only functional difference between you and somewhere like kayak.com or padmapper is business relationships.
  • ralusek 10y ago
    Mind giving a quick description of getting PhantomJS to work?
• danso 10y ago
  This good list of tactics underscores, for me, how the state of the Web has made it a lot more difficult to teach web scraping as a fun exercise for newbie programmers. It used to be you could get by with an assumption that what you see in the browser is what you get when you download the raw HTML...but that's increasingly less common the case. So now you have to teach how to debug via the console and network panel, on top of basic HTTP concepts (such as query parameters). (Even more problematic is that college kids today seem to have a decaying understanding of what a URL is, given how much web navigation we do through the omnibar or apps, particularly on mobile, but that's another issue). I've been archiving a few government sites to preserve them for web scraping exercises [0] (the Texas death penalty site is a classic, for both being relatively simple at first, and being incredibly convoluted depending on what level of detail you want to scrape [1])). But I imagine even government sites will move more toward AJAX/app-like sites, if the trend at the federal level means anything. That said, I think the analytics.usa.gov site is a great place to demonstrate the difference between server-generated HTML and client-rendered HTML. But as someone who just likes doing web-scraping, I feel the tools have mostly kept up with the changes to the web. It's been relatively easy, for example, to run Selenium through Python to mimic user action [2]. Same with PhantomJS through node, which has vastly improved how accurately it renders pages for screenshots compared to what I remember a few years back [0] https://github.com/wgetsnaps https://github.com/wgetsnaps [1] https://github.com/wgetsnaps/tdcj-state-tx-us--death_row https://github.com/wgetsnaps/tdcj-state-tx-us--death_row [2] https://gist.github.com/dannguyen/8a6fa49253c1d6a0eb92 https://gist.github.com/dannguyen/8a6fa49253c1d6a0eb92
  • niftich 10y ago
    It's unfortunate that nearly every webpage these days is a Javascript State Machine which you have to execute in a sandbox and inspect its internal state to get stuff out of. On a blog post by Paul Kinlan ('Open Web Advocate' at Google and Chromium) [1], I lamented that we ended up here instead of the semantic web because the semantic web was hard to execute. Instead, every web page is a black-box, only navigable by an intelligent and/or sufficiently persuadable human. But this is also why I don't buy ethical arguments against scraping. Sure, legally any company can unilaterally set any TOS prohibition against behavior they don't want, and these terms may be tested in court. But navigating a page in an automated manner that's designed to resemble interactions of humans (ie. through Selenium) is in my opinion ethical, because it merely time-shifts a user's activity. [1] https://news.ycombinator.com/item?id=12206846 https://news.ycombinator.com/item?id=12206846
• XCSme 10y ago
  Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails ", this just shows a lot of immaturity.
  • franciskim 10y ago
    Thanks for your feedback, I do appreciate it.
    • kentt 10y ago
      Not many people can take criticism is stride like that. You are awesome.
      • franciskim 10y ago
        :)
    • state 10y ago
      As a counterpoint: I think this article is fantastic. API restrictions are incredibly annoying when they pertain to what I consider to be my data. Should data and interface be so tightly joined? Of course not.
      • ProAm 10y ago
        What is your data? EDIT - Completely serious, you mean data you put on other peoples servers, using their services and expect them to let you have it back when and however you want it? Let's be serious here. You're lucky they let you access it at all.
  • deleted 10y ago
    [deleted]
• elchief 10y ago
  To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.
  • franciskim 10y ago
    Haven't come across those yet but yes I guess it could be painful.
  • deleted 10y ago
    [deleted]
  • elmigranto 10y ago
    > To fight scrapers, we show some values as images that look like text You are fighting screen readers more than anything; as well as legitimate plugins, form autofills, etc. If this is for captcha, you are fighting all the users as well. > And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. Legitimate browser plugins, etc. I'd just use electron or selenium with `nth-child`, `:visible`, `[class*="…"]`, etc. What you effectively doing is wasting time on useless stuff. This is even more useless than trying to prevent copying of DVDs or pirating games.
    • elchief 10y ago
      Except traffic from known scrapers (or what appear to be) is down 20% Sure, xpath and css selector experts can figure it out, but that's not everyone
      • comboy 10y ago
        I don't understand, why only 20%? If the traffic is from known scrapers why can't you just render "scrap off" ie easily get rid of them? And traffic from good scrapers is of course pretty much impossible to measure so you don't know how big percentage of scrapers you got rid of in total.
• KennyCason 10y ago
  As someone who does a lot of scraping, I was happy to learn about Antigate :)
  • KennyCason 10y ago
    Just joking as I don't scrape unless scraping is allowed. :)
• fake-name 10y ago
  I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for message passing middleware, SaltStack for automated VM provisioning, and python everywhere for everything else. Using some randomization, and a list of the top n user agents, I can randomly generate about ~800K unique but valid-looking UAs. Selenium+PhantomJS gets you through non-capcha cloudflare. Backing storage is Postgres. Database triggers do row versioning, and I wind up with what is basically a mini internet-archive of my own, with periodic snapshots of a site over time. Additionally, I have a readability-like processing layer that re-writes the page content in hopes of making the resulting layout actually pleasant to read on, with pluggable rulesets that determine page element decomposition. At this point, I have a system that is, as far as I can tell, definitionally a botnet. The only things is I actually pay for the hosts. --- Scaling something like this up to high volume is really an interesting challenge. My hosts are physically distributed, and just maintaining the RabbitMQ socket links is hard. I've actually had to do some hacking on the RabbitMQ library to let it handle the various ways I've seen a socket get wedged, and I still have some reliability issues in the SaltStack-DigitalOcean interface where VM creation gets stuck in a infinite loop, leading to me bleeding all my hosts. I also had to implement my own message fragmentation on top of RabbitMQ, because literally no AMQP library I found could reliably handle large (>100K) messages without eventually wedging. There are other fun problems too, like the fact that I have a postgres database that's ~700 GB in size, which means you have to spend time considering your DB design and doing query optimization too. I apparently have big data problems in my bedroom (My home servers are in my bedroom closet). --- It's all on github, FWIW: Manager: https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy Agent and salt scheduler: https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever
  • franciskim 10y ago
    take my money!
  • chmars 10y ago
• rch 10y ago
  There is so much that's missing from this. What about gathering tokens from customers vs. paying for social data feeds? How about canned services like 80legs?
  • franciskim 10y ago
    Hmm yeah there are a lot of other things that I could write about. 80legs seem like another Scrapy type of SaaS? Not sure what you mean about gathering tokens from customers.
    • rch 10y ago
      I've heard of companies that scrape on behalf of customers who will walk marketing people through the process of creating an API token to help mitigate rate limiting.
• zzzcpan 10y ago
  > But if you are automating your exact actions that happen via a browser, can this be blocked? Of course it can! You won't be able to defeat even the simplest attempt on anti-scraping based on statistical data. Like even keeping a list of individual rate-limits for /16 subnets of actual visiting users and you are in trouble.
• novacole 10y ago
  All of this could be accomplished with buildabot studio buildabot is.com
• novacole 10y ago
  All of this could be accomplished with buildabot studio buildabotnow.com
• stupidcar 10y ago
  I wrote a fairly complex spidering and scraping script in Node a few months ago. I found downcache[1] to be absolutely invaluable, particularly as I was debugging my parsing scripts, a I was able to rerun them relatively quickly over the cached responses. However, when the network was no longer a bottleneck, I found that the speed and single-threaded nature of Node became one. It wasn't really that slow, relatively speaking, but I had a few hundred gigs of HTML to chew through every time I made a correction, so it was important to keep the turnaround as fast as possible. I eventually managed to manually partition the task so I could launch separate Node scripts to handle different parts of it, but it wasn't a perfect split, and there was a fair bit of duplicated work, where a shared cache would have helped a great deal. In retrospect, I should have thrown my JS away and started again in something with easy threading like Java or C#. But -- familiar story -- I'd underestimated the complexity of the task to begin with, and by the time I understood, I'd sunk a lot of time into writing my JS parsing code and didn't fancy converting it all to another language, particular when it always seemed like "just one more" correction to the parsing would make everything work right. In the end, what was supposed to take a weekend took about three months of work, off and on, to finish. [1] https://www.npmjs.com/package/downcache https://www.npmjs.com/package/downcache
  • ralusek 10y ago
    Threading in node is very easy, just use clusters. Alternatively, take any of the CPU intensive activity, like parsing the HTML and formatting as JSON, and just put that on an AWS lambda. You can invoke as many lambdas from your application as you want in parallel and you're not going to be bottlenecked by your CPU :)
    • stupidcar 10y ago
      Clustering in Node creates isolated child processes, not threads. I needed to have shared queues, in-memory caches, and hashes to coordinate workers and avoid them doing duplicate work. I'm did consider using clustering and having some master process coordinate everything, and using some shared-memory caching library. But it would not be "easy" to set up, especially compared to something like Java where you get thread pools and synchronized thread-safe collections out of the box. And Lambda would have been totally impractical. As I said, I had hundred of gigs of data to process. If I'd been uploading this over my puny ADSL upstream every time, I'd still be waiting for a single run to complete. I'm not trashing Node. I like it. There's a reason I used in the first place, after all. But for this particular use-case, I didn't find it was very good fit.
• rezashirazian 10y ago
  When I was building liisted.com I scraped using Selenium and it worked great.
• loourr 10y ago
  Interesting that right after announcing this, Elon announced buying 65M in solar city bonds http://investors.solarcity.com/secfiling.cfm?filingid=1193125-16-688693&CIK=1408356 http://investors.solarcity.com/secfiling.cfm?filingid=119312...
• skeletonjelly 10y ago
  Hooray Melbourne! Would be interested seeing this at a meetup group if you were thinking of presenting.
  • danieltrembath 10y ago
    Another 3000'nder. Would be great to see this turned into a talk somewhere.
    • skeletonjelly 10y ago
      For sure. Trying to think which ones. Probably the MelbJS one and maybe dddmelb? You could modify it to talk at the OWASP one perhaps. Which ones have you been to?
• deleted 10y ago
  [deleted]
• IANAD 10y ago
  > But if you are automating your exact actions that happen via a browser, can this be blocked? Yes, by checking times between actions and number of actions in a time period, and blocking atypical activity. I was IP banned from a site once for a few months, after trying to scrape it too much and hitting links on the site that were hidden from humans. The random wait settings specified in the post are better than nothing, but still too flimsy. You would need to put hours between requests, only request during a certain 15 hour periods, take days off, and eventually you aren't scraping regularly enough to do much good. Scraping is not an API, and I should know- I used to do it for a living. Its unreliable. It requires constant maintenance. APIs can break too, but they are meant for the sort of consumption you are trying for. If you scrape for a living, only do it as a side job.
  • deejbee 10y ago
    Not scraping but banks don't even do this for their security which I found surprising. I just finished building a chrome extension (https://chrome.google.com/webstore/detail/uyp-free-blasts-through-m/kblhemffnhfhjianafepcclpakocicgg https://chrome.google.com/webstore/detail/uyp-free-blasts-th...) that auto-logins into pretty much any bank or financial web site without having to type anything. The key difference to other password managers is it can auto-fill pretty much anything. I guess it's part password manager (it stores passwords encrypted in browser storage, not remotely) and part automation wizard :)
  • red_blobs 10y ago
    It really depends on the data you are scraping. My main business relies on scraping and my data mining application has been running for over 5 years. If you have enough IP addresses available to you, it becomes almost impossible to distinguish it from normal users hitting the site...and bandwidth has gotten so cheap, the overhead is very affordable. I've noticed that most sites actually don't change that often. I deal with changes once or twice every 3 months. "If you scrape for a living, only do it as a side job." This is true if you are scraping the low hanging fruit. I scrape 40+ sources (I do have access to a few APIs as well) and then have to extract the patterns/data I need to then integrate it into my business model. This is all automatic now and I only work on upgrading for speed and efficiency. If you have to scan millions of urls daily from 1 site, it's probably not going to work out. You need to figure out clever ways of getting the data and using it without breaking any laws or pissing off the site owner.
• headmelted 10y ago
  I actually love Selenium for this purpose, for much the same reasons the author mentions here. It's almost impossible for a website to reliably detect that a client web browser is being automated, and I find I can make Selenium scripts much more adaptable to breaking changes in websites when they occur than I can when hooking up my code directly. I actually disagree with the contention that Selenium is slower than directly scraping though. The Firefox driver has always been lightning fast for me and the bottleneck is almost always server requests that would have been necessary either way.
• ge95 10y ago
  How do you push a button like hit next on a paginated page?
  • oli5679 10y ago
    Right click on the 'next' button in chrome and use 'inspect element' to find its id/class/css selector and then: browser.findElement(webdriverio.By.id('#Next')).click();
• kingkool68 10y ago
  It's trivial to scrape public Instagram URLs... https://github.com/kingkool68/zadieheimlich/blob/master/functions/instagram.php#L421-L428 https://github.com/kingkool68/zadieheimlich/blob/master/func...
• pault 10y ago
  I don't know why more people don't use chrome extensions for scraping. Using a boilerplate[1], you can get a scraper up and running in minutes. Start a node server that serves up urls and stores parsed data, and run the scraper in the browser. Best of all, you can watch it running and debug if something goes wrong. I know it doesn't scale well if you're running a SaaS, but for personal projects and research/data normalization it's the lowest barrier to entry, in my opinion. [1] http://extensionizr.com http://extensionizr.com
• nreece 10y ago
  At Feedity (https://feedity.com https://feedity.com), we "index" webpages to generate custom feeds. Over the years, we've designed our system to use a mix of technologies like .NET (C#) and node.js, and implemented a bunch of tweaks and optimizations for seamless & scalable access to public content.
  • deleted 10y ago
    [deleted]
  • LunaSea 10y ago
    Any tips and tricks you are able to share about the technologies you guys developed? It would be especially interesting to see what you use for text extraction from HTML.
• mack73 10y ago
  Corporations will abuse your personal integrity whenever they get a chance, while abiding the law. Corporations will cry like babies when their publicly available data (their livelyhood) gets scraped. They will take you to court. They consider their data to be theirs, even though they published it on the internet. They consider your data (your personal integrity) to be theirs as well, because how can you assume personal integrity when you are surfing the internet? I have high hopes that the judicial system some time not too far from now will realize that since the law should be a reflection of the current moral standings it will always be behind, trying to catch up with us and that those who break the law while not breaking the current moral standings are still "good citizens" unworthy of prison or fines. I guess Google won this iteration of the internet because of the double-standars site owners stand by, to allow Google to scrape anything while hindering any competitors from doing the same. There will only be a true competitor to Google when we in the next iteration of the internet realize that searching vast amounts of data (the internet) is a solved problem, that anyone can do as good a job as Google, and move on to the next quirk, around wich there will be competition, and in the end that quirk will be solved, we'll have a winner, signaling that is it time to move on to the next iteration.
  • nerdponx 10y ago
    The difference is that Google didn't agree to not scrape your data. You, as per their TOS, agreed not to scrape theirs, as part of the condition of using their service.
    • Bootvis 10y ago
      Which TOS? I might have accepted terms when I created a Google Account but in no way do I agree to a TOS by visiting a URL.
      • jacquesm 10y ago
        Try using google from a fresh install, they´ll force you to accept their TOS.
• kevindeasis 10y ago
  Does cheerio account for single page apps? In any case thanks for the tutorial! Anyways I added your stuff here along with other data mining resource: https://github.com/kevindeasis/awesome-fullstack#web-scraping https://github.com/kevindeasis/awesome-fullstack#web-scrapin...
• lamby 10y ago
  Whilst they mean well, I find this a fundamentally deceptive — the arduous parts of "real world" scraping simply aren't in the parsing and extraction of data from the target page, the typical focus of these "scrape the web with X" articles. The difficulties are invariably in "post-processing"; working around incomplete data on the page, handling errors gracefully and retrying in some (but not all) situations, keeping on top of layout/URL/data changes to the target site, not hitting your target site too often, logging into the target site if necessary and rotating credentials and IP addresses, respecting robots.txt, target site being utterly braindead, keeping users meaningfully informed of scraping progress if they are waiting of it, target site adding and removing data resulting in a null-leaning database schema, sane parallelisation in the presence of prioritisation of important requests, difficulties in monitoring a scraping system due to its implicitly non-deterministic nature, and general problems associated with long-running background processes in web stacks. Et cetera. In other words, extracting the right text on the page is the easiest and trivial part by far, with little practical difference between an admittedly cute jQuery-esque parsing library or even just using a blunt regular expression. It would be quixotic to simply retort that sites should provide "proper" APIs but I would love to see more attempts at solutions that go beyond the superficial.
  • dismantlethesun 10y ago
    > the arduous parts of "real world" scraping simply aren't in the parsing and extraction of data from the target page, the typical focus of these "scrape the web with X" articles. I can agree with this after having written a scraper as part of core business functionality (we paid a company for access, but access was just to bare HTML blobs and CVS and not an actual API). However, to what degree you want to do all this is negotiable whereas the 'core' of screen-scraping is not---all scrapers have to first figure out how to get text, parse it, then stick it back in their system. An example of what I mean when I say 'negotiable' is.... > working around incomplete data on the page Deciding how to do this depends on your problem domain. Sometimes, we'd get bad computed data from our source but not care because it just meant more work putting more work in calculating it from a more raw source. > not hitting your target site too often If they publish how often you are allowed to scrape, this isn't too difficult. If not, then trial and error is the only solution. On occasion, a site simply just doesn't know/care. For example, in my case, the site was static content behind a CDN, so that if we were anywhere under 200 req/second then no flags would ever be raised. For most smaller sites, that you are unofficially scraping, you may be limited to 1 request every 2 seconds.
• Twisell 10y ago
  What bother me the most is that recently I wanted to extract and archive of all the threads I participated in from an Internet forum. The webmaster told me that the BBS he use don't provide such a function and that I just had to download each thread manually... (300+ thread in my case). He then say that it don't bother him if I scrape theses thread. And I'm currently figuring out how to manage his site's cookie protected search feature, so that my painstaking effort (I'm not a dev, more a DB guy) could be reproducible more easily by other users of this service. But this shouldn't appen in the first place because all post of this service are stored in a cleanly organized MySQl DB. Yet as no method is provided the only way to get back structured data is by scrapping (as the webmaster told me that no, he won't run custom SQL because "he don't want to mess his DB"). So even if all the data is publicly available through the internet forum only a geek can download a personal archive... or google because google scrape and store everything.
  • CalRobert 10y ago
    It's overkill for most things, but I have found that on occasion the best way to scrape stuff behind annoying frontends is with Selenium. pysaunter is a useful library that's one layer of abstraction higher, if you're familiar with Python.
    • CalRobert 10y ago
      Well I see now that I'm really late to the party with that comment.
• xcopy 10y ago
  Like
• etatoby 10y ago
  Does anybody know what the author means by "lead" (noun)? I don't think it's any of the regular meanings: http://www.ldoceonline.com/search/?q=Lead http://www.ldoceonline.com/search/?q=Lead But it doesn't seem to be any of these slang terms either: http://www.urbandictionary.com/define.php?term=lead http://www.urbandictionary.com/define.php?term=lead
  • stedaniels 10y ago
    https://en.wikipedia.org/wiki/Lead_generation https://en.wikipedia.org/wiki/Lead_generation
• frostymarvelous 10y ago
  While everyone is busy debating whether scraping is bad or legal, I just can't stop thinking a out Antigate. Of the sweatshops that must have been setup to deliver this service. That, is to me the true horror of this story.
• unixhero 10y ago
  And from the trenches: - rails application - scraping with nokogiri gem on Ruby - simple models doing the scraping in rails app - some scraping is parsed with CSS selectors - nokogiri - some scraping is parsed with regex - nokogiri - persisting to DB, Text, even Google docs - presentation on web, text, pdf, xls Boom