8 ms·

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools l

Really interesting use of data extraction.... For developers and managers out there, do you prefer to build your own in-house scrapers or use Scrapy or tools like Mozenda instead? What about import.io and kimono? I'm asking because lot of developers seem to be adamant against using web scraping tools they didn't develop themselves. Which seems counter productive because you are going into technical debt for an already solved problem. So developers, what is the perfect web scraping tool you envision? And it's always a fine balance between people who want to scrape Linkedin to spam people, others looking to do good with the data they scrape, and website owners who get aggressive and threatening when they realize they are getting scraped. It seems like web scraping is a really shitty business to be in and nobody really wants to pay for it.

austinhutch 11y ago
After Kimono got shut down, I think a self-hosted open source version would be extremely popular. I want to build my own solution, but the API functionality and pagination / AJAX loaded data would be too difficult.
- jorgecurio 11y ago
  interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade? Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data. Meanwhile I see Mozenda charging like 5 cents for each page extracted, do you think this is a fair model or does it not matter?
  - toomuchtodo 11y ago
    Would need to Kickstart or Patreon it.
  - unsettledtck 11y ago
    So for Scrapy and Portia, they are both free as in beer, specifically because we believe in the power of open source. Scrapy actually precedes Scrapinghub and was certainly not developed as a marketing tool. Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans), datasets, and for us to handle complex crawls for companies outsourcing to us. Our model is that there is something for everyone whether you are looking to dip your toes into web scraping (free), use it occasionally (usually journalists) or dependent on web crawling for your business.
    - vram22 11y ago
      >Scrapy actually precedes Scrapinghub Right. I had first come across Scrapy, while browsing the web for Python software tools, some years ago, on the site of a company in Uruguay called Insophia. It was in the list of products developed by them, and that they worked on. Scrapinghub came later.
  - malka 11y ago
    by proposing paid hosting and support for companies that don't want the burden to manage it themselves ? There could be some additional features with the paid version also
    - jorgecurio 11y ago
      I'm wondering about this but how realistic do you expect someone to pay when it's already free? I would imagine only large enterprise users so essentially you are supporting free users by charging enterprise users that may pay you for support. Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.
      - predius 11y ago
        We don't expect anyone to pay for Scrapy or Portia! We provide the best Platform (as a Service) to run Scrapy or Portia spiders, and will soon be supporting most standard web scraping technologies. This is free for light users, but we charge for people who need extra or dedicated computing or network resources. We also provide help to startups or enterprise orgs looking to get help in building a web data harvesting system (more than just parsing pages!), either by building it ourselves or by helping our partners train their engineers in using our technologies. This has worked so far, and we're very healthy from a revenue perspective – more than doubling every year for a few years now, and good enough to grow to become the largest fully distributed company outside of the US. We're pretty happy with being a brand that gives to the community, it tends to get repaid 10x in the long run.
        jorgecurio 11y ago
        how much are you generating in terms of revenue? are you venture funded?
- unsettledtck 11y ago
  Not sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-projects-to-portia/ https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-... Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.
iamdave 11y ago
It doesn't scale very well, unless you have a lot of patience...but I've had immense success using the importxml() function in Google sheets to compile raw election data while I did some freelance work for the Texas Libertarian party a couple of years ago. Outside of that, I did often find myself building my own tools with a combination of ruby, nokogiri and mechanize. Partly out of a desire to learn something new, and partly many of my use scenarios didn't require anything more complex than "Go to these pages, get the data within these elements and throw a CSV file over there".
unsettledtck 11y ago
Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code. Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The publicized perception of web scraping is fairly negative, but doesn't take into account the benefits of data used in machine-learning or democratized data extraction (as in the case of this article or for building public service apps like transportation notifications), or the simple realities of competitive pricing and monitoring the activities of resellers. Researchers, academics, data scientists, marketers, the list goes on for those who use web scraping daily. Glad you enjoyed the article! I'm hoping that more examples of ethical data extraction will start to turn the tide of public perception.
- cookiecaper 11y ago
  Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost millions of dollars to successfully defend a CFAA suit.
  - deleted 11y ago
    [deleted]
  - predius 11y ago
    Hi! This is definitely not legal advice, so consult with a lawyer and do your own research if you are thinking of applying this to your own practices. I work for Scrapinghub as well and try to understand the law around this. I can help with some pointers to why I think some web scraping isn't illegal… there are of courses some limits to this. When the data scraped is "is publicly available on the Internet, without requiring any login, password, or other individualized grant of access", the Eastern District Court of Virginia in Cvent vs Eventbrite (https://casetext.com/case/cvent-inc-v-eventbrite https://casetext.com/case/cvent-inc-v-eventbrite) ruled one could not be deemed to be exceeding unauthorized access. There are two ways, that I know of, that courts have ruled you can exceed your authorization: - When the site owner has contacted you and removed your authorization in a written manner, as happened on Craigslist vs 3taps. - By accepting the terms of service and agreeing against scraping. You have to do this through a "clickwrap" ToS, rather than a "browsewrap". You can read about the differences here: https://termsfeed.com/blog/browsewrap-clickwrap/ https://termsfeed.com/blog/browsewrap-clickwrap/ As a matter of policy, we don't scrape any site with a ToS with clear anti-scraping language and which forces us to create an account or "constructively agree" as part of the use of the site. Any user wishing to revoke authorization for anyone using our platform can make an abuse report on our site– we tend to handle these within 24 hours and haven't had a single claim go further than this stage, as we aim to be reasonable and look for a way to provide value to both sides.
emluque 11y ago
I recently did a website, that mines Argentinian Central Bank statistics daily and generates graphics and reports: http://estadisticasbcra.com/en http://estadisticasbcra.com/en ( The data that I'm mining is published here: http://www.bcra.gov.ar/Estadisticas/estprv010000.asp http://www.bcra.gov.ar/Estadisticas/estprv010000.asp ) On this case, some scripts using Beautiful Soup were enough to get the job done, but I was completely unaware of Scrapy, seems like a fantastic tool, if I would have known about it I probably would have used it.
logn 11y ago
> an already solved problem It's a hard problem to generalize. > balance between people who want to scrape Linkedin to spam people, others looking to do good with the data they scrape, and website owners who get aggressive and threatening when they realize they are getting scraped Agreed. No one wants to be the bad guy and most clients looking to spam people are awful clients to have anyhow. Btw scraping LinkedIn is fairly difficult/expensive and they like to sue people.
predius 11y ago
I agree, this is definitely a solved problem! If you need to build a solid web scraping stack which is going to be maintained by many people and is critical to your business, you have two options… to use Scrapy or to build something yourself. Scrapy has been tried and tested over 6-7 years of community development, as well as being the base infrastructure for a number of >$1B businesses. Not only this, but there is a suite of tools which you have been built around it – Portia for one, but also other lots of useful open source libraries: http://scrapinghub.com/opensource/ http://scrapinghub.com/opensource/). Right now most people still have the issue of having to use xpath or css selectors to run your crawl or get the data, but not for too long. There's more and more ways of skipping this step and getting at data automatically: https://github.com/redapple/parslepy/wiki/Use-parslepy-with-scrapy https://github.com/redapple/parslepy/wiki/Use-parslepy-with-... https://speakerdeck.com/amontalenti/web-crawling-and-metadata-extraction-in-python?slide=28 https://speakerdeck.com/amontalenti/web-crawling-and-metadat... https://github.com/scrapy/loginform https://github.com/scrapy/loginform https://github.com/TeamHG-Memex/Formasaurus https://github.com/TeamHG-Memex/Formasaurus https://github.com/scrapy/scrapely https://github.com/scrapy/scrapely https://github.com/scrapinghub/webpager https://github.com/scrapinghub/webpager https://moz.com/devblog/benchmarking-python-content-extraction-algorithms-dragnet-readability-goose-and-eatiht/ https://moz.com/devblog/benchmarking-python-content-extracti... Scrapy (and also lots of python tools, likely a majority of them created by people using it and BeautifulSoup) have lowered the cost of building web data harvesting systems to the point where one guy can build crawlers for an entire industry in a couple of months.

austinhutch 11y ago

After Kimono got shut down, I think a self-hosted open source version would be extremely popular. I want to build my own solution, but the API functionality and pagination / AJAX loaded data would be too difficult.

jorgecurio 11y ago
interesting, how would a self-hosted open source version make money tho in order to support itself and continue to upgrade? Is this even a realistic business model? Seems like this is what Scrapy is doing and what Import.io is doing. Make the tool free in order to get free marketing and then charge people willing to pay money to extract data. Meanwhile I see Mozenda charging like 5 cents for each page extracted, do you think this is a fair model or does it not matter?
- toomuchtodo 11y ago
  Would need to Kickstart or Patreon it.
- unsettledtck 11y ago
  So for Scrapy and Portia, they are both free as in beer, specifically because we believe in the power of open source. Scrapy actually precedes Scrapinghub and was certainly not developed as a marketing tool. Charges come with large scale crawls (above certain limits on our platform), additional products like Crawlera (our smart downloader that routes requests from a crawl through a pool of IP addresses to avoid bans), datasets, and for us to handle complex crawls for companies outsourcing to us. Our model is that there is something for everyone whether you are looking to dip your toes into web scraping (free), use it occasionally (usually journalists) or dependent on web crawling for your business.
  - vram22 11y ago
    >Scrapy actually precedes Scrapinghub Right. I had first come across Scrapy, while browsing the web for Python software tools, some years ago, on the site of a company in Uruguay called Insophia. It was in the list of products developed by them, and that they worked on. Scrapinghub came later.
- malka 11y ago
  by proposing paid hosting and support for companies that don't want the burden to manage it themselves ? There could be some additional features with the paid version also
  - jorgecurio 11y ago
    I'm wondering about this but how realistic do you expect someone to pay when it's already free? I would imagine only large enterprise users so essentially you are supporting free users by charging enterprise users that may pay you for support. Horrible industry imho when you have to give away things for free just to be competitive. I just don't get why people would expect software to be free.
    - predius 11y ago
      We don't expect anyone to pay for Scrapy or Portia! We provide the best Platform (as a Service) to run Scrapy or Portia spiders, and will soon be supporting most standard web scraping technologies. This is free for light users, but we charge for people who need extra or dedicated computing or network resources. We also provide help to startups or enterprise orgs looking to get help in building a web data harvesting system (more than just parsing pages!), either by building it ourselves or by helping our partners train their engineers in using our technologies. This has worked so far, and we're very healthy from a revenue perspective – more than doubling every year for a few years now, and good enough to grow to become the largest fully distributed company outside of the US. We're pretty happy with being a brand that gives to the community, it tends to get repaid 10x in the long run.
      - jorgecurio 11y ago
        how much are you generating in terms of revenue? are you venture funded?
unsettledtck 11y ago
Not sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-projects-to-portia/ https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-... Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.

iamdave 11y ago

It doesn't scale very well, unless you have a lot of patience...but I've had immense success using the importxml() function in Google sheets to compile raw election data while I did some freelance work for the Texas Libertarian party a couple of years ago. Outside of that, I did often find myself building my own tools with a combination of ruby, nokogiri and mechanize. Partly out of a desire to learn something new, and partly many of my use scenarios didn't require anything more complex than "Go to these pages, get the data within these elements and throw a CSV file over there".

unsettledtck 11y ago

Full disclosure, I work for Scrapinghub. Our tools are Scrapy and Portia, both open source and both free as in beer. Scrapy is for those who want fine-tuned manual control and who have a background in Python. Portia is the visual web scraper for those who are non-technical to technical but don't want to bother with code. Web scraping is everywhere, even if it's not necessarily spoken openly about or acknowledged. The publicized perception of web scraping is fairly negative, but doesn't take into account the benefits of data used in machine-learning or democratized data extraction (as in the case of this article or for building public service apps like transportation notifications), or the simple realities of competitive pricing and monitoring the activities of resellers. Researchers, academics, data scientists, marketers, the list goes on for those who use web scraping daily. Glad you enjoyed the article! I'm hoping that more examples of ethical data extraction will start to turn the tide of public perception.

cookiecaper 11y ago
Every time I see a scrapinghub post I ask the same question: what's your strategy for dealing with CFAA suits that arise from use of your platform? Most web scraping is illegal in the United States. I completely accept how important scraping is as a data source, but that doesn't make it any more legal. It's in a space right now where only big companies can take unmitigated advantage of the tool, because it'd cost millions of dollars to successfully defend a CFAA suit.
- deleted 11y ago
  [deleted]
- predius 11y ago
  Hi! This is definitely not legal advice, so consult with a lawyer and do your own research if you are thinking of applying this to your own practices. I work for Scrapinghub as well and try to understand the law around this. I can help with some pointers to why I think some web scraping isn't illegal… there are of courses some limits to this. When the data scraped is "is publicly available on the Internet, without requiring any login, password, or other individualized grant of access", the Eastern District Court of Virginia in Cvent vs Eventbrite (https://casetext.com/case/cvent-inc-v-eventbrite https://casetext.com/case/cvent-inc-v-eventbrite) ruled one could not be deemed to be exceeding unauthorized access. There are two ways, that I know of, that courts have ruled you can exceed your authorization: - When the site owner has contacted you and removed your authorization in a written manner, as happened on Craigslist vs 3taps. - By accepting the terms of service and agreeing against scraping. You have to do this through a "clickwrap" ToS, rather than a "browsewrap". You can read about the differences here: https://termsfeed.com/blog/browsewrap-clickwrap/ https://termsfeed.com/blog/browsewrap-clickwrap/ As a matter of policy, we don't scrape any site with a ToS with clear anti-scraping language and which forces us to create an account or "constructively agree" as part of the use of the site. Any user wishing to revoke authorization for anyone using our platform can make an abuse report on our site– we tend to handle these within 24 hours and haven't had a single claim go further than this stage, as we aim to be reasonable and look for a way to provide value to both sides.

emluque 11y ago

I recently did a website, that mines Argentinian Central Bank statistics daily and generates graphics and reports: http://estadisticasbcra.com/en http://estadisticasbcra.com/en ( The data that I'm mining is published here: http://www.bcra.gov.ar/Estadisticas/estprv010000.asp http://www.bcra.gov.ar/Estadisticas/estprv010000.asp ) On this case, some scripts using Beautiful Soup were enough to get the job done, but I was completely unaware of Scrapy, seems like a fantastic tool, if I would have known about it I probably would have used it.

logn 11y ago

> an already solved problem It's a hard problem to generalize. > balance between people who want to scrape Linkedin to spam people, others looking to do good with the data they scrape, and website owners who get aggressive and threatening when they realize they are getting scraped Agreed. No one wants to be the bad guy and most clients looking to spam people are awful clients to have anyhow. Btw scraping LinkedIn is fairly difficult/expensive and they like to sue people.

predius 11y ago

I agree, this is definitely a solved problem! If you need to build a solid web scraping stack which is going to be maintained by many people and is critical to your business, you have two options… to use Scrapy or to build something yourself. Scrapy has been tried and tested over 6-7 years of community development, as well as being the base infrastructure for a number of >$1B businesses. Not only this, but there is a suite of tools which you have been built around it – Portia for one, but also other lots of useful open source libraries: http://scrapinghub.com/opensource/ http://scrapinghub.com/opensource/). Right now most people still have the issue of having to use xpath or css selectors to run your crawl or get the data, but not for too long. There's more and more ways of skipping this step and getting at data automatically: https://github.com/redapple/parslepy/wiki/Use-parslepy-with-scrapy https://github.com/redapple/parslepy/wiki/Use-parslepy-with-... https://speakerdeck.com/amontalenti/web-crawling-and-metadata-extraction-in-python?slide=28 https://speakerdeck.com/amontalenti/web-crawling-and-metadat... https://github.com/scrapy/loginform https://github.com/scrapy/loginform https://github.com/TeamHG-Memex/Formasaurus https://github.com/TeamHG-Memex/Formasaurus https://github.com/scrapy/scrapely https://github.com/scrapy/scrapely https://github.com/scrapinghub/webpager https://github.com/scrapinghub/webpager https://moz.com/devblog/benchmarking-python-content-extraction-algorithms-dragnet-readability-goose-and-eatiht/ https://moz.com/devblog/benchmarking-python-content-extracti... Scrapy (and also lots of python tools, likely a majority of them created by people using it and BeautifulSoup) have lowered the cost of building web data harvesting systems to the point where one guy can build crawlers for an entire industry in a couple of months.