13 ms·

The State of Web Scraping 2022

291 points by Ian_Kerins 5y ago

joe_91 5y ago
I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.
- Ian_Kerins 5y ago
  This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/ https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playw...
- emptysea 5y ago
  Is the “pretty optimized headless browser” an off the shelf thing, or something custom? Are you using playwright/puppeteer to drive it?
  - mycall 5y ago
    Headless Chrome [0] and alpine-Chrome [1] are pretty popular. Some variations also include V2Ray, Shadowsocks and other VPNs. [0] https://hub.docker.com/r/justinribeiro/chrome-headless/ https://hub.docker.com/r/justinribeiro/chrome-headless/ [1] https://github.com/Zenika/alpine-chrome https://github.com/Zenika/alpine-chrome
  - rozenmd 5y ago
    There are plugins for Puppeteer: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth https://github.com/berstend/puppeteer-extra/tree/master/pack...
- nanna 5y ago
  I'm finding that Cloudflare is even blocking my RSS reader from requesting feeds behind their service. It's not even just scrapers at this point.
- temp8964 5y ago
  I think it will eventually goes to like stock trading. If you have a good strategy, you don't want to share with the world, because it will render your strategy useless.
- nsonha 5y ago
  > optimised headless browser with good proxies instead are you saying you only had problem because you didn't use headless browser before and now with both headless and proxy it generally suffices to not be seen as scrapper?
- valar_m 5y ago
  Do you have any recommendations for the "good proxies" you mentioned?
mellosouls 5y ago
With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should. It would be better I think if there was a treatment of the ethics here, rather than a seemingly "ra-ra go bots" attitude, as though the only consideration is commercial.
- bryanrasmussen 5y ago
  this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.
- Ian_Kerins 5y ago
  100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time). - If you can get cached data from somewhere that works, then use that. Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need. The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
  - travisporter 5y ago
    Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”
    - RobSm 5y ago
Ian_Kerins 5y ago
If anyone has anything else they think was missed or should be included then let me know!
newsbinator 5y ago
Like most here, I am very good at web scraping and automated form fills. I keep trying to figure out a profitable side project or business idea to make out of it and keep coming up with nothing that works. Any good ideas?
- Fantosism 5y ago
  I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.
  - mschuster91 5y ago
    I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers?!
    - pjc50 5y ago
      Same as NFTs: hype and resale value. At least you can wear the sneakers once you've stopped flipping them.
    - nichch 5y ago
      Economic value and actual usage.
      - mschuster91 5y ago
        What economic value (other than hoping for the value to increase, aka tulip mania) can I derive out of ultra-rare sneakers? What usage that goes above "it looks cooler than an unbranded, otherwise identical sneaker"? For me, this kind of product is part of the "bullshit economy" - similar to "bullshit jobs", this kind of product has no reason to exist other than vanity, as almost all of these "collectibles" won't ever be used. We are using up valuable, finite resources to create and distribute this kind of useless "bullshit product", we are using up valuable human time and IT resources on developing websites capable to resist (D)DoS attacks and on developing snipers to bypass the anti-bot technologies employed by the shops, and we are creating a lot of demand for all kinds of sneaker-related crime - and there's a lot of that: theft and robberies from stores, theft and robberies in the supply chain, ebay/classifieds scams, credit card fraud, robberies on broad daylight [1]. Seriously, fuck all that shit. No one needs hundreds of dollars worth of sneakers that only incentivize crime and bullshit. [1]: https://www.google.com/search?q=man+robbed+because+of+sneaker https://www.google.com/search?q=man+robbed+because+of+sneake...
blantonl 5y ago
I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opinion of it is that those that are doing it to my platforms are doing so solely to steal data and build businesses on top of other's hard work.
- KieranMac 5y ago
  There are pro-social and anti-social uses of web scraping. If you have ever used Kayak or any other price discovery or price comparison website, you've relied on web scraping to provide you a service.
  - zffr 5y ago
    Also google or any other search engine
  - tyingq 5y ago
    I believe Kayak has agreements with the sites they scrape though. So it's a different type of "scraping", really.
- usbqk 5y ago
  I think if you make intangible things public you shouldn’t consider them to be only yours anymore.
- xrendan 5y ago
  I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remittance process: ethical - Scraping faces to build a surveillance-tech company: disgusting - Scraping your own website because your internal processes are so broken you can't get it any other way: ethical - Scraping to just copy someone's data they worked hard to generate to go and resell: unethical
KieranMac 5y ago
As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespass to chattels. In my opinion, 2021 was a bad year for the law as it relates to web scraping. The Supreme Court remanded hiQ Labs, and many high-profile lower-court cases ended badly for web scrapers. It's a darker shade of gray than it was in 2020. It can be navigated, but it's tricky.
- Ian_Kerins 5y ago
  Interesting!...I'm not a lawyer, so the content for this piece was based on commentary in the below article. Was written by their lawyer, but would love to hear your counter point to it. Always good to get multiple viewpoints on something. https://www.zyte.com/blog/van-buren-a-victory-for-web-scrapers/ https://www.zyte.com/blog/van-buren-a-victory-for-web-scrape...
  - KieranMac 5y ago
    The Zyte article isn't inaccurate; it's just a simplified assessment of a complicated issue. If you'd like a more nuanced perspective on this, please read my guest post of Prof. Goldman's blog. https://blog.ericgoldman.org/archives/2021/06/more-perspectives-about-van-buren-v-us-guest-blog-post.htm https://blog.ericgoldman.org/archives/2021/06/more-perspecti...
- faizshah 5y ago
  Is there a good blog or something that tracks these cases?
  - KieranMac 5y ago
    Prof. Eric Goldman's blog is probably the #1 site historically on scraping and the law. I've contributed to it a few times. https://blog.ericgoldman.org/archives/2021/06/more-perspectives-about-van-buren-v-us-guest-blog-post.htm https://blog.ericgoldman.org/archives/2021/06/more-perspecti... The name of my firm is McCarthy Garber Law. I write about scraping there when I have time (which I rarely do)!
ok_coo 5y ago
Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/ https://commoncrawl.org/
- mycall 5y ago
  I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org
  - wumpus 5y ago
    web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit https://github.com/cocrawler/cdx_toolkit
    - kevinsundar 5y ago
      Hey! I was using your tool a couple months ago. It was super helpful for my project.
      - wumpus 5y ago
        Thanks! I rarely hear from users, great to hear from you!
  - kevinsundar 5y ago
    They do and its better than common crawl's by my testing.
- dewey 5y ago
  I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.
NDizzle 5y ago
I still have a daily job running a web scraper I first wrote with Scrapy back in 2017. I think I've had to update it 3 times over the years for changes to the site and web standards. Good old government sites - rarely change!
cblconfederate 5y ago
Cloudflare's blocks get in the way of many websites who are simply trying to get a "link preview" of the page, even if it is only a single request from a new IP. I wish they would offer some kind of alternative for the pages they serve instead of a captcha block.
bobblywobbles 5y ago
Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.
- tommek4077 5y ago
  Because those terms are the law and cant be ignored in almost all the rest of the world...
- akersten 5y ago
  > many terms of service prohibit interacting with their website in an automated fashion, Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data!), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable, or do I have your permission to let Chrome automate that for me?
  - RobSm 5y ago
    This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.
    - nsonha 5y ago
      are you a lawyer? Your opinion doesn't really mean anything if you still lose the case at the end. By your logic there isn't a clear way to define DDoS either. Sounds like there is though?
      - Chris2048 5y ago
fareesh 5y ago
My toolbox of choice for web scraping is either Nokogiri or puppeteer Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?
- edmundsauto 5y ago
  One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.
  - fareesh 5y ago
    Ah interesting, I end up doing this manually, i.e. File.write followed by what I want to scrape
    - edmundsauto 5y ago
      I believe scrapy has somewhat intelligent cache control options - maybe it could be recreated in a few dozens of lines of code, maybe a few hundred. But there are a huge number of these types of features - it’s basically a Swiss Army knife. Examples include rotating proxies, rotating user agent headers. Hooks to add in middleware for processing pipelines. CLI switches to change your data output format. Nice debugging and logging. Other large scale features include distributed crawlers. Scheduling. Monitoring UI so you can see progress via a web UI. It’s what I reach for first, because you can be up and running with your first scraper in an hour. By hand, that’s maybe 10 minutes - but if you want to iterate, and your first scraper is a v1 rather than final effort… i think it’s definitely worth it.
coverj 5y ago
I have been interested in web scraping lately but never really dived too deep. Did anyone have more indepth resources (github projects, blogs, forums, etc) than the tutorials that are basically install beautiful soup and get data from a tag?
- JimBlackwood 5y ago
  Genuine question but, what more do you need?
JJxFile 5y ago
The web scraping ecosystem is growing, with more libraries, frameworks and products available than ever before to simplify our web scraping headaches so the future is looking bright.
gmanis 5y ago
What does HN think of web scraping for the purpose of price comparison? I’m asking this because I run a small side project to show prices across retailers for a very small niche. The users are very very happy. Even the vendors started contacting to be listed on the comparison. But I am unable to make a business out of it other than few affiliate commission.
- magixx 5y ago
  I worked for a company that did exactly this many years ago. (They were even able to parter with some retailer). Their product worked well yet they still went out of business long ago. To be honest, I don't see much value in such a service, not that it doesn't exist, it's just hard to justify paying for this data.
darepublic 5y ago
Separate from web scraping, there is the use of automation to perform normal allowable user actions on the site. That should be considered distinct from large scale data extraction no
slvrspoon 5y ago
for those in this thread with super-serious experience scraping and automating at scale, looking for work (ethical!) please contact me directly.

291 points by Ian_Kerins 5y ago

I'm scraping about 30 sites for work at the moment, but have a few that are using Cloudflare which has been a b*tch to deal with. Tried numerous libraries and different proxy providers, but reliability is patchy. Previous fixes like https://github.com/Anorov/cloudflare-scrape https://github.com/Anorov/cloudflare-scrape don't seem to work anymore after Cloudflare updates, so I've switched to using a pretty optimised headless browser with good proxies instead.

Ian_Kerins 5y ago
This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-scraping/ https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playw...
emptysea 5y ago
Is the “pretty optimized headless browser” an off the shelf thing, or something custom? Are you using playwright/puppeteer to drive it?
- mycall 5y ago
  Headless Chrome [0] and alpine-Chrome [1] are pretty popular. Some variations also include V2Ray, Shadowsocks and other VPNs. [0] https://hub.docker.com/r/justinribeiro/chrome-headless/ https://hub.docker.com/r/justinribeiro/chrome-headless/ [1] https://github.com/Zenika/alpine-chrome https://github.com/Zenika/alpine-chrome
- rozenmd 5y ago
  There are plugins for Puppeteer: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth https://github.com/berstend/puppeteer-extra/tree/master/pack...
nanna 5y ago
I'm finding that Cloudflare is even blocking my RSS reader from requesting feeds behind their service. It's not even just scrapers at this point.
temp8964 5y ago
I think it will eventually goes to like stock trading. If you have a good strategy, you don't want to share with the world, because it will render your strategy useless.
nsonha 5y ago
> optimised headless browser with good proxies instead are you saying you only had problem because you didn't use headless browser before and now with both headless and proxy it generally suffices to not be seen as scrapper?
valar_m 5y ago
Do you have any recommendations for the "good proxies" you mentioned?

mellosouls 5y ago

With the right combination of proxies, user agents and browsers, you can scrape every website. Even those that seem unscrapable. : This outcome was great news for web scrapers, as it means that so long as a websites has made their data public you are not in violation of the CFAA when you scrape the data even if it is prohibited in some other way (T&Cs, robots.txt, etc). Just because you can, doesn't mean you should. It would be better I think if there was a treatment of the ethics here, rather than a seemingly "ra-ra go bots" attitude, as though the only consideration is commercial.

bryanrasmussen 5y ago
this sort of implies that the 'ethics' would end up meaning that you shouldn't scrape if it is not wanted, although I suppose there can be ethics or other than commercial requirements that mean that you should.
Ian_Kerins 5y ago
100% agree, when scraping it should always be done respectfully. - If they provide a API, then use it. - Don't slam a website, ideally spread it out over hours of the day when there target audience is least active (night time). - If you can get cached data from somewhere that works, then use that. Most developers are respectful and only scrape what they really need, not only from an ethical point of view but also a cost and resources point of view. Scraping data is resource intensive and proxy costs can quickly rise to $1,000-$10,000 per month. So most only scrape the minimum they need. The other thing here as well, is that a lot of the most popular sites being scraped, are also massive scrapers themselves. The big ecommerce sites are being scraped, but they are also scraping their competitors too.
- travisporter 5y ago
  Don’t get my home address, name, family members names, salary, cell phone number, aggregate and sell them and claim “it’s all publically available anyway”
  - RobSm 5y ago

Ian_Kerins 5y ago

If anyone has anything else they think was missed or should be included then let me know!

newsbinator 5y ago

Like most here, I am very good at web scraping and automated form fills. I keep trying to figure out a profitable side project or business idea to make out of it and keep coming up with nothing that works. Any good ideas?

Fantosism 5y ago
I know many people that follow limited/exclusive releases for things like Yeezy/Air Jordan sneakers as well as PS5's and graphics cards. They pay $500/mo for access to a bot that will allow them to make these purchases. Most of the community lives on discord.
- mschuster91 5y ago
  I understand people using bots to snipe PS5s and GPUs, these have real economic value and actual usage. But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers?!
  - pjc50 5y ago
    Same as NFTs: hype and resale value. At least you can wear the sneakers once you've stopped flipping them.
  - nichch 5y ago
    Economic value and actual usage.
    - mschuster91 5y ago
      What economic value (other than hoping for the value to increase, aka tulip mania) can I derive out of ultra-rare sneakers? What usage that goes above "it looks cooler than an unbranded, otherwise identical sneaker"? For me, this kind of product is part of the "bullshit economy" - similar to "bullshit jobs", this kind of product has no reason to exist other than vanity, as almost all of these "collectibles" won't ever be used. We are using up valuable, finite resources to create and distribute this kind of useless "bullshit product", we are using up valuable human time and IT resources on developing websites capable to resist (D)DoS attacks and on developing snipers to bypass the anti-bot technologies employed by the shops, and we are creating a lot of demand for all kinds of sneaker-related crime - and there's a lot of that: theft and robberies from stores, theft and robberies in the supply chain, ebay/classifieds scams, credit card fraud, robberies on broad daylight [1]. Seriously, fuck all that shit. No one needs hundreds of dollars worth of sneakers that only incentivize crime and bullshit. [1]: https://www.google.com/search?q=man+robbed+because+of+sneaker https://www.google.com/search?q=man+robbed+because+of+sneake...

blantonl 5y ago

I fail to understand why Web Scraping isn't almost universally viewed as unethical and a terrible and nasty business practice. In almost all cases I view Web scraping as people who are trying to build businesses on top of other people's innovation and data. I know this isn't a popular opinion, so change my mind, but at the same time, I'm one of those business owners that fights with Web scraping constantly and my opinion of it is that those that are doing it to my platforms are doing so solely to steal data and build businesses on top of other's hard work.

KieranMac 5y ago
There are pro-social and anti-social uses of web scraping. If you have ever used Kayak or any other price discovery or price comparison website, you've relied on web scraping to provide you a service.
- zffr 5y ago
  Also google or any other search engine
- tyingq 5y ago
  I believe Kayak has agreements with the sites they scrape though. So it's a different type of "scraping", really.
usbqk 5y ago
I think if you make intangible things public you shouldn’t consider them to be only yours anymore.
xrendan 5y ago
I think it really depends on the application of web scraping. (As someone who does, what is in my mind, ethical web scraping) - Scraping public information from government websites to do analysis: ethical, it's the public's data - Scraping to help some companies customers more effectively use that companies product, for example scraping a medical office's insurance claims to help them automate their insurance remittance process: ethical - Scraping faces to build a surveillance-tech company: disgusting - Scraping your own website because your internal processes are so broken you can't get it any other way: ethical - Scraping to just copy someone's data they worked hard to generate to go and resell: unethical

KieranMac 5y ago

As a lawyer whose primary focus is in web scraping, this article is in many ways misleading and inaccurate. While it is true that the Van Buren case is generally positive for web scraping, the overall legal landscape is still murky. The main battleground for web scraping legal issues is shifting from the CFAA to breach of contract and various state-law issues, including misappropriation, unjust enrichment, and trespass to chattels. In my opinion, 2021 was a bad year for the law as it relates to web scraping. The Supreme Court remanded hiQ Labs, and many high-profile lower-court cases ended badly for web scrapers. It's a darker shade of gray than it was in 2020. It can be navigated, but it's tricky.

Ian_Kerins 5y ago
Interesting!...I'm not a lawyer, so the content for this piece was based on commentary in the below article. Was written by their lawyer, but would love to hear your counter point to it. Always good to get multiple viewpoints on something. https://www.zyte.com/blog/van-buren-a-victory-for-web-scrapers/ https://www.zyte.com/blog/van-buren-a-victory-for-web-scrape...
- KieranMac 5y ago
  The Zyte article isn't inaccurate; it's just a simplified assessment of a complicated issue. If you'd like a more nuanced perspective on this, please read my guest post of Prof. Goldman's blog. https://blog.ericgoldman.org/archives/2021/06/more-perspectives-about-van-buren-v-us-guest-blog-post.htm https://blog.ericgoldman.org/archives/2021/06/more-perspecti...
faizshah 5y ago
Is there a good blog or something that tracks these cases?
- KieranMac 5y ago
  Prof. Eric Goldman's blog is probably the #1 site historically on scraping and the law. I've contributed to it a few times. https://blog.ericgoldman.org/archives/2021/06/more-perspectives-about-van-buren-v-us-guest-blog-post.htm https://blog.ericgoldman.org/archives/2021/06/more-perspecti... The name of my firm is McCarthy Garber Law. I write about scraping there when I have time (which I rarely do)!

ok_coo 5y ago

Time for me to advocate again for people to use Common Crawl. Please don't slam peoples' websites, look for alternatives before scraping. There are probably other, better options. APIs, data set downloads, etc. https://commoncrawl.org/ https://commoncrawl.org/

mycall 5y ago
I wish web.archive.org had an index by someone like common crawl. There is lots of great stuff on archive.org
- wumpus 5y ago
  web.archive.org has a CDX index, similar to Common Crawl. Since I use both of these archives together, I wrote this code to iron out the differences between them: https://github.com/cocrawler/cdx_toolkit https://github.com/cocrawler/cdx_toolkit
  - kevinsundar 5y ago
    Hey! I was using your tool a couple months ago. It was super helpful for my project.
    - wumpus 5y ago
      Thanks! I rarely hear from users, great to hear from you!
- kevinsundar 5y ago
  They do and its better than common crawl's by my testing.
dewey 5y ago
I'd guess that for the many popular scraping uses cases this is not really useful as it's usually about being quick and up to date (job postings, availability information, e-commerce, serps,...) not about having a big corpus of historic data.

NDizzle 5y ago

I still have a daily job running a web scraper I first wrote with Scrapy back in 2017. I think I've had to update it 3 times over the years for changes to the site and web standards. Good old government sites - rarely change!

cblconfederate 5y ago

Cloudflare's blocks get in the way of many websites who are simply trying to get a "link preview" of the page, even if it is only a single request from a new IP. I wish they would offer some kind of alternative for the pages they serve instead of a captcha block.

bobblywobbles 5y ago

Not a lawyer, but many terms of service prohibit interacting with their website in an automated fashion, as well as collecting their data. In my understanding, scraping a site with these terms already puts you in the wrong.

tommek4077 5y ago
Because those terms are the law and cant be ignored in almost all the rest of the world...
akersten 5y ago
> many terms of service prohibit interacting with their website in an automated fashion, Ignoring the fact that I didn't agree to anything just by virtue of requesting a page from a webserver (and, your server sent me the data!), that's such a meaningless phrase that it's certainly unenforceable. What is an automated fashion? Do I have to manually craft my HTTP request by hand-pulsing a voltage on an Ethernet cable, or do I have your permission to let Chrome automate that for me?
- RobSm 5y ago
  This is so exactly. People do not realize that when they use chrome to view website, chrome is their 'scraper'. And the goal of webs craping is not to get illegal data, but to have efficiency and performance by not doing something manually but letting computer do the repetitive tasks. It's a productivity tool. You can't make something illegal just because it's an automation instead of 'manual' operation.
  - nsonha 5y ago
    are you a lawyer? Your opinion doesn't really mean anything if you still lose the case at the end. By your logic there isn't a clear way to define DDoS either. Sounds like there is though?
    - Chris2048 5y ago

fareesh 5y ago

My toolbox of choice for web scraping is either Nokogiri or puppeteer Can someone sell me on beautiful soup or scrapy or any of the others? Do they provide any advantages or features that I'd be missing out on?

edmundsauto 5y ago
One great scrapy feauture is caching the page content. So you can essentially write a crawler, and when that’s running, you write your extraction code. Then, if you want to go back, you can add more extractors and run it against your local copy.
- fareesh 5y ago
  Ah interesting, I end up doing this manually, i.e. File.write followed by what I want to scrape
  - edmundsauto 5y ago
    I believe scrapy has somewhat intelligent cache control options - maybe it could be recreated in a few dozens of lines of code, maybe a few hundred. But there are a huge number of these types of features - it’s basically a Swiss Army knife. Examples include rotating proxies, rotating user agent headers. Hooks to add in middleware for processing pipelines. CLI switches to change your data output format. Nice debugging and logging. Other large scale features include distributed crawlers. Scheduling. Monitoring UI so you can see progress via a web UI. It’s what I reach for first, because you can be up and running with your first scraper in an hour. By hand, that’s maybe 10 minutes - but if you want to iterate, and your first scraper is a v1 rather than final effort… i think it’s definitely worth it.

coverj 5y ago

I have been interested in web scraping lately but never really dived too deep. Did anyone have more indepth resources (github projects, blogs, forums, etc) than the tutorials that are basically install beautiful soup and get data from a tag?

JimBlackwood 5y ago
Genuine question but, what more do you need?

JJxFile 5y ago

The web scraping ecosystem is growing, with more libraries, frameworks and products available than ever before to simplify our web scraping headaches so the future is looking bright.

gmanis 5y ago

What does HN think of web scraping for the purpose of price comparison? I’m asking this because I run a small side project to show prices across retailers for a very small niche. The users are very very happy. Even the vendors started contacting to be listed on the comparison. But I am unable to make a business out of it other than few affiliate commission.

magixx 5y ago
I worked for a company that did exactly this many years ago. (They were even able to parter with some retailer). Their product worked well yet they still went out of business long ago. To be honest, I don't see much value in such a service, not that it doesn't exist, it's just hard to justify paying for this data.

darepublic 5y ago

Separate from web scraping, there is the use of automation to perform normal allowable user actions on the site. That should be considered distinct from large scale data extraction no

slvrspoon 5y ago

for those in this thread with super-serious experience scraping and automating at scale, looking for work (ethical!) please contact me directly.