25 ms·
Lessons learned scraping 100B product pages
- blattimwind 8y ago> Multi-threading is a must, when scraping at scale. The more concurrent requests your spiders can make the better your performance - simple. Intuitively I would think that this sort of problem would profit from using asynchronous ingestion at the edge pushing unprocessed contents to a multi-threaded/multi-process backend. (Because I'd expect that network latencies mean you need lots of threads to saturate I/O, which I'd expect would conflict with effectively using the available CPU power to do the actual document processing).
- skate22 8y agoI would expect that you would use both async requests & multithreading. I might restrict each thread to operate on an exclusive set of domains to throttle the request rate to any given domain at a point in time
- mgliwka 8y agoThat's been exactly my experience. Most time is spent connecting or waiting for the server response (TTFB). Using an async I/O event loop approach in combination with EPOLL/KQUEUE you can handle thousands of concurrent connections. You then push the response to your worker nodes, which process the data in a multi-threaded fashion. Stream Processing Frameworks like Apache Spark or Storm work great for that.
- fareesh 8y agoI've always wondered if it makes more sense to render the page as a jpeg and run some kind of machine learning to identify and read off the relevant details
- jacquesm 8y agoI'd go the other way and say that pages that no longer contain relevant information in a normally digestible format should be dropped from search engines and other automated indices. After all, the web was built on accessibility of information, not on purposeful obfuscation. If you go so far as to essentially flatten the webpage to the point where you might as well print it out and then do OCR on it then you've thrown out the baby with the bathwater, you had all that information when you started. Or at least, you should have had it. Otherwise we might as well kiss HTML goodbye and render the web as pdfs, with or without links.
- zzzcpan 8y agoThe biggest search engine doesn't have your best interests at heart and has been trying to make HTML and accessibility of information obsolete for years. Some pages now render only with javascript or require solving javascript challenge to even get to the rendering (hello cloudflare) and essentially kissed HTML goodbye.
- deleted 8y ago[deleted]
- nerdponx 8y agoOCR tech is good but it's not there yet.
- vinceguidry 8y agoI want machine learning, but definitely not to identify information off of a gif. Websites are too dynamic for that to really be feasible. Say I want to scrape plane ticket prices. Getting the info off the page once you're there is the easy part. The hard part is figuring out how to navigate the website. I don't want to do the annoying manual work of programming page traversal. I want to be able to make a ML bot that does it for me. That way no matter how often they change the interface, all I have to do is re-run the ML.
- dredmorbius 8y agoDumping rendered text, aswith lynx or w3m, is another option. There's a lot of very bad HTML out there.
- tegansnyder 8y agoMaybe someone can chime in, but I'm pretty sure Diffbot does something similar.
- dwynings 8y agoHey Tegan, Answered on the parent, but it's somewhat similar.
- dwynings 8y agoFair warning: I work at Diffbot. Essentially that's what Diffbot (https://www.diffbot.com/ https://www.diffbot.com/) does, except we don't the render pages as an image nor do OCR. Diffbot renders the page in a headless browser, and uses computer vision to automatically identify the key page attributes and extract normalized data for specific page types (Articles, Products, Discussions, Profiles, Images, and Videos). This approach enables us to work in any language and on sites that we've never come across before automatically with better than human level accuracy.
- ainiriand 8y agoJust by chance we experienced a scraper bot on the site past week and we discovered some performance problems thanks to it. It literally fried our ancient caching system and we finally took the step towards using cdn for static delivery and redis for api responses. I wonder if there were those guys because it was some solid scraping.
- jacquesm 8y agoBadly behaved scrapers should be blocked, not accommodated.
- ameister14 8y agoRealistically, both should happen. That way you don't have any losses when it happens again from another scraper.
- Theodores 8y agoBut, as the article says, they have proxies to get around that. The thing is that most ecommerce websites of note generate shopping feeds in easily machine readable formats (JSON, XML) for Google Shopping, Facebook and the like. These feeds also go down to SKU level. The URL might not be advertised but it won't be blocked or protected with a user/password API key. If buying a T Shirt, the product page might list all sizes and all colours only showing a master 'variant' SKU (that is not a real SKU) and the backend might then add the actual size/colour specific SKU to the basket, of which there could be twenty on the 'variant' product page. Meanwhile the product feed will list every SKU variant, complete with latest pricing and other pertinent information, e.g. barcode, product image etc. I am sure that most retailers would prefer to just point the scraping party to the feed rather than have them grind the site to a crawl with multi-threaded crawlers hiding behind proxies. So that is how these scrapers can be 'accommodated'. The sitemaps that go with the ecommerce game are also pretty reliable, these are high up the SEO checklist and will say when products were last updated. Then there are rich snippets - or whatever they are called now. The trend in these is to have some JSON-LD attached to the page in some format GoogleBot likes. Not hard to ingest. Sites that don't have their act together for Google Shopping and SEO really are not worth scraping, they will never make it to the Google top 100 search results unless they are selling something that nobody else sells, e.g. 'Tibetan Monkey Stones' where you probably don't need to compete. To me it sounds like these scraping concerns just need to pay a bit extra for ecommerce developers to show them how the 'puzzle was made' and to stop abusing people's business websites that are not built to be scraped on a daily basis by some random third party on the other side of the globe. Also the plain old telephone helps. If your brand owning ecommerce team get a call from an interested party saying that they would really like to get a list of their products for their comparison/whatever site then they just might say yes, here is the URL for the feed, oh and here is the one for locale_en_xy. But people would prefer to hack away at some hacky spider rather than just pick up the phone and ask.
- afandian 8y agoCan I piggy-back off this submission to ask HN if you're running a scraper, have the recent wave of GDPR splash-screens caused you issues? How are you dealing with them? https://news.ycombinator.com/item?id=17471599 https://news.ycombinator.com/item?id=17471599
- moltar 8y agoYou can just remove them from DOM or hide with CSS.
- blattimwind 8y agoExcept when the backend redirects you to consent.someco.example unless you're logged in or send a consent cookie along.
- afandian 8y agoWhat about e.g. http://discourseontheotter.tumblr.com/ http://discourseontheotter.tumblr.com/ ? Edit: In the UK I see this: https://imgur.com/a/zlWOByh https://imgur.com/a/zlWOByh
- jklein11 8y agoI don't see a GDPR challenge on this page?
- afandian 8y agoThat's probably because it's being inconsistently applied and you're not in Europe. If that's the case it's all the more insidious! Or you accepted the Tumblr terms in the past. This is what I get in the UK: https://imgur.com/a/zlWOByh https://imgur.com/a/zlWOByh
- YouKnowBetter 8y ago:s/Europe/EU/ I'm in Switzerland (Europe), don't get the https://imgur.com/a/zlWOByh https://imgur.com/a/zlWOByh
- baxtr 8y agoI had to stop reading the otherwise interesting sounding piece when at the left bottom corner a "Get the Enterprise Web Scraping Guide" box popped up (second paragraph or so). Maybe I'll give it a second chance later.
- moltar 8y agoIt was a pretty thin self promoting post. You didn’t miss much.
- livando 8y ago"Multi-threading is a must, when scraping at scale." I disagree on this point. Starting with a single threaded model allowed my team to scale quickly and with little additional overhead. What we have lost with performance we gained in simplicity and developer productivity. That being said tuning and porting portions of the app to a multi-threaded system is slotted to take place within the next year. Start with single threaded and simple, move to multi-threaded scrapers when the juice is worth the squeeze.
- pdimitar 8y agoOr use a language where fully utilizing all CPU cores is transparent, like Elixir? There's zero complexity, you basically add 4-5 lines of code and that's it. Honestly, not exaggerating. I've done several very amateur scrapers in the last several years, I am never going back to languages with a global interpreter lock, ever.
- detaro 8y agoAny further information on this? Last I looked (which was a while ago), the infrastructure like HTML parsers seemed surprisingly tricky in Elixir.
- pdimitar 8y agoThe only complication is if you want to use Meeseks (https://github.com/mischov/meeseeks https://github.com/mischov/meeseeks) which requires the Rust compiler and runtime be installed because it has native bindings. Meeseks is useful because it's a bit faster than the default Floki (https://github.com/philss/floki https://github.com/philss/floki) and because it can handle very malformed HTML. As for Elixir itself, here's a quick example: ``` # Assume this contains 1000 URLs urls = [....] # This will utilize 100 threads; if the second parameter is omitted, it will use threads equal to CPU cores. For I/O bound tasks however it's pretty safe to use much more. results = Task.async_stream(&YourScrapingModule.your_scraping_function/1, max_concurrency: 100) ``` It's honestly that simple in Elixir. For finer grained control the line count is little bigger -- but little. Not hundreds of lines for sure.
- cmjqol 8y agoI always assumed Web Scraping wasn't something particularly challenging because of how many libraries existed for this purpose. This article made me realize I assumed wrong.
- mipmap04 8y agoIt gets especially difficult with dynamic content or when trying to scrape sites written on very heavy frameworks like ASP.NET Webforms that require passing the view state with every request. I made a calendar aggregator for adult hockey times in my area that scrapes rink websites[0] and it was far more difficult than I had thought it would be because of the fact that the rinks all used Telerik Webforms controls to do their calendars. It turned a 30 minute job into a 2 hour job. [0]: http://dpscschedule.azurewebsites.net/ http://dpscschedule.azurewebsites.net/
- taitems 8y agoI hope you too got a chuckle from reading their anti-bot counter measures section only to see their form protected by Google’s “I’m not a robot” CAPTCHA plugin.
- apeace 8y ago> As most companies need to extract product data on a daily basis, waiting a couple days for your engineering team to fix any broken spiders isn’t an option. When these situations arise, Scrapinghub uses a machine learning based data extraction tool that we’ve developed as a fallback until the spider has been repaired. I once worked on a spider that crawled article content and I ran into the same problem. I always wanted to try the following solution to it but never had the chance. Assume you have a database of URLs and the fields you've scraped from them in the past (title, author, date, etc). If you ever fail to scrape one of those values from a new URL, here's what you do: - Go back to one of the old URLs where you already have the correct value (let's say it's the title). - Walk through the whole DOM until you find that known title. At each node you will have to remove child nodes except for text, to deal with titles like "Foo <span>Bar</span>" which you want to match against "Foo Bar". So this is going to be an expensive search. - Generate several possible selectors which match the node you walked to (maybe you have ".title", ".title h2", ".content .top h2", etc). - Test each new selector on several other already-crawled pages. If any of the selectors work 100% of the time, there's your new selector. Any thoughts on whether something like this would work?
- moepstar 8y agoI've had the same idea, it kind of sounds like the obvious thing to do... That said, i haven't found a need to do that yet to verify the idea itself. Maybe that is similar to what they do using their "ML approach" mentioned?
- RossM 8y agoTheir ML approach is most likely based on their https://github.com/scrapy/scrapely https://github.com/scrapy/scrapely project, which uses instance-based learning (as I understand it, lightweight ML) to scrape other pages from a few examples.
- AznHisoka 8y agoThat sounds like a very creative idea. What I do is call a regression test every x minutes. If it fails, set a flag to save/store the html everytime we crawl pages. Now we can go back and process these saved pages when we fix our crawler
- polskibus 8y agoWouldn't such scraping be under TOS of scraped sites?
- merinowool 8y agoBot cannot consent to or understand TOS...
- jd20 8y agoWhat if the site requires you to create an account and login? Can the bot create it's own account, and still claim to not consent?
- merinowool 8y agoIs bot aware of what it is doing? I don't think so.
- Exuma 8y ago> However, our recommendation is to go with a proxy provider who can provide a single endpoint for proxy configuration and hide all the complexities of managing your proxies. Can you provide an example of such service? THanks!
- danni 8y agoI run https://www.scraperapi.com https://www.scraperapi.com which does this!
- Ian_Kerins 8y agoCrawlera (https://scrapinghub.com/crawlera https://scrapinghub.com/crawlera) is the one Scrapinghub developed
- misterbowfinger 8y ago> A large proportion of these bot countermeasures use javascript to determine if the request is coming from a crawler or a human (Javascript engine checks, font enumeration, WebGL and Canvas, etc.). How effective are scraping countermeasures anyway?
- jd20 8y agoThey work pretty well for any scraper that's not using an actual browser with JavaScript engine. It keeps the riff-raff out. A dedicated person will eventually work his way around all available counter-measures, though.
- jjeaff 8y agoAs a side note, I have had quite a bit of experience trying to block automated scraping services. And I found that the best way is to quietly attempt to detect scraping. Then, serve up tainted data. In our case, competitors were scraping pricing data in order to competitively price their products without having to do the work. So we just randomly start to give them incorrect prices on every few products. Not only would it make the whole data set useless, they had no way of figuring out which data was correct without manually checking and since we didn't do it to everything and started at random intervals, it made it too difficult for them to figure out when their ip had actually be quietly blacklisted.
- elorant 8y agoSomething like that could only work if prices are scraped from one source only. If multiple sources are used they could just compare prices and exclude the ones that fall way off. So my guess is your site is an edge case.
- throwawaymath 8y agoWhat do you do if you can't detect the scraping? And if you do detect scraping, how do you ensure the data you provide them is both invalid and consistent?
- jjeaff 8y agoI used a multiplier that was calculated using the date, a static secret, and a seed hashed from the sku. So it was consistent but the offset was different product to product. So that even if you manually went in and figured out the offset for a specific product you couldn't just offset all scraped prices. But once they lose a bunch of money the first time, they tend to stop trying. We tracked down one competitor that was mirroring our prices on an hourly basis. So we waited until late at night, tanked our price on a few expensive items, then placed orders on the competitors site. The human touch tends to scare off scrapers faster than a technological fence anyway.
- ikeboy 8y agoWhat's ironic is that most of the sites with anti scraping protection also do scraping of their own. E.g. Amazon and Walmart both do a lot of their own scraping.
- ChuckMcM 8y agoAh yes, Challenge 4 (anti-bot measures). At Blekko I developed a number of ways to deal with people that tried to scrape the web site for web results. The three most effective ways are blackholing (your web site vanishes as far as these folks are concerned), hang holding (basically using a crafted TCP/IP stack that does the syn/ack sequence but then never sends data so the client hangs forever), and data poisoning (returning a web page that has the same format as the one they are requesting but filling it with incorrect data). We had a couple of funny triggers of the anti-bot stuff during the run, once when a presenter on stage showed an example query and enough people in the audience typed it into their phones/tablets/laptops and it all came from the same router that it looked like a bot. The other where the entire country was behind a single router and a school had all all of their students making the same sort of query at the same time (in both cases the trigger was a rapid query rate for the exact same search query from an address). In Blekko's case since bots either never clicked on ads or always clicked on the same ad (in both cases we got no revenue) keeping bot traffic off the site was measurable in terms of income.
- jd20 8y agoI thought the number one anti-bot measure was a cease and desist letter :) seriously though, some of these websites clearly don't want to be scraped, what's stopping them from sending scrapehub a C&D letter and forcing them to comply?
- ChuckMcM 8y agoSure, if you can make a reasonable assumption it is them scraping you. As they point out in the article they invest in proxy networks to make their requests appear to come from a bunch of addresses that don't lead back to them. One of the things we learned at Blekko was that people that run botnets often sell 'proxy service' as a thing, we identified several made out users of the Time Warner "road runner" service. That put us as a web site in a bind in that the proxy service that was running on an infected computer was violating our terms of service but the user might be completely unaware. If they were also a customer and we black holed their IP it would also cut off legitimate traffic. Since we didn't keep a logs that could identify these relations over time (privacy issues) we had to rely on other methods. We never got enough penetration into the search market to make this a huge concern however so the problem remained largely theoretical. We started a program of doing exponential banning where an IP would be banned and then an hour later unbanned, and if it resumed its bad behavior banned for 2 hours then 4 Etc. Once you get to 1024 hours it is pretty safe to assume they are lawfully evil as it were. These guys fake their user agent, mask their IP addresses, and generally work hard to defeat anti-bot measures. They know they are over the line, but the law has yet to catch up to them.
- rosha 8y agoI tried several different queue systems best version I got is using Erlang Queue, Elixir & Kafka on top for doing high concurrent crawler, the project was to develop a realtime Amazon product ASIN price monitoring system for our company as a challenger prototype. Our main problem was basically proxies, we stopped buying them as managing thousands of proxies is a huge effort that we did not want to take, also lack of data means our Hadoop clusters gets thirsty and machines stops learning properly. Currently we are using a third party https://proxycrawl.com https://proxycrawl.com on very high tiers > 10B with a great discount and we are happy to get that part solved. Other lessons learnt are like sometimes things fail and logs help a lot so you will need a highly available Logging and monitoring system.
- jd20 8y agoBack when I worked for a very large tech company, building their web crawler, I had good success with Golang. On four servers, with 10 GigE interconnect and SSD, and a very fast pipe to the Internet, I was able to push about 10K pages / second sustained. At any given time, there were probably several million connections open concurrently. I've played with Elixir as well, and it's also great for this type of thing. proxycrawl.com looks very cool, I'm actually looking for a proxy service for my current scraping project. Are they also a good choice if you're doing lower tiers (like thousands of requests a day)?
- rosha 8y agoGolang is a good choice too but in my experience its nothing compared to what you can do with Erlang Queue and Elixir. Regarding your question about proxycrawl, I do not know honestly, I tested the service for few days on some few millions per day and it was great too. I would say they are good for a very high volume, we are still using it, so that should be a good signal to try them.