5 ms·

Sorry guys, hit by traffic - just scaling my EC2 at the moment.

• franciskim 10y ago
  ok on m4.4xlarge now :)
• niftich 10y ago
  No worries, we had your page scraped just in case ;) Google Cache link: http://webcache.googleusercontent.com/search?q=cache:https://franciskim.co/2016/08/24/dont-need-no-stinking-api-web-scraping-2016-beyond/ http://webcache.googleusercontent.com/search?q=cache:https:/... Archive.is link: http://archive.is/DQccs http://archive.is/DQccs
  • franciskim 10y ago
    haha :)
• fareesh 10y ago
  Is it common for developers in the eCommerce space to use scrapers as a means to aggressively push automated price-match algorithms? I've been asked to do this a number of times, was just curious as to how prevalent it is.
  • hk__2 10y ago
    Yes, everybody scrapes the prices of the others.
    • rckclmbr 10y ago
      Most companies will use resources not in their datacenter and not identifiable. Executives know it's sketchy, but they do it anyway.
    • fareesh 10y ago
      With stuff like Facebook opengraph (e.g. og:price) and other meta tags meant to help search engines and social networks get this sort of data to display inline, do you think it's inevitable that complex scraping will no longer be needed in a practical sense since everyone will be inadvertently optimizing their markup in a way that you could write a really simple parser to grab the data?
      • greglindahl 10y ago
        Given that FB's bot identifies itself, no, eventually some websites will present og: markup only to FB's bot.
        • fareesh 10y ago
          Couldn't you just spoof the user agent though? Or is there some other mechanism that you can use to verify the bot is Facebook's?
          • greglindahl 10y ago
            IP address.
• ksahin 10y ago
  I just like to know, how much traffic did you got ?