3 ms·

Out of curiosity, what options beyond nokogiri, mechanize, etc. are you referring to? Something open source, commercial, or custom?

• 3pt14159 9y ago
  Well I've done it a bunch of different ways, and granted this take is about three or four years out of date, so take it with a grain of salt. Selenium / headless Firefox with JS running in the browser waiting for things to load / resolve. This is the worst case scenario. Sometimes you can use a layered approach, first do a normal mechanize request then if the page doesn't 404 you can try to parse it there. If you detect that its relying on infinite scroll try to run the JS command via Ruby Racer then if that isn't [semi-]automatically possible put the URL in the "requires full DOM queue" all of this on a bunch of code that lights up different servers to evade log analysis. Some URLs are behind a CAPTCHA, and some people will then add those urls to the "Use a human to do the work" queue, but I've never really felt right about doing that. To me a CAPTCHA is basically someone saying, "no seriously, bugger off" plus I don't want have some poor soul waste their life copying and pasting text. It's easier to just convince a client that last 0.01% of data isn't worth getting. What's really funny is the meta game that stores have. There is no "price" any more for many of these people. There is "the price you happen to see" and they use a bunch of tricks to try to figure out if the traffic is real or not because they know that other places are dynamically setting their price to undercut them by some optimal amount. It's just an arms race, but you know what arms races are really good for? Arms manufacturers.