3 ms·

Yes it does, but a little goes a looooong way to Google. I've also been starting to notice that having custom HTML / CSS is boosting things more than before fo

Yes it does, but a little goes a looooong way to Google. I've also been starting to notice that having custom HTML / CSS is boosting things more than before for smaller sites. Also, even if using JS for SPAs doesn't matter to Google there is a reasonable long tail of other web crawlers that may or may not be helpful to your ends and you'd be surprised at how many of them just use mechanize or nokogiri instead of something more robust. Basically it comes down to cost, last time I was in the custom crawler trade it was around 50x more expensive to do things with a virtual dom than it was to either straight parse the HTML or figure out the main API calls JS was making from the browser that actually had the data I needed and just run those.

devin 9y ago
Out of curiosity, what options beyond nokogiri, mechanize, etc. are you referring to? Something open source, commercial, or custom?
- 3pt14159 9y ago
  Well I've done it a bunch of different ways, and granted this take is about three or four years out of date, so take it with a grain of salt. Selenium / headless Firefox with JS running in the browser waiting for things to load / resolve. This is the worst case scenario. Sometimes you can use a layered approach, first do a normal mechanize request then if the page doesn't 404 you can try to parse it there. If you detect that its relying on infinite scroll try to run the JS command via Ruby Racer then if that isn't [semi-]automatically possible put the URL in the "requires full DOM queue" all of this on a bunch of code that lights up different servers to evade log analysis. Some URLs are behind a CAPTCHA, and some people will then add those urls to the "Use a human to do the work" queue, but I've never really felt right about doing that. To me a CAPTCHA is basically someone saying, "no seriously, bugger off" plus I don't want have some poor soul waste their life copying and pasting text. It's easier to just convince a client that last 0.01% of data isn't worth getting. What's really funny is the meta game that stores have. There is no "price" any more for many of these people. There is "the price you happen to see" and they use a bunch of tricks to try to figure out if the traffic is real or not because they know that other places are dynamically setting their price to undercut them by some optimal amount. It's just an arms race, but you know what arms races are really good for? Arms manufacturers.
wolfgang42 9y ago
What do you mean by 'custom HTML / CSS'? Not using common WordPress templates or Bootstrap themes?
- 3pt14159 9y ago
  Well, I've generally found that the more "compressible" your site's HTML structure or CSS is with other websites the worse it does. I suspect its because Google is fighting Shopify / SquareSpace fly-by-night stores. Even if a store objectively looks worse, if it's all hand coded HTML and it's got an array of products, it's less likely to be a scam or shoddy material.

devin 9y ago

Out of curiosity, what options beyond nokogiri, mechanize, etc. are you referring to? Something open source, commercial, or custom?

3pt14159 9y ago
Well I've done it a bunch of different ways, and granted this take is about three or four years out of date, so take it with a grain of salt. Selenium / headless Firefox with JS running in the browser waiting for things to load / resolve. This is the worst case scenario. Sometimes you can use a layered approach, first do a normal mechanize request then if the page doesn't 404 you can try to parse it there. If you detect that its relying on infinite scroll try to run the JS command via Ruby Racer then if that isn't [semi-]automatically possible put the URL in the "requires full DOM queue" all of this on a bunch of code that lights up different servers to evade log analysis. Some URLs are behind a CAPTCHA, and some people will then add those urls to the "Use a human to do the work" queue, but I've never really felt right about doing that. To me a CAPTCHA is basically someone saying, "no seriously, bugger off" plus I don't want have some poor soul waste their life copying and pasting text. It's easier to just convince a client that last 0.01% of data isn't worth getting. What's really funny is the meta game that stores have. There is no "price" any more for many of these people. There is "the price you happen to see" and they use a bunch of tricks to try to figure out if the traffic is real or not because they know that other places are dynamically setting their price to undercut them by some optimal amount. It's just an arms race, but you know what arms races are really good for? Arms manufacturers.

wolfgang42 9y ago

What do you mean by 'custom HTML / CSS'? Not using common WordPress templates or Bootstrap themes?

3pt14159 9y ago
Well, I've generally found that the more "compressible" your site's HTML structure or CSS is with other websites the worse it does. I suspect its because Google is fighting Shopify / SquareSpace fly-by-night stores. Even if a store objectively looks worse, if it's all hand coded HTML and it's got an array of products, it's less likely to be a scam or shoddy material.