3 ms·
Why is Google telling people to run a headless browser just for their crawler? Sounds way too complicated for any small-scale website. Why doesn't Google run
by jimmybot 17y ago
Why is Google telling people to run a headless browser just for their crawler? Sounds way too complicated for any small-scale website. Why doesn't Google run the headless browser themselves? It sounds like they are offloading parsing costs onto websites, but then they have a cheating problem they have to deal with.
Somewhat related: It's not AJAX, but I remember hearing Microsoft Research a couple of years ago talk about parsing CSS. They wanted the page as an image so that they could analyze pictorially what was a menu, what was a header, what was main content, etc. It seemed fairly neat for the time, although I think there are much simpler heuristics for when you are after just the main article type content of a page that will work for most blogs/CMS's out there.
(No, they didn't use IE. I asked. Yes, they tried. They said it was slow and crashed too much.)