5 ms·
I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()
by anaolykarpov 11y ago
I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()
- deleted 11y ago[deleted]
- biokoda 11y agoMaybe they trigger timers immediately
- kuschku 11y agoAnd how about server-side state? Imagine you use setInterval to load a new paragraph from a server, and the server only provides a new parapgraph 1200ms after the first polling?
- fridek 11y agoWaiting for all one-time async operations to finish is not unfeasible. Async loops (like pinging server for possible updates) is a tad harder.
- KMag 11y agoEach time we re-analyzed the page, we got more data and more URLs for the crawler to grab and have waiting for us the next time we analyzed that particular page. Of course, session-dependent content would get badly messed up, but that generally doesn't make useful (or at least repeatable) search results anyway. Source: I primarily did JavaScript execution for Google's indexing pipeline 2006 to 2010.
- deleted 11y ago[deleted]
- andybak 11y agoA better example would be "how does Google index a page where an element changes sufficiently slowly that a user would see it post-change but a faster-than-real-time script would have a different experience" Think of a long page where you need to read a few things as you scroll. You could game Google by timing it so that most humans would see content x but any script that ran at an unnatural speed would see y.
- Rygu 11y agoSeriously, they don't have to. That kind of timeout makes the DOM element irrelevant.
- netnichols 11y agoThey probably don't care about that content. My first guess would be that they snapshot the DOM in the JS tick immediately after window.onload completes. Maybe they have a short pause to let any fast timeouts or callbacks complete, but there's got to be a cutoff at some point (e.g. to stop an infinite wait for pages that continuously update a relative date). Of course, with their own JS engine, I bet they can get really fancy with the heuristics to determine when to take that snapshot.
- KMag 11y agoActually, we did care about this content. I'm not at liberty to explain the details, but we did execute setTimeouts up to some time limit. If they're smart, they actually make the exact timeout a function of a HMAC of the loaded source, to make it very difficult to experiment around, find the exact limits, and fool the indexing system. Back in 2010, it was still a fixed time limit. Source: executing JavaScript in Google's indexing pipeline was my job from 2006 to 2010.
- blumkvist 11y agoWhat about AJAX? Does it load/read/index data after the fact?