4 ms·

This seems rather contradictory with the general AdSense historic guidance to run up to 3 ad units and place them generally to the top and left side of the page

by robk 15y ago

This seems rather contradictory with the general AdSense historic guidance to run up to 3 ad units and place them generally to the top and left side of the page (per various pieces of heatmap research often cited). Seems like this could be pretty easily gamed too depending on how much of a CSS-wizard you are, as I suppose the algorithms are somewhat limited in analyzing CSS vs plain HTML layout or Javascript.

TylerE 15y ago
I wouldn't assume that at all. What's keep them from running a GUI-less version of webkit on their server farm. In fact, I'd be surprised if they _didn't_.
- eitland 15y ago
  Agree. 4 years ago I was offered a position at a Norwegian company that used a modified version of Firefox to read various news sources and compare them, so this was definitely possible back then.
jonknee 15y ago
Why would you assume that they don't fully analyze web pages? Google makes most advanced Javascript engine and the fastest growing browser, it's beyond belief that they aren't rendering every page. (For starters, how do you think they get all those preview images?)
- mike-cardwell 15y ago
  The Googlebot doesn't fetch external css or javascript. How can it be rendering the full page if it doesn't have those resources?
  - jonknee 15y ago
    Googlebot definitely fetches Javascript. How else would it get AJAX pages? http://searchenginewatch.com/article/2122137/Googlebot-Learns-to-Read-AJAXJavaScript-Comments http://searchenginewatch.com/article/2122137/Googlebot-Learn... It also grabs CSS and has for years.
    - mike-cardwell 15y ago
      My mistake. I just checked my server logs and can see Googlebot fetching these resources now.
AdamTReineke 15y ago
There was an article a while back about how Googlebot is likely a version of Chrome that fully renders a page to process it. [1] You're right, without a full browser engine, you could pull a whole host of tricks to make the page appear clean but then put in ads later. Since Googlebot is probably a full featured browser, it becomes a lot harder to trick. [1] http://ipullrank.com/googlebot-is-chrome/ http://ipullrank.com/googlebot-is-chrome/
- PaulHoule 15y ago
  I've seen a lot of evidence that Googlebot evaluates Javascript in a page. For one thing, I've got a site that has Facebook Connect on it, and I see Facebook's bot following on the heels of Googlebot. They work together just like those crabs that work in teams to cut up starfish...
- HNatWORK 15y ago
  There are definitely at least two different, shall we say classes of crawlers: The Googlebot and the Google Web Preview crawler. I don't know the extent of Googlebot's javascript parsing but the web preview crawler appears to parse javascript like plain webkit. Here's a link where they show the UA for the Google Web Preview crawler: http://www.webmasterworld.com/search_engine_spiders/4353651.htm http://www.webmasterworld.com/search_engine_spiders/4353651.... I believe this crawler renders your pages for the preview snippets you get in search results when you hover over the arrow that appears on the right side of a result. The previews I've seen would only look that way if javascript was being rendered and allowed to run for ~10-20s by my estimation--based on the progress of an animation that was previewed.
prawn 15y ago
I regularly get emails from Google's AdSense program encouraging me to add more ad blocks, and as you said, they include a template that encourages you to place those ads high on the page. Got one of these emails just yesterday funnily enough.

by robk 15y ago

TylerE 15y ago

I wouldn't assume that at all. What's keep them from running a GUI-less version of webkit on their server farm. In fact, I'd be surprised if they _didn't_.

eitland 15y ago
Agree. 4 years ago I was offered a position at a Norwegian company that used a modified version of Firefox to read various news sources and compare them, so this was definitely possible back then.

jonknee 15y ago

Why would you assume that they don't fully analyze web pages? Google makes most advanced Javascript engine and the fastest growing browser, it's beyond belief that they aren't rendering every page. (For starters, how do you think they get all those preview images?)

mike-cardwell 15y ago
The Googlebot doesn't fetch external css or javascript. How can it be rendering the full page if it doesn't have those resources?
- jonknee 15y ago
  Googlebot definitely fetches Javascript. How else would it get AJAX pages? http://searchenginewatch.com/article/2122137/Googlebot-Learns-to-Read-AJAXJavaScript-Comments http://searchenginewatch.com/article/2122137/Googlebot-Learn... It also grabs CSS and has for years.
  - mike-cardwell 15y ago
    My mistake. I just checked my server logs and can see Googlebot fetching these resources now.

AdamTReineke 15y ago

There was an article a while back about how Googlebot is likely a version of Chrome that fully renders a page to process it. [1] You're right, without a full browser engine, you could pull a whole host of tricks to make the page appear clean but then put in ads later. Since Googlebot is probably a full featured browser, it becomes a lot harder to trick. [1] http://ipullrank.com/googlebot-is-chrome/ http://ipullrank.com/googlebot-is-chrome/

PaulHoule 15y ago
I've seen a lot of evidence that Googlebot evaluates Javascript in a page. For one thing, I've got a site that has Facebook Connect on it, and I see Facebook's bot following on the heels of Googlebot. They work together just like those crabs that work in teams to cut up starfish...
HNatWORK 15y ago
There are definitely at least two different, shall we say classes of crawlers: The Googlebot and the Google Web Preview crawler. I don't know the extent of Googlebot's javascript parsing but the web preview crawler appears to parse javascript like plain webkit. Here's a link where they show the UA for the Google Web Preview crawler: http://www.webmasterworld.com/search_engine_spiders/4353651.htm http://www.webmasterworld.com/search_engine_spiders/4353651.... I believe this crawler renders your pages for the preview snippets you get in search results when you hover over the arrow that appears on the right side of a result. The previews I've seen would only look that way if javascript was being rendered and allowed to run for ~10-20s by my estimation--based on the progress of an animation that was previewed.

prawn 15y ago

I regularly get emails from Google's AdSense program encouraging me to add more ad blocks, and as you said, they include a template that encourages you to place those ads high on the page. Got one of these emails just yesterday funnily enough.