11 ms·

Accidental SEO: How I got 400,000 page views without even knowing it

107 points by jsomers 16y ago

ddemchuk 16y ago
It's well known that traffic and SERP CTR has a significant impact on rankings. About a year ago, I wrote a post on my blog about how to write a bot in Ruby using Watir, and I submitted it here. Within 24 hours, I had gotten over 1000 uniques and had a front page ranking for the term "ruby bot" and "ruby web bot". I run Google Analytics on my blog, and Google has provisions in its algorithm to account for "breaking news" and because my site rapidly received traffic and links, they chucked it up onto the front page for these terms. I've maintained the rankings because of the links I received during that traffic spike as well as the traffic coming in through the search engine results. What better way for Google to test whether a site is actually decent or not than to rank it quickly because it's getting traffic, then keep it there or drop it based on the CTR and Bounce Rate of the site? It's possible to game these metrics, but not as easily as something like link or content spam. For reference, the blog post is: http://www.layeredthoughts.com/automation/how-to-write-your-first-ruby-web-bot-in-watir-scraping-weather-com http://www.layeredthoughts.com/automation/how-to-write-your-...
- _delirium 16y ago
  I can believe that CTR has a significant impact, but how could traffic have an impact? Matt Cutts says explicitly that search doesn't touch any data from Analytics in any way: http://www.youtube.com/watch?v=CgBw9tbAQhU http://www.youtube.com/watch?v=CgBw9tbAQhU
  - epoxyhockey 16y ago
    It's not from Analytics.. Log out of your gmail account, clear your cache, do a google search for "it turns out" and then right-click -> copy the URL of the first hit. Then, paste that into notepad.
    - _delirium 16y ago
      Isn't that just how they count CTR / from-Google traffic? I thought the previous poster was suggesting that traffic from HN to his blog was upping his Google ranking (in addition to CTR), because Google detected a traffic spike indicating some variety of trending/breaking news. But if Google doesn't use Analytics data for ranking, how could they know that his blog was getting a spike of traffic from sources other than Google search?
      - bigiain 16y ago
        Note the post you're discussing explicitly says "SERP CTR": "It's well known that traffic and SERP CTR has a significant impact on rankings." We can see they're measuring it, it's obviously at least worth investigating as a signal, I'd be absolutely astounded if they aren't using it. (I'm also a little surprised at Matt's continued insistence that the search team don't use CTR data from the Adwords and/or Analytics teams data. As a user-of-search, I'd like them to use _all_ available signals, and I suspect the Google Analytics data contains information which would noticeably improve the quality of search results).
        ig1 16y ago
        It would probably be illegal. Google is the dominant player in both the analytics and the search space, using it's dominance in one area to strengthen in dominance in another would practically be asking for an anti-trust suit. Plus it's incredibly easy to rig google analytics data, you can just modify the GA javascript you're serving.
  - ddemchuk 16y ago
    The golden rule if you want to become a good SEO is to not take Matt Cutts (or anyone at the big 3 for that matter) at face value when they say things. Test test test. Traffic (and Analytics data) absolutely has an effect on your rankings. As mentioned above, it's part of the QDF effect
PaulHoule 16y ago
This reminds me the time that I ranked #1 for something in the news because I wrote an article about the incident before it happened.
- rkudeshi 16y ago
  Ok, I'll bite - what incident/article?
auxbuss 16y ago
This is great writing. It might not satisfy those looking for a studied approach to the topic, but it's definitely real world. Made me laugh. Thanks for sharing.
jonknee 16y ago
I had something similar happen--found out a small feature I wrote six years earlier was getting nearly 3 million page views a month from almost exclusively iPhone users. Still not sure how they all found it (if it was just search engines I can't imagine the usage would end up being 99% iPhone users considering it's not iPhone focused).
- avichal 16y ago
  What did you write? 3 million pageviews a month is amazing.
  - jonknee 16y ago
    A mobile version of the Drudge Report. Before the days of modern smartphone browsers it was an especially frustrating site to visit and I wrote a scraper that reformatted out all the long lists of unchanging links. Didn't think about it (or use it much really--it was more of an academic exercise) and then years later realized it had gained [significant] popularity. The news out of Japan has increased traffic. 140k page views yesterday and 3.4m in the last rolling month period according to Google Analytics.
    - alanfalcon 16y ago
      It would be amusing if someone was somehow selling your website as a simple web view app on the App Store.
      - jonknee 16y ago
        Bing bing. That's what it is. There's both a free and paid version. Time to contact the developer...
        sireat 16y ago
        So what are you going to do? The easy thing would be to just shut it off, but more amusing thing would be to make the news links nonsensical (ie go to Onion etc). Of course, the best thing would be to get your own cut of the revenue.
glenstansberry 16y ago
Were you able to capitalize on any of the traffic? Get signups to the RSS feed at least? That's the problem with junk traffic: most don't really provide any value other than raw pageviews. Content is still king though.
- jsomers 16y ago
  To a certain extent, yes. After the "it turns out" post I think some people went around to other parts of my site and started submitting different posts to HN. There was one point where I had three posts simultaneously on the front page (it was a glorious moment). That converted into about a doubling of my RSS subscriber base to just over 100. But culling RSS subscribers seems to be mostly a game of posting and linking and getting buzz on close a daily basis, whereas I'll occasionally have one or two months of inactivity. Otherwise, somewhere between 0.5 and 1% of visitors ever check out other posts.
InfinityX0 16y ago
This incidence is an example of "QDF", or "Query Deserves Freshness". When a certain competitive keyword gets temporal spikes in links, tweets, and etc, Google uses QDF to give immediate boosts to these keywords on SERPs where it might not otherwise have been able to assault for years. QDF is something that's supposed to hypothetically "turn off", but with the new valuation of tweets and shares, it seems increasingly likely that these social mentions are actually beginning to count (and persist) much like links. See SEOMoz's study of how a similar thing happened to them for the term "Beginner's Guide" when Smashing Magazine tweeted it out suddenly even when it had existed (and not ranked for the term) for months: http://www.seomoz.org/blog/tweets-effect-rankings-unexpected-case-study http://www.seomoz.org/blog/tweets-effect-rankings-unexpected...
- kgtm 16y ago
  The 400K pageviews he is referring to were for "Feynman Lectures on Physics", which was a PDF, which had zero backlinks and wasn't mentioned anywhere. You are of course correct about QDF. The first half of his post is indeed relevant to that, but the major point is further down.
jaysonelliot 16y ago
In the interest of never letting the brilliant light of Douglas Adams fade, I'd like to drop his original observation of the phrase "it turns out." The words of the sadly late, eternally great Douglas Adams: “Incidentally, am I alone in finding the expression ‘it turns out’ to be incredibly useful? It allows you to make swift, succinct, and authoritative connections between otherwise randomly unconnected statements without the trouble of explaining what your source or authority actually is. It’s great. It’s hugely better than its predecessors ‘I read somewhere that...’ or the craven ‘they say that...’ because it suggests not only that whatever flimsy bit of urban mythology you are passing on is actually based on brand new, ground breaking research, but that it’s research in which you yourself were intimately involved. But again, with no actual authority anywhere in sight.”
- arctangent 16y ago
  I used this exact phrase in a presentation today and it helped me gloss over technical details that my non-technical audience did not need to listen to. DNA's books are full of other good advice, although I have yet to try his approach to flying ;-)
  - jaysonelliot 16y ago
    I have to confess to, just once, when I was about twelve, trying to "throw myself at the ground and miss."
  - alexfarran 16y ago
    When I first understood how spacecraft stay in orbit I remember thinking "Throw yourself at the ground and miss"
- deleted 16y ago
  [deleted]
- psykotic 16y ago
  Try to do a word count for 'it turns out' in Wolfram's A New Kind of Science. Anyone who's read that doorstop knows the prose is dreadful, and that phrase is especially conspicuous.
almost 16y ago
Isn't it very easy to rank for phrases that no one else wants to rank for? Sure you got some traffic, but it was probably almost entirely worthless traffic. Pick most any random phrase that no one else would ever want to rank for and you can probably get near the top of the search results for it.
Tichy 16y ago
Is it safe to admit publicly to handing out some copyrighted thing 400000 times? I don't think Feynman has been dead long enough for the copyright to expire? Thanks for sharing the story, just wondering...
- true_religion 16y ago
  He said it was accidental, and he meant to put it in a private unindexed directory.
  - Tichy 16y ago
    Does that help (who pays for damages in accidents - copyright owner could claim loss of 400000 sales)? Also even in the private directory he probably wasn't allowed to own it. I am not judging him, I am only concerned for him.
csomar 16y ago
AwStat is unreliable, or at least, I don't understand how it works. I'm getting something like 50,000 unique a month, but with Google Analytics they are only 2,500. Not even close. I doubt the author does even get 10,000 unique a month. The keyword "it turns out" turns out to be a not so popular keyword with only 8.1K search per month. (Data From Google Adwords Keyword Tool)
- eli 16y ago
  It's bots. Google Analytics starts out by only counting clients that run javascript, so that eliminates most automated queries right there. AwStats looks at actual requests from server logs, which is going to count a lot of requests from poorly written comment spam botnets, email scrapers, and other non-vistors.
gohat 16y ago
Bad juju to post something that is widely read about how you promoted hundreds of thousands of times the illegal distribution of a book.
solipsist 16y ago
Like others have posted, I have also had a surprising number of page views without knowing it. The secret for me was Google Images. Got an iPhone image from Apple's PR site and uploaded it to my homepage. Now my page shows up in image search on Google, Yahoo, and Bing. Once I realized what was happening, I added some code to my webpage so that removes itself from the frame it's embedded in. It's essential the same as finding it on a web search, but it's page rank is a lot higher on the image search due to the iPhone image.
DanielBMarkham 16y ago
Note, for what it's worth, I could not find his site on the first page of Google using any of the phrases mentioned. I was going to check backlinks and the site isn't there to check. The reason I was going to check is that there's no way he's getting that many visits on such an obscure term. There's simply not that many searches. Also without some kind of link, there's no way for a bot to even know your site exists. Perhaps it's just me. I must be missing something. Maybe the article was a bit over-stated?
fedd 16y ago
now you taught SEO professionals put feynman lectures online in pdf with ads injected in feynman content! :)
ecaradec 16y ago
Last year my browser extension ranked on the first page of bing and yahoo for the keyword 'google' : It resulted thousands of impressions, but it fade out the next days of course. It's not very useful trafic of course.

107 points by jsomers 16y ago

It's well known that traffic and SERP CTR has a significant impact on rankings. About a year ago, I wrote a post on my blog about how to write a bot in Ruby using Watir, and I submitted it here. Within 24 hours, I had gotten over 1000 uniques and had a front page ranking for the term "ruby bot" and "ruby web bot". I run Google Analytics on my blog, and Google has provisions in its algorithm to account for "breaking news" and because my site rapidly received traffic and links, they chucked it up onto the front page for these terms. I've maintained the rankings because of the links I received during that traffic spike as well as the traffic coming in through the search engine results. What better way for Google to test whether a site is actually decent or not than to rank it quickly because it's getting traffic, then keep it there or drop it based on the CTR and Bounce Rate of the site? It's possible to game these metrics, but not as easily as something like link or content spam. For reference, the blog post is: http://www.layeredthoughts.com/automation/how-to-write-your-first-ruby-web-bot-in-watir-scraping-weather-com http://www.layeredthoughts.com/automation/how-to-write-your-...

_delirium 16y ago
I can believe that CTR has a significant impact, but how could traffic have an impact? Matt Cutts says explicitly that search doesn't touch any data from Analytics in any way: http://www.youtube.com/watch?v=CgBw9tbAQhU http://www.youtube.com/watch?v=CgBw9tbAQhU
- epoxyhockey 16y ago
  It's not from Analytics.. Log out of your gmail account, clear your cache, do a google search for "it turns out" and then right-click -> copy the URL of the first hit. Then, paste that into notepad.
  - _delirium 16y ago
    Isn't that just how they count CTR / from-Google traffic? I thought the previous poster was suggesting that traffic from HN to his blog was upping his Google ranking (in addition to CTR), because Google detected a traffic spike indicating some variety of trending/breaking news. But if Google doesn't use Analytics data for ranking, how could they know that his blog was getting a spike of traffic from sources other than Google search?
    - bigiain 16y ago
      Note the post you're discussing explicitly says "SERP CTR": "It's well known that traffic and SERP CTR has a significant impact on rankings." We can see they're measuring it, it's obviously at least worth investigating as a signal, I'd be absolutely astounded if they aren't using it. (I'm also a little surprised at Matt's continued insistence that the search team don't use CTR data from the Adwords and/or Analytics teams data. As a user-of-search, I'd like them to use _all_ available signals, and I suspect the Google Analytics data contains information which would noticeably improve the quality of search results).
      - ig1 16y ago
        It would probably be illegal. Google is the dominant player in both the analytics and the search space, using it's dominance in one area to strengthen in dominance in another would practically be asking for an anti-trust suit. Plus it's incredibly easy to rig google analytics data, you can just modify the GA javascript you're serving.
- ddemchuk 16y ago
  The golden rule if you want to become a good SEO is to not take Matt Cutts (or anyone at the big 3 for that matter) at face value when they say things. Test test test. Traffic (and Analytics data) absolutely has an effect on your rankings. As mentioned above, it's part of the QDF effect

PaulHoule 16y ago

This reminds me the time that I ranked #1 for something in the news because I wrote an article about the incident before it happened.

rkudeshi 16y ago
Ok, I'll bite - what incident/article?

auxbuss 16y ago

This is great writing. It might not satisfy those looking for a studied approach to the topic, but it's definitely real world. Made me laugh. Thanks for sharing.

jonknee 16y ago

I had something similar happen--found out a small feature I wrote six years earlier was getting nearly 3 million page views a month from almost exclusively iPhone users. Still not sure how they all found it (if it was just search engines I can't imagine the usage would end up being 99% iPhone users considering it's not iPhone focused).

avichal 16y ago
What did you write? 3 million pageviews a month is amazing.
- jonknee 16y ago
  A mobile version of the Drudge Report. Before the days of modern smartphone browsers it was an especially frustrating site to visit and I wrote a scraper that reformatted out all the long lists of unchanging links. Didn't think about it (or use it much really--it was more of an academic exercise) and then years later realized it had gained [significant] popularity. The news out of Japan has increased traffic. 140k page views yesterday and 3.4m in the last rolling month period according to Google Analytics.
  - alanfalcon 16y ago
    It would be amusing if someone was somehow selling your website as a simple web view app on the App Store.
    - jonknee 16y ago
      Bing bing. That's what it is. There's both a free and paid version. Time to contact the developer...
      - sireat 16y ago
        So what are you going to do? The easy thing would be to just shut it off, but more amusing thing would be to make the news links nonsensical (ie go to Onion etc). Of course, the best thing would be to get your own cut of the revenue.

glenstansberry 16y ago

Were you able to capitalize on any of the traffic? Get signups to the RSS feed at least? That's the problem with junk traffic: most don't really provide any value other than raw pageviews. Content is still king though.

jsomers 16y ago
To a certain extent, yes. After the "it turns out" post I think some people went around to other parts of my site and started submitting different posts to HN. There was one point where I had three posts simultaneously on the front page (it was a glorious moment). That converted into about a doubling of my RSS subscriber base to just over 100. But culling RSS subscribers seems to be mostly a game of posting and linking and getting buzz on close a daily basis, whereas I'll occasionally have one or two months of inactivity. Otherwise, somewhere between 0.5 and 1% of visitors ever check out other posts.

InfinityX0 16y ago

This incidence is an example of "QDF", or "Query Deserves Freshness". When a certain competitive keyword gets temporal spikes in links, tweets, and etc, Google uses QDF to give immediate boosts to these keywords on SERPs where it might not otherwise have been able to assault for years. QDF is something that's supposed to hypothetically "turn off", but with the new valuation of tweets and shares, it seems increasingly likely that these social mentions are actually beginning to count (and persist) much like links. See SEOMoz's study of how a similar thing happened to them for the term "Beginner's Guide" when Smashing Magazine tweeted it out suddenly even when it had existed (and not ranked for the term) for months: http://www.seomoz.org/blog/tweets-effect-rankings-unexpected-case-study http://www.seomoz.org/blog/tweets-effect-rankings-unexpected...

kgtm 16y ago
The 400K pageviews he is referring to were for "Feynman Lectures on Physics", which was a PDF, which had zero backlinks and wasn't mentioned anywhere. You are of course correct about QDF. The first half of his post is indeed relevant to that, but the major point is further down.

jaysonelliot 16y ago

In the interest of never letting the brilliant light of Douglas Adams fade, I'd like to drop his original observation of the phrase "it turns out." The words of the sadly late, eternally great Douglas Adams: “Incidentally, am I alone in finding the expression ‘it turns out’ to be incredibly useful? It allows you to make swift, succinct, and authoritative connections between otherwise randomly unconnected statements without the trouble of explaining what your source or authority actually is. It’s great. It’s hugely better than its predecessors ‘I read somewhere that...’ or the craven ‘they say that...’ because it suggests not only that whatever flimsy bit of urban mythology you are passing on is actually based on brand new, ground breaking research, but that it’s research in which you yourself were intimately involved. But again, with no actual authority anywhere in sight.”

arctangent 16y ago
I used this exact phrase in a presentation today and it helped me gloss over technical details that my non-technical audience did not need to listen to. DNA's books are full of other good advice, although I have yet to try his approach to flying ;-)
- jaysonelliot 16y ago
  I have to confess to, just once, when I was about twelve, trying to "throw myself at the ground and miss."
- alexfarran 16y ago
  When I first understood how spacecraft stay in orbit I remember thinking "Throw yourself at the ground and miss"
deleted 16y ago
[deleted]
psykotic 16y ago
Try to do a word count for 'it turns out' in Wolfram's A New Kind of Science. Anyone who's read that doorstop knows the prose is dreadful, and that phrase is especially conspicuous.

almost 16y ago

Isn't it very easy to rank for phrases that no one else wants to rank for? Sure you got some traffic, but it was probably almost entirely worthless traffic. Pick most any random phrase that no one else would ever want to rank for and you can probably get near the top of the search results for it.

Tichy 16y ago

Is it safe to admit publicly to handing out some copyrighted thing 400000 times? I don't think Feynman has been dead long enough for the copyright to expire? Thanks for sharing the story, just wondering...

true_religion 16y ago
He said it was accidental, and he meant to put it in a private unindexed directory.
- Tichy 16y ago
  Does that help (who pays for damages in accidents - copyright owner could claim loss of 400000 sales)? Also even in the private directory he probably wasn't allowed to own it. I am not judging him, I am only concerned for him.

csomar 16y ago

AwStat is unreliable, or at least, I don't understand how it works. I'm getting something like 50,000 unique a month, but with Google Analytics they are only 2,500. Not even close. I doubt the author does even get 10,000 unique a month. The keyword "it turns out" turns out to be a not so popular keyword with only 8.1K search per month. (Data From Google Adwords Keyword Tool)

eli 16y ago
It's bots. Google Analytics starts out by only counting clients that run javascript, so that eliminates most automated queries right there. AwStats looks at actual requests from server logs, which is going to count a lot of requests from poorly written comment spam botnets, email scrapers, and other non-vistors.

gohat 16y ago

Bad juju to post something that is widely read about how you promoted hundreds of thousands of times the illegal distribution of a book.

solipsist 16y ago

Like others have posted, I have also had a surprising number of page views without knowing it. The secret for me was Google Images. Got an iPhone image from Apple's PR site and uploaded it to my homepage. Now my page shows up in image search on Google, Yahoo, and Bing. Once I realized what was happening, I added some code to my webpage so that removes itself from the frame it's embedded in. It's essential the same as finding it on a web search, but it's page rank is a lot higher on the image search due to the iPhone image.

DanielBMarkham 16y ago

Note, for what it's worth, I could not find his site on the first page of Google using any of the phrases mentioned. I was going to check backlinks and the site isn't there to check. The reason I was going to check is that there's no way he's getting that many visits on such an obscure term. There's simply not that many searches. Also without some kind of link, there's no way for a bot to even know your site exists. Perhaps it's just me. I must be missing something. Maybe the article was a bit over-stated?

fedd 16y ago

now you taught SEO professionals put feynman lectures online in pdf with ads injected in feynman content! :)

ecaradec 16y ago

Last year my browser extension ranked on the first page of bing and yahoo for the keyword 'google' : It resulted thousands of impressions, but it fade out the next days of course. It's not very useful trafic of course.