3 ms·

dotbot is one of the data sources that SEOmoz uses for their Linkscape crawler. Not sure why it was submitted here but I suspect promotional motives. It is int

by meatbag 18y ago

dotbot is one of the data sources that SEOmoz uses for their Linkscape crawler. Not sure why it was submitted here but I suspect promotional motives. It is interesting data, but when people build crawlers to index the entire www, especially if the data is intended as an SEO intelligence tool, certain issues arise. Some background on this particular issue: http://incredibill.blogspot.com/2008/10/seomozs-new-linkscape-creates-webmaster.html http://incredibill.blogspot.com/2008/10/seomozs-new-linkscap...

inovica 18y ago
Are you sure that Linkscape use the submitted organisations bot? I agree that it could just be promotional - as they are only giving access to some of their content (around 10% at my calc) but I'm not sure we're talking about the same bot in this instance
- meatbag 18y ago
  from http://www.seomoz.org/linkscape/help/sources http://www.seomoz.org/linkscape/help/sources * Dotnetdotcom.org * Grub.org/Wikia * Page-Store.com * Amazon/Alexa’s crawl and internet archive resources * Exalead’s commercially available data * Gigablast’s commercially available data * Yahoo!’s BOSS API and other data sources * Microsoft’s Live API and other data sources * Google’s API and other data sources * Ask.com’s API and other data sources * Additional crawls from open source, commercial and academic projects In my experience, the single most useful feature (main selling point) of the Linkscape tool is that it reports http status codes (for a price) so SEOs can detect 301 redirects, etc. AFAIK, dotnetdotcom.org has the only free, publicly available crawl data which also includes http status codes. Not sure about Exalead and Gigablast but I am pretty sure the other SEs don't release this information. To clarify: I don't have any proof, and things may have changed, but I've read some intelligent speculation (smarter than me) which claims that dotbot/dotnetdotcom.org provides the majority of the data (especially the unique info, like status codes) for the Linkscape tool.
daleharvey 18y ago
It was posted here because I thought it was interesting hacker news, I am in no way affiliated.
- meatbag 18y ago
  I very much agree that it is an interesting bit of news, I've been meaning to download the dataset and play with it for a while now. The only point I wanted to make was that there is another side to this story. An adequate summary can be found here: http://sphinn.com/story/80142 http://sphinn.com/story/80142 - I read the HN terms of use and didn't see anything that specifically forbids promotional stuff, but this tool is associated with a company that develops linkbait for a living. I don't mean to imply that this would make dotbot less noteworthy/interesting, but I am suspicious of linkbait. I lurk on HN because, frankly, much of the information sources in my industry are crapped up with "social media optimized" thin content. Again, sorry if I overreacted. This is my second post here and I don't mean to make a nuisance of myself.

by meatbag 18y ago

inovica 18y ago

Are you sure that Linkscape use the submitted organisations bot? I agree that it could just be promotional - as they are only giving access to some of their content (around 10% at my calc) but I'm not sure we're talking about the same bot in this instance

meatbag 18y ago
from http://www.seomoz.org/linkscape/help/sources http://www.seomoz.org/linkscape/help/sources * Dotnetdotcom.org * Grub.org/Wikia * Page-Store.com * Amazon/Alexa’s crawl and internet archive resources * Exalead’s commercially available data * Gigablast’s commercially available data * Yahoo!’s BOSS API and other data sources * Microsoft’s Live API and other data sources * Google’s API and other data sources * Ask.com’s API and other data sources * Additional crawls from open source, commercial and academic projects In my experience, the single most useful feature (main selling point) of the Linkscape tool is that it reports http status codes (for a price) so SEOs can detect 301 redirects, etc. AFAIK, dotnetdotcom.org has the only free, publicly available crawl data which also includes http status codes. Not sure about Exalead and Gigablast but I am pretty sure the other SEs don't release this information. To clarify: I don't have any proof, and things may have changed, but I've read some intelligent speculation (smarter than me) which claims that dotbot/dotnetdotcom.org provides the majority of the data (especially the unique info, like status codes) for the Linkscape tool.

daleharvey 18y ago

It was posted here because I thought it was interesting hacker news, I am in no way affiliated.

meatbag 18y ago
I very much agree that it is an interesting bit of news, I've been meaning to download the dataset and play with it for a while now. The only point I wanted to make was that there is another side to this story. An adequate summary can be found here: http://sphinn.com/story/80142 http://sphinn.com/story/80142 - I read the HN terms of use and didn't see anything that specifically forbids promotional stuff, but this tool is associated with a company that develops linkbait for a living. I don't mean to imply that this would make dotbot less noteworthy/interesting, but I am suspicious of linkbait. I lurk on HN because, frankly, much of the information sources in my industry are crapped up with "social media optimized" thin content. Again, sorry if I overreacted. This is my second post here and I don't mean to make a nuisance of myself.