5 ms·

They serve their robots.txt as a "application/octet-stream" not "text/plain". Might be confusing some search engines? Also according to Internet Archive their

They serve their robots.txt as a "application/octet-stream" not "text/plain". Might be confusing some search engines? Also according to Internet Archive their robots.txt has changed in the last months. This was a version that was archived: http://web.archive.org/web/20080104044407/http://www.kogan.com/robots.txt http://web.archive.org/web/20080104044407/http://www.kogan.c... At a glance, they are whitelisting some bots, including Googlebot, but not including Bingbot, and disallowing the rest. User-agent: Googlebot Disallow: User-agent: * Disallow: / Another marketing stunt?

UnoriginalGuy 14y ago
Your link is pointing at 2008. That is four years ago. Hardly proof of anything. You are right about the content type however.
- blauwbilgorgel 14y ago
  Correct. It doesn't proof much, but that they previously had a buggy robots.txt. Usually these posts are accompanied by a declining traffic graph for Bing referrers. That would give us more proof and a timeframe to work with. The post gives two options for their absence in Bing results: a fault at Microsoft or as a punishment. With their history I think a fault on their own part is a more likely option. I can't rule that out.
  - Yver 14y ago
    You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+ known crawlers. It doesn't include Bing, which was launched in 2009. In 2008, kogan.com was a parked domain. It has nothing to do with the current owner(s).
    - blauwbilgorgel 14y ago
      The choice for linking to a 2008 file was not the best. At least their current robots.txt is buggy too, so it isn't all that relevant for proving a track record. I thought kogan.com was around in 2008 as the blog goes back to 2008. The robots.txt is whitelisting bots with agents like "No" and "Due to a deficiency in Java it's not currently possible to set the User-agent." but then blocks all other known crawlers (like MSNbot and Yahoo Slurp). I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is the fault or an evil plan of Microsoft, when you don't present anything to substantiate your claim, and it is a fact the webmaster is at fault in the vast majority of these cases.

UnoriginalGuy 14y ago

Your link is pointing at 2008. That is four years ago. Hardly proof of anything. You are right about the content type however.

blauwbilgorgel 14y ago
Correct. It doesn't proof much, but that they previously had a buggy robots.txt. Usually these posts are accompanied by a declining traffic graph for Bing referrers. That would give us more proof and a timeframe to work with. The post gives two options for their absence in Bing results: a fault at Microsoft or as a punishment. With their history I think a fault on their own part is a more likely option. I can't rule that out.
- Yver 14y ago
  You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+ known crawlers. It doesn't include Bing, which was launched in 2009. In 2008, kogan.com was a parked domain. It has nothing to do with the current owner(s).
  - blauwbilgorgel 14y ago
    The choice for linking to a 2008 file was not the best. At least their current robots.txt is buggy too, so it isn't all that relevant for proving a track record. I thought kogan.com was around in 2008 as the blog goes back to 2008. The robots.txt is whitelisting bots with agents like "No" and "Due to a deficiency in Java it's not currently possible to set the User-agent." but then blocks all other known crawlers (like MSNbot and Yahoo Slurp). I was not trying to be deceitful. IMO: Deceitful is a PR stunt arguing it is the fault or an evil plan of Microsoft, when you don't present anything to substantiate your claim, and it is a fact the webmaster is at fault in the vast majority of these cases.