5 ms·

Anyone else thinking they added a robots.txt entry for bing to disallow, waited to get de-listed then removed the offending robots.txt entry, posted this news a

Anyone else thinking they added a robots.txt entry for bing to disallow, waited to get de-listed then removed the offending robots.txt entry, posted this news article knowing bing would pick them back up in a couple of days?

jbigelow76 14y ago
If that's what they did (no telling really) that would be pretty clever PR wise.
- rmc 14y ago
  Clever… and dishonest.
  - taligent 14y ago
    Very much Kogan's style.
michaelt 14y ago
The HTTP last modified header for www.kogan.com/robots.txt has Last-Modified: Wed, 06 Jun 2012 06:14:35 GMT It's possible they did this and they were cunning enough to turn back the last-modified clock.
- reinhardt 14y ago
  How did you get that? Curl says May 4 instead: $ curl -I http://www.kogan.com/robots.txt HTTP/1.1 200 OK Server: nginx/1.0.14 Date: Mon, 09 Jul 2012 18:48:34 GMT Content-Type: application/octet-stream Content-Length: 79 Last-Modified: Fri, 04 May 2012 18:58:02 GMT Connection: keep-alive Accept-Ranges: bytes
  - michaelt 14y ago
    I get Last-Modified: Fri, 04 May 2012 18:58:02 GMT from my home machine (and you obviously do too) where I connect to 74.118.68.224 - but I got Wed, 06 Jun 2012 06:14:37 GMT from work (and via http://web-sniffer.net/ http://web-sniffer.net/ and http://www.rexswain.com/cgi-bin/httpview.cgi http://www.rexswain.com/cgi-bin/httpview.cgi) which was looking at 108.166.39.66. Perhaps we get DNS load balanced or something similar?