4 ms·

DNT is like putting a note atop your plain text file asking unintended recipients to ignore the contents. What browsers need is something akin to encryption, wh

by hexis 15y ago

DNT is like putting a note atop your plain text file asking unintended recipients to ignore the contents. What browsers need is something akin to encryption, where you're not asking for compliance, you're denying access.

azakai 15y ago
> DNT is like putting a note atop your plain text file asking unintended recipients to ignore the contents. Well, robots.txt is a simple text file that politely asks robots to not spider parts of your website. It is completely voluntary and unenforceable, but works very well.
- qjz 15y ago
  How can you tell it works? How do you know the bot owners aren't returning to crawl those parts with User-Agent strings that impersonate ordinary users?
- sbierwagen 15y ago
  It is completely voluntary and unenforceable, but works very well. Ha! It works very well: for Google, and Yahoo. They use informative user-agents, and respect robots.txt directives. They have to, they're large corporations, with shareholders. Everybody else ignores it. Why would they listen? Live example: 216.55.185.45 - - [11/Sep/2011:06:50:56 -0400] "GET / HTTP/1.0" 200 4835 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" 216.55.185.45 - - [11/Sep/2011:06:50:57 -0400] "GET / HTTP/1.0" 200 4835 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" 216.55.185.45 - - [11/Sep/2011:06:50:57 -0400] "GET /blog/ HTTP/1.0" 200 114535 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" 216.55.185.45 - - [11/Sep/2011:06:51:01 -0400] "GET /blog/archives/2010/01/12/creating_an_account_on_the_scp_wiki_is_like_pissing_glass/ HTTP/1.0" 200 10490 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" This is all the traffic from this IP, with no lines skipped. You might notice that it doesn't request CSS, or any of the linked images, or favicon.ico. Nor does it populate the referral header fields, which of course it would be doing if it was actually Firefox, and there was actually a human clicking on these links. It doesn't even bother requesting a robots.txt, which I don't have anyway.[1] Do a whois check on the IP, and we get: Codero CODERO1999A (NET-216-55-176-0-1) 216.55.176.0 - 216.55.187.255 Codero's a dedicated server host. This is a spambot, looking for email addresses. Visit the IP in a browser, and you see a site selling fake Tiffany jewelry. 1: http://www.archiveteam.org/index.php?title=Robots.txt http://www.archiveteam.org/index.php?title=Robots.txt
  - deleted 15y ago
    [deleted]
  - icebraining 15y ago
    Maybe, but most tracking services are large corporations, most websites wouldn't embed analytics scripts from spammers-r-us.com.
deleted 15y ago
[deleted]
shabble 15y ago
And yet, it seems to be some sort of meme in the enterprise world to have these sorts of things in every single email: "Please note - This e-mail and any files transmitted with it may contain confidential or privileged information which is protected by legislation, copyright & the code of practice covering personal information. It is intended solely for the use of the individual or entity to whom it is addressed. If you have received this e-mail in error, please notify the originator. Any disclosure, copying, alteration, distribution, publication or the taking of action in reliance on the contents may be an offence." as a random example. Bonus points for using it it in a footer, where almost by definition, you've read the email before seeing it. I'm sure there's some legal justification for doing so, but it still strikes me as pretty stupid.

by hexis 15y ago

azakai 15y ago

> DNT is like putting a note atop your plain text file asking unintended recipients to ignore the contents. Well, robots.txt is a simple text file that politely asks robots to not spider parts of your website. It is completely voluntary and unenforceable, but works very well.

qjz 15y ago
How can you tell it works? How do you know the bot owners aren't returning to crawl those parts with User-Agent strings that impersonate ordinary users?
sbierwagen 15y ago
It is completely voluntary and unenforceable, but works very well. Ha! It works very well: for Google, and Yahoo. They use informative user-agents, and respect robots.txt directives. They have to, they're large corporations, with shareholders. Everybody else ignores it. Why would they listen? Live example: 216.55.185.45 - - [11/Sep/2011:06:50:56 -0400] "GET / HTTP/1.0" 200 4835 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" 216.55.185.45 - - [11/Sep/2011:06:50:57 -0400] "GET / HTTP/1.0" 200 4835 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" 216.55.185.45 - - [11/Sep/2011:06:50:57 -0400] "GET /blog/ HTTP/1.0" 200 114535 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" 216.55.185.45 - - [11/Sep/2011:06:51:01 -0400] "GET /blog/archives/2010/01/12/creating_an_account_on_the_scp_wiki_is_like_pissing_glass/ HTTP/1.0" 200 10490 "-" "Mozilla/5.0 (Windows NT 6.1; rv:5.0) Gecko/20100101 Firefox/5.0" This is all the traffic from this IP, with no lines skipped. You might notice that it doesn't request CSS, or any of the linked images, or favicon.ico. Nor does it populate the referral header fields, which of course it would be doing if it was actually Firefox, and there was actually a human clicking on these links. It doesn't even bother requesting a robots.txt, which I don't have anyway.[1] Do a whois check on the IP, and we get: Codero CODERO1999A (NET-216-55-176-0-1) 216.55.176.0 - 216.55.187.255 Codero's a dedicated server host. This is a spambot, looking for email addresses. Visit the IP in a browser, and you see a site selling fake Tiffany jewelry. 1: http://www.archiveteam.org/index.php?title=Robots.txt http://www.archiveteam.org/index.php?title=Robots.txt
- deleted 15y ago
  [deleted]
- icebraining 15y ago
  Maybe, but most tracking services are large corporations, most websites wouldn't embed analytics scripts from spammers-r-us.com.

deleted 15y ago

[deleted]

shabble 15y ago

And yet, it seems to be some sort of meme in the enterprise world to have these sorts of things in every single email: "Please note - This e-mail and any files transmitted with it may contain confidential or privileged information which is protected by legislation, copyright & the code of practice covering personal information. It is intended solely for the use of the individual or entity to whom it is addressed. If you have received this e-mail in error, please notify the originator. Any disclosure, copying, alteration, distribution, publication or the taking of action in reliance on the contents may be an offence." as a random example. Bonus points for using it it in a footer, where almost by definition, you've read the email before seeing it. I'm sure there's some legal justification for doing so, but it still strikes me as pretty stupid.