4 ms·
I got frustrated with this one time when I tried to look up a defunct website in the Wayback Machine. The website was gone, but had been replaced by a domain sq
by Rebelgecko 7y ago
I got frustrated with this one time when I tried to look up a defunct website in the Wayback Machine. The website was gone, but had been replaced by a domain squatter with a restrictive robots.txt. Because of the new roboots.txt, all of the old site was retroactively made unavailable. As time passes and more websites change hands, it would be a nice compromise if the default behavior was only to make retroactive deletions that go back to the last change of ownership in WHOIS data.
- foota 7y agoWould be cool if this could be tied to SSL keys. Being able to serve requests with the same SSL key could be used as verification. (obviously some edge cases like people that are behind some kind of common SSL cert like a subdomain with shared hosting)
- loa_in_ 7y agoThat's impossible due to the way assymetric encryption works. It would help to indicate to the user, and maybe allow searching/grouping by certificate hash to transcend domain names
- deleted 7y ago[deleted]
- foota 7y agoIsn't that what cert pinning does though?
- chousuke 7y agoI think it would make more sense to tie the behaviour to the domain's whois record. If that changes significantly, chances are the domain has changed owners.
- foota 7y agoThis is true, but the whois isn't really a highly official source.
- dredmorbius 7y agoActually, it is. It's the TLD registrar's record. The usefulness of WHOIS has been declining precipitously, largely as useful information has been horribly misused and abused.
- dredmorbius 7y agoIA are specifically refusing to follow robots.txt directives largely on account of just this behaviour. https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea... https://archive.org/post/184024/robotstxt-policy-is-a-failure https://archive.org/post/184024/robotstxt-policy-is-a-failur...
- Rebelgecko 7y agoThat's great news. Is there any update to the 2017 blog post? Is the new policy still only for .mil and .gov sites?
- dredmorbius 7y agoI don't know, though you could research via their blogsite.