3 ms·

Yeah, I definitely expect this to bite some people, if I'm understanding correctly. A plausible scenario (among many) would be: soft launch a site, show it to s

by JackC 6y ago

Yeah, I definitely expect this to bite some people, if I'm understanding correctly. A plausible scenario (among many) would be: soft launch a site, show it to some early stakeholders, have Wayback archive everything via Always Online, fix embarrassing screwups or oversharing in soft-launched version, publicize site more broadly, everyone in the world can rewind to version zero, regrets. I don't think the existing warnings really make clear that a soft launch is now a forever launch.

jgrahamc 6y ago
The solution to this is... robots.txt. Otherwise your site might turn up in Google etc. Since it's archive.org that's doing the crawling and they respect robots.txt it won't get archived.
- symfoniq 6y ago
  Archive.org does not respect robots.txt IIRC. I’ve run into this problem before with them. Ironically, I ended up blocking Internet Archive’s ASN using Cloudflare. EDIT: Internet Archive started ignoring robots.txt in 2017: https://www.digitaltrends.com/computing/internet-archive-robots-txt/ https://www.digitaltrends.com/computing/internet-archive-rob...
  - kalleboo 6y ago
    They only started ignoring robots.txt on US government websites (as that article also says)
    - symfoniq 6y ago
      That is not what the article says. It says Internet Archive had already started ignoring robots.txt on US government websites. Now (since 2017) they ignore it on all websites.
capableweb 6y ago
I think that's fine. The reason we fix screwups is so the next people who arrive don't see them. We don't fix screwups to hide that sometimes we fuck up. If someone goes out of their way to find old screwups, then so be it. As long as not the majority of people see it, we're mostly fine.
renewiltord 6y ago
Most people password-protect this. It's very common. If you contract a webdev for something, he will recommend it for you 100%. Not the basic auth thing, just a shared secret. Something trivial.
deleted 6y ago
[deleted]

by JackC 6y ago

jgrahamc 6y ago

The solution to this is... robots.txt. Otherwise your site might turn up in Google etc. Since it's archive.org that's doing the crawling and they respect robots.txt it won't get archived.

symfoniq 6y ago
Archive.org does not respect robots.txt IIRC. I’ve run into this problem before with them. Ironically, I ended up blocking Internet Archive’s ASN using Cloudflare. EDIT: Internet Archive started ignoring robots.txt in 2017: https://www.digitaltrends.com/computing/internet-archive-robots-txt/ https://www.digitaltrends.com/computing/internet-archive-rob...
- kalleboo 6y ago
  They only started ignoring robots.txt on US government websites (as that article also says)
  - symfoniq 6y ago
    That is not what the article says. It says Internet Archive had already started ignoring robots.txt on US government websites. Now (since 2017) they ignore it on all websites.

capableweb 6y ago

I think that's fine. The reason we fix screwups is so the next people who arrive don't see them. We don't fix screwups to hide that sometimes we fuck up. If someone goes out of their way to find old screwups, then so be it. As long as not the majority of people see it, we're mostly fine.

renewiltord 6y ago

Most people password-protect this. It's very common. If you contract a webdev for something, he will recommend it for you 100%. Not the basic auth thing, just a shared secret. Something trivial.

deleted 6y ago

[deleted]