14 ms·
Cloudflare and the Wayback Machine, joining forces for a more reliable Web
- superkuh 6y agoNormally I'd be upset about Cloudflare getting involved in anything good and pure like archive.org but this relationship, just suggesting new URLs to archive, seems harmless enough.
- syshum 6y ago>>>seems harmless enough. The number of things that started out with those words and then went on to be come huge problems is non-trivial I am sure But I want to believe...
- deleted 6y ago[deleted]
- cj 6y ago> I'd be upset about Cloudflare getting involved in anything good and pure Why? At least by FAANG standards, Cloudflare stands out to me as one of the good guys.
- deleted 6y ago[deleted]
- NikolaeVarius 6y agoYou fool, you have opened up a can of worms, no good will come out of this resulting flame war https://en.wikipedia.org/wiki/Cloudflare#Controversies https://en.wikipedia.org/wiki/Cloudflare#Controversies
- Dirlewanger 6y agoWow, zero coverage about their attempts to centralize key parts of the Internet under their control and how many people got outraged about that, but plenty on the trivial outrage of CF supplying services to customers they found offensive. I'm kind of tempted to update this section.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- core-questions 6y agoWhy establish a false dichotomy? One can be just as mad about them abusing their position to attack free speech in a censorious way, as one is about their other efforts to create a monopoly over a large swathe of internet traffic. I get it, nobody wants to defend badthink, but it's a hell of a lot easier philosophically to defend all political speech regardless of content than it is to try to pick and choose and make some moral case.
- johnklos 6y agoSet aside the free speech stuff which involves groups inciting violence - while we can agree that it's bad, deciding what to do will inevitably lead to wildly divergent ideas. Cloudflare says that it's "free speech" for their customers to claim to be Adobe offering a "Flash Updater", or for their customers to claim to be Bank of America. That is how evil Cloudflare is - they allow clear and unambiguous crime for pennies.
- meowface 6y agoThey do? I'm pretty sure they've always banned phishing pages. Do you have a source on that?
- slipheen 6y agoAny sufficiently large and centralized infrastructure (like Cloudflare is becoming) has an incentive to engage in user-hostile behaviors. So far, I personally think most people would say they've been mostly benevolent. But they are a public company, and a new CEO can have different priorities. For example, (fairly or not) the public perception of Google has slowly been diminishing over time.
- forgotmypw17 6y ago>So far, I personally think most people would say they've been mostly benevolent. They're denying access to no-JS users to many sites.
- aeyes 6y agoThe owner of the site is denying access, Cloudflare just provides a service. The owner can turn the feature off.
- forgotmypw17 6y agoNo, I can't even get through to the site itself past Cloudflare's gate...
- yandie 6y agoThe gate is configured by the website owner. Not to defend the frustrating experience - but I personally prefer my website to not be attacked by bot. I'm just a happy Cloudflare customer.
- kortilla 6y agoCloudflare being the enabler of a user-hostile technology doesn’t make them benevolent, no matter how happy the customers.
- 6y ago
- therealmarv 6y agoJust a reminder: Cloudflare with its standard settings is breaking the second and third world countries internet with their captchas on websites. This is discrimination in my opinion. As long as you are only in a first world country you will never notice.
- freedomben 6y agoMinor correction: As long as you are only in a first world country [and you don't use a VPN service or Tor] you will never notice. I point this out because I think it's an anti-privacy "feature" and I wish CF would stop.
- meowface 6y agoThey could just block these IPs outright. A massive percentage of attack traffic runs over these networks. Before, site owners would often have to block Tor completely after dealing with enough spam/attacks from exit nodes, and now they can allow actual humans without any sort of complex setup. I think it's a net win for everyone involved, personally.
- judge2020 6y agoIsn't this because of ISP-grade NAT where one bad actor (either maliciously or via running something like hola vpn) can kill a few hundred users' IP reputation?
- peteretep 6y agoI do almost all of my internet access from a 2nd/3rd world country and hadn’t noticed?
- andreareina 6y agoDepends on the country/ISP probably. From the Philippines, Firefox with uBlockOrigin and PrivacyBadger hit captcha walls all the time; all that stopped as soon as I moved to Singapore.
- ffpip 6y ago
- surround 6y agoWhile Cloudflare actions have been mostly positive, the sheer size of their network is a concern for decentralization. We can’t trust that they’ll always make good decisions. I’m glad that the Internet Archive is independent and hope it always remains that way.
- johnklos 6y agoThey allow and support known spammers and scammers. They make reporting of spammers and scammers arduous. They play dumb when asked about their barriers and repeat inane responses rather than answering questions. In other words, they have a history of clearly, unambiguously showing themselves to be unapologetic assholes.
- britmob 6y agoWow, this is awesome to see. I hope this doesn’t put a lot of load on the IA, though..
- M4v3R 6y agoI would assume that when a site goes offline Cloudflare fetches a snapshot from IA only once and then serves this copy to all further visitors, unless I'm missing something? Here's a more detailed description of the service from Cloudflare support pages: https://support.cloudflare.com/hc/en-us/articles/200168436 https://support.cloudflare.com/hc/en-us/articles/200168436
- tiffanyh 6y agoIs this basically Archive.org becoming a customer of Cloudflare CDN to reduce load off their servers?
- toomuchtodo 6y agoIt's Archive.org being provided URL telemetry for archiving public sites they have not yet found through traditional means (crawling or users submitting requests through the Wayback Save page) by a Cloudflare product. The next step would be for Cloudflare to point to Archive.org Wayback links when an origin isn't available (similar to browser extensions that point to Archive.org when sites 404 or are down, but in Cloudflare's core). Cool stuff. Thanks Cloudflare folks.
- GekkePrutser 6y agoI really doubt their customers would want that. Usually when a page is 404 it's because the company in question wants to forget about it :)
- deleted 6y ago[deleted]
- jedberg 6y agoYou would return the archived page for a 5xx error, not a 4xx error.
- GekkePrutser 6y agoAh I see. But this is precisely a usecase for cloudflare's own caching service. It wouldn't be fair to use archive.org's community-sponsored resources for propping up businesses which are too cheap to pay for proper IT :)
- pronoiac 6y agoWhile it's not explicitly mentioned, I think Cloudflare is providing financial support to the Internet Archive.
- booleanbetrayal 6y agoThis is actually a really good symbiotic relationship that should foster the archival of a ton more content. Hoping to see this toggle enabled by default at some point.
- luckylion 6y agoI don't like the idea of "we're tacking this onto an existing service lots of people have enabled". CF bit me recently by suddenly taking away proxied dns wildcards from free zones, as it's now a premium feature (breaking the security promise in the process by changing the wild card entry to non-proxied). I don't like surprises and opt-out changes in critical infrastructure. It's one thing to use CF's Always-On service - you're a customer, you know you can remove your data from it. It's another to get the Internet Archive involved, who may or may not remove your data, and may or may not honor robots.txt.
- stubish 6y agoSending the details seems to be tied into clicking the 'Update' button in the Cloud Flare UI, which documents that clicking it you agree. So they might not be sending your PII to a 3rd party until they get your permission. Hopefully any automated updates are not violating customers wishes. Yes, it is annoying the features have been tied together for people who choose to have as little interaction with IA as possible.
- lgats 6y agoGreat, until your admin panel is archived...
- scrollaway 6y agoCloudflare just gives more discovery, it doesn't give IA access to anything that was previously more secure... "COVID tests: great, until you find a positive result"
- lgats 6y ago| As new URLs are added to sites that use that service they are submitted for archiving to the Wayback Machine. Yes, this would prevent most order-confirmation pages or otherwise private must-be-logged-in pages from being archived, but it will expose presumed-private URLs that are thought to be unique (tracking numbers, files uploaded with unique names, unique/private image urls that are otherwise publicly accessible) If you've made efforts to your systems to prevent enumeration-attacks, this could partly bypass them.
- jlgaddis 6y agoWhat I'm hearing is "don't rely on security by obscurity", which I wholeheartedly agree with.
- james412 6y agoWas worried about CF getting their claws dug into archive.org, but on reading, this is a decidedly non-evil deal, actually it sounds wonderful. Still, I worry if there might be some unseen long term interest in the archive. Never forget Dejanews
- ttul 6y agoKeep in mind how Cloudflare makes most of their money: They sell a web proxy service with security and performance features including a CDN. Cloudflare's interests are furthered by improving that service in ways that help its customers. Keeping the Web Archive healthily stocked with content is aligned with their long term revenue growth.
- james412 6y agoT+10 years I very much expect CloudFlare's core business to have expanded significantly. I remember that time my Googler friend told me they were about to release that one thing they'd absolutely never do, Chrome came out a few weeks later, now look at Firefox You need to pay attention to the silent positioning of these companies to even guess at where they might go, so deals with things like archive.org may have some unseen substance to them that might only become obvious much later
- adventured 6y agoAs a business they absolutely are not going to stay in the CDN lane as a primary. Akamai has $3b in sales and an $18b market cap. Cloudflare has $348m in sales and an $10.8b market cap. Akamai is their maximum ceiling if they focus primarily on the CDN segment. Cloudflare is rapidly approaching their valuation ceiling if they stick to CDN as their core (and they'd have to start killing Akamai just to get there; the CDN business is increasingly a slower growth segment in the larger cloud industry). Companies all around them in the cloud are growing faster, yet few are more important than Cloudflare. Zero question Cloudflare will continue to aggressively branch out, leveraging their critical positioning. In the not-so-distant future CDN will not be the center of their business. CDN is and will remain a springboard for them, a gateway drug, milk at the back of the grocery store.
- varbhat 6y agoGood News. I also recommend using Internet archive addon in browser. Clicking on it would archive the website. That way, you can archive pages you visit.
- surround 6y agoOr use this bookmarklet: javascript:window.location="https://web.archive.org/save/"+location.href
- sp332 6y agoI use https://web.archive.org/web/submit?url=%s https://web.archive.org/web/submit?url=%s and set the keyword to "rez". That way I can type "rez example.com" and it will send me to the archived version.
- jwilk 6y agoWhy "rez"?
- null0pointer 6y agoShort for "resurrect" maybe?
- sp332 6y agoYup!
- OkGoDoIt 6y agoAnother nice feature is when you hit a page that is a 404, it will automatically try to load it from Wayback if available
- buildbuildbuild 6y agoThis should be made very clear to Cloudflare users, ideally a warning next to the Always Online checkbox. "Always Online" now can mean "Archive Forever" - even when a site is pre-launch.
- judge2020 6y agoFrom the blog post, an image of the checkbox: https://lh6.googleusercontent.com/J42AtNZv8xNcyQPPefVywiAGEhJyNbtqEE1pclEjIEwdDKdrNri9IFx8QK21QKo5wZD1B96CplW24pAEXJlpIZJ-LQesFDkcCeyIF3ITXhv-owHH63Az-zIptuJk1acDeWacjfMO https://lh6.googleusercontent.com/J42AtNZv8xNcyQPPefVywiAGEh...
- deleted 6y ago[deleted]
- jorams 6y agoIt always has. If your site is publicly available and you don't disallow bots through robots.txt, they can crawl it at any time. Even if the site is "pre-launch", because that doesn't mean anything on its own.
- zeta0134 6y agoAnd of course, remember that robots.txt is only a signal to benevolent bots which respect it. If you have secrets to keep, don't put them online in the first place.
- josefresco 6y agoSide rant: Sure would be nice if the Wayback Machine showed actual snapshots of web pages, instead of "hybrid" snapshots where they combine old with new (maybe it's a setting?). I recently horked a website, and thought to check the Wayback Machine. Curiously, an edit I made that day was showing on snapshots dating back several years. Until I discovered how the WBM worked, I was pulling my hair out.
- judge2020 6y agoThis is probably due to XHR's. The IA loads all JS, so if a website hard-codes the URL or does other complicated XHR stuff the IA might not be equipped to save the response for those, if they do at all.
- vaccinator 6y agoIf only we could get the NSA to publish their archive of public data.
- est31 6y agoThere is hope. With the help of then-senator Al Gore, the CIA made photographs available to researchers it had made of the polar regions to search for soviet nuclear installations. They became valuable for climate research later on.
- throwawaysea 6y agoHas Cloudflare clarified their stance on which content they allow and disallow? They’ve wavered in the past and given how their service is basically critical infrastructure for the internet, I really want them to commit to free speech, avoid deplatforming, and avoid exceeding legal minimums.
- nikisweeting 6y agoNext step, have CloudFlare start mirroring IA on their own servers so we have some redundancy in case IA ever goes bankrupt. Ideally it would be a non-profit that does it, but as a last resort CF is one of the few companies I'd trust to do it right and do it transparently.
- deleted 6y ago[deleted]
- sebow 6y agoUnlike the Wayback Machine, Archive.is for example does not censor & remove certain pages because of "hate-speech" and other motives that are more or less political. Shame really, but i guess compromises are necessary to stay in "business", even though an internet archive should be exactly that, updated but never removed.
- SimeVidas 6y agoJust how big are Internet Archive’s servers? I can’t fathom how they’re able to store so much of the web in so many versions.
- 013 6y agoThe Wayback Machine uses 9.6 PetaBytes. Total storage is 50 PetaBytes. https://archive.org/web/petabox.php https://archive.org/web/petabox.php
- dualboot 6y agoCloudflare is really neat unless you find yourself mysteriously blacklisted by them as a user. Then suddenly the web is a much smaller place.
- pabs3 6y agoYou can use archive.org to bypass the Cloudflare blocklist, especially considering the save page feature.
- amelius 6y agoPerhaps they can start archiving YouTube.
- zmix 6y agoWhatever fills web.archive.org is good! Though, it would be nice if someone invented technology, that can erase all the 404 pages and redirects, that are archived, as well, as soon the page goes offline. Maybe a job for AI?
- feralimal 6y agoNo. Keep historical revisionism out of archive.org.
- mcdevilkiller 6y agoStrange that the blog doesn't have https redirection.
- j45 6y agoEncouraging to see this kind of partnership this day in age. May it never forget why it started and only improve on it.
- stubish 6y agoInteresting to find that, when I checked to see if I was using the feature, I had already agreed to the supplemental terms saying my information will be shared with IA. (For others who need to opt out, https://support.cloudflare.com/hc/en-us/articles/200168436-Understanding-Cloudflare-Always-Online https://support.cloudflare.com/hc/en-us/articles/200168436-U... describes how to disable "Always Online". There doesn't seem to be a way to turn off just the information sharing.)
- feralimal 6y agoPerhaps we are getting a sugared pill? Perhaps CF are genuinely being useful here, but in order to gain trust to act nefariously in future? I don't feel comfortable with their ability to switch off parts of the internet, nor in this case, that they have their hands near what is preserved for posterity. As they say: "Cloudflare has become core infrastructure for the Web, and we are glad we can be helpful in making a more reliable web for everyone." They are indeed powerful. I'm concerned that they are becoming gatekeepers to information, under the guise of providing a better internet service. They are able to operate at a level deeper than the odious restrictions youtube, facebook et al enforce on free speech.
- feralimal 6y agoI'm being downvoted - but we have seen major 'book burnings' on youtube, etc where billions of comments and videos have been purged. These are private platforms and can do what they like, so in a way that's acceptable as it is within their terms of service. CF is a level deeper than that. This is a company that can effectively shut down the internet for companies and individuals. And now they are involved with archive.org? Should we be concerned about online historical revisionism as that relationship matures? I feel uncomfortable that CF seems to be positioning itself as a guardian to all information - not at an application level, but at an architectural level. Cloudflare is shaping up to be a key tool that an authoritarian government requires. And I'm concerned about it!
- josefrichter 6y agoCould someone please automatically activate this for all content linked from HN? It happens all the time that many of the first page links are down due to traffic spike.
- raybb 6y agoFrom what I can tell, all links submitted are automatically archived.
- nomercy400 6y agoWho owns Cloudflare? And what are they valued at? I mean, these deals make them look cool and altruistic, but what happens when BigCompany offers them enough money to sell?
- borrame 6y agoCloudflare is not vpn friendly. I'm a privacy concerned vpn user and in my daily browsing I have to deal dozens of times a day with cloudflare captchas or in some cases with cloudflare total blocking.
- keepingscore 6y agoIs using this chrome/fx addon a option for your use case? https://support.cloudflare.com/hc/en-us/articles/115001992652-Using-Privacy-Pass-with-Cloudflare https://support.cloudflare.com/hc/en-us/articles/11500199265...
- jamescampbell 6y agoMy heart shuddered when I read the headline. I can’t be alone in the fear.
- resynth1943 6y agoThis clearly isn't to create some utopic 'more reliable Web'. In fact, Cloudflare severely undermines that, by pushing their centralised view of what the internet should be. I was hopeful, but after reading this: > “The Internet Archive’s Wayback Machine has an impressive infrastructure that can archive the web at scale,” said Matthew Prince, co-founder and CEO of Cloudflare. “By working together, we can take another step toward making the Internet more resilient by stopping server issues for our customers and in turn from interrupting businesses and users online.” It's plain to see that this is a money-making venture for Cloudflare. While I do like the added functionality, I personally can't see how this 'improves' the Wayback Machine. It's just going to place more load on it.