20 ms·

Thank you for helping us increase our bandwidth

677 points by edward 7y ago

mostlysimilar 7y ago
archive.org feels like an irreplaceable treasure, the Wayback Machine alone is a time capsule of our digital history. I donate to them monthly and know a lot of other people do as well, so I don't worry much about their financial stability. I'm more worried about external pressures taking content down. I hope the data is backed up six ways to sunday, and that somewhere there's a plan to make it all accessible if Internet Archive can't continue to play the role it does.
- chrisseaton 7y ago
  > archive.org feels like an irreplaceable treasure Yes I save some dead sites that I think are critically important resources to tarballs that I'm storing separately in case the Wayback Machine disappears.
  - aesh2Xa1 7y ago
    That's useful for personal use. I think that the IPFS is designed for this purpose as well, but meant to be accessible to anyone.
    - chrisseaton 7y ago
      Well the idea is I can provide it publicly (not sure how... don't know how they get the copyright to do it) myself in the future if needed.
  - im3w1l 7y ago
    While that may work for your personal usecase, there is a significant issue around trust, if you were to share these tarballs with anyone else. I wouldn't trust you to give me originals in the same way I trust the wayback machine. So even if there are backups lying around here and there, their utility is decreased a lot.
    - fermienrico 7y ago
      By your philosophy, just get off the internet. At some point you gotta trust HN, your keyboard driver, CPU firmware, motherboard, ISP, Google or whatever service you're using. I am not even mentioning your mailman and your amazon delivery driver. Your stance is pretty extreme - please don't discourage people from making backups.
      - bigyikes 7y ago
        I don’t think their stance is extreme as you make it seem. Insert their username into the same list you just provided: im3w1l, HN, your keyboard driver, CPU, firmware, motherboard, ISP, Google One of these things is not like the others.
- dmd 7y ago
  I'm very worried about its backups and what happens when the next Big One hits SF. As far as I've ever been able to determine from talking to anyone at IA (e.g., Kahle, Scott), they don't really have any sort of backups that could actually be restored from in a disaster situation.
  - britmob 7y ago
    There is http://iabak.archiveteam.org http://iabak.archiveteam.org, but it’s not exactly large.
    - microcolonel 6y ago
      If I'm reading that correctly, it would only cost a bit over 500 bucks a month to host that whole archive on BackBlaze B2. Furthermore it would not be so hard to translate Archive.org items to IPFS objects, if there were an effort to pin a significant number of them to storage and network.
      - dmd 6y ago
        You're reading it correctly, but IABAK backs up 0.2% of the Internet Archive.
      - Endlessly 6y ago
        Since numbers are not 100% clear... (50 petabytes * 0.2% = 100 terabytes) [$0.005 ($/GB/Month) BackBlaze cost] [(50 petabytes) / (1 gigabyte) = 50,000,000] (50,000,000 * $0.005 = $250,000 US$) ————— Meaning based on my numbers, that is $250,000 USD a month to host 50 petabytes of data on BackBlaze.
        ramraj07 6y ago
        That's a fraction of the AWS bills for many startups arguably doing absolutely nothing
        6y ago
- erichocean 6y ago
  They should team up with Backblaze...
- city41 6y ago
  They have lots of copyrighted content. Such as virtually every commercial retro video game ever made it seems. I personally think it’s great but surely companies aren’t too pleased? How does archive.org avoid being sued into oblivion?
  - judge2020 6y ago
    It's a non-profit library and they don't publicly provide a way to download most of the copyrighted content, so rights holders aren't too concerned. I imagine most people, even copyright lawyers, personally support the free archiving of everything made in the current age as long as it doesn't detract from current business operations and the copyrights are respected for their duration.
    - city41 6y ago
      It's very easy to download the copyrighted content. For example, here is a full NES ROM set: https://archive.org/details/NESrompack https://archive.org/details/NESrompack
  - beerandt 6y ago
    As far as I know, they limit public access to basically any material with a complaint or request, but keep copies. They may do more, but that seems to be the default response. Which seems smart enough: minimize litigation costs while not losing content permanently via court orders or dmcas or similar threats. But it's frustrating for the wayback machine- where sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles). I don't know what the right answer is, but even if they didn't display the page, it would be nice to see the diffs (ala wikipedia), or at least show if and when changes to a page were made. Maybe that's beyond their mission scope, though.
    - TeMPOraL 6y ago
      > sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles) Is there any way to interpret this behavior as anything other than straight up admitting to being dishonest? Especially in context of accusations that could be easily disproved using the Wayback Machine... unless there is something to those accusations, that is.
- Aeolun 6y ago
  It’s the only organization I actually donate to. Partially because it was so easy to set up a monthly donation ;)
- Cthulhu_ 6y ago
  They could set up a torrent that people could download parts of, I think it's an ideal system for a distributed backup. But it probably needs tweaking for that purpose; for one you need to ensure that the data is evenly distributed, and second you're dealing with data that is appended to regularly. But I think it can be done.
  - Qub3d 6y ago
    It exists! The ArchiveTeam had a project, you can see its status here: http://iabak.archiveteam.org/ http://iabak.archiveteam.org/ Here's an overview: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Sadly, it seems the project has fallen out of repair.
- bane 6y ago
  I've thought for a long time why I absolutely agree with this sentiment and the best I can come up with is that the Internet Archive feels to me like the embodiment of the old internet, not this marketing driven, data steal-and-sell, VC backed cyberpunk dystopia Internet that we have everywhere else. It's the same sort of feeling as I why I enjoy this site, wikipedia, and so on. Bonus, IA is also a bit of an organizational mess, but rewards the adventurer with rich treasure. (I was recently on a Korean history kick and came across not only 1, but several entirely different first hand books written by visitors in the late 19th and early 20th century, scanned in, freely accessible/downloadable, in a variety of formats, and with an excellent on-web reader. These books are so out of print I checked with three local counties for copies and none of them even have references to any of them in their catalog -- treasure!)
- lunchables 6y ago
  There are only two sites I donate to: wikipedia and archive.org. I honestly don't know what I'd do without them. I have been able to find web sites from 20 years ago on archive.org, it's an absolute treasure.
SteveNuts 7y ago
I'm really surprised they don't use more CDN for this. Anyone know the reason why it isn't served by something like cloudflare?
- ajayyy 7y ago
  It could be philosophical
  - 1f60c 7y ago
    That's what I'm thinking. If they wanted to use Cloudflare, I'm pretty sure they'd jump at the opportunity.
- adtac 7y ago
  Please don't make the whole internet basically cloudflare. they've banned my VPN endpoint (Hetzner server) and as a result a huge chunk of websites already don't work for me despite me having done nothing wrong. I've heard reports of Tor users being restricted as well.
  - SteveNuts 7y ago
    Okay, any one of the other CDNs then. It seems like the bandwidth costs and equipment are prohibitively expensive for them. There must be a reason they don't use them which is what I want to know.
    - dylz 6y ago
      Any of the other CDNs are absurdly prohibitively expensive, pay-per-GB, etc. And would still have to download from their source.
  - judge2020 7y ago
    It's more likely that website owners are blocking the ASNs of hosting providers since those are often used for content scraping and exploit type attacks [since they're cheap]. There are public and private lists of hosting provider ASNs you can use to block them all if you generally only want visitors with a business/residential IP.
chx 7y ago
I have revoked my donations until they close their pirate library. Sorry but let's call a spade a spade. They admit: > multiple readers can access a digital book simultaneously with the only caveat being it is borrowed for two weeks but let's face it, most value of a book comes from its first reading -- and what stops you from "borrowing" it again, anyways. It's been a gigantic disappointment for me to see them do this and not back but try to placate the authors with weasel words and an opt out.
- toomuchtodo 7y ago
  I’m sorry to hear this. I will make an additional donation on your behalf. I can appreciate how authors feel, but they must also realize copyright laws are asymmetrical (tilted heavily towards copyright owners) and in these times, many have no access to their local public library. They are essentially robbed of access to the book materials their tax dollars (which pays an author) have paid for during this time. It’s been a gigantic disappointment to see authors respond negatively to this effort (considering this once-in-a-century event), and I will never buy a book again from one of these authors.
  - chx 7y ago
    I also find it amusing that sofware developers do not recognize any more how we are on the same side in this fight. Imagine if we were still selling boxed software and they decided that in light of COVID-19 they just hand it out with a two week limit. It's an imperfect comparison but the current situation is not a reason to just discard the law. It might not be the best law, the place to fight that is in Congress and not on the back of authors most of whom are not in the best financial state to say the least.
    - toomuchtodo 7y ago
      You assume authors are losing revenue from this effort. It is likely this revenue would never have been realized regardless of the Archive’s efforts. A piece of content copied doesn’t mean someone would’ve paid for it. As an aside, many SaaS products have given away their product for free due to COVID and widespread forced WFH. https://www.entrepreneur.com/article/347840 https://www.entrepreneur.com/article/347840
jeromegv 7y ago
The wayback machine has been essential through COVID as many government just publish the numbers and data "of the day" and the only way to compare to the day before is to look at IA.
- makomk 7y ago
  Governments aren't the only ones overwriting old information with new either. The BBC has developed an annoying habit of completely overwriting old articles about Covid-19 with new, semi-related ones, meaning that the only way to see what they were saying earlier in the pandemic is through sites like the Internet Archive. There's also at least one outright correction that only seems to exist on the Internet Archive now: https://web.archive.org/web/20200428214306/https://www.bbc.co.uk/news/health-51943612 https://web.archive.org/web/20200428214306/https://www.bbc.c... "Correction 25 April 2020: An earlier version of this article incorrectly said that France had conducted just under 140,000 tests a day by 21 April. The figure of just under 140,000 refers instead to the number of tests it had carried out weekly."
  - endgame 7y ago
    Here's another one: "some of our old tweets were wrong so we just quietly deleted them lol": https://twitter.com/voxdotcom/status/1242537366620966912 https://twitter.com/voxdotcom/status/1242537366620966912
    - airstrike 7y ago
      Related https://twitter.com/nyt_diff https://twitter.com/nyt_diff
    - gxon 6y ago
      Twitter ought to have some kind of strikethrough feature. Allow users to mark that they no longer stand behind a tweet without completely deleting it.
      - tommica 6y ago
est31 7y ago
60 Gbit/s of continuous traffic is a lot. If I'm reading the graphs right, Wikipedia "only" has 13.4 Gbit/s of outbound traffic [1]. Of course still well below the single-digit Tbit/s traffic of large internet exchanges [2] [3], but still unexpectedly large. [1]: adding up the outbound numbers for each datacenter: 1.888 + 8.003 + 810 + 1.958 + 807: https://grafana.wikimedia.org/d/000000605/datacenter-global-overview?orgId=1 https://grafana.wikimedia.org/d/000000605/datacenter-global-... [2]: https://en.wikipedia.org/wiki/List_of_Internet_exchange_points_by_size https://en.wikipedia.org/wiki/List_of_Internet_exchange_poin... [3]: Not 100% correct as thanks to Corona, ix.br now has passed 10 Tbit/s peaks. https://ix.br/noticia/releases/ix-br-reaches-mark-of-10-tb-s-of-peak-internet-traffic/ https://ix.br/noticia/releases/ix-br-reaches-mark-of-10-tb-s...
- dpcx 7y ago
  I wonder what the comparison between served data is for IA vs Wikipedia - anecdotally, I feel like most of the data for Wikipedia is text based (plus some number of images), whereas IA includes (as mentioned in the article) audio, video, etc.
- cperciva 7y ago
  60 Gbps is about 1/3 of the traffic served by a single Netflix CDN node.
  - notRobot 7y ago
    Source?
  - est31 7y ago
    You are comparing to one of the largest bandwidth using services world-wide. Of course archive.org will lose out. I should have added: 60 Gbit/s is a lot for a donation run service like archive.org. Not sure if there are donation run services on the internet with more traffic.
  - toomuchtodo 7y ago
    > single Netflix CDN node. Looks like they max out at 100 Gbps, or as low as 40 Gbps, depending on appliance model and link aggregation configuration. No argument either way, just thought it was cool info. https://openconnect.zendesk.com/hc/en-us/articles/360034538352 https://openconnect.zendesk.com/hc/en-us/articles/3600345383...
simonmales 7y ago
For the video and audio content. Could they take advantage of something like Webtorrent.
- toomuchtodo 7y ago
  Every item has a torrent file, and can be retrieved with a BitTorrent client. https://help.archive.org/hc/en-us/articles/360004715251-Archive-BitTorrents https://help.archive.org/hc/en-us/articles/360004715251-Arch...
  - simonmales 6y ago
    Brilliant!
Someone1234 7y ago
I donate to the Internet Archive (recurring). I just hope they stick to their core mission, and don't get sidetracked like Wikipedia has. They offer a valuable service and I believe will help historians in the future understand this point in the internet's life. PS - Don't forget to pick an Amazon Smile charity and use the Smile.Amazon sub-domain. Donated almost $20 to the EFF that way.
- notRobot 7y ago
  How has Wikipedia got sidetracked?
  - suyash 7y ago
    I believe the Author is referring to the phenomenon why Wikipedia has lost core contributors due to poor management. https://en.wikipedia.org/wiki/Wikipedia:Why_is_Wikipedia_losing_contributors_-_Thinking_about_remedies https://en.wikipedia.org/wiki/Wikipedia:Why_is_Wikipedia_los...
  - gruez 6y ago
    Most of wikimedia's budget goes towards non-wikipedia related expenses: https://www.washingtonpost.com/news/the-intersect/wp/2015/12/02/wikipedia-has-a-ton-of-money-so-why-is-it-begging-you-to-donate-yours/ https://www.washingtonpost.com/news/the-intersect/wp/2015/12...
- Alex3917 7y ago
  > They offer a valuable service and I believe will help historians in the future understand this point in the internet's life. It's also essential for non-historians. A huge percentage of the info people need just to do their jobs or pursue their hobbies is only found on people's personal webpages, and those presumably all go away once the person dies.
- PaywallBuster 6y ago
  They already kind of have no? They go from archiving the web (simple websites), itself a huge feat, to archiving a bunch of videos from justin.tv as long as they have 10 views? There's more cases of websites shutting down, and AI feeling the need to archive everything. Do we really need it?
tgsovlerkhgsel 7y ago
They increased the bandwidth and traffic immediately filled the new capacity. While this could be just a faster web site attracting more users / users were giving up due to slow loads before, it could also indicate that they have a lot of automated traffic that will consume whatever resources are available and can be throttled without endangering their mission. I hope they'll look into it, find a way to identify that traffic, and then (ideally) classify it accordingly and continue to serve it only after all human traffic has been served (i.e. if they run into a bandwidth crunch due to bots, only the bots suffer, and they don't need to upgrade as quickly).
- manquer 6y ago
  It could also be they are limited by the capacity, For me Wayback machine is always slow , although my bandwidth is otherwise not been a bottleneck. Perhaps increased capacity is getting utilized by both increased usage and higher performance of their services .
- textfiles 6y ago
  We don't save logs of who hits us, but we watch who hits us. It's not bots. But good thinking.
  - twicetwice 6y ago
    How do you know? Not doubting you, just curious as to how you figure out if a request is from a bot or not.
    - textfiles 6y ago
      Well, obviously, a sneaky bot is a sneaky sneak and acts like a person. But conversely, people act in a way that could be like a bot if anyone casually looked at them - mass downloads, no theme or meaning to content. In general, however, it's all pretty much people. Millions of people a day.
ufo 7y ago
The article doesn't mention why archive.org demand went up so much after COVID. Does anyone know?
- gkoberger 7y ago
  I'm not sure, but this article suggests people might just be online more and reading more? http://blog.archive.org/2020/05/11/what-it-means-to-be-a-library-during-covid-19/ http://blog.archive.org/2020/05/11/what-it-means-to-be-a-lib...
- manquer 6y ago
  Not just reading, general increase in internet use, for example they have a lot of old archived games, many official covid health pages only track current numbers so wayback machine is the only way to track first / second derivative data points etc.
- sp332 6y ago
  I don't know if this explains all of it, but they launched a National Emergency Library, where they pooled the DRM'd versions of ebooks from lots of libraries and removed one little bit of the DRM restrictions. http://blog.archive.org/2020/03/24/announcing-a-national-emergency-library-to-provide-digitized-books-to-students-and-the-public/ http://blog.archive.org/2020/03/24/announcing-a-national-eme...
social_quotient 7y ago
What if there was a feature to load the pages without images? I use IA a lot but I rarely need the media stuff, at least initially. Speaking for myself I’d be happy to get the page without images and then click to optionally get the images. (Lazy lazy loading)
- gojomo 7y ago
  Inline images (even across all Wayback Machine usage) are likely a tiny factor, compared to the rich-media downloads (audio, video) dominating this usage.
suyash 7y ago
A suggestion around Donation, see if your employer will match it that way you can double the contribution: https://help.archive.org/hc/en-us/articles/360002059712-How-can-I-get-my-company-to-match-my-donation- https://help.archive.org/hc/en-us/articles/360002059712-How-...
afpx 7y ago
What's the easiest way to seed the most popular content ala torrent?
- __s 7y ago
  ipfs: https://betanews.com/2018/08/09/decentralized-archive-org https://betanews.com/2018/08/09/decentralized-archive-org
  - ghastmaster 6y ago
    I would love to donate bandwidth/storage. I haven't a clue how. I wish there was software I could throw in my server, set how much storage and bandwidth I can donate and run 24/7.
    - betamaxthetape 6y ago
      Every item on archive.org has a bit-torrent file, I believe. This doesn't solve the problem of working out which items are the most popular, but if you can figure out that then you would be able to manage storage and bandwidth. I suppose the real question is: how many other people use bit-torrent to download from archive.org, as opposed to direct downloads. I (sadly) suspect it's a very small fraction. Semi-unrelated, but if you're looking for ways to help and have a spare server, Archive Team [1] is always looking for additional capacity. Although Archive Team != archive.org, they do grabs of at-risk content which (almost always) get uploaded to archive.org. [disclaimer: I help out with various Archive Team projects, the most recent of which was the backup of Yahoo Groups). [1] https://www.archiveteam.org/ https://www.archiveteam.org/
    - sergiotapia 6y ago
      This should be the future of internet media as a whole. You set your limits, and billions of people do the same, suddenly no more single point of failure. Someday
- lowwave 6y ago
  What if to have every item on archive.org also have a webtorrent. I know that webtorrent leaks your private IP, but if an item is requested a lot then it will at least save them bandwidth and be shared amongst people who are requesting the content.
wglane 6y ago
Donated.
VectorLock 6y ago
What a refreshingly simple and wholesome announcement.
x3blah 6y ago
Archive.org works surprisingly well as a general purpose web proxy. Just prefix the URL, e.g., http://example.com http://example.com, with https://web.archive.org/save/ https://web.archive.org/save/, e.g., https://web.archive.org/save/http://example.com https://web.archive.org/save/http://example.com The aesthetic intrusiveness of the archive.org header and footer are minimal since I use a text-only browser that has no Javascript engine. Sometimes I get "This url is not available on the live web or can not be archived." However this happens for only a surprisingly small minority of websites. Rarely I find that /save is unsuccessful in which case I can still find past copies using something like curl -o 1.txt "https://web.archive.org/cdx/search/cdx?url=http://www.example.net&fl=timestamp,original" ; sed -i '/^[12][0-9]* h/!d;/^[12][0-9]* h/{s/^/http:\/\/web.archive.org\/web\//;s/ /\//;s/\r//;}' 1.txt The limitation with past copies versus /save is that archive.org will not usually crawl past page one on websites with many successive pages, e.g., http://example.com/?page=2 http://example.com/?page=2, http://example.com/?page=3 http://example.com/?page=3, etc. Has anyone ever considered mirroring archive.org, or parts of it, to other geographic locations. Could this be done. Why or why not.
- Springtime 6y ago
  I use a custom browser keyword search to find existing archived pages before saving one, personally. Eg: ar <URL> for: https://wayback.archive.org/web/*/%S I'd imagine it would be useful for IA to implement some message for scenarios where a page has already been saved within a certain timespan and provide both a link to the already saved version and offer to save again. As this would mitigate mass savings of an identical page that can occur when some popular link is accidentally shared with the /save/ URL instead of the static URL or when it's a popular page that people want to archive. Archive.is displays such a message (to the effect of, 'this page was archived <date>, if it looks outdated click save') and also redirects to the most recent copy.
  - x3blah 6y ago
    That's a good point. I mainly use it for browsing websites that change daily as well as ones with many successive pages, e.g., 1, 2, 3, etc. that IA does automatically not crawl. Thus, not many existing copies if any. HAproxy changes the Host header and modifies the URL. I can either use the text-only browser's http-proxy option or I can direct the request to the web.archive.org backend by adding a custom HTTP header to the request. If I am not mistaken, the so-called "modern" browsers do not have built-in capability to add headers.
S_A_P 6y ago
Not trying to brag or virtue signal here but this is one cause I really feel great supporting. This is potentially our archive of human history and collective knowledge.
fireattack 6y ago
Kinda off-topic, but I noticed that it seems the owners of the sites can ask Archive.org to remove existing archives. I've lost a few I "created" myself this way. Not say it's not reasonable, but how do we work around this? Any alternative services that are more "resilient" in this regard?
mmmBacon 6y ago
Maybe my sense of scale is warped but I’m amazed at how little BW the archive uses. Since this data appears to be averaged over a 1 minute window, I wonder what their p99 usage is. The article says they bought a 2nd router. Were they only using a single router before (no redundancy or protected BW)? I realize the Archive probably has a small budget but one can buy a 3.2T switch relatively cheaply these days.
teekert 6y ago
I clicked this article to look for ways to contribute.. bandwidth. But there is no torrent/ipfs like method to help, just cash. I'd be happy to run a service that would help Wikipedia, archive.org, ddg (any service I believe in) decentralize. I always keep my Ubuntu torrents open for launch days and some beyond. I have a server running and I don't need my 50/50 fiber all the time. If I was a better software dev, I'd try to make a daemon which I can feed a list of websites which I want to support with my caches that can be upgraded (like the DNS system?). Like a folding at home for bandwidth.
- TeMPOraL 6y ago
  > just cash This is almost always the best and most efficient way you can help a cause. You donating infrastructure means donating them extra work - they'll need to integrate and keep track of it, as well as manage a relationship with you (in particular, a risk of problems with you or your service). Meanwhile, with extra cash, they can buy what they need and what integrates well, or pay the most effective specialists they need. It's an universal principle. Cash is the best gift, if you're gifting to help. That's why e.g. Red Cross frowns at people donating stuff - it creates a huge logistical problem for them as well as depriving them of opportunity to boost the markets in the disaster-struck area. Or why it's better for you to donate money to your local homeless shelter rather than volunteer to work in it - if you work your job for extra hours instead, and donate that money, they'll hire workers better skilled for that task.
  - teekert 6y ago
    But I have bandwidth to share, not cash. I'm already at the lowest subscription level with my ISP. I understand the argument, but if there was a general technical solution this would be nice, right?
    - TeMPOraL 6y ago
      That's true. Does such solution exist, though? I had some hopes for Filecoin (i.e. a P2P system with monetary incentives attached to backing up other people's data, from one of the very few groups in the crypto space that don't look like scammers to me), but I haven't heard anything about it in a while.
PudgePacket 6y ago
I'm curious about the outgoing bandwidth, for crawling :) !
sergiotapia 6y ago
What's the endgame for archive.org? Are they expecting for there to be a breakthrough in storage technology? I read somewhere that data creation is exceeding storage solutions' pace. Is this true? What about a mesh of some kind where every person who install an application hosts bits and pieces of random data and serves it to whoever asks for it?
jl6 6y ago
Mixed feelings here... I love the IA but I feel people should not use them as a streaming service. Their mission is preservation, and access is obviously part of that, but I would feel bad hammering them every day for general TV watching purposes.
- sradman 6y ago
  I assumed that their bandwidth needs were due to their web crawler rather than serving cached content. I guess I’m basing that assumption on my own usage patterns.
lihaciudaniel 6y ago
Sometimes I wonder if people keep track of internet history. For example, we have the writing how Rome and Republic of Rome was only because of Cicero otherwise we wouldn't be able to comprehend. I think the modern day Herodotus is this website right here (assuming it will survive centuries hopefully)
lkramer 6y ago
Crazy idea: Why don't ICANN hand over the running of .org to archive.org? It's a bit outside their expertise, but they do have knowhow on running tech to scale, and the extra revenue could be used to fund a very good cause. Also, I'd trust these guys not to mishandle .org (yeah, I know it's not going to happen)
- jonah-archive 6y ago
  > Also, I'd trust these guys not to mishandle .org As much as I like the idea of running a TLD, if anyone gives me a TLD, I'm gonna put an MX record on it and be jonah@org. That's a promise. EDIT: To be fair, I would probably be overwhelmed with the usual sense of responsibility and not do this. But, the temptation
  - Jleagle 6y ago
    Would be a nightmare with all the crappy email validators out there..
    - myself248 6y ago
      I recall a similar story involving amateur radio. By ITU convention, amateur callsigns are [prefix][digit][suffix], like W1AW. So one night, a new ham is operating his friend's fancier station, which has the HF equipment to talk clear around the world. And they make contact with another ham identifying himself as JY1. No suffix. Well that's weird, so they look it up, the JY prefix is Jordan, okay. So the guy on the mic just asks, "why doesn't your callsign have any letters after the digit?" and the reply comes back with a chuckle, "Oh, because I am the King." That was the late King Hussein. about whom much has been written. I don't know if his call ever caused trouble with software validators, but I'd certainly believe it.
      - lb1lf 6y ago
        I know of a callsign causing trouble with most log programs - RAEM, which Russian amateurs dust off every now and then to commemorate polar pioneer Ernst Krenkel. Took a while to get ARRL Logbook of the World credit for that one.
sktrdie 6y ago
Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html https://sqlite.org/fts5.html Then put the file in a torrent. Let the users seed it. Users can use sqltorrent (https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. Every time a new dump is published by internet archive, the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated. I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 Would save Internet Archive lots of bandwidth and hassle
- contravariant 6y ago
  Seems that you're being downvoted at the moment because it looks like you're making fun of archive.org, but that article is actually rather interesting. How does this compare to something like IPFS?
- Aeolun 6y ago
  A 50PB sqlite database? I don’t think any filesystem would be happy with that.
  - sktrdie 6y ago
    Indeed. But nothing nothing stops them from storing torrents of torrents. The search index would be just that, the text index. Which would point to another torrent storing the actual content. Assets would point to yet other torrents. Would be interesting to learn how it's currently partitioned. I would mimic the same portioning system but use torrent instead so users can help with hosting. And use sqltorrent to serve queries efficiently.
sersi 6y ago
I wish that Archive.org would start collecting funds to finance an external backup site. I strongly feel that archive.org is an irreplaceable treasure and that making sure that it would resist natural disasters should be a priority...
- driverdan 6y ago
  That doesn't require a special fund, donate to them. They'd make backups if they had enough donations to cover it.
  - sersi 6y ago
    I actually already donate about 250 usd a year to them... But would much prefer if there was a special fund I would donate that would make that happen in the future
deleted 6y ago
[deleted]
oknoorap 6y ago
We need an open source competitor of archive.org using IPFS technology
- iofiiiiiiiii 6y ago
  Or an IPFS transfer/mirror mechanism built into archive.org.
whanamura 6y ago
Hey,for those who would like to help the Internet Archive with their excess storage, we are ginning up a project with Filecoin.ai to store some of our open data collections like Prelinger films, .Gov data, some audio and texts. You can get an update on decentralized storage at our DWeb Meetup tomorrow (5/13 at 10 AM pacific, 5 PM UTC) https://www.eventbrite.com/e/dweb-meet-up-virtual-decentralized-storage-comes-of-age-tickets-104783963656 https://www.eventbrite.com/e/dweb-meet-up-virtual-decentrali...

677 points by edward 7y ago

archive.org feels like an irreplaceable treasure, the Wayback Machine alone is a time capsule of our digital history. I donate to them monthly and know a lot of other people do as well, so I don't worry much about their financial stability. I'm more worried about external pressures taking content down. I hope the data is backed up six ways to sunday, and that somewhere there's a plan to make it all accessible if Internet Archive can't continue to play the role it does.

chrisseaton 7y ago
> archive.org feels like an irreplaceable treasure Yes I save some dead sites that I think are critically important resources to tarballs that I'm storing separately in case the Wayback Machine disappears.
- aesh2Xa1 7y ago
  That's useful for personal use. I think that the IPFS is designed for this purpose as well, but meant to be accessible to anyone.
  - chrisseaton 7y ago
    Well the idea is I can provide it publicly (not sure how... don't know how they get the copyright to do it) myself in the future if needed.
- im3w1l 7y ago
  While that may work for your personal usecase, there is a significant issue around trust, if you were to share these tarballs with anyone else. I wouldn't trust you to give me originals in the same way I trust the wayback machine. So even if there are backups lying around here and there, their utility is decreased a lot.
  - fermienrico 7y ago
    By your philosophy, just get off the internet. At some point you gotta trust HN, your keyboard driver, CPU firmware, motherboard, ISP, Google or whatever service you're using. I am not even mentioning your mailman and your amazon delivery driver. Your stance is pretty extreme - please don't discourage people from making backups.
    - bigyikes 7y ago
      I don’t think their stance is extreme as you make it seem. Insert their username into the same list you just provided: im3w1l, HN, your keyboard driver, CPU, firmware, motherboard, ISP, Google One of these things is not like the others.
dmd 7y ago
I'm very worried about its backups and what happens when the next Big One hits SF. As far as I've ever been able to determine from talking to anyone at IA (e.g., Kahle, Scott), they don't really have any sort of backups that could actually be restored from in a disaster situation.
- britmob 7y ago
  There is http://iabak.archiveteam.org http://iabak.archiveteam.org, but it’s not exactly large.
  - microcolonel 6y ago
    If I'm reading that correctly, it would only cost a bit over 500 bucks a month to host that whole archive on BackBlaze B2. Furthermore it would not be so hard to translate Archive.org items to IPFS objects, if there were an effort to pin a significant number of them to storage and network.
    - dmd 6y ago
      You're reading it correctly, but IABAK backs up 0.2% of the Internet Archive.
    - Endlessly 6y ago
      Since numbers are not 100% clear... (50 petabytes * 0.2% = 100 terabytes) [$0.005 ($/GB/Month) BackBlaze cost] [(50 petabytes) / (1 gigabyte) = 50,000,000] (50,000,000 * $0.005 = $250,000 US$) ————— Meaning based on my numbers, that is $250,000 USD a month to host 50 petabytes of data on BackBlaze.
      - ramraj07 6y ago
        That's a fraction of the AWS bills for many startups arguably doing absolutely nothing
        6y ago
erichocean 6y ago
They should team up with Backblaze...
city41 6y ago
They have lots of copyrighted content. Such as virtually every commercial retro video game ever made it seems. I personally think it’s great but surely companies aren’t too pleased? How does archive.org avoid being sued into oblivion?
- judge2020 6y ago
  It's a non-profit library and they don't publicly provide a way to download most of the copyrighted content, so rights holders aren't too concerned. I imagine most people, even copyright lawyers, personally support the free archiving of everything made in the current age as long as it doesn't detract from current business operations and the copyrights are respected for their duration.
  - city41 6y ago
    It's very easy to download the copyrighted content. For example, here is a full NES ROM set: https://archive.org/details/NESrompack https://archive.org/details/NESrompack
- beerandt 6y ago
  As far as I know, they limit public access to basically any material with a complaint or request, but keep copies. They may do more, but that seems to be the default response. Which seems smart enough: minimize litigation costs while not losing content permanently via court orders or dmcas or similar threats. But it's frustrating for the wayback machine- where sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles). I don't know what the right answer is, but even if they didn't display the page, it would be nice to see the diffs (ala wikipedia), or at least show if and when changes to a page were made. Maybe that's beyond their mission scope, though.
  - TeMPOraL 6y ago
    > sites like Snopes and some newspapers have opted out of having their history published (after being accused of ghost edits to articles) Is there any way to interpret this behavior as anything other than straight up admitting to being dishonest? Especially in context of accusations that could be easily disproved using the Wayback Machine... unless there is something to those accusations, that is.
Aeolun 6y ago
It’s the only organization I actually donate to. Partially because it was so easy to set up a monthly donation ;)
Cthulhu_ 6y ago
They could set up a torrent that people could download parts of, I think it's an ideal system for a distributed backup. But it probably needs tweaking for that purpose; for one you need to ensure that the data is evenly distributed, and second you're dealing with data that is appended to regularly. But I think it can be done.
- Qub3d 6y ago
  It exists! The ArchiveTeam had a project, you can see its status here: http://iabak.archiveteam.org/ http://iabak.archiveteam.org/ Here's an overview: https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Sadly, it seems the project has fallen out of repair.
bane 6y ago
I've thought for a long time why I absolutely agree with this sentiment and the best I can come up with is that the Internet Archive feels to me like the embodiment of the old internet, not this marketing driven, data steal-and-sell, VC backed cyberpunk dystopia Internet that we have everywhere else. It's the same sort of feeling as I why I enjoy this site, wikipedia, and so on. Bonus, IA is also a bit of an organizational mess, but rewards the adventurer with rich treasure. (I was recently on a Korean history kick and came across not only 1, but several entirely different first hand books written by visitors in the late 19th and early 20th century, scanned in, freely accessible/downloadable, in a variety of formats, and with an excellent on-web reader. These books are so out of print I checked with three local counties for copies and none of them even have references to any of them in their catalog -- treasure!)
lunchables 6y ago
There are only two sites I donate to: wikipedia and archive.org. I honestly don't know what I'd do without them. I have been able to find web sites from 20 years ago on archive.org, it's an absolute treasure.

SteveNuts 7y ago

I'm really surprised they don't use more CDN for this. Anyone know the reason why it isn't served by something like cloudflare?

ajayyy 7y ago
It could be philosophical
- 1f60c 7y ago
  That's what I'm thinking. If they wanted to use Cloudflare, I'm pretty sure they'd jump at the opportunity.
adtac 7y ago
Please don't make the whole internet basically cloudflare. they've banned my VPN endpoint (Hetzner server) and as a result a huge chunk of websites already don't work for me despite me having done nothing wrong. I've heard reports of Tor users being restricted as well.
- SteveNuts 7y ago
  Okay, any one of the other CDNs then. It seems like the bandwidth costs and equipment are prohibitively expensive for them. There must be a reason they don't use them which is what I want to know.
  - dylz 6y ago
    Any of the other CDNs are absurdly prohibitively expensive, pay-per-GB, etc. And would still have to download from their source.
- judge2020 7y ago
  It's more likely that website owners are blocking the ASNs of hosting providers since those are often used for content scraping and exploit type attacks [since they're cheap]. There are public and private lists of hosting provider ASNs you can use to block them all if you generally only want visitors with a business/residential IP.

chx 7y ago

I have revoked my donations until they close their pirate library. Sorry but let's call a spade a spade. They admit: > multiple readers can access a digital book simultaneously with the only caveat being it is borrowed for two weeks but let's face it, most value of a book comes from its first reading -- and what stops you from "borrowing" it again, anyways. It's been a gigantic disappointment for me to see them do this and not back but try to placate the authors with weasel words and an opt out.

toomuchtodo 7y ago
I’m sorry to hear this. I will make an additional donation on your behalf. I can appreciate how authors feel, but they must also realize copyright laws are asymmetrical (tilted heavily towards copyright owners) and in these times, many have no access to their local public library. They are essentially robbed of access to the book materials their tax dollars (which pays an author) have paid for during this time. It’s been a gigantic disappointment to see authors respond negatively to this effort (considering this once-in-a-century event), and I will never buy a book again from one of these authors.
- chx 7y ago
  I also find it amusing that sofware developers do not recognize any more how we are on the same side in this fight. Imagine if we were still selling boxed software and they decided that in light of COVID-19 they just hand it out with a two week limit. It's an imperfect comparison but the current situation is not a reason to just discard the law. It might not be the best law, the place to fight that is in Congress and not on the back of authors most of whom are not in the best financial state to say the least.
  - toomuchtodo 7y ago
    You assume authors are losing revenue from this effort. It is likely this revenue would never have been realized regardless of the Archive’s efforts. A piece of content copied doesn’t mean someone would’ve paid for it. As an aside, many SaaS products have given away their product for free due to COVID and widespread forced WFH. https://www.entrepreneur.com/article/347840 https://www.entrepreneur.com/article/347840

jeromegv 7y ago

The wayback machine has been essential through COVID as many government just publish the numbers and data "of the day" and the only way to compare to the day before is to look at IA.

makomk 7y ago
Governments aren't the only ones overwriting old information with new either. The BBC has developed an annoying habit of completely overwriting old articles about Covid-19 with new, semi-related ones, meaning that the only way to see what they were saying earlier in the pandemic is through sites like the Internet Archive. There's also at least one outright correction that only seems to exist on the Internet Archive now: https://web.archive.org/web/20200428214306/https://www.bbc.co.uk/news/health-51943612 https://web.archive.org/web/20200428214306/https://www.bbc.c... "Correction 25 April 2020: An earlier version of this article incorrectly said that France had conducted just under 140,000 tests a day by 21 April. The figure of just under 140,000 refers instead to the number of tests it had carried out weekly."
- endgame 7y ago
  Here's another one: "some of our old tweets were wrong so we just quietly deleted them lol": https://twitter.com/voxdotcom/status/1242537366620966912 https://twitter.com/voxdotcom/status/1242537366620966912
  - airstrike 7y ago
    Related https://twitter.com/nyt_diff https://twitter.com/nyt_diff
  - gxon 6y ago
    Twitter ought to have some kind of strikethrough feature. Allow users to mark that they no longer stand behind a tweet without completely deleting it.
    - tommica 6y ago

est31 7y ago

60 Gbit/s of continuous traffic is a lot. If I'm reading the graphs right, Wikipedia "only" has 13.4 Gbit/s of outbound traffic [1]. Of course still well below the single-digit Tbit/s traffic of large internet exchanges [2] [3], but still unexpectedly large. [1]: adding up the outbound numbers for each datacenter: 1.888 + 8.003 + 810 + 1.958 + 807: https://grafana.wikimedia.org/d/000000605/datacenter-global-overview?orgId=1 https://grafana.wikimedia.org/d/000000605/datacenter-global-... [2]: https://en.wikipedia.org/wiki/List_of_Internet_exchange_points_by_size https://en.wikipedia.org/wiki/List_of_Internet_exchange_poin... [3]: Not 100% correct as thanks to Corona, ix.br now has passed 10 Tbit/s peaks. https://ix.br/noticia/releases/ix-br-reaches-mark-of-10-tb-s-of-peak-internet-traffic/ https://ix.br/noticia/releases/ix-br-reaches-mark-of-10-tb-s...

dpcx 7y ago
I wonder what the comparison between served data is for IA vs Wikipedia - anecdotally, I feel like most of the data for Wikipedia is text based (plus some number of images), whereas IA includes (as mentioned in the article) audio, video, etc.
cperciva 7y ago
60 Gbps is about 1/3 of the traffic served by a single Netflix CDN node.
- notRobot 7y ago
  Source?
- est31 7y ago
  You are comparing to one of the largest bandwidth using services world-wide. Of course archive.org will lose out. I should have added: 60 Gbit/s is a lot for a donation run service like archive.org. Not sure if there are donation run services on the internet with more traffic.
- toomuchtodo 7y ago
  > single Netflix CDN node. Looks like they max out at 100 Gbps, or as low as 40 Gbps, depending on appliance model and link aggregation configuration. No argument either way, just thought it was cool info. https://openconnect.zendesk.com/hc/en-us/articles/360034538352 https://openconnect.zendesk.com/hc/en-us/articles/3600345383...

simonmales 7y ago

For the video and audio content. Could they take advantage of something like Webtorrent.

toomuchtodo 7y ago
Every item has a torrent file, and can be retrieved with a BitTorrent client. https://help.archive.org/hc/en-us/articles/360004715251-Archive-BitTorrents https://help.archive.org/hc/en-us/articles/360004715251-Arch...
- simonmales 6y ago
  Brilliant!

Someone1234 7y ago

I donate to the Internet Archive (recurring). I just hope they stick to their core mission, and don't get sidetracked like Wikipedia has. They offer a valuable service and I believe will help historians in the future understand this point in the internet's life. PS - Don't forget to pick an Amazon Smile charity and use the Smile.Amazon sub-domain. Donated almost $20 to the EFF that way.

notRobot 7y ago
How has Wikipedia got sidetracked?
- suyash 7y ago
  I believe the Author is referring to the phenomenon why Wikipedia has lost core contributors due to poor management. https://en.wikipedia.org/wiki/Wikipedia:Why_is_Wikipedia_losing_contributors_-_Thinking_about_remedies https://en.wikipedia.org/wiki/Wikipedia:Why_is_Wikipedia_los...
- gruez 6y ago
  Most of wikimedia's budget goes towards non-wikipedia related expenses: https://www.washingtonpost.com/news/the-intersect/wp/2015/12/02/wikipedia-has-a-ton-of-money-so-why-is-it-begging-you-to-donate-yours/ https://www.washingtonpost.com/news/the-intersect/wp/2015/12...
Alex3917 7y ago
> They offer a valuable service and I believe will help historians in the future understand this point in the internet's life. It's also essential for non-historians. A huge percentage of the info people need just to do their jobs or pursue their hobbies is only found on people's personal webpages, and those presumably all go away once the person dies.
PaywallBuster 6y ago
They already kind of have no? They go from archiving the web (simple websites), itself a huge feat, to archiving a bunch of videos from justin.tv as long as they have 10 views? There's more cases of websites shutting down, and AI feeling the need to archive everything. Do we really need it?

tgsovlerkhgsel 7y ago

They increased the bandwidth and traffic immediately filled the new capacity. While this could be just a faster web site attracting more users / users were giving up due to slow loads before, it could also indicate that they have a lot of automated traffic that will consume whatever resources are available and can be throttled without endangering their mission. I hope they'll look into it, find a way to identify that traffic, and then (ideally) classify it accordingly and continue to serve it only after all human traffic has been served (i.e. if they run into a bandwidth crunch due to bots, only the bots suffer, and they don't need to upgrade as quickly).

manquer 6y ago
It could also be they are limited by the capacity, For me Wayback machine is always slow , although my bandwidth is otherwise not been a bottleneck. Perhaps increased capacity is getting utilized by both increased usage and higher performance of their services .
textfiles 6y ago
We don't save logs of who hits us, but we watch who hits us. It's not bots. But good thinking.
- twicetwice 6y ago
  How do you know? Not doubting you, just curious as to how you figure out if a request is from a bot or not.
  - textfiles 6y ago
    Well, obviously, a sneaky bot is a sneaky sneak and acts like a person. But conversely, people act in a way that could be like a bot if anyone casually looked at them - mass downloads, no theme or meaning to content. In general, however, it's all pretty much people. Millions of people a day.

ufo 7y ago

The article doesn't mention why archive.org demand went up so much after COVID. Does anyone know?

gkoberger 7y ago
I'm not sure, but this article suggests people might just be online more and reading more? http://blog.archive.org/2020/05/11/what-it-means-to-be-a-library-during-covid-19/ http://blog.archive.org/2020/05/11/what-it-means-to-be-a-lib...
manquer 6y ago
Not just reading, general increase in internet use, for example they have a lot of old archived games, many official covid health pages only track current numbers so wayback machine is the only way to track first / second derivative data points etc.
sp332 6y ago
I don't know if this explains all of it, but they launched a National Emergency Library, where they pooled the DRM'd versions of ebooks from lots of libraries and removed one little bit of the DRM restrictions. http://blog.archive.org/2020/03/24/announcing-a-national-emergency-library-to-provide-digitized-books-to-students-and-the-public/ http://blog.archive.org/2020/03/24/announcing-a-national-eme...

social_quotient 7y ago

What if there was a feature to load the pages without images? I use IA a lot but I rarely need the media stuff, at least initially. Speaking for myself I’d be happy to get the page without images and then click to optionally get the images. (Lazy lazy loading)

gojomo 7y ago
Inline images (even across all Wayback Machine usage) are likely a tiny factor, compared to the rich-media downloads (audio, video) dominating this usage.

suyash 7y ago

A suggestion around Donation, see if your employer will match it that way you can double the contribution: https://help.archive.org/hc/en-us/articles/360002059712-How-can-I-get-my-company-to-match-my-donation- https://help.archive.org/hc/en-us/articles/360002059712-How-...

afpx 7y ago

What's the easiest way to seed the most popular content ala torrent?

__s 7y ago
ipfs: https://betanews.com/2018/08/09/decentralized-archive-org https://betanews.com/2018/08/09/decentralized-archive-org
- ghastmaster 6y ago
  I would love to donate bandwidth/storage. I haven't a clue how. I wish there was software I could throw in my server, set how much storage and bandwidth I can donate and run 24/7.
  - betamaxthetape 6y ago
    Every item on archive.org has a bit-torrent file, I believe. This doesn't solve the problem of working out which items are the most popular, but if you can figure out that then you would be able to manage storage and bandwidth. I suppose the real question is: how many other people use bit-torrent to download from archive.org, as opposed to direct downloads. I (sadly) suspect it's a very small fraction. Semi-unrelated, but if you're looking for ways to help and have a spare server, Archive Team [1] is always looking for additional capacity. Although Archive Team != archive.org, they do grabs of at-risk content which (almost always) get uploaded to archive.org. [disclaimer: I help out with various Archive Team projects, the most recent of which was the backup of Yahoo Groups). [1] https://www.archiveteam.org/ https://www.archiveteam.org/
  - sergiotapia 6y ago
    This should be the future of internet media as a whole. You set your limits, and billions of people do the same, suddenly no more single point of failure. Someday
lowwave 6y ago
What if to have every item on archive.org also have a webtorrent. I know that webtorrent leaks your private IP, but if an item is requested a lot then it will at least save them bandwidth and be shared amongst people who are requesting the content.

wglane 6y ago

Donated.

VectorLock 6y ago

What a refreshingly simple and wholesome announcement.

x3blah 6y ago

Archive.org works surprisingly well as a general purpose web proxy. Just prefix the URL, e.g., http://example.com http://example.com, with https://web.archive.org/save/ https://web.archive.org/save/, e.g., https://web.archive.org/save/http://example.com https://web.archive.org/save/http://example.com The aesthetic intrusiveness of the archive.org header and footer are minimal since I use a text-only browser that has no Javascript engine. Sometimes I get "This url is not available on the live web or can not be archived." However this happens for only a surprisingly small minority of websites. Rarely I find that /save is unsuccessful in which case I can still find past copies using something like curl -o 1.txt "https://web.archive.org/cdx/search/cdx?url=http://www.example.net&fl=timestamp,original" ; sed -i '/^[12][0-9]* h/!d;/^[12][0-9]* h/{s/^/http:\/\/web.archive.org\/web\//;s/ /\//;s/\r//;}' 1.txt The limitation with past copies versus /save is that archive.org will not usually crawl past page one on websites with many successive pages, e.g., http://example.com/?page=2 http://example.com/?page=2, http://example.com/?page=3 http://example.com/?page=3, etc. Has anyone ever considered mirroring archive.org, or parts of it, to other geographic locations. Could this be done. Why or why not.

Springtime 6y ago
I use a custom browser keyword search to find existing archived pages before saving one, personally. Eg: ar <URL> for: https://wayback.archive.org/web/*/%S I'd imagine it would be useful for IA to implement some message for scenarios where a page has already been saved within a certain timespan and provide both a link to the already saved version and offer to save again. As this would mitigate mass savings of an identical page that can occur when some popular link is accidentally shared with the /save/ URL instead of the static URL or when it's a popular page that people want to archive. Archive.is displays such a message (to the effect of, 'this page was archived <date>, if it looks outdated click save') and also redirects to the most recent copy.
- x3blah 6y ago
  That's a good point. I mainly use it for browsing websites that change daily as well as ones with many successive pages, e.g., 1, 2, 3, etc. that IA does automatically not crawl. Thus, not many existing copies if any. HAproxy changes the Host header and modifies the URL. I can either use the text-only browser's http-proxy option or I can direct the request to the web.archive.org backend by adding a custom HTTP header to the request. If I am not mistaken, the so-called "modern" browsers do not have built-in capability to add headers.

S_A_P 6y ago

Not trying to brag or virtue signal here but this is one cause I really feel great supporting. This is potentially our archive of human history and collective knowledge.

fireattack 6y ago

Kinda off-topic, but I noticed that it seems the owners of the sites can ask Archive.org to remove existing archives. I've lost a few I "created" myself this way. Not say it's not reasonable, but how do we work around this? Any alternative services that are more "resilient" in this regard?

mmmBacon 6y ago

Maybe my sense of scale is warped but I’m amazed at how little BW the archive uses. Since this data appears to be averaged over a 1 minute window, I wonder what their p99 usage is. The article says they bought a 2nd router. Were they only using a single router before (no redundancy or protected BW)? I realize the Archive probably has a small budget but one can buy a 3.2T switch relatively cheaply these days.

teekert 6y ago

I clicked this article to look for ways to contribute.. bandwidth. But there is no torrent/ipfs like method to help, just cash. I'd be happy to run a service that would help Wikipedia, archive.org, ddg (any service I believe in) decentralize. I always keep my Ubuntu torrents open for launch days and some beyond. I have a server running and I don't need my 50/50 fiber all the time. If I was a better software dev, I'd try to make a daemon which I can feed a list of websites which I want to support with my caches that can be upgraded (like the DNS system?). Like a folding at home for bandwidth.

TeMPOraL 6y ago
> just cash This is almost always the best and most efficient way you can help a cause. You donating infrastructure means donating them extra work - they'll need to integrate and keep track of it, as well as manage a relationship with you (in particular, a risk of problems with you or your service). Meanwhile, with extra cash, they can buy what they need and what integrates well, or pay the most effective specialists they need. It's an universal principle. Cash is the best gift, if you're gifting to help. That's why e.g. Red Cross frowns at people donating stuff - it creates a huge logistical problem for them as well as depriving them of opportunity to boost the markets in the disaster-struck area. Or why it's better for you to donate money to your local homeless shelter rather than volunteer to work in it - if you work your job for extra hours instead, and donate that money, they'll hire workers better skilled for that task.
- teekert 6y ago
  But I have bandwidth to share, not cash. I'm already at the lowest subscription level with my ISP. I understand the argument, but if there was a general technical solution this would be nice, right?
  - TeMPOraL 6y ago
    That's true. Does such solution exist, though? I had some hopes for Filecoin (i.e. a P2P system with monetary incentives attached to backing up other people's data, from one of the very few groups in the crypto space that don't look like scammers to me), but I haven't heard anything about it in a while.

PudgePacket 6y ago

I'm curious about the outgoing bandwidth, for crawling :) !

sergiotapia 6y ago

What's the endgame for archive.org? Are they expecting for there to be a breakthrough in storage technology? I read somewhere that data creation is exceeding storage solutions' pace. Is this true? What about a mesh of some kind where every person who install an application hosts bits and pieces of random data and serves it to whoever asks for it?

jl6 6y ago

Mixed feelings here... I love the IA but I feel people should not use them as a streaming service. Their mission is preservation, and access is obviously part of that, but I would feel bad hammering them every day for general TV watching purposes.

sradman 6y ago
I assumed that their bandwidth needs were due to their web crawler rather than serving cached content. I guess I’m basing that assumption on my own usage patterns.

lihaciudaniel 6y ago

Sometimes I wonder if people keep track of internet history. For example, we have the writing how Rome and Republic of Rome was only because of Cicero otherwise we wouldn't be able to comprehend. I think the modern day Herodotus is this website right here (assuming it will survive centuries hopefully)

lkramer 6y ago

Crazy idea: Why don't ICANN hand over the running of .org to archive.org? It's a bit outside their expertise, but they do have knowhow on running tech to scale, and the extra revenue could be used to fund a very good cause. Also, I'd trust these guys not to mishandle .org (yeah, I know it's not going to happen)

jonah-archive 6y ago
> Also, I'd trust these guys not to mishandle .org As much as I like the idea of running a TLD, if anyone gives me a TLD, I'm gonna put an MX record on it and be jonah@org. That's a promise. EDIT: To be fair, I would probably be overwhelmed with the usual sense of responsibility and not do this. But, the temptation
- Jleagle 6y ago
  Would be a nightmare with all the crappy email validators out there..
  - myself248 6y ago
    I recall a similar story involving amateur radio. By ITU convention, amateur callsigns are [prefix][digit][suffix], like W1AW. So one night, a new ham is operating his friend's fancier station, which has the HF equipment to talk clear around the world. And they make contact with another ham identifying himself as JY1. No suffix. Well that's weird, so they look it up, the JY prefix is Jordan, okay. So the guy on the mic just asks, "why doesn't your callsign have any letters after the digit?" and the reply comes back with a chuckle, "Oh, because I am the King." That was the late King Hussein. about whom much has been written. I don't know if his call ever caused trouble with software validators, but I'd certainly believe it.
    - lb1lf 6y ago
      I know of a callsign causing trouble with most log programs - RAEM, which Russian amateurs dust off every now and then to commemorate polar pioneer Ernst Krenkel. Took a while to get ARRL Logbook of the World credit for that one.

sktrdie 6y ago

Why don't they put their data dumps in an SQLite database and indexed for full-text-search https://sqlite.org/fts5.html https://sqlite.org/fts5.html Then put the file in a torrent. Let the users seed it. Users can use sqltorrent (https://github.com/bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. Every time a new dump is published by internet archive, the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated. I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4 Would save Internet Archive lots of bandwidth and hassle

contravariant 6y ago
Seems that you're being downvoted at the moment because it looks like you're making fun of archive.org, but that article is actually rather interesting. How does this compare to something like IPFS?
Aeolun 6y ago
A 50PB sqlite database? I don’t think any filesystem would be happy with that.
- sktrdie 6y ago
  Indeed. But nothing nothing stops them from storing torrents of torrents. The search index would be just that, the text index. Which would point to another torrent storing the actual content. Assets would point to yet other torrents. Would be interesting to learn how it's currently partitioned. I would mimic the same portioning system but use torrent instead so users can help with hosting. And use sqltorrent to serve queries efficiently.

sersi 6y ago

I wish that Archive.org would start collecting funds to finance an external backup site. I strongly feel that archive.org is an irreplaceable treasure and that making sure that it would resist natural disasters should be a priority...

driverdan 6y ago
That doesn't require a special fund, donate to them. They'd make backups if they had enough donations to cover it.
- sersi 6y ago
  I actually already donate about 250 usd a year to them... But would much prefer if there was a special fund I would donate that would make that happen in the future

deleted 6y ago

[deleted]

oknoorap 6y ago

We need an open source competitor of archive.org using IPFS technology

iofiiiiiiiii 6y ago
Or an IPFS transfer/mirror mechanism built into archive.org.

whanamura 6y ago

Hey,for those who would like to help the Internet Archive with their excess storage, we are ginning up a project with Filecoin.ai to store some of our open data collections like Prelinger films, .Gov data, some audio and texts. You can get an update on decentralized storage at our DWeb Meetup tomorrow (5/13 at 10 AM pacific, 5 PM UTC) https://www.eventbrite.com/e/dweb-meet-up-virtual-decentralized-storage-comes-of-age-tickets-104783963656 https://www.eventbrite.com/e/dweb-meet-up-virtual-decentrali...