9 ms·
At this point, we need a service that "offers" an 8-bay (with 12TB? 14TB? drives) full with the whole ~80TB Anna's Archive. It's essentially all of human knowle
by helloworld42024 2y ago
At this point, we need a service that "offers" an 8-bay (with 12TB? 14TB? drives) full with the whole ~80TB Anna's Archive. It's essentially all of human knowledge and to be frank it belongs to no one - rather...everyone.
People can store this at their house, keep it offline. Just to have these seeds of knowledge everywhere.
...I suppose LLM's trained on this data, essentially their model weights and tokenization are a much more efficient way of storing and condensing this 80TB archive?
- falsaberN1 2y agoThe problem with the later is reliability, or rather it's efficient but unreliable. I'd rather overdo my offline storage and figure out some way to script/code my way into searching it in a convenient way.
- deleted 2y ago[deleted]
- jononor 2y agoIf the content is to be trustworthy then using LLMs to compress it makes no sense.
- deleted 2y ago[deleted]
- miloignis 2y agoIt's possible to do lossless compression with LLMs, basically using the LLM as a predictor and then storing differences when the LLM would have predicted incorrectly. The incredible Fabrice Bellard actually implemented this idea: https://bellard.org/ts_zip/ https://bellard.org/ts_zip/
- rustcleaner 2y agoCan we do this in physics? Use a universal function approximator to approximate the universe, seek Erf(x)>threshold, interrogate universe for fresh data, retrain new universal approximator, ... loop previous ... , universe in a bottle.
- A_D_E_P_T 2y agoYou can do that sort of thing with a toy universe -- in fact Stephen Wolfram has a number of ongoing projects along broadly similar lines -- but you can't do it in our physical universe. Among other reasons, the universe is to all appearances infinite and simultaneously very complex, therefore it is incompressible and cannot be described by anything smaller than itself, nor can it be encapsulated in any encoding. You can make statistical statements about it -- with, e.g., Ramsey Theory -- but you can never capture its totality in a way that would enable its use in computation. For another thing, toy model universes tend to be straightforwardly deterministic, which is not clearly the case with our physical universe. (It is likely deterministic in ways that are not straightforward from our frame of reference.)
- squigz 2y agoYou dropped a 0. Anna's Archive is currently 862.4 TB
- RachelF 2y agoYes, too much for one person, but collectively it is possible to keep it alive. If anyone wishes to help, you can generate a chunk in 1TB units and seed via BitTorrent here: https://annas-archive.gs/torrents https://annas-archive.gs/torrents
- CamperBob2 2y agoHonestly, if I can't have the whole thing, I'm not going to bother mirroring a 1TB fragment that's worthless by itself to everybody except copyright attorneys. As ndriscoll points out, the only feasible way to distribute an archive of this size is with physical hard drives. I sure wish they would find a reasonably-trustworthy way to offer that.
- squigz 2y agoYou are never going to have a physical copy of the archive. It's nearly a petabyte in size.
- ranger_danger 2y agoI know several datahoarders that have at least 1PB, also archive.org grows by that much at least every day
- squigz 2y agoI assumed that GP was an average person who doesn't have a storage array sitting at home. I'm not really sure why the IA is relevant here
- CamperBob2 2y ago
- xk_id 2y agoI’d rather copy the whole thing down by hand, than rely on a bullshit generator for my access to knowledge.
- GolfPopper 2y agoWhen it comes to the evils (and goods) of copyright, it is hard to go wrong with Thomas Babington Macaulay's address to the House of Commons in 1841[1]: "At present the holder of copyright has the public feeling on his side. Those who invade copyright are regarded as knaves who take the bread out of the mouths of deserving men. Everybody is well pleased to see them restrained by the law, and compelled to refund their ill-gotten gains. No tradesman of good repute will have anything to do with such disgraceful transactions. Pass this law: and that feeling is at an end. Men very different from the present race of piratical booksellers will soon infringe this intolerable monopoly. Great masses of capital will be constantly employed in the violation of the law. Every art will be employed to evade legal pursuit; and the whole nation will be in the plot. On which side indeed should the public sympathy be when the question is whether some book as popular as “Robinson Crusoe” or the “Pilgrim’s Progress” shall be in every cottage, or whether it shall be confined to the libraries of the rich for the advantage of the great-grandson of a bookseller who, a hundred years before, drove a hard bargain for the copyright with the author when in great distress? Remember too that, when once it ceases to be considered as wrong and discreditable to invade literary property, no person can say where the invasion will stop. The public seldom makes nice distinctions. The wholesome copyright which now exists will share in the disgrace and danger of the new copyright which you are about to create. And you will find that, in attempting to impose unreasonable restraints on the reprinting of the works of the dead, you have, to a great extent, annulled those restraints which now prevent men from pillaging and defrauding the living." He was decrying the increase in term of copyright to life of the author + 50 years. 1.https://www.thepublicdomain.org/2014/07/24/macaulay-on-copyright/ https://www.thepublicdomain.org/2014/07/24/macaulay-on-copyr...
- matheusmoreira 2y agoThat is a powerful address, indeed. Good thing to know even in 1841 people saw what copyright would become today: an intolerable monopoly granted by the government, functionally infinite. Seriously, this text is so great. I read the entire thing. It's nearly two hundred years old and contains everything one needs to know about copyright in 2024. Thank you for posting it.
- jerojero 2y agoPeople didn't want to buy apples new headset because it was too expensive at 3500 dollars. You think anyone would spend 3000 dollars on such a thing? I doubt it.
- account42 2y agoThere are plenty of gadgets that people are willing to spend over 3k dollars on. So it just depends on what value you thing you can get out of it.