3 ms·
Anna’s Archive approaching 1 petabyte
- futureshock 2y agoThis project is now all the more interesting given what we now know about LLM scaling laws. It will take trillions of high quality, factual tokens to train the next generation of LLMs. Copyright is also appearing to be beside the point. A petabyte of copyrighted data could be run through an LLM and rephrased/summarized in a way that retains the essential knowledge but creates a public domain dataset for further training.
- shrimp_emoji 2y ago> It will take trillions of high quality, factual tokens to train the next generation of LLMs. This is like saying it will take trillions of man-hours to chisel through a mounting with a spoon. You might want to think outside of that box.
- pilimi_anna 2y agoAnna here. In some sense copyright is appearing to be beside the point, but for the exact same reason copyright holders are becoming more defensive, and trying to expand the scope of copyright. That's one of the reasons why this is such a critical window for preservation!
- glimshe 2y agoYou probably hear this a lot, but thank you for fighting the good fight and working towards advancing humanity. Your work is incredibly important.
- dtx1 2y agoI am immensely curious about everything AA is doing but I guess all my questions better remain unanswered for the sake of the project. But perhaps, if you are interested, you might chose to answer some of them. How are you holding up? Do you notice the persecutocutorial pressure on a day to day basis? How is the Project being financed? How is the financing being protected from the legal pressures? Just having one petabyte of data somewhere is expensive. Are KYC an issue for you? I would think that any large enough law enforcement would be able to take down not just your domains but also your servers. Why is it that enforcement seems to focus on domains not on server seizures? What protective measures and backup plans do you have in place? Any Opsec tips to share for small time pirates?
- amy-petrik-214 2y agoAnna's cousin, Amy ZLib here. Anna is doing well, I cannot explicitly "out" her identity but let's say she does well for herself and in a quite safe and stable country. Financing again is a topic where she is strongly positioned but generally asks that people respect her privacy. She does work with a small team of engineers regarding distributed data storages, and is very open to people who want to be mirrors. A petabyte is nothing to her, perhaps a few dozen drives. Underlying system is known as "AAFLOW" I think of it as the next higher elevated version of bittorrent KYC is not an issue for Anna, she prefers Chik-fil-A or Popeye's Law enforcement typically focuses on domain seizure and not server seizures because a domain may trivially be pointed to a backup server, whereas it is non-trivial to point a million end-users to a new domain. Anna doesn't wish to speak in great deal regarding protective measures and backup plans, but let's say there is person after person system after system, plan within plan, at the ready if anything should happen. With regards to opsec tips for small time pirates, don't disclose anything personal on any social channels, or if you do, make it up. Generally it's best to have a labtop you purchased in cash in a non-traceable way, camera and microphone and bluetooth and wifi physically removed, and keep it in a farraday bag generally, but when you need internet access find a public access point. Anna usually runs a pringles can antenna a ways from a coffee shop in an old concrete structure, with a modded usb wifi dongle, but she tries to be random in terms of timing and location when she hits the internet. this type of opsec has served her well.
- dotancohen 2y agoThank you for all that you do. In the fine article, "10,000" is mistakenly written as "1,0000".
- deleted 2y ago[deleted]
- boramalper 2y agoMay want to change its title to the original: The critical window of shadow libraries
- edflsafoiewq 2y agoYes please. The content has vastly larger scope than the current title implies.
- throwaway2562 2y agoAgreed
- carlosjobim 2y agoOther interesting news from the link: They expect improvements in OCR to make it possible for them in the coming years to apply it to their entire library. This would liberate an enormous amount of knowledge for easier access.
- moonchild 2y agothey are talking about treating ocr as lossy. i wonder about making a lossless compression algorithm for text scans based on an ocr; in effect, use the ocr to predict which text will show up and how, and then encode the pixel-level differences on top of that
- eulgro 2y agoDjVu does this to some extent, identifying identifical glyph bitmaps and reusing them for compression. See https://en.m.wikipedia.org/wiki/DjVu#Compression https://en.m.wikipedia.org/wiki/DjVu#Compression
- benxh 2y agoI am assuming this will be solved this year.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- RecycledEle 2y agoA lot of their content is duplicated. De-duping similar uploads of the same version of the same book would cut petabyte down quite a bit.
- ffsm8 2y agoA lot of seemingly duplicate content isn't actually. There are often multiple book publications/revisions that are hard to tell apart as the name/cover stays the same. You only find out by reading the small print at the start were the publisher info/ISBN is