9 ms·

How to become a pirate archivist

579 points by pilimi_anna 4y ago

MasJ 4y ago
As the founder of emuparadise some 22 years ago, I can relate. I got into retrogames because I never got to play those games growing up in India. I thought, well let me archive these games and make them available for everyone else to play. It was wildly successful. At it's zenith EmuParadise was ranked 700 or so as per Alexa on the entire internet. We're talking millions of visitors per day and thousands of active users every single second. I ran it all by myself with an entire team of moderators, contributors, etc. It did have ads. Heck, our server bills were in the range of tens of thousands of dollars a month. How could I pay for that without having ads on the site? Then we're in commercial copyright infringement territory. Basically if you get sued, you can go to prison, and you will be bankrupted for sure. At the time there were no torrents, no IPFS, no distributed hosting solutions in any case. As time went by the stress became enormous. Of course threatening letters and DMCA takedown notices were the norm. And the fact that the site was hugely popular and government agencies such as the FBI could get involved at the behest of Nintendo et al just made it worse. But also keeping it online, through various CDNs, trying to keep it anonymously run at all times (my OpSec was terrible starting out, it started in the year 2000), keeping servers online and uptime to almost 100% and bandwidth flowing and hard drives spinning and RAID arrays working. It was a whole lot of everything all at once and I was just one guy doing it all. After another website Loveroms got sued by Nintendo in 2018 (for $12MM) I decided I had had enough. Reading stories like the kickasstorrents guy getting arrested while on holiday with his wife and kid, loveroms getting sued, I decided that this was the end of the road for me. I pulled all the games from the site. Eighteen years of work down the drain. My mental health had suffered tremendously, I was depressed and anxious almost all the time. The sight of a police officer on the street would set me panicking. The cost was too high. Was it a blast? Oh yes it was. I used to receive thousands of emails from grateful people. Cancer patients who reminisced in their last days playing video games from their childhood, soldiers at war whose only escape was a few rounds of Bomberman (the irony is not lost on me), and so many more beautiful stories of nostalgia and connection. But current copyright law is going to destroy all this art and culture. There is no real legal way to preserve it. And people like me may do it for a long while, but at what cost to ourselves? I firmly believe that a 7-10 year copyright (extendible even somehow? debatable) would be fair and would let authors get what they need out of their creations. It would help us preserve all this beautiful art and culture that we have enjoyed and share it with future generations. I would love for a human kid living on a distant exoplanet in the far future be able to play Chrono Trigger and wonder about the history of the earth and our stories.
- justin_ 4y ago
  I just want to say thank you for creating emuparadise. As a kid (~2005) searching for ROMs online, I remember finding a ton of ad-ridden fake sites, endless demands for "voting" on link aggregator sites, and malware downloads. Emuparadise was like a breath of fresh air compared to those sites, and it basically instantly became my favorite ROM site. While not perfect, it actually had all the games I was looking for, and the community actually seemed to care. I was able to play so many classic games that way that I otherwise never would have had access to. (Including Chrono Trigger :) Emuparadise is also the site that introduced me to BitTorrent, and my very first torrent was downloaded from there. That would get me interested in file sharing. In some way, it's partly responsible for why I'm interested in archiving and links like the OP. I'm sure I'm not the only one. So, thank you for creating such a wonderful library and community back then! It was a great part of my childhood and adolescence, and it showed me how important preservation and sharing can be.
  - MasJ 4y ago
    Oh yes, I always tried to keep it friendly towards our users. There were too many sites out there that just kept you going in loops forever to get to whatever you wanted. Thanks for noticing that :) Our bittorrent tracker was very short lived. It did well and had some pretty good sets on it! But in 2010 when bittorrent was getting a really bad name right after The Pirate Bay case it was easy to get torrent trackers shut down. One day I got a downtime alert, I think it was mid-2010. I checked the site, gone. The server, unresponsive. I got in touch with the host. He said he'll check with the data center. After a while he got back to me and said: "German police came in and seized your servers." There had been no notice, no warning, no nothing. Just boom, and gone. I asked him: "How can they do this? What do we do?". He said: "Nothing, they just come and take whatever they want every now and then." I hired a lawyer in Frankfurt to go and check on the case. He said that they had closed the case with no further process because the person in question was unknown. And he ended that email with: "But Nintendo may try something else". Until that moment, I had no idea that Nintendo was behind the server seizure. I was relieved that the case was closed. Anyway, I still went ahead and resurrected the site sans bittorrent tracker. YOLO and all that. For the next 8 years, we never really had much trouble after that except the usual DMCA takedown notice here and there and a threatening legal letter sometimes. But pressure kept piling up. I did consider myself small and unimportant fish to fry (compared to say, The Pirate Bay or even current gen videogame piracy websites) but that didn't stop them from going after LoveROMs. There was always the chance that one day they would just catch me at an airport or immigration (like the kickasstorrents dude) or something and that would be it. Or the police would just knock at my door. I mean, they would have to know who I was provably but I don't think it would be that hard for a government agency. It was just a matter of time that the powers that be would need to lobby the government to get at me. I didn't want to live my life like that any more.
yamrzou 4y ago
«That secrecy, however, comes with a psychological cost. Most people love being recognized for the work that they do, and yet you cannot take any credit for this in real life.» Feels like the anonymous torrent seeder who keeps seeding a file for years just for the sake of keeping it alive. It's not easy, but some people seem to be able to derive full pleasure from accomplishing the task itself, whether recognition happens or not.
- jrochkind1 4y ago
  I think you can still get a reddit account without giving them even an email address, and definitely not a phone number.
  - midoridensha 4y ago
    Or you can use a fake email address like from mailinator.
    - Freak_NL 4y ago
      I've found Mailinator to be pretty much useless nowadays, despite having been the go to place for, what, a decade? It limited its free domain to only mailinator.com, which is mostly blocked anyway. There are plenty of alternatives out there though.
      - Brooksy3000 4y ago
        You should check out Mailinator's new free tier... You'll get a private domain, access to the API, and avoid the filters that come with the public mailinator.com domain. Here's a link: https://bit.ly/3cs4v5E https://bit.ly/3cs4v5E
  - deleted 4y ago
    [deleted]
  - steve_john 4y ago
    Yes you are right reddit account can get without phone number
ynno 4y ago
I think Alexandra Elbakyan actually did not want to be revealed as the librarian behind Sci-Hub, it was her poor opsec that led to her being identified. Basically her servers were set up to emit detailed error messages from PHP, including full path of faulting source file, which was under directory /home/ringo-ring, which could be traced to a username she had online on an unrelated site, attached to her real name. Before this revelation, she was anonymous.
- pilimi_anna 4y ago
  Did not know that detail. Will add to the post, thank you.
  - userbinator 4y ago
    And from your addendum: So, use random usernames on the computers you use for this stuff, in case you misconfigure something. ...or a username that is so common as to be meaningless, like "user", "Administrator", or even "root".
    - jtsuken 4y ago
      The next step is to change your legal name to Zhang Wei or at least a John Smith.
      - Fnoord 4y ago
        差不多 (chabuduo / chàbuduō)
    - mgdlbp 4y ago
      You get a little obscurity with random though
    - bigiain 4y ago
      "ec2-user"
    - rakoo 4y ago
the-printer 4y ago
Feels as though an organization such as this should have more domain appropriate points of contact than Twitter or Reddit. A very interesting thing nonetheless.
O__________O 4y ago
>> That secrecy, however, comes with a psychological cost. Being acknowledged for someone concerned about OpSec is minor, if not completely unimportant issue. Grind of maintaining OpSec for most is mind numbing in my experience, especially over an extended duration. One minor slip ends it all - and risk of slipping increases relative significance of the related operations, since more eyes increases odds someone will notice something, they’ll be forced into unfamiliar situations, etc. Beyond that, research shows that odds of being discovered grow as more people know: https://www.bbc.com/news/science-environment-35411684.amp https://www.bbc.com/news/science-environment-35411684.amp
mdaniel 4y ago
I'm not in the pirate archivist space, but sections 3 and 5 are relevant to my interests. I've had great luck with ZAP (https://github.com/zaproxy/zaproxy#readme https://github.com/zaproxy/zaproxy#readme) glued to a copy of Firefox (because it allows monkeying with the _browser_'s proxy without having to alter the system one as other browsers do) for archiving all content seen while surfing around a site. It even achieves the stated goal of preserving the HTML (etc) in a database since ZAP uses hsqldb Then, section 5 reads like an advertisement for Scrapy since it is just stellar at following all pagination links and then either emitting the extracted payload as your own data structure and/or by telling Scrapy you want to download some media as-is. It will, by default, put the local content in a directory of your choice and hash the url to make the local filename. A separate json file serves as the "accounting" between the things it downloaded and their hashed on-disk filename Scrapy is also able to glue 3 and 5 together because it has a pluggable (everything, heh) dupe detection hook and also HTTP cache support that can be backed by anything, including the aforementioned hsqldb operating in network mode. Scrapy is also very test friendly, since each method accepts a well known python object and emits either a follow-on request, zero or more extracted objects, or nothing if pagination has ended. Thus, one need not rerun the whole scraping process just to test if a bug has been fixed, or during development I can appreciate there may be other scraping frameworks, but of the ones I've tried Scrapy makes everything that I've asked it to do simple and transparent
- genewitch 4y ago
  This is extremely relevant to my interests. I will try ZAP with a VM that has the explicit purpose of mirroring all content i view "online" within that VM. there's the web archival projects - that i cannot remember right now - that have some sort of proxy front end, but realistically, it should be possible to record the "content" portion of all web interactions, without relying on such dalliances as OCR and screengrabs or even OBS studio or a screen recorder. Sometimes i go on a deep dive of some concept, and when i am done i feel i have a decent enough understanding to explain the concept to an adult, and sometimes i do a deep enough dive to explain to a 6 year old. I'd like to archive the entire "session" that got me there. Ideally as plaintext, but never have i wanted video documentation. I only ever use video to prove to someone that their service is acting up, since audio/visual desktop captures can do that, without cheating and provably.
chatterhead 4y ago
While reading this I realized that the first impression for 'Pirate Archivists' that I was exposed to were the bums in Fahrenheit 451 who memorize books so they can't be burned. I never realized that was my first true introduction to piracy. Really enjoyed the write up!
- totetsu 4y ago
  200 years ago the Grimm brothers collected tales that had been memorized and shared for generations in the German oral tradition and made them into a book of fairy tales. 70 years Walt Disney made some animated movies based on these fairy tales. Today sharing a copy of Cinderella is Piracy.
  - justsomehnguy 4y ago
    > sharing a copy of Cinderella is Piracy with Disney artwork.
    - justshowpost 4y ago
      Moreover, Grimms' copyright has been expired while Disney's copyright is set to never expire.
    - chatterhead 4y ago
      Disney being the artists whose labor and skill they labelled their own and claimed ownership of in perpetuity. Sure, it's "Disney's" artwork.
      - justsomehnguy 4y ago
        I have a problem with Disney's > claimed ownership of in perpetuity but I don't have a problem with labour and skill - D. paid for that labour and skill. Awfully low and abusive, but just like every other corporate, especially in USA.
        4y ago
jancsika 4y ago
I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet connection, one could start downloading the lib right now and see if they win the race against the copyright holders. On the other hand, the What.cd/Redacted approach seems to use Bittorrent ratios to create a private-tracker economy. New users get a few gigs free download on joining. But apparently because a) there's a 1:1 upload/download ratio, and b) a few first-mover fat cats are sitting on enormous ratios, this means there is a scramble by everyone else to upload new FLACs to build up their ratio so they can continue to be able to download FLACs. It seems that would mean the library-in-its-entirety cannot be easily replicated at will. Yet the tracker was apparently already nuked off the internet as What.cd and reappeared later as Redacted. Was any data lost between the two services? Oh yeah, there's also apparently another approach in rutracker, which seems to be blitzkrieg to add content and publish, at the (apparent?) cost of quality of content. It's really a shame that the nerdy, completist domain of digital archiving through torrents isn't covered by fair use. Perhaps we could exclude the most recent 10 years of music so that the hopeful young musician streamers can get paid a few hundred dollars for millions of streams and then receive the silver lining of fair use protection against a label refusing to release one of their albums.
- soulofmischief 4y ago
  Unfortunately, a lot of data is currently yet to be restored. Some of that music might not surface again for decades to come.
- MontyCarloHall 4y ago
  Sci-Hub was interesting because until 2021, it could automatically add any article not in its database by querying proxy servers set up at universities that actually subscribed to the journals, which would then download the PDF from the journal’s website and forward it to Sci-Hub. (This was the approach most academics took in the pre-Sci-Hub days; they’d email friends at other universities and ask them if they had access to a given article.) Sadly, Sci-Hub took down this “magic proxy” to try and win a court case in India, which its creator thinks might legitimize it elsewhere. It’s a huge shame, because it means that many obscure articles are now inaccessible via Sci-Hub. Ironically, a pure proxy-based Sci-Hub that didn’t host any articles on its own might actually be legal in certain countries, since it’s not actually hosting any copyrighted content itself. It definitely would be a lot easier to host and a lot harder to shut down; indeed, it could be completely decentralized.
deleted 4y ago
[deleted]
imhoguy 4y ago
As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, even partially complete. In super extra option the app could even notify me to load DVD because somebody asked for a file which I indexed and advertised previously. For years my program preferences changed, file locations changed - I have moved files around, made them offline, burned on DVDs, deleted some parts of torrent just to keep interesting stuff. Now these torrents are lost, at least my seeding contribution is gone. But I almost never change the content of these files, their checksum stays the same forever, so they could be still discoverable. The digital preservation needs better distribution system.
- londons_explore 4y ago
  The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.
  - chhs 4y ago
    A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there's no lookup mechanism for a "pieces root" to know that torrent B exists and contains file C. If you were to make a v2 torrent of your entire drive, other clients won't know to download from your torrent. They'd need to have the contents of the metadata to know it would contain a file they are interested in, and have no way of knowing which metadata contains the desired "pieces root" entries without downloading all of them. I'm very interested in this problem space, if you are aware of clients/mechanisms that allow for this I would love to hear them.
justshowpost 4y ago
I just dropped here to praise archivists and their merit in general. I treasure content (regardless of its perceived quality) preservation much more than legal or even ethical problems associated with it. Anecdote: Remember when Microsoft Corp. declared what they love open source software and launched CodePlex platform, and then lost their business interest in it (when they bought GitHub) so they completely erased CodePlex archives? I was able to reach several long forgotten project I was interested in thanks to invaluable work of independent volunteer archivists. (It was quite tough manual job for me, I had to d/l database then locate desired archive segment and only then could transfer required files via bittorrent proto)
mrfinn 4y ago
I keep feeling we shouldn't accept the term "piracy" anymore. The problem, the big problem is on the so-called "legal" side, and the purpose of this system is not about retrieving authors anymore, is about some big economic groups hoarding goods (and power by doing that). But that's heavily against the common interest. I met quite a few years ago with a member of my country's senate with a solid proposition to end the "piracy" problem. Got an email asking for more info about my proposal. That was the end of it. PS. Maybe instead "pirates", we should call ourselves "keepers".
- genewitch 4y ago
  The "DNA" of pirate-ish-ness is baked into the US cultural zeitgeist since the 1860s. Freebooters, filibuster, the US excursions into Central America in the 1850s, and the overall Dutch influence on our language and culture still captures some part of us, whether we want it to or not. We're adverse to pirates, even though pirates are universally bad. Unless it's Jack Sparrow. I guess.
standup 4y ago
Having a local copy of an entire ebook archive is one way we can find information without having to use the Internet. Thus we can avoid being subjected to mass surveillance, which is excellent. I wonder if the archive is full text searchable? Finally an alternative to this Orwellian nightmare we call the Internet. Can't wait to have a copy at home, and there will probably be times where I'll be pulling the plug on the router with relief. And it's one more step towards reducing my Internet usage, thus keeping the government and corporations out of my life.
pwdisswordfish0 4y ago
Would you be open to using an apostrophe ’ in your header instead of the straight single quote? I dig the Comic Sans, but it would look so much better. Cheers lol
tenacious_tuna 4y ago
This article has me curious as to most people's "op-sec" around personal piracy practices, e.g. torrenting. Do people take requests from family members? How restrictive are you with these behaviors, especially when backed by something like Plex (which presumably just directly erodes any other opsec you may be practicing).

579 points by pilimi_anna 4y ago

MasJ 4y ago

As the founder of emuparadise some 22 years ago, I can relate. I got into retrogames because I never got to play those games growing up in India. I thought, well let me archive these games and make them available for everyone else to play. It was wildly successful. At it's zenith EmuParadise was ranked 700 or so as per Alexa on the entire internet. We're talking millions of visitors per day and thousands of active users every single second. I ran it all by myself with an entire team of moderators, contributors, etc. It did have ads. Heck, our server bills were in the range of tens of thousands of dollars a month. How could I pay for that without having ads on the site? Then we're in commercial copyright infringement territory. Basically if you get sued, you can go to prison, and you will be bankrupted for sure. At the time there were no torrents, no IPFS, no distributed hosting solutions in any case. As time went by the stress became enormous. Of course threatening letters and DMCA takedown notices were the norm. And the fact that the site was hugely popular and government agencies such as the FBI could get involved at the behest of Nintendo et al just made it worse. But also keeping it online, through various CDNs, trying to keep it anonymously run at all times (my OpSec was terrible starting out, it started in the year 2000), keeping servers online and uptime to almost 100% and bandwidth flowing and hard drives spinning and RAID arrays working. It was a whole lot of everything all at once and I was just one guy doing it all. After another website Loveroms got sued by Nintendo in 2018 (for $12MM) I decided I had had enough. Reading stories like the kickasstorrents guy getting arrested while on holiday with his wife and kid, loveroms getting sued, I decided that this was the end of the road for me. I pulled all the games from the site. Eighteen years of work down the drain. My mental health had suffered tremendously, I was depressed and anxious almost all the time. The sight of a police officer on the street would set me panicking. The cost was too high. Was it a blast? Oh yes it was. I used to receive thousands of emails from grateful people. Cancer patients who reminisced in their last days playing video games from their childhood, soldiers at war whose only escape was a few rounds of Bomberman (the irony is not lost on me), and so many more beautiful stories of nostalgia and connection. But current copyright law is going to destroy all this art and culture. There is no real legal way to preserve it. And people like me may do it for a long while, but at what cost to ourselves? I firmly believe that a 7-10 year copyright (extendible even somehow? debatable) would be fair and would let authors get what they need out of their creations. It would help us preserve all this beautiful art and culture that we have enjoyed and share it with future generations. I would love for a human kid living on a distant exoplanet in the far future be able to play Chrono Trigger and wonder about the history of the earth and our stories.

justin_ 4y ago
I just want to say thank you for creating emuparadise. As a kid (~2005) searching for ROMs online, I remember finding a ton of ad-ridden fake sites, endless demands for "voting" on link aggregator sites, and malware downloads. Emuparadise was like a breath of fresh air compared to those sites, and it basically instantly became my favorite ROM site. While not perfect, it actually had all the games I was looking for, and the community actually seemed to care. I was able to play so many classic games that way that I otherwise never would have had access to. (Including Chrono Trigger :) Emuparadise is also the site that introduced me to BitTorrent, and my very first torrent was downloaded from there. That would get me interested in file sharing. In some way, it's partly responsible for why I'm interested in archiving and links like the OP. I'm sure I'm not the only one. So, thank you for creating such a wonderful library and community back then! It was a great part of my childhood and adolescence, and it showed me how important preservation and sharing can be.
- MasJ 4y ago
  Oh yes, I always tried to keep it friendly towards our users. There were too many sites out there that just kept you going in loops forever to get to whatever you wanted. Thanks for noticing that :) Our bittorrent tracker was very short lived. It did well and had some pretty good sets on it! But in 2010 when bittorrent was getting a really bad name right after The Pirate Bay case it was easy to get torrent trackers shut down. One day I got a downtime alert, I think it was mid-2010. I checked the site, gone. The server, unresponsive. I got in touch with the host. He said he'll check with the data center. After a while he got back to me and said: "German police came in and seized your servers." There had been no notice, no warning, no nothing. Just boom, and gone. I asked him: "How can they do this? What do we do?". He said: "Nothing, they just come and take whatever they want every now and then." I hired a lawyer in Frankfurt to go and check on the case. He said that they had closed the case with no further process because the person in question was unknown. And he ended that email with: "But Nintendo may try something else". Until that moment, I had no idea that Nintendo was behind the server seizure. I was relieved that the case was closed. Anyway, I still went ahead and resurrected the site sans bittorrent tracker. YOLO and all that. For the next 8 years, we never really had much trouble after that except the usual DMCA takedown notice here and there and a threatening legal letter sometimes. But pressure kept piling up. I did consider myself small and unimportant fish to fry (compared to say, The Pirate Bay or even current gen videogame piracy websites) but that didn't stop them from going after LoveROMs. There was always the chance that one day they would just catch me at an airport or immigration (like the kickasstorrents dude) or something and that would be it. Or the police would just knock at my door. I mean, they would have to know who I was provably but I don't think it would be that hard for a government agency. It was just a matter of time that the powers that be would need to lobby the government to get at me. I didn't want to live my life like that any more.

yamrzou 4y ago

«That secrecy, however, comes with a psychological cost. Most people love being recognized for the work that they do, and yet you cannot take any credit for this in real life.» Feels like the anonymous torrent seeder who keeps seeding a file for years just for the sake of keeping it alive. It's not easy, but some people seem to be able to derive full pleasure from accomplishing the task itself, whether recognition happens or not.

jrochkind1 4y ago
I think you can still get a reddit account without giving them even an email address, and definitely not a phone number.
- midoridensha 4y ago
  Or you can use a fake email address like from mailinator.
  - Freak_NL 4y ago
    I've found Mailinator to be pretty much useless nowadays, despite having been the go to place for, what, a decade? It limited its free domain to only mailinator.com, which is mostly blocked anyway. There are plenty of alternatives out there though.
    - Brooksy3000 4y ago
      You should check out Mailinator's new free tier... You'll get a private domain, access to the API, and avoid the filters that come with the public mailinator.com domain. Here's a link: https://bit.ly/3cs4v5E https://bit.ly/3cs4v5E
- deleted 4y ago
  [deleted]
- steve_john 4y ago
  Yes you are right reddit account can get without phone number

ynno 4y ago

I think Alexandra Elbakyan actually did not want to be revealed as the librarian behind Sci-Hub, it was her poor opsec that led to her being identified. Basically her servers were set up to emit detailed error messages from PHP, including full path of faulting source file, which was under directory /home/ringo-ring, which could be traced to a username she had online on an unrelated site, attached to her real name. Before this revelation, she was anonymous.

pilimi_anna 4y ago
Did not know that detail. Will add to the post, thank you.
- userbinator 4y ago
  And from your addendum: So, use random usernames on the computers you use for this stuff, in case you misconfigure something. ...or a username that is so common as to be meaningless, like "user", "Administrator", or even "root".
  - jtsuken 4y ago
    The next step is to change your legal name to Zhang Wei or at least a John Smith.
    - Fnoord 4y ago
      差不多 (chabuduo / chàbuduō)
  - mgdlbp 4y ago
    You get a little obscurity with random though
  - bigiain 4y ago
    "ec2-user"
  - rakoo 4y ago

the-printer 4y ago

Feels as though an organization such as this should have more domain appropriate points of contact than Twitter or Reddit. A very interesting thing nonetheless.

O__________O 4y ago

>> That secrecy, however, comes with a psychological cost. Being acknowledged for someone concerned about OpSec is minor, if not completely unimportant issue. Grind of maintaining OpSec for most is mind numbing in my experience, especially over an extended duration. One minor slip ends it all - and risk of slipping increases relative significance of the related operations, since more eyes increases odds someone will notice something, they’ll be forced into unfamiliar situations, etc. Beyond that, research shows that odds of being discovered grow as more people know: https://www.bbc.com/news/science-environment-35411684.amp https://www.bbc.com/news/science-environment-35411684.amp

mdaniel 4y ago

I'm not in the pirate archivist space, but sections 3 and 5 are relevant to my interests. I've had great luck with ZAP (https://github.com/zaproxy/zaproxy#readme https://github.com/zaproxy/zaproxy#readme) glued to a copy of Firefox (because it allows monkeying with the _browser_'s proxy without having to alter the system one as other browsers do) for archiving all content seen while surfing around a site. It even achieves the stated goal of preserving the HTML (etc) in a database since ZAP uses hsqldb Then, section 5 reads like an advertisement for Scrapy since it is just stellar at following all pagination links and then either emitting the extracted payload as your own data structure and/or by telling Scrapy you want to download some media as-is. It will, by default, put the local content in a directory of your choice and hash the url to make the local filename. A separate json file serves as the "accounting" between the things it downloaded and their hashed on-disk filename Scrapy is also able to glue 3 and 5 together because it has a pluggable (everything, heh) dupe detection hook and also HTTP cache support that can be backed by anything, including the aforementioned hsqldb operating in network mode. Scrapy is also very test friendly, since each method accepts a well known python object and emits either a follow-on request, zero or more extracted objects, or nothing if pagination has ended. Thus, one need not rerun the whole scraping process just to test if a bug has been fixed, or during development I can appreciate there may be other scraping frameworks, but of the ones I've tried Scrapy makes everything that I've asked it to do simple and transparent

genewitch 4y ago
This is extremely relevant to my interests. I will try ZAP with a VM that has the explicit purpose of mirroring all content i view "online" within that VM. there's the web archival projects - that i cannot remember right now - that have some sort of proxy front end, but realistically, it should be possible to record the "content" portion of all web interactions, without relying on such dalliances as OCR and screengrabs or even OBS studio or a screen recorder. Sometimes i go on a deep dive of some concept, and when i am done i feel i have a decent enough understanding to explain the concept to an adult, and sometimes i do a deep enough dive to explain to a 6 year old. I'd like to archive the entire "session" that got me there. Ideally as plaintext, but never have i wanted video documentation. I only ever use video to prove to someone that their service is acting up, since audio/visual desktop captures can do that, without cheating and provably.

chatterhead 4y ago

While reading this I realized that the first impression for 'Pirate Archivists' that I was exposed to were the bums in Fahrenheit 451 who memorize books so they can't be burned. I never realized that was my first true introduction to piracy. Really enjoyed the write up!

totetsu 4y ago
200 years ago the Grimm brothers collected tales that had been memorized and shared for generations in the German oral tradition and made them into a book of fairy tales. 70 years Walt Disney made some animated movies based on these fairy tales. Today sharing a copy of Cinderella is Piracy.
- justsomehnguy 4y ago
  > sharing a copy of Cinderella is Piracy with Disney artwork.
  - justshowpost 4y ago
    Moreover, Grimms' copyright has been expired while Disney's copyright is set to never expire.
  - chatterhead 4y ago
    Disney being the artists whose labor and skill they labelled their own and claimed ownership of in perpetuity. Sure, it's "Disney's" artwork.
    - justsomehnguy 4y ago
      I have a problem with Disney's > claimed ownership of in perpetuity but I don't have a problem with labour and skill - D. paid for that labour and skill. Awfully low and abusive, but just like every other corporate, especially in USA.
      - 4y ago

jancsika 4y ago

I'm curious how Sci-hub's approach compares to the What.cd/Redacted approach. IIUC Sci-hub has scooped up science docs through a good enough UX that it was able to leverage the goodwill of science folks to upload docs (plus whatever other methods it has used to scoop up docs), and it uses a public blitzkrieg-style distribution mechanism. I.e., I guess if one had a big enough harddrive and a fast enough internet connection, one could start downloading the lib right now and see if they win the race against the copyright holders. On the other hand, the What.cd/Redacted approach seems to use Bittorrent ratios to create a private-tracker economy. New users get a few gigs free download on joining. But apparently because a) there's a 1:1 upload/download ratio, and b) a few first-mover fat cats are sitting on enormous ratios, this means there is a scramble by everyone else to upload new FLACs to build up their ratio so they can continue to be able to download FLACs. It seems that would mean the library-in-its-entirety cannot be easily replicated at will. Yet the tracker was apparently already nuked off the internet as What.cd and reappeared later as Redacted. Was any data lost between the two services? Oh yeah, there's also apparently another approach in rutracker, which seems to be blitzkrieg to add content and publish, at the (apparent?) cost of quality of content. It's really a shame that the nerdy, completist domain of digital archiving through torrents isn't covered by fair use. Perhaps we could exclude the most recent 10 years of music so that the hopeful young musician streamers can get paid a few hundred dollars for millions of streams and then receive the silver lining of fair use protection against a label refusing to release one of their albums.

soulofmischief 4y ago
Unfortunately, a lot of data is currently yet to be restored. Some of that music might not surface again for decades to come.
MontyCarloHall 4y ago
Sci-Hub was interesting because until 2021, it could automatically add any article not in its database by querying proxy servers set up at universities that actually subscribed to the journals, which would then download the PDF from the journal’s website and forward it to Sci-Hub. (This was the approach most academics took in the pre-Sci-Hub days; they’d email friends at other universities and ask them if they had access to a given article.) Sadly, Sci-Hub took down this “magic proxy” to try and win a court case in India, which its creator thinks might legitimize it elsewhere. It’s a huge shame, because it means that many obscure articles are now inaccessible via Sci-Hub. Ironically, a pure proxy-based Sci-Hub that didn’t host any articles on its own might actually be legal in certain countries, since it’s not actually hosting any copyrighted content itself. It definitely would be a lot easier to host and a lot harder to shut down; indeed, it could be completely decentralized.

deleted 4y ago

[deleted]

imhoguy 4y ago

As an active hoarder I think there is problem with "6. Distribution: Packaging it up in torrents, announcing it somewhere, getting people to spread it.". I miss a p2p application with torrent packaging and Kademlia like per-file advertising and discovery, where I could point it to my hefty NAS directory of random things and they could be wired to released torrents. This way we could make torrents live much longer, even partially complete. In super extra option the app could even notify me to load DVD because somebody asked for a file which I indexed and advertised previously. For years my program preferences changed, file locations changed - I have moved files around, made them offline, burned on DVDs, deleted some parts of torrent just to keep interesting stuff. Now these torrents are lost, at least my seeding contribution is gone. But I almost never change the content of these files, their checksum stays the same forever, so they could be still discoverable. The digital preservation needs better distribution system.

londons_explore 4y ago
The "v2" torrent file format allows most of this. Some clients have support already. All you'd have to do is make a torrent of your whole hard drive and then seed that. You don't need to publish the torrent anywhere. If anyone else in the world is downloading any other torrent that happens to contain a file you have, they will end up connecting to your machine to download it.
- chhs 4y ago
  A v2 torrent allows clients to identify duplicate files across torrents via the "pieces root" entry in the metadata, so if they're downloading from torrent A and B, and each share file C, they can utilize peers from either swarm. But there's no way for other clients to know that there exists another torrent containing the file they are interested in if they only have the metadata for torrent A. In other words, there's no lookup mechanism for a "pieces root" to know that torrent B exists and contains file C. If you were to make a v2 torrent of your entire drive, other clients won't know to download from your torrent. They'd need to have the contents of the metadata to know it would contain a file they are interested in, and have no way of knowing which metadata contains the desired "pieces root" entries without downloading all of them. I'm very interested in this problem space, if you are aware of clients/mechanisms that allow for this I would love to hear them.

justshowpost 4y ago

I just dropped here to praise archivists and their merit in general. I treasure content (regardless of its perceived quality) preservation much more than legal or even ethical problems associated with it. Anecdote: Remember when Microsoft Corp. declared what they love open source software and launched CodePlex platform, and then lost their business interest in it (when they bought GitHub) so they completely erased CodePlex archives? I was able to reach several long forgotten project I was interested in thanks to invaluable work of independent volunteer archivists. (It was quite tough manual job for me, I had to d/l database then locate desired archive segment and only then could transfer required files via bittorrent proto)

mrfinn 4y ago

I keep feeling we shouldn't accept the term "piracy" anymore. The problem, the big problem is on the so-called "legal" side, and the purpose of this system is not about retrieving authors anymore, is about some big economic groups hoarding goods (and power by doing that). But that's heavily against the common interest. I met quite a few years ago with a member of my country's senate with a solid proposition to end the "piracy" problem. Got an email asking for more info about my proposal. That was the end of it. PS. Maybe instead "pirates", we should call ourselves "keepers".

genewitch 4y ago
The "DNA" of pirate-ish-ness is baked into the US cultural zeitgeist since the 1860s. Freebooters, filibuster, the US excursions into Central America in the 1850s, and the overall Dutch influence on our language and culture still captures some part of us, whether we want it to or not. We're adverse to pirates, even though pirates are universally bad. Unless it's Jack Sparrow. I guess.

standup 4y ago

Having a local copy of an entire ebook archive is one way we can find information without having to use the Internet. Thus we can avoid being subjected to mass surveillance, which is excellent. I wonder if the archive is full text searchable? Finally an alternative to this Orwellian nightmare we call the Internet. Can't wait to have a copy at home, and there will probably be times where I'll be pulling the plug on the router with relief. And it's one more step towards reducing my Internet usage, thus keeping the government and corporations out of my life.

pwdisswordfish0 4y ago

Would you be open to using an apostrophe ’ in your header instead of the straight single quote? I dig the Comic Sans, but it would look so much better. Cheers lol

tenacious_tuna 4y ago

This article has me curious as to most people's "op-sec" around personal piracy practices, e.g. torrenting. Do people take requests from family members? How restrictive are you with these behaviors, especially when backed by something like Plex (which presumably just directly erodes any other opsec you may be practicing).