9 ms·
The Geocities Torrent (~1TB of awesomeness)
- pavel_lishin 16y agoI'd love to be able to search those contents. I'm pretty sure I had a few Geocities sites, but I'm not going to download a terabyte to see if it's in there.
- corin_ 16y agoI imagine at least somebody will download to a server and host them all there. Might grab a new 1TB drive into one of my servers and do it if I'm bored enough...
- jacquesm 16y agoI've been doing that for the last 12 months, and the collection is significantly larger than that. http://www.reocities.com/ http://www.reocities.com/
- bobf 16y agoGreat idea and name!
- pavel_lishin 16y agoHave you grabbed the torrent they're offering?
- jacquesm 16y agoI would be very surprised if there is anything in there that is not in reocities yet but I will certainly do a comparison. I'll have to run that against all the deletion requests as well for this content because chances are fairly large that lots of it has already been removed at the request of the owners.
- a1k0n 16y agoHave you tried de-duping files to save space? I imagine there's a finite number of MIDI files and "under construction" GIFs that could just be symlinked to save a ton of space...
- jacquesm 16y agoSpace is not the problem, I've got plenty of that. The bigger problem is that the underlying filesystem is having some problems dealing with the total number of files. Total disk space usage on the array that holds reocities is about 10.5 T (that includes a master copy though) and runs to 233 million files.
- jacquesm 16y agoOn second reading, I don't actually see a torrent. Did I miss something?
- arst 16y agoWhile I think the web is probably a better place with reocities than without it, slapping ads on this content feels a tad bit slimy to me. I'm curious to hear your take on the ethical and legal implications of it.
- jacquesm 16y agoThe ads are actually a boost for a HN'er. And let's suppose for a second that they were not (as they have not been in the past), reocities has cost a fairly large amount of money to date (instead of made money, as you suggest), not a single person that has asked me to remove the content has ever commented on the presence of the ads, and neither has anybody that has found their stuff again because I backed it up. On the contrary, the reactions have been almost 100% positive with a very few exceptions.
- arst 16y ago(instead of made money, as you suggest) I apologize if that's how you read my post; I certainly didn't intend to suggest that or to belittle the amount of work and money that you've put into this project.
- jacquesm 16y agoIf you put up half of the money that went in to it sight unseen then we'll call it even , deal ? ;)
- pbhjpbhj 16y agoThis is from the Google stable of morals - if people don't know you're profiting from their copyright then it's OK. Presumably you have no legal right to redistribute any of the Geocities stuff? The fact you spent time and money on it is neither here nor there, that's not a moral|legal argument. Presumably there have been some exceptions (you intimate that), what sort of percentage does that amount to? If you calculate that as a true reflection of the whole population then how many people's copyright do you suppose you've infringed, knowingly, against their wishes? If someone takes your published material and slaps ads on it and republishes without your knowledge is that all good with you? (I'm guessing you may say yes here!). It's not just about economics, it's probably as much about moral rights. People believe that their Geocities content died and was buried. Of course they are pretty minor "offences" (or at least appears so) across a large population - akin to being a spammer or somesuch. Actually scratch that this is simply like copying others blog posts on a massive scale.
- vyrotek 16y agoAh ha! So here's where you've all been hiding. http://www.reocities.com/TimesSquare/1266/ http://www.reocities.com/TimesSquare/1266/
- samatman 16y agoThat page is by a young 'hacker' named Simon Liu. I wonder if it is this Simon Liu: http://www.computer.org/portal/web/csdl/doi/10.1109/MITP.2009.109 http://www.computer.org/portal/web/csdl/doi/10.1109/MITP.200... "Surviving Distributed Denial of Service Attacks" See? This is why reocities is awesome. The Simon Lius of the world should have access to their old Descent scores.
- mhd 16y agoAnd after only two random clicks, I'm at a (bad) foot fetish site. Well, it was either that, or a "Mr. T ate my <censored>"…
- brc 16y agoI checked, my very first website is thus far not there. I'll wait until you're finished, and if it doesn't appear I can send the offline backup. I've been carting it around for 10 years, laughing at the animated graphics and frames.
- a_m_kelly 16y agoI'd love to know what sort of infrastructure you're running this on. I'm in a course for my Masters for library school that deals with similar sorts of problems in maintaining and preserving for the "long term" digital materials. From your site it sounds like you wrote a script using wget to harvest the files and another to check them against versions that were still up. What do you do on the server end now to ensure that the files are still working correctly? Are you running periodic checksums on them or the like? Finally, are you looking for any help from an interested novice?
- jacquesm 16y agoI have a large database table that stores the md5 hashes of all the files and there is a script that can compare all of the contents of the site with the hashes in the files (and with a second copy if that's what it would come to). Some bitrot is inevitable but I think it's under control for now. As for help, yes, but right now I'm pretty swamped in other stuff, the next round of work on reocities will likely come after the new year.
- daeken 16y agoHave you considered using something like MogileFS? It'd be perfect for this sort of situation. Let me know if you're interested in this or have any questions -- I've dealt a good bit with systems like this in the past, and would love to give you a hand.
- jacquesm 16y agoYou know where to find me :) And yes, of course I'm interested. But right now no time.
- afterburner 16y agoIt is truly amazing to see one of my old sites reappear in this form. Also, there appears to be another mirror, oocities.com
- flipbrad 16y agoare you worried about copyright infringement suits?
- jacquesm 16y agoNot one bit. It was a situation where the common good outweighed the benefits of respecting copyright law handily, besides that the only thing that changed is the machine where the content was hosted. I do not pass it off as my own, have a very clearly established procedure for removing the data at the request of the copyright holder. Think of it as a hosting provider going out of business and a new one taking over the corpse of the old one. That it happened without contacting the owners is simply because the vast majority of the owners does not have contact information to begin with. See the reocities faq for some more info on this: http://www.reocities.com/newhome/faq.html http://www.reocities.com/newhome/faq.html
- sp332 16y agoYou can replace most geocities.com URLs with reocities.com and it will appear.
- jbail 16y agoWow. This site is amazing. Tons of gems: http://www.textfiles.com/underconstruction/ http://www.textfiles.com/underconstruction/
- tibbon 16y agoHis documentaries (Get Lamp and BBS) are also pretty interesting and well worth watching/purchase. I for one will make this one torrent that I will perpetually seed until my ISP tries to stop me.
- naz 16y agoWas under construction the first web meme? All these people couldn't have arrived at the same metaphor individually (as obvious as it is).
- hartror 16y agoI remember back in `95 building websites that were basically just animated gifs, (no judging i was a kid). I used some under construction ones because I had seen it everywhere and I was constructing!
- lorax 16y agoHard to say if it was the first or not. It may have been "Here's my list of interesting web pages (most of them were probably links to other peoples lists of interesting pages, very recursive.)" I don't think the word meme existed then either.
- enf 16y agoThe word was coined in 1976 by Richard Dawkins, but I don't know when it became well known.
- britta 16y agoCheck out http://www.metafilter.com/85695/Please-Be-Patient-This-Page-is-Under-Construction#2774563 http://www.metafilter.com/85695/Please-Be-Patient-This-Page-... for a person's story of how he made the first animated GIFs.
- mhd 16y agoHow many "web rings" fit into 1 TB?
- roadnottaken 16y agoWow, when did those disappear? That phrase immediately brought me back to ~1994. :)
- mhd 16y agoDisappear? webring.com is still there. I dare you to enter "goth poetry" in its search window. Double dare, as a matter of fact.
- noonespecial 16y agoOh the teenage humanity! Curse you mhd! Curse you!
- _exec 16y agoSkeletal Lovers Two dead people Embraced even in death They lie there for eternity Together forever Memories turned to dust Laughter and sin forgotten Nothing but pale white bone Nothing to complain about Only the two of them Forever together ...
- vaksel 16y agoI don't think they disappeared, they just went into "blog rolls"
- kellishaver 16y agoI had someone email me a few days ago asking for code for an old webring script I'd written in 1998. I was a little amazed that 1. anyone would still want such a thing, and B. someone had found a listing for my 12yr old PHP script and thought it worth using. I haven't had the code in my possession for at least 10 years, and being so old, I'm sure it was full of all kinds of security loopholes. No idea where he came across it, but apparently some resource site somewhere on the internet still lists it.
- cnlwsu 16y agoSounds like a good candidate for an Amazon public data set (http://aws.amazon.com/publicdatasets/ http://aws.amazon.com/publicdatasets/). Although they do peak at 1 TB.
- TamDenholm 16y ago1TB of internet nostalgia.
- Luyt 16y agoArgh, they're gonna revive and store the stuff I was so ashamed for that I hoped it would never resurface.
- Devilboy 16y agoWhat happens on the internet... stays on the internet.
- tiles 16y agoOn reocities.com you can request to have information removed. If you want sites removed from this torrent... better ask now, before the torrent's up.
- pbhjpbhj 16y agoYou can sue them, it is copyright infringement and your moral rights are probably more important than any commercial rights in this case.
- mhansen 16y agoFor those that missed it, here's jacquesm's epic story of backing up geocities. http://reocities.com/newhome/makingof.html http://reocities.com/newhome/makingof.html and the discussion: http://news.ycombinator.com/item?id=903567 http://news.ycombinator.com/item?id=903567 Does this torrent contain the fruits of the collaboration planed in that discussion?
- steve19 16y agoThis may be the largest public torrent ever created!
- Dramatize 16y agoFor anyone in Australia, it would be cheaper to pay them to put the data on a HDD, wrap it in bacon and hand deliver it - rather than download the torrent.
- deleted 16y ago[deleted]
- noonespecial 16y agoReally? I'm seeding it from Adelaide. There are options besides whatever Telstra shovels your way in most areas, you just have to poke around. How would you like your bacon cooked?
- Dramatize 16y agoI like it crispy.
- mahmud 16y agoDude, get yourself a $50/mo TPG account. Unlimited. (sometimes including landline phone!) That was in Sydney though, last year.
- rwhitman 16y agoI like the concept of "Digital Heritage" as they call it. I think preserving digital heritage may become an important issue in the next few years. As the world begins to recognize the historic legacy of the web, it may someday become as important as preserving physical historical landmarks. Its time to accept that the internet may be the greatest legacy 21st century civilization leaves behind. It likely isn't going anywhere, and is the most complete archive of our lives, that may live on for generations after we're dead. For all we know, the things we type here could be preserved longer than the pyramids of giza should someone make sure to back it up regularly. I hope that someone does.
- stevenbedrick 16y agoCharles Stross has an excellent novel, "Glasshouse", a major plot point of which is that due to shoddy digital preservation practices, far-future historians know virtually nothing about ≈1950 through ≈2050 or so. What little they know is pieced together from fragmentary bits and pieces of evidence, with results that are at turns hilarious and horrifying when "put into practice" by historical re-enactors. Kind of reminded me of David Macauly's "Motel of the Mysteries" in some ways.
- SoftwareMaven 16y agoI'll second the positive review of "Glasshouse." The lack of knowledge portrayed in the book is also an interesting commentary on one of the often unsung evils of DRM. That license server certainly won't be running 300 years from now.
- eru 16y agoHistorians learn most about ancient societies by digging through their trash. We have left more trash that takes ages to degrade than all prior societies put together. Only if they get serious about recycling, history will be at stake.
- Jach 16y agoSounds like a great excuse to test those "Unlimited Diskspace!" and "Unlimited Bandwidth!" claims of shared providers.
- vaksel 16y agothey don't work, the second you start getting any serious traffic you get a warning. I had a video(flash movie) that got popular, and they sent me a warning after only 2 gigs of bandwidth. Granted those 2 gigs were used up in something like 5 minutes, but never the less..you'd think they'd give you a little more to play with.
- Mrcurious 16y agoThe a51 tables used for cracking gsm crypto are 2TB of awesomeness. http://reflextor.com/trac/a51/wiki/TableStructure http://reflextor.com/trac/a51/wiki/TableStructure
- wenbert 16y agoI remember starting out in Geocities and then in Angelfire. Those where the days when you had to submit your site to the Yahoo directories :D I actually made money selling ads from Commission Junction back then. It wasn't much but it felt great.
- mrschwabe 16y agoYEAH my first website lives on!
- scrame 16y agoThis is fantastic. The early web is a treasure trove of an interesting time in history. It was the first time average people could just write public documents to express themselves. Naturally the pages were terrible, covered in things that look good the first time you see it, pointless opinions and personal shrines to obscure relics of pop-culture. The web is still the same, but more everyday. Companies work day and night to have a web presence, and "using the internet" is synonymous with replying to status and 'liking' things. Geocities, AOL Homepages, and tripod are landmarks of the first time in history someone could just make a page about themselves, or something they liked and _anyone_ could see it. It was society making paintings on caves. Unfortunately, these sites don't produce revenue, and never will, so from a corporate point of view, they are worthless. The early era of the web is like trying to find rare music. Of course there is a modern site, a torrent, or some convenient way to find most of what you want. What you find is at best, the same thing everyone else finds. The old web is full of non-technical people earnestly trying to make something, not a startup, not to sell a book, just trying to put something together which is largely lost in the ease of "List your favorite bands" Not that it was better, or more insightful, simply that it is a huge body of primitive work that is unlikely to be recreated. These things should be stored, if for no other reason than we can see the bloviated opinions of mensans, the C-style poetry of 90's sysadmins, or just the insane ramblings of people who think like Gene Ray, but don't have the perseverence to keep up timecube. The sites are a labor of love, no matter the revenue, and it annoys me to no end that AOL or Yahoo has the power to simply delete these old sites because they don't make business sense, to businesses that don't even know what they are doing. Anyway, as someone who mirrored a few old HomePages and Geocities sites, and backs up pieces of the old internet whenever I can find them, this is a breath of fresh air.
- rakkhi 16y agoI remember creating my first site on geocities. A southpark fan site with links to download episodes (linked to another site hosting the of course). That obviously became the most popular feature. Think will have to get the torrent or at least part of it for a trip down memory lane. It is the digital equvilent of the 80's haircut
- liedra 16y agoI'm actually sad that I didn't put my first website on geocities now! I had my own web hosting from my ISP, so I had a fabulously easy to remember url of "homepages.tig.com.au/~liedra" which was lost as soon as my family upgraded to a cable connection from dialup. And no, archive.org didn't manage to catch it :( I think it had a page devoted to Nick Cave and some terrible poetry! Go go websites of a 17 year old! :) The interesting thing was that at the time my friends and I (who had ISP-based homepages) looked down on Geocities because it was "lame" comparatively. Now I'm sad that I don't have any records of that original page (possibly on an ancient CD-R though? but most of those early ones have degraded now...)
- kingofspain 16y agoMy first pages were on my ISP which offered a subdomain! I paid £2 a month extra (on top of call charges) for the privelige. My friends thought I was crazy but I showed them when that very same subdomain impressed someone enough to give me a web dev job ("You have a subdomain? Impressive"). I also looked down on geocities/angelfire sites and I still think I got the better deal out of it - my first stuff was too embarrassing to live on for eternity in the depths of a torrent.
- micheljansen 16y agoYay for the preservation of a generation. My early attempts of hacking together something in Notepad as a kid should be in there somewhere. Now if only Yahoo donated the geocities.com domain to whoever has the guts to keep this archive online...
- jbillingsley 16y agoI can't for the life of me remember what I put out on geocities but it was probably something to do with Star Wars. I think my username back then was Fett82. Good times...
- textfiles 16y agoHey, everyone, Jason Scott (the textfiles.com guy) here. Just wanted to address that reocities.com has even more than I do, and more than what's in the torrent. If you want to browse geocities, like ye old days, go visit reocities. This data release is never meant to be "all of geocities" just "a lot of geocities" (and all I have). I am ALL for a 2.0 from jacquesm. :)
- sz 16y agoDid Alex Stefanov's collection of free online math resources make it in? It was an awesomely comprehensive list.