11 ms·

Academic Torrents: A distributed system for sharing enormous datasets

552 points by iamjeff 10y ago

tombert 10y ago
I'm pretty glad that torrents have started to break out of the "it's only for warez" stereotype. It's a useful technology, regardless of what made it popular.
- NelsonMinar 10y ago
  That happened around 2008. Or even earlier; Linux distros were being distributed via Bittorrent back in 2006. https://torrentfreak.com/popular-linux-distro-torrents/ https://torrentfreak.com/popular-linux-distro-torrents/
  - swiley 10y ago
    When I was younger I would only download these via bittorent because our connection was extremely unreliable and bittorent was always restartable.
    - rck404 10y ago
      Even now I do the same. In india direct downloads are slightly slower. I guess it's to do with the mirror location or something else. Torrent makes it much faster
      - kochthesecond 10y ago
        Probably because tcp or http or both are throttled somewhere. Bit torrent uses any port and lots of connections so throttling is much less effective.
      - mads 10y ago
        Same in China. The Chinese firewall throttles TCP connections making direct downloads very slow. Not uncommon to have an unbearable slow connection while browsing web pages and then when starting a torrent download, you immediately get 2 Megabytes per second download speeds here. A tip is to use a tool like axel (https://github.com/eribertomota/axel https://github.com/eribertomota/axel) to do direct downloads. This will usually speed it up.
        voltagex_ 10y ago
        >My name is Eriberto and I am not a C developer. I imported Axel from its old repository[1] to GitHub (the original homepage and developers are inactive) Uh oh. Thanks to the repo owner for updating the README, but that's not a good situation. I'd also like to suggest aria2c for this purpose: https://aria2.github.io/ https://aria2.github.io/
  - jamesrcole 10y ago
    I think the grandparent was talking about the general perception of torrenting
  - tombert 10y ago
    Oh, I know that programmers and hackers and whatnot knew torrents were cool; I'm talking about non-technical people. For example, until fairly recently, if I mentioned a "torrent" to my non-technical mom, she would assume I meant ThePirateBay or something like that. Nowadays, she knows it as just another means to download files.
    - NelsonMinar 10y ago
      Not sure Academic Torrents is the best example of a use of torrents for "non-technical people".
      - tombert 10y ago
        That's a fair point; I guess what I was trying to get at is that when I worked for NYU, I think it would have been an incredibly tough sell to use torrents in any capacity, because of the stigma of piracy. However, I think if I were to pitch it now, there would be serious consideration.
  - revelation 10y ago
    Actually, World of Warcraft was released in 2004 and I think it used BitTorrent for patch distribution from the beginning. So very legitimate uses date quite some time back (as you would expect).
  - gens 10y ago
    I remember a game called "GunZ" (the 2005 one, i think) internally using torrents to propagate updates. But it was a long time ago.
    - Houshalter 10y ago
      I saw a game do that recently. It makes a lot of sense to use torrents and reduce bandwidth on their servers.
    - dansze 10y ago
      A huge number of games used torrents for patching since around that time, notably in the MMO scene. WoW and every Nexon game come to mind. AFAIK the Battle.net launcher still downloads updates via torrent.
      - dsl 10y ago
        The Blizzard updater is actually a very cool download utility, worth hacking/poking at. AFAIK it pioneered the concept of "web seeds", using HTTP GETs with a Range: header to download specific chunks from a CDN that were not healthy/available in the swarm.
    - tombert 10y ago
      I completely forgot about GunZ doing that. I guess I was wrong; torrents have been taken seriously for awhile :D
  - PhasmaFelis 10y ago
    None of that changed the popular stereotype. And Linux distros, etc. have never been more than a fraction of the volume of warez torrents.
- adamnemecek 10y ago
  Isn't this technically warez?
  - apetresc 10y ago
    How so? It's meant to be used by researchers distributing the datasets for their own papers. (Notwithstanding the fact that people seem to already be uploading textbooks/lectures, which seems against the stated intent).
    - adamnemecek 10y ago
      Some of the articles seem to be journal articles. Don't get me wrong, I'm not against it, but isn't this partially what Aaron Swartz was indicted for?
  - swang 10y ago
    Since it isn't illegally acquired and distributed software, no.
- metaphor 10y ago
  Agreed...although it is rather ironic that "CrackStation's Password Cracking Dictionary"[1] tops their list of most downloaded by a respectable 9% over 2nd most downloaded "Arizona State University Twitter Data Set"[2]. [1] http://academictorrents.com/details/fd62cc1d79f595cbe1de6356fb13c2165994e469 http://academictorrents.com/details/fd62cc1d79f595cbe1de6356... [2] http://academictorrents.com/details/2399616d26eeb4ae9ac3d05c7fdd98958299efa9 http://academictorrents.com/details/2399616d26eeb4ae9ac3d05c...
- radarsat1 10y ago
  I wish it was use more for distributing Linux package updates, a la DebTorrent. https://wiki.debian.org/DebTorrent https://wiki.debian.org/DebTorrent This could take off it only a big player like Ubuntu pushed it. I don't see why we depend on a set of centralized servers for a bunch of files that a huge number of people download on a very regular basis. And yet, the idea has been stagnant for years. Edit: By which I mean, it works, but not enough people use it.
  - tombert 10y ago
    I totally agree; what really bothers me about current OS's is that they all depend on something centralized, so if, for example, Canonical went broke, I wouldn't be able to install new packages on my Ubuntu laptop. I'm aware that you can swap out the PPAs as needed, but I would really like something distributed and decentralized.
  - rakoo 10y ago
    It's all listed in your link, but the biggest reason is that Bittorrent was inherently built to distribute static content, and the packages is an ever-changing list. Which means that you'd have upstream servers constantly calculating new torrent files and distributing them. Moreover Bittorrent makes it real easy for an observer to know what packages you're installing, and thus what version you have. See this alternative that uses similar idea but not real Bittorrent, they worked around the 1st problem: http://www.camrdale.org/apt-p2p/ http://www.camrdale.org/apt-p2p/
    - radarsat1 10y ago
      So not entirely stagnant ;) I still don't see why Ubuntu/Debian/et al don't take this (or something like it) up in a more official manner. I can see why it's not a default of course, but it could be made a question during installation for example. An additional benefit would be that you'd be able to source packages from machines on your local network, with fallback to the internet, and it would all be pretty much automatic and configuration-free.
      - 10y ago
- teej 10y ago
  At least back in 2010, Facebook and Twitter were both reportedly using torrents to do code deploys. https://blog.twitter.com/2010/murder-fast-datacenter-code-deploys-using-bittorrent https://blog.twitter.com/2010/murder-fast-datacenter-code-de... https://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/ https://torrentfreak.com/facebook-uses-bittorrent-and-they-l...
- vesinisa 10y ago
  Are you serious? This is the original purpose of BitTorrent. It was not invented as a "warez technology", but as a way for service providers to save on bandwidth costs when serving huge files. Back when BitTorrent spec was written hosting a collection of large datasets like this meant paying thousands of bucks in hosting expenses, making it prohibitively expensive for most independent developers.
- sametmax 10y ago
  Yes. It should be built in in browsers.
  - yread 10y ago
    Opera used to have it, but it was much slower than a dedicated client
- paavokoya 10y ago
  And yet, Comcast still illegally throttles torrent traffic..
- rck404 10y ago
  Torrents and mesh networks were breaking out to be the future of internet during mid and late 90s (Also why Skype was P2P and why it was successfull). I'm not sure what happened to it though.
- kilotaras 10y ago
  Torrent (or p2p in general) when used for distribution a la Blizzard can be a win-win-win scenario if setuped correctly. 1. Consumers can get higher speed in ISP network. 2. ISPs get lower external bandwith usage 3. Lower resource usage for distributor. Relevant HN discussion - https://news.ycombinator.com/item?id=12380797 https://news.ycombinator.com/item?id=12380797
mtgx 10y ago
.com? Is that wise? I imagine the domain will be seized as soon as the site becomes popular enough. They should at least have some contingency plan to deal with that.
- ORioN63 10y ago
  Torrents don't mean illegal content.
  - deleted 10y ago
    [deleted]
  - sandworm101 10y ago
    In his defence, it doesn't really matter whether the content is legal or not. There are people out there who see the torrents as evil. These people have political power and have been known to disrupt and seize .com domains on the scantiest of evidence. A more resilient tld would be a good idea.
    - hueving 10y ago
      Do you have an example of a domain being seized for hosting legitimate torrents?
- Ianvdl 10y ago
  Why? Nothing indicates that this is illegal?
- angusp 10y ago
  If it is as they claim well academically supported, you'd assume they'd be easily able to get a subhuman on someone's .edu or.ac.uk or whatever
  - jedanbik 10y ago
    I believe a .org would be most appropriate here. Thinking about http://opendap.org/node/305 http://opendap.org/node/305 for context. [What is OPeNDAP: http://opendap.org/about http://opendap.org/about]
rakoo 10y ago
If only WebTorrent (https://github.com/feross/webtorrent https://github.com/feross/webtorrent) worked with standard bittorrent protocol instead of a custom one on top of WebRTC, we would have live access to all the papers and "displayable" data directly instead of firing up a torrent downloader just for a small file.
- sktrdie 10y ago
  I don't think you understand how WebTorrent works. WebTorrent in fact works with the regular BitTorrent network if you run it from node, and falls back to use WebRTC when used in the browser. So you can seed those torrents directly in the browser with something like instant.io.
  - rakoo 10y ago
    I do know how WebTorrent works. The problem is that it creates a completely parallel network of nodes, which on the surface happen to exchange the same framing of messages, so: - When WebTorrent runs on the standard bittorrent network from node, that doesn't change anything: it's still not available from the browser. - When WebTorrent runs on the WebRTC network through instant.ion or anything else, it will only work if somebody else is also seeding the same torrent in their browser. Which they can only have in the browser if they first got it somewhere else. Oh and I'm willing to bet that none of the nodes who currently have the content (ie on the bittorrent network) also share it on the WebTorrent network. I don't expect classic bittorrent peers to ever implement the mess that is WebRTC just to accommodate browsers, unfortunately.
    - forgotpwtomain 10y ago
      Webtorrent on Node talks to both types of peers.
      - nathcd 10y ago
        That's not quite correct. From https://github.com/feross/webtorrent-hybrid https://github.com/feross/webtorrent-hybrid: > In node.js, the webtorrent package only connects to normal TCP/UDP peers, not WebRTC peers. That's why there's this webtorrent-hybrid client which runs a hidden electron process to communicate with WebRTC peers and normal TCP/UDP peers. According to the readme, there's (understandably; it's running chromium in the background) a lot of overhead with this method so they're working toward a non-electron version of WebRTC in Node.
Philipp__ 10y ago
This is amazing! Thank you for sharing this!
degenerate 10y ago
I already knew about Academic Torrents, but didn't know about Kaggle, which was linked[1] from one of the 'popular' data sets on AT: https://www.kaggle.com/c/yelp-restaurant-photo-classification https://www.kaggle.com/c/yelp-restaurant-photo-classificatio... It looks like one of those logo-design-competition-sites, but for big data. Anyone compete in one of these? [1]: http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0d36f812e0f1b87bc http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0...
jakub_h 10y ago
I'm wondering if torrents as such are actually useful for this. I'd figure some kind of virtual file system (perhaps based on BitTorrent) would be very useful. You'd simply pass a file path to an open() routine in your scientific code and data would get opened transparently. You currently have this with URLs and HTTP but there's no useful caching or data distribution.
- rakoo 10y ago
  If you have the torrent file, you can use one of the things that mount them as virtual file systems, like btfs (https://github.com/johang/btfs https://github.com/johang/btfs).
- seanp2k2 10y ago
  You might like https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/about.rst https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/ab... , https://urbit.org/docs/using/filesystem/ https://urbit.org/docs/using/filesystem/ , or https://ipfs.io https://ipfs.io BTSync and SyncThing are also tools to do this, and I'm sure there are FUSE things to work with BT and block chains ("bittorrent fuse" google results look promising).
  - espadrine 10y ago
    With the same principles, http://infinit.sh/ http://infinit.sh/ adds security and editability to the mix.
  - jakub_h 10y ago
    Lots of interesting things to read, thank you! (To the others as well.)
- zielmicha 10y ago
  IPFS (https://ipfs.io https://ipfs.io) does that.
- Eridrus 10y ago
  Filesystems have distribution problems. You don't always have permissions to install a new filesystem.
babak_ap 10y ago
Anyone looking for Wireless Data, I suggest taking a look at: http://crawdad.org/ http://crawdad.org/ and http://www.cise.ufl.edu/~helmy/MobiLib.htm#traces http://www.cise.ufl.edu/~helmy/MobiLib.htm#traces
danielmorozoff 10y ago
This is fantastic. I am so glad someone built this.
rkda 10y ago
You might also be interested in dat. They're trying to solve this problem as well. http://dat-data.com/ http://dat-data.com/ "Dat is a decentralized data tool for distributing data small and large."
WhitneyLand 10y ago
API suggestions: 1). Don't use hard coded values for types >GET /apiv2/entries?cat=6 -- List entries that are datasets >GET /apiv2/entries?cat=5 -- List entries that are papers These could be written as: /apiv2/entries/datasets /apiv2/entries/papers 2). You may not need path elements like entries, entry, collection, and collection name. For example further simplification would leave /datasets /papers 3). Don't use capitals like this, switch to lowercase: /apiv2/entry/INFOHASH 4). Use HTTP verbs in a standard, semantic way. For example this POST /apiv2/collection -- create an collection POST /apiv2/collection/collection-name/update POST /apiv2/collection/collection-name/delete POST /apiv2/collection/collection-name/add POST /apiv2/collection/collection-name/remove This could all be collapsed into one form and would in turn be more familiar to developers.
- jnbiche 10y ago
  I strongly agree with these suggestions, if the creator(s) happen to read this. And while these suggestions may seem like nitpicking, they are not. While not everything around REST APIs is entirely standard, there's a great deal of agreement about proper resource naming and use of HTTP verbs. If you follow these widely-used standards, not only will developers find it much easier to interact with your API, but also there is a lot of tooling and client libraries out there built specifically to work with APIs that use HTTP verbs and URIs in the standardized ways.
  - iamjeff 10y ago
    Hi. I found this site late last night while looking for material for a hobby project. Really, really awesome site. I also found the creators here [1] and here [2]...maybe you can help shoot them an e-mail so that they can acknowledge a lot of the praise on this thread and the flood of helpful suggestions. Additional contributors can also be found here [3] as well as related publications/presentations [4] including a massively interesting Reddit discussion [5]. [1]- Joseph Paul Cohen https://twitter.com/josephpaulcohen https://twitter.com/josephpaulcohen joseph@josephpcohen.com http://josephpcohen.com/w/ http://josephpcohen.com/w/ https://www.linkedin.com/in/josephpcohen https://www.linkedin.com/in/josephpcohen https://github.com/ieee8023 https://github.com/ieee8023 https://www.reddit.com/user/ieee8023 https://www.reddit.com/user/ieee8023 [2]- Henry Z Lo henryzlo@cs.umb.edu https://www.linkedin.com/in/henry-z-lo-6a11461b https://www.linkedin.com/in/henry-z-lo-6a11461b https://github.com/henryzlo https://github.com/henryzlo [3] http://academictorrents.com/about.php http://academictorrents.com/about.php [4]- Academic Torrents: A Community-Maintained Distributed Repository (http://dl.acm.org.sci-hub.cc/citation.cfm?id=2616528&dl=ACM&coll=DL&CFID=661488745&CFTOKEN=24433760 http://dl.acm.org.sci-hub.cc/citation.cfm?id=2616528&dl=ACM&...) - Academic Torrents - Simple Pitch (https://docs.google.com/presentation/d/1JC2d1g9U6HaenGSn_Xvkl7FXyNo5LgVpLFQqCrNgF6k/edit#slide=id.p https://docs.google.com/presentation/d/1JC2d1g9U6HaenGSn_Xvk...) - Academic Torrents: Scalable Data Distribution (http://arxiv.org/pdf/1603.04395v1.pdf http://arxiv.org/pdf/1603.04395v1.pdf) [5]- I have every publicly available Reddit comment for research. ~ 1.7 billion comments @ 250 GB compressed. Any interest in this? (https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...)
wodenokoto 10y ago
A lot of these "data sets" appears to be coursera courses. I'm not sure if those are legal to redistribute. It also clutters the browse function since a lot of results aren't data sets
- Inthenameofmine 10y ago
  Archive.org is now legally providing the coursera videos. So it should be legal to distribute this way as well.
- ieee8023 10y ago
  This is fixed! We added a new category! http://academictorrents.com/browse.php?cat=7 http://academictorrents.com/browse.php?cat=7
WhitneyLand 10y ago
It's a great initiative. One more step to help bring science collaboration into the modern Internet world. How much data do you have? How much storage do you project is needed? I'm wondering how practical it would have been to use centralized storage, which has its own advantages.
_lpa_ 10y ago
I like this idea! In my research we deal with relatively large amounts of sequence data, all of which needs to be associated with geo (https://www.ncbi.nlm.nih.gov/gds/ https://www.ncbi.nlm.nih.gov/gds/). While geo is in many ways a good thing, it is not the most pleasant to use - I would love it if we could use something like torrents instead. I feel like there is a danger, however, that using torrents would facilitate the thousands of nonstandard (often redundant) formats bioinformaticians seem to create.
arxpoetica 10y ago
How does this compare with, say, noms? https://github.com/attic-labs/noms https://github.com/attic-labs/noms
- aboodman 10y ago
  Several immediate things come to mind: - this is a mechanism for sharing files and directories (e.g., zipped csv files), whereas noms defines its own structured data model that is much more granular - noms has versioning built-in, you can track the history of a particular dataset - this is firmly based on bittorrent. you could maybe run noms on top of bittorrent, but it's more intended to be run like git, where you talk directly to a server that you want to collaborate with
edraferi 10y ago
How does this compare to IPFS (https://ipfs.io/ https://ipfs.io/)? That project maintains a number of archival datasets, including arXiv: https://ipfs.io/ipfs/QmZBuTfLH1LLi4JqgutzBdwSYS5ybrkztnyWAfRBP729WB/archives/ https://ipfs.io/ipfs/QmZBuTfLH1LLi4JqgutzBdwSYS5ybrkztnyWAfR... Seems like an opportunity to combine efforts.
jakeogh 10y ago
Similar project: http://911datasets.org http://911datasets.org
danso 10y ago
So, what are the rights and licenses for this data? I see that one of them is Yelp photo data from a Kaggle contest [0]. Yelp distributes another Academic data set, but you have to fill out a form and agree to their TOS [1]. So they're OK with the data being available like this? Another random datapoint: When EdX/Harvard released a dataset showing how students performed/dropped out, I uploaded a copy to my S3 to mirror and linked to it from HN. I got a polite email the next day asking for it to be taken down. Academics are (rightfully, IMO) protective of their data and its distribution (particularly its attribution). One thing I would love to see on here is stuff from ICPRS, such as its mirror of the FBI's National Incident-Based Reporting System [2]. As far as I can tell, it's free for anyone to download after you fill out a form. But it also should be free to distribute in the public domain, but for all I know, ICPSR has an agreement with the FBI to only distribute that data with an academic license. (The FBI website has the data in aggregate form, but not the gigabytes that ICPSR does) [0] http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0d36f812e0f1b87bc http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0... [1] https://www.yelp.com/dataset_challenge/dataset https://www.yelp.com/dataset_challenge/dataset [2] https://www.icpsr.umich.edu/icpsrweb/NACJD/NIBRS/ https://www.icpsr.umich.edu/icpsrweb/NACJD/NIBRS/
xchaotic 10y ago
Is there anything preventing the abuse of this systems to share copyrighted content? Just call it "Population.Data.torrent" where the actual data is a movie?
amai 10y ago
See also https://www.globus.org/ https://www.globus.org/.

552 points by iamjeff 10y ago

I'm pretty glad that torrents have started to break out of the "it's only for warez" stereotype. It's a useful technology, regardless of what made it popular.

NelsonMinar 10y ago
That happened around 2008. Or even earlier; Linux distros were being distributed via Bittorrent back in 2006. https://torrentfreak.com/popular-linux-distro-torrents/ https://torrentfreak.com/popular-linux-distro-torrents/
- swiley 10y ago
  When I was younger I would only download these via bittorent because our connection was extremely unreliable and bittorent was always restartable.
  - rck404 10y ago
    Even now I do the same. In india direct downloads are slightly slower. I guess it's to do with the mirror location or something else. Torrent makes it much faster
    - kochthesecond 10y ago
      Probably because tcp or http or both are throttled somewhere. Bit torrent uses any port and lots of connections so throttling is much less effective.
    - mads 10y ago
      Same in China. The Chinese firewall throttles TCP connections making direct downloads very slow. Not uncommon to have an unbearable slow connection while browsing web pages and then when starting a torrent download, you immediately get 2 Megabytes per second download speeds here. A tip is to use a tool like axel (https://github.com/eribertomota/axel https://github.com/eribertomota/axel) to do direct downloads. This will usually speed it up.
      - voltagex_ 10y ago
        >My name is Eriberto and I am not a C developer. I imported Axel from its old repository[1] to GitHub (the original homepage and developers are inactive) Uh oh. Thanks to the repo owner for updating the README, but that's not a good situation. I'd also like to suggest aria2c for this purpose: https://aria2.github.io/ https://aria2.github.io/
- jamesrcole 10y ago
  I think the grandparent was talking about the general perception of torrenting
- tombert 10y ago
  Oh, I know that programmers and hackers and whatnot knew torrents were cool; I'm talking about non-technical people. For example, until fairly recently, if I mentioned a "torrent" to my non-technical mom, she would assume I meant ThePirateBay or something like that. Nowadays, she knows it as just another means to download files.
  - NelsonMinar 10y ago
    Not sure Academic Torrents is the best example of a use of torrents for "non-technical people".
    - tombert 10y ago
      That's a fair point; I guess what I was trying to get at is that when I worked for NYU, I think it would have been an incredibly tough sell to use torrents in any capacity, because of the stigma of piracy. However, I think if I were to pitch it now, there would be serious consideration.
- revelation 10y ago
  Actually, World of Warcraft was released in 2004 and I think it used BitTorrent for patch distribution from the beginning. So very legitimate uses date quite some time back (as you would expect).
- gens 10y ago
  I remember a game called "GunZ" (the 2005 one, i think) internally using torrents to propagate updates. But it was a long time ago.
  - Houshalter 10y ago
    I saw a game do that recently. It makes a lot of sense to use torrents and reduce bandwidth on their servers.
  - dansze 10y ago
    A huge number of games used torrents for patching since around that time, notably in the MMO scene. WoW and every Nexon game come to mind. AFAIK the Battle.net launcher still downloads updates via torrent.
    - dsl 10y ago
      The Blizzard updater is actually a very cool download utility, worth hacking/poking at. AFAIK it pioneered the concept of "web seeds", using HTTP GETs with a Range: header to download specific chunks from a CDN that were not healthy/available in the swarm.
  - tombert 10y ago
    I completely forgot about GunZ doing that. I guess I was wrong; torrents have been taken seriously for awhile :D
- PhasmaFelis 10y ago
  None of that changed the popular stereotype. And Linux distros, etc. have never been more than a fraction of the volume of warez torrents.
adamnemecek 10y ago
Isn't this technically warez?
- apetresc 10y ago
  How so? It's meant to be used by researchers distributing the datasets for their own papers. (Notwithstanding the fact that people seem to already be uploading textbooks/lectures, which seems against the stated intent).
  - adamnemecek 10y ago
    Some of the articles seem to be journal articles. Don't get me wrong, I'm not against it, but isn't this partially what Aaron Swartz was indicted for?
- swang 10y ago
  Since it isn't illegally acquired and distributed software, no.
metaphor 10y ago
Agreed...although it is rather ironic that "CrackStation's Password Cracking Dictionary"[1] tops their list of most downloaded by a respectable 9% over 2nd most downloaded "Arizona State University Twitter Data Set"[2]. [1] http://academictorrents.com/details/fd62cc1d79f595cbe1de6356fb13c2165994e469 http://academictorrents.com/details/fd62cc1d79f595cbe1de6356... [2] http://academictorrents.com/details/2399616d26eeb4ae9ac3d05c7fdd98958299efa9 http://academictorrents.com/details/2399616d26eeb4ae9ac3d05c...
radarsat1 10y ago
I wish it was use more for distributing Linux package updates, a la DebTorrent. https://wiki.debian.org/DebTorrent https://wiki.debian.org/DebTorrent This could take off it only a big player like Ubuntu pushed it. I don't see why we depend on a set of centralized servers for a bunch of files that a huge number of people download on a very regular basis. And yet, the idea has been stagnant for years. Edit: By which I mean, it works, but not enough people use it.
- tombert 10y ago
  I totally agree; what really bothers me about current OS's is that they all depend on something centralized, so if, for example, Canonical went broke, I wouldn't be able to install new packages on my Ubuntu laptop. I'm aware that you can swap out the PPAs as needed, but I would really like something distributed and decentralized.
- rakoo 10y ago
  It's all listed in your link, but the biggest reason is that Bittorrent was inherently built to distribute static content, and the packages is an ever-changing list. Which means that you'd have upstream servers constantly calculating new torrent files and distributing them. Moreover Bittorrent makes it real easy for an observer to know what packages you're installing, and thus what version you have. See this alternative that uses similar idea but not real Bittorrent, they worked around the 1st problem: http://www.camrdale.org/apt-p2p/ http://www.camrdale.org/apt-p2p/
  - radarsat1 10y ago
    So not entirely stagnant ;) I still don't see why Ubuntu/Debian/et al don't take this (or something like it) up in a more official manner. I can see why it's not a default of course, but it could be made a question during installation for example. An additional benefit would be that you'd be able to source packages from machines on your local network, with fallback to the internet, and it would all be pretty much automatic and configuration-free.
    - 10y ago
teej 10y ago
At least back in 2010, Facebook and Twitter were both reportedly using torrents to do code deploys. https://blog.twitter.com/2010/murder-fast-datacenter-code-deploys-using-bittorrent https://blog.twitter.com/2010/murder-fast-datacenter-code-de... https://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/ https://torrentfreak.com/facebook-uses-bittorrent-and-they-l...
vesinisa 10y ago
Are you serious? This is the original purpose of BitTorrent. It was not invented as a "warez technology", but as a way for service providers to save on bandwidth costs when serving huge files. Back when BitTorrent spec was written hosting a collection of large datasets like this meant paying thousands of bucks in hosting expenses, making it prohibitively expensive for most independent developers.
sametmax 10y ago
Yes. It should be built in in browsers.
- yread 10y ago
  Opera used to have it, but it was much slower than a dedicated client
paavokoya 10y ago
And yet, Comcast still illegally throttles torrent traffic..
rck404 10y ago
Torrents and mesh networks were breaking out to be the future of internet during mid and late 90s (Also why Skype was P2P and why it was successfull). I'm not sure what happened to it though.
kilotaras 10y ago
Torrent (or p2p in general) when used for distribution a la Blizzard can be a win-win-win scenario if setuped correctly. 1. Consumers can get higher speed in ISP network. 2. ISPs get lower external bandwith usage 3. Lower resource usage for distributor. Relevant HN discussion - https://news.ycombinator.com/item?id=12380797 https://news.ycombinator.com/item?id=12380797

mtgx 10y ago

.com? Is that wise? I imagine the domain will be seized as soon as the site becomes popular enough. They should at least have some contingency plan to deal with that.

ORioN63 10y ago
Torrents don't mean illegal content.
- deleted 10y ago
  [deleted]
- sandworm101 10y ago
  In his defence, it doesn't really matter whether the content is legal or not. There are people out there who see the torrents as evil. These people have political power and have been known to disrupt and seize .com domains on the scantiest of evidence. A more resilient tld would be a good idea.
  - hueving 10y ago
    Do you have an example of a domain being seized for hosting legitimate torrents?
Ianvdl 10y ago
Why? Nothing indicates that this is illegal?
angusp 10y ago
If it is as they claim well academically supported, you'd assume they'd be easily able to get a subhuman on someone's .edu or.ac.uk or whatever
- jedanbik 10y ago
  I believe a .org would be most appropriate here. Thinking about http://opendap.org/node/305 http://opendap.org/node/305 for context. [What is OPeNDAP: http://opendap.org/about http://opendap.org/about]

rakoo 10y ago

If only WebTorrent (https://github.com/feross/webtorrent https://github.com/feross/webtorrent) worked with standard bittorrent protocol instead of a custom one on top of WebRTC, we would have live access to all the papers and "displayable" data directly instead of firing up a torrent downloader just for a small file.

sktrdie 10y ago
I don't think you understand how WebTorrent works. WebTorrent in fact works with the regular BitTorrent network if you run it from node, and falls back to use WebRTC when used in the browser. So you can seed those torrents directly in the browser with something like instant.io.
- rakoo 10y ago
  I do know how WebTorrent works. The problem is that it creates a completely parallel network of nodes, which on the surface happen to exchange the same framing of messages, so: - When WebTorrent runs on the standard bittorrent network from node, that doesn't change anything: it's still not available from the browser. - When WebTorrent runs on the WebRTC network through instant.ion or anything else, it will only work if somebody else is also seeding the same torrent in their browser. Which they can only have in the browser if they first got it somewhere else. Oh and I'm willing to bet that none of the nodes who currently have the content (ie on the bittorrent network) also share it on the WebTorrent network. I don't expect classic bittorrent peers to ever implement the mess that is WebRTC just to accommodate browsers, unfortunately.
  - forgotpwtomain 10y ago
    Webtorrent on Node talks to both types of peers.
    - nathcd 10y ago
      That's not quite correct. From https://github.com/feross/webtorrent-hybrid https://github.com/feross/webtorrent-hybrid: > In node.js, the webtorrent package only connects to normal TCP/UDP peers, not WebRTC peers. That's why there's this webtorrent-hybrid client which runs a hidden electron process to communicate with WebRTC peers and normal TCP/UDP peers. According to the readme, there's (understandably; it's running chromium in the background) a lot of overhead with this method so they're working toward a non-electron version of WebRTC in Node.

Philipp__ 10y ago

This is amazing! Thank you for sharing this!

degenerate 10y ago

I already knew about Academic Torrents, but didn't know about Kaggle, which was linked[1] from one of the 'popular' data sets on AT: https://www.kaggle.com/c/yelp-restaurant-photo-classification https://www.kaggle.com/c/yelp-restaurant-photo-classificatio... It looks like one of those logo-design-competition-sites, but for big data. Anyone compete in one of these? [1]: http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0d36f812e0f1b87bc http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0...

jakub_h 10y ago

I'm wondering if torrents as such are actually useful for this. I'd figure some kind of virtual file system (perhaps based on BitTorrent) would be very useful. You'd simply pass a file path to an open() routine in your scientific code and data would get opened transparently. You currently have this with URLs and HTTP but there's no useful caching or data distribution.

rakoo 10y ago
If you have the torrent file, you can use one of the things that mount them as virtual file systems, like btfs (https://github.com/johang/btfs https://github.com/johang/btfs).
seanp2k2 10y ago
You might like https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/about.rst https://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/docs/ab... , https://urbit.org/docs/using/filesystem/ https://urbit.org/docs/using/filesystem/ , or https://ipfs.io https://ipfs.io BTSync and SyncThing are also tools to do this, and I'm sure there are FUSE things to work with BT and block chains ("bittorrent fuse" google results look promising).
- espadrine 10y ago
  With the same principles, http://infinit.sh/ http://infinit.sh/ adds security and editability to the mix.
- jakub_h 10y ago
  Lots of interesting things to read, thank you! (To the others as well.)
zielmicha 10y ago
IPFS (https://ipfs.io https://ipfs.io) does that.
Eridrus 10y ago
Filesystems have distribution problems. You don't always have permissions to install a new filesystem.

babak_ap 10y ago

Anyone looking for Wireless Data, I suggest taking a look at: http://crawdad.org/ http://crawdad.org/ and http://www.cise.ufl.edu/~helmy/MobiLib.htm#traces http://www.cise.ufl.edu/~helmy/MobiLib.htm#traces

danielmorozoff 10y ago

This is fantastic. I am so glad someone built this.

rkda 10y ago

You might also be interested in dat. They're trying to solve this problem as well. http://dat-data.com/ http://dat-data.com/ "Dat is a decentralized data tool for distributing data small and large."

WhitneyLand 10y ago

API suggestions: 1). Don't use hard coded values for types >GET /apiv2/entries?cat=6 -- List entries that are datasets >GET /apiv2/entries?cat=5 -- List entries that are papers These could be written as: /apiv2/entries/datasets /apiv2/entries/papers 2). You may not need path elements like entries, entry, collection, and collection name. For example further simplification would leave /datasets /papers 3). Don't use capitals like this, switch to lowercase: /apiv2/entry/INFOHASH 4). Use HTTP verbs in a standard, semantic way. For example this POST /apiv2/collection -- create an collection POST /apiv2/collection/collection-name/update POST /apiv2/collection/collection-name/delete POST /apiv2/collection/collection-name/add POST /apiv2/collection/collection-name/remove This could all be collapsed into one form and would in turn be more familiar to developers.

jnbiche 10y ago
I strongly agree with these suggestions, if the creator(s) happen to read this. And while these suggestions may seem like nitpicking, they are not. While not everything around REST APIs is entirely standard, there's a great deal of agreement about proper resource naming and use of HTTP verbs. If you follow these widely-used standards, not only will developers find it much easier to interact with your API, but also there is a lot of tooling and client libraries out there built specifically to work with APIs that use HTTP verbs and URIs in the standardized ways.
- iamjeff 10y ago
  Hi. I found this site late last night while looking for material for a hobby project. Really, really awesome site. I also found the creators here [1] and here [2]...maybe you can help shoot them an e-mail so that they can acknowledge a lot of the praise on this thread and the flood of helpful suggestions. Additional contributors can also be found here [3] as well as related publications/presentations [4] including a massively interesting Reddit discussion [5]. [1]- Joseph Paul Cohen https://twitter.com/josephpaulcohen https://twitter.com/josephpaulcohen joseph@josephpcohen.com http://josephpcohen.com/w/ http://josephpcohen.com/w/ https://www.linkedin.com/in/josephpcohen https://www.linkedin.com/in/josephpcohen https://github.com/ieee8023 https://github.com/ieee8023 https://www.reddit.com/user/ieee8023 https://www.reddit.com/user/ieee8023 [2]- Henry Z Lo henryzlo@cs.umb.edu https://www.linkedin.com/in/henry-z-lo-6a11461b https://www.linkedin.com/in/henry-z-lo-6a11461b https://github.com/henryzlo https://github.com/henryzlo [3] http://academictorrents.com/about.php http://academictorrents.com/about.php [4]- Academic Torrents: A Community-Maintained Distributed Repository (http://dl.acm.org.sci-hub.cc/citation.cfm?id=2616528&dl=ACM&coll=DL&CFID=661488745&CFTOKEN=24433760 http://dl.acm.org.sci-hub.cc/citation.cfm?id=2616528&dl=ACM&...) - Academic Torrents - Simple Pitch (https://docs.google.com/presentation/d/1JC2d1g9U6HaenGSn_Xvkl7FXyNo5LgVpLFQqCrNgF6k/edit#slide=id.p https://docs.google.com/presentation/d/1JC2d1g9U6HaenGSn_Xvk...) - Academic Torrents: Scalable Data Distribution (http://arxiv.org/pdf/1603.04395v1.pdf http://arxiv.org/pdf/1603.04395v1.pdf) [5]- I have every publicly available Reddit comment for research. ~ 1.7 billion comments @ 250 GB compressed. Any interest in this? (https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...)

wodenokoto 10y ago

A lot of these "data sets" appears to be coursera courses. I'm not sure if those are legal to redistribute. It also clutters the browse function since a lot of results aren't data sets

Inthenameofmine 10y ago
Archive.org is now legally providing the coursera videos. So it should be legal to distribute this way as well.
ieee8023 10y ago
This is fixed! We added a new category! http://academictorrents.com/browse.php?cat=7 http://academictorrents.com/browse.php?cat=7

WhitneyLand 10y ago

It's a great initiative. One more step to help bring science collaboration into the modern Internet world. How much data do you have? How much storage do you project is needed? I'm wondering how practical it would have been to use centralized storage, which has its own advantages.

_lpa_ 10y ago

I like this idea! In my research we deal with relatively large amounts of sequence data, all of which needs to be associated with geo (https://www.ncbi.nlm.nih.gov/gds/ https://www.ncbi.nlm.nih.gov/gds/). While geo is in many ways a good thing, it is not the most pleasant to use - I would love it if we could use something like torrents instead. I feel like there is a danger, however, that using torrents would facilitate the thousands of nonstandard (often redundant) formats bioinformaticians seem to create.

arxpoetica 10y ago

How does this compare with, say, noms? https://github.com/attic-labs/noms https://github.com/attic-labs/noms

aboodman 10y ago
Several immediate things come to mind: - this is a mechanism for sharing files and directories (e.g., zipped csv files), whereas noms defines its own structured data model that is much more granular - noms has versioning built-in, you can track the history of a particular dataset - this is firmly based on bittorrent. you could maybe run noms on top of bittorrent, but it's more intended to be run like git, where you talk directly to a server that you want to collaborate with

edraferi 10y ago

How does this compare to IPFS (https://ipfs.io/ https://ipfs.io/)? That project maintains a number of archival datasets, including arXiv: https://ipfs.io/ipfs/QmZBuTfLH1LLi4JqgutzBdwSYS5ybrkztnyWAfRBP729WB/archives/ https://ipfs.io/ipfs/QmZBuTfLH1LLi4JqgutzBdwSYS5ybrkztnyWAfR... Seems like an opportunity to combine efforts.

jakeogh 10y ago

Similar project: http://911datasets.org http://911datasets.org

danso 10y ago

So, what are the rights and licenses for this data? I see that one of them is Yelp photo data from a Kaggle contest [0]. Yelp distributes another Academic data set, but you have to fill out a form and agree to their TOS [1]. So they're OK with the data being available like this? Another random datapoint: When EdX/Harvard released a dataset showing how students performed/dropped out, I uploaded a copy to my S3 to mirror and linked to it from HN. I got a polite email the next day asking for it to be taken down. Academics are (rightfully, IMO) protective of their data and its distribution (particularly its attribution). One thing I would love to see on here is stuff from ICPRS, such as its mirror of the FBI's National Incident-Based Reporting System [2]. As far as I can tell, it's free for anyone to download after you fill out a form. But it also should be free to distribute in the public domain, but for all I know, ICPSR has an agreement with the FBI to only distribute that data with an academic license. (The FBI website has the data in aggregate form, but not the gigabytes that ICPSR does) [0] http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0d36f812e0f1b87bc http://academictorrents.com/details/19c3aa2166d7bfceaf3d76c0... [1] https://www.yelp.com/dataset_challenge/dataset https://www.yelp.com/dataset_challenge/dataset [2] https://www.icpsr.umich.edu/icpsrweb/NACJD/NIBRS/ https://www.icpsr.umich.edu/icpsrweb/NACJD/NIBRS/

xchaotic 10y ago

Is there anything preventing the abuse of this systems to share copyrighted content? Just call it "Population.Data.torrent" where the actual data is a movie?

amai 10y ago

See also https://www.globus.org/ https://www.globus.org/.