14 ms·
BTFS – mount any .torrent file or magnet link as directory
- toomuchtodo 6y agoThis is cool. I can mount most of the Internet Archive as a directory with this. Would love this to grow into what your consumer cloud storage services are but for torrents (a global file system you can mount locally, using torrent data as inodes, with visual representations for availability status similar to the Dropbox green check boxes). Perhaps even with support for web seeding from a storage system of last resort (Backblaze, S3, etc).
- rakoo 6y agoI know it's not good spirit, but mounting torrents into the filesystem has been done a few times already, there are many projects that do that already. Also, it's not really advertised, but S3 can act as a torrent node: https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.ht...
- ShinTakuya 6y agoSounds like they've dropped support for newer regions, though. Which is a shame. Seems like a decent tool.
- toomuchtodo 6y agoThe cool part is you can have S3 serve a torrent, and then disable access to the object with permissions once it reaches an adequate seed level. You’re no longer charged for outbound data, but S3 will continue to act as a tracker for the swarm.
- fiatjaf 6y agoWow, that was very unexpected.
- MR4D 6y agoJust checked Amazon's S3 page linked to by the PP. Question - how do the following "Notes" that Amazon has on it's page affect that? "Amazon S3 does not support the BitTorrent protocol in AWS Regions launched after May 30, 2016." "You can only get a torrent file for objects that are less than 5 GBs in size."
- deleted 6y ago[deleted]
- res0nat0r 6y agoLikely just can only seed/torrent via older s3 regions is all.
- toomuchtodo 6y agoBlessed be thy us-east-1.
- anacrolix 6y agoFWIW the S3 BitTorrent support is very primitive and limited. It hasn't been updated since 2006 from what I can tell.
- rakoo 6y agoWhile you're here I have to thank you for your go torrent library, it's really nice ! I have never used S3's torrent capability, but at this point bittorrent is pretty much "done" -- there's not much to add to make the protocol better without breaking compatibility, which means there's not much support to maintain a working software. But I'm not sure about it, maybe they don't participate in the DHT or don't use PEX, so that could be something they can improve.
- yjftsjthsd-h 6y agoWell that is neat, albeit an unfortunate name (1-char off from BTRFS). Wonder what performance is like - does it try to preemptively download? Cache? I would expect it to take a while to run `cp ~/mnt_btfs/big_file ~/Downloads/`, but how about `find ~/mnt_btfs`? If it's performant, it could be really nifty:)
- shuringai 6y agovery unfortunate indeed as there's another project with this name: https://github.com/TRON-US/go-btfs https://github.com/TRON-US/go-btfs I dont know whether the maintainer is a supporter of crypto (as in tokens) but having a common name with a Tron product (or cryptocoins in general) can backfire.
- heavyset_go 6y agoIt looks like the BTFS in the OP was the "original", with its first commit in July of 2015. Could be wrong, because the GitHub interface doesn't make it easy to get to the beginning of the 13k commits for go-btfs.
- jrimbault 6y ago9d0e6a7ffb5deea2c8c8e555d7bf6bcab6fdc6ac 2014-06-26 Big things have small beginnings. (I have an alias specifically for finding that out) Personnally, I don't care about any of the crytpo-mining-coining and I think the two are sufficiently distinct.
- heavyset_go 6y agoThanks for digging in.
- lgierth 6y agoThat's when go-ipfs was started - see my other comment for more info
- lgierth 6y ago
- eqtn 6y agoBTFS with mutable torrents would make a nice piracy suite
- ksec 6y agoAnd with the increase of Subscription fatigue, torrent may possibly make a come back.
- rakoo 6y agoMutable "torrents" with fuse access exists, it's called hyperdrive (https://github.com/hypercore-protocol/hyperdrive https://github.com/hypercore-protocol/hyperdrive). See the daemon in particular (https://github.com/hypercore-protocol/hyperdrive-daemon https://github.com/hypercore-protocol/hyperdrive-daemon)
- namibj 6y agoNo. Hyperdrive is not like the official BEP-conform "mutable DHT item"-based updatable torrents. They essentially say "download the most recent version signed by this pubkey", which ia notably distinct from hyperdrive's append-only log.
- rakoo 6y ago"download the most recent version signed by this pubkey" is literally how hyperdrive works. The append-only log is hypercore, one of the building blocks of hyperdrive. Hyperdrive is a layer above and gives you a view of how the archive is at any point in time, with preferential access to the latest version. TBF what I understood from the request of "mutable torrents" wasn't the literal feature, but a way to have decentralized nodes exchanging updatable content. Bittorent with the Mutable Torrents BEP does work, but it's nowhere near as developed as what hyperdrive is today. And as sad as it may sound I don't think it will ever be.
- namibj 6y agoYeah. My point was, that BEP 46[0] doesn't have any archive/log functionality. [0]: https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html
- yunruse 6y agoThis is a brilliant concept of a tool. I’ve never worked with monorepos before, but something like this with rw capacity would be an absolutely amazing pipeline, if every developer in the office and beyond are full nodes in the system.
- david_draco 6y agoTorrents can't write though, or are you referring to a shared filesystem like NFS?
- adtac 6y agoLet's create a chain of immutable blocks with back pointers on top of this to create new versions of the data. Maybe even call it a... blockchain.
- rakoo 6y agoCheck out hyperdrive (https://github.com/hypercore-protocol/hyperdrive https://github.com/hypercore-protocol/hyperdrive), specifically the daemon. Torrents are notorious for being read-only, but hyperdrive builds on bittorrent ideas and is making read/write shared spaces a breeze
- waynesonfire 6y agoWow, what a great concept.
- grawprog 6y agoThis looks pretty awesome. Does it work with private trackers?
- pxx 6y agoErr. Why would it not? edit: more constructively, this seems to just use libtorrent in the backend, which supports the 'private' flag https://www.libtorrent.org/features.html https://www.libtorrent.org/features.html . Though even if you didn't, it would still 'work' in the sense of being able to download chunks.
- dewey 6y agoIt also depends on which client ID it exposes and if that one is whitelisted on trackers. But that's not really a technology problem.
- dcrn 6y agoThat answer probably lies more so with the tracker's software and approved clients than this project's implementation.
- torrmentor 6y agoWhy not torrentfs? What are the needs for obfuscation or association with btrfs ('better try remaking fs'? )
- w4hirelife99 6y ago“Why this project updated yesterday and not the project last updated 3 years ago?” Unless you mean a different one since you didn’t link source
- JoshuaRLi 6y agoMaybe https://github.com/anacrolix/torrent/tree/master/cmd/torrentfs https://github.com/anacrolix/torrent/tree/master/cmd/torrent...?
- manjalyc 6y agoThe source code is far smaller that I would've expected, tiny in fact - no seriously, its probably <2000 lines of actual code and looks extremely simple to audit and fix bugs. It seems the only real external dependency is libtorrent. Honestly, if your not already amazed if not at the simplicity of this project, than you should be by the simplicity of fuse filesystems and the capabilities it exposes.
- wbkang 6y agoYou are right, you encouraged me to look at it and the actual interface into fuse is here whic is at least understandable: https://github.com/johang/btfs/blob/master/src/btfs.cc#L968 https://github.com/johang/btfs/blob/master/src/btfs.cc#L968
- calvinmorrison 6y agoIf you think that's excellent and are interested in fuse, you're probably intereted in 9front which takes the concept to another level. Ftpfs does what it says, but webfs handles http and forms and the like, upas/fs can show your imap inbox, it's splendid and simple and allows your tooling to be the familiar nix-like utilities like awk and sed. Even capture your screenshot by catting /dev/screen to a file
- so_tired 6y agoWhich lib/approach would you recommend if i need run a remote batch process using local files? For example, on the local system: remote-execute gcc localfile.c -o localfile.o And on the remote system, there is a gcc and a remote agent which receives and retransmits the files
- toomuchtodo 6y agoSSH for job running and management, NFS for filesystem access. Have remote systems mount the local file system. If NFS isn’t available, have the remote job pull files over rsync from your local system, then rsync back the resulting output.
- IncRnd 6y agoAwesome! People can now use standard tools for adding malware into stolen archives! /s
- boardwaalk 6y ago> With BTFS, you can mount any .torrent file or magnet link and then use it as any /read-only/ directory in your file tree.
- IncRnd 6y agoAt first I wasn't sure if you were being too serious or just literal when that wasn't needed, but then I realized that you didn't read my full comment.
- yjftsjthsd-h 6y ago? At this writing, the full comment is: > Awesome! People can now use standard tools for adding malware into stolen archives! /s But the most charitable reading I can think of is that you could use BTFS to pirate something, and very slightly reduce the number of steps when modifying it? I guess? But that's no different than just using a normal torrent client and then modifying the files once downloaded. It's just a different interface. So... what context am I missing?
- IncRnd 6y agoYou missed the /s at the end.
- yjftsjthsd-h 6y agoAh, so I did. Not... totally sure how that affects the read, but you might have a point.
- IncRnd 6y ago
- deleted 6y ago[deleted]
- shric 6y agoNeat idea, but it seems like abuse of the torrent protocol. If peers don't grab chunks randomly then it partially defeats the purpose of bittorrent.
- antpls 6y agoStreaming services have shown that BitTorrent is practical for streaming in some setup too
- oh_sigh 6y agoEvery torrent client I've used recently has an "enable sequential download" option. The only torrents this would realistically affect are maybe ones where the seeder may go offline when a bunch of clients have been doing sequential downloads and no one has the final parts of the content.
- keyserzose 6y agoWhich would be the same as no one would have those random parts.
- gruez 6y agoI wonder how intelligently this handles piece download order. Torrent clients usually use rarest-first download algorithm, but if you've mounted a movie and you're playing it with VLC, you'd probably want it to download sequentially so there's as little time spent buffering as possible. You could download the pieces on demand (whatever piece the application is trying to read), but I suspect that will provide a suboptimal experience, especially if your download speed is barely above the bitrate of the movie.
- kccqzy 6y agoI'm not sure clients actually download rarest-first. It's supposed to be random. From The BitTorrent Protocol Specification, http://bittorrent.org/beps/bep_0003.html http://bittorrent.org/beps/bep_0003.html: > Downloaders generally download pieces in random order, which does a reasonably good job of keeping them from having a strict subset or superset of the pieces of any of their peers. However clients can certainly choose to download in whatever order they wish. Like Popcorn Time.
- calvinmorrison 6y agolibtorrent has repeatedly ignored requests for sequential download because of the silly spec. There's a patch out there that quite easily enables it. It's a bit absurd to me because there's a very good use case for sequential download and it's not clear there are any actual side effects.
- Hamuko 6y agoDoesn't sequential downloading degrade the swarm performance?
- londons_explore 6y agoYes, but only 2nd order... If all users downloaded in order, then users downloading the file cannot exchange chunks with other downloaders. That "exchange" is necessary to encourage users not to limit their upload speeds. In today's world where the vast majority of users just use the default settings for their torrent client, it probably doesn't matter much.
- 6510 6y agoGreat stuff, thanks!
- inyorgroove 6y agoTools[1] like this have existed for at least 6 years. Don't know if the older tools still work though. [1] https://github.com/mafintosh/torrent-mount https://github.com/mafintosh/torrent-mount
- ngcc_hk 6y agoIf you make your own torrent files, how to prevent it accidentally share with the world? Is there anything in the torrent files that can prevent this? (Program and configuration can make mistake.)
- anacrolix 6y agoYes, there are BEPs that describe not announcing or accepting connections in certain circumstances. There are also peer discovery mechanisms that are designed to work on local networks. Of course none of this matters if you mess up (and someone externally cares enough to take your data), or if your client doesn't support any of this stuff.
- saurik 6y agoThis is effectively what IPFS is, but IPFS takes it to the next level by using the equivalent of torrent files for subdirectories within the mounted torrent files, so you can "traverse" entire networks of the pretty-much-a-torrent files without first having to download and mount them. And like, you could implement similar functionality here by having torrent files that contain files named .torrent have those files be treated as directories, but at that point IPFS is going to be much more feature complete; even like, if you have some giant file, on IPFS that might be represented by trees of hashes as opposed to merely a giant array of hashes so you can incrementally and efficiently access the parts you need. So like, while this is awesome work, if you find it inspiring, maybe channel that inspiration into learning about IPFS and extending that ecosystem, rather than doubling down on trying to build on torrent files.
- dereg 6y agoUsually, an upvote should suffice, but this is the best ELIF of IPFS that I've read.
- trevyn 6y agoAre there any design changes that could create organic adoption of IPFS? Or, conversely, what design decisions do you think have hindered adoption?
- dafoex 6y agoI've had a breif look at IPFS, but not enough to say anything with any amount of confidence. That said, the thing that immediately put me off was that everything seemed to want to be browser based, like starting some service that makes a local web server that acts as a GUI - there was even a chrome extension that tried to make this easy for you. A feature that might exist, but I haven't discovered, that would make IPFS more appealing to me, would be a way to mount the IPFS like it were an FTP server, and navigate it with whatever file manager you prefer and not have to worry about managing the backend stuff. That is to say, I'd like BTFS for IPFS.
- 6y ago
- fuzzybear3965 6y agoCan someone explain why I might use this instead of downloading a select group of files using a torrent file/magnet link? Is the advantage that it selectively downloads files on an as-needed basis? Or, is the product niche somehow related to the fact that the torrent file/magnet link is mounted as a filesystem, instead of only as a collection of files and folders?
- kchr 6y agoThe latter - it lets you navigate the file structure of a torrent as well as read (stream) file content using any of your usual applications, without having to download the files first.
- fuzzybear3965 6y agoStreaming. I thought about this in the interfile sense, but not the intrafile sense. This makes sense.
- anacrolix 6y agoSee https://github.com/anacrolix/torrent/tree/master/fs https://github.com/anacrolix/torrent/tree/master/fs and https://github.com/anacrolix/torrent/blob/master/cmd/torrentfs/main.go https://github.com/anacrolix/torrent/blob/master/cmd/torrent... for a Go FUSE implementation of this. (Looks like it was started in early 2014.)
- Randor 6y agoWell, I wrote a device driver for Windows 8 that would mount a .torrent file as a virtual disk way back in 2011. We pitched the intellectual property around to several large companies. The only interest we had in the technology was video game companies... where we could mark file pieces as 'high priority' and allow the player to play the game even when the game disk was partially downloaded. The IP was sold to a company developing their next-generation console video game system as part of a 24 million dollar package and never saw the light of day. I suspect what they really wanted was the associated patents.
- jtl999 6y agoOkay. I'm afraid to ask. What happens in the case of a) a torrent with no seeders or b) Has seeders with enough data to get a directory listing but either the seeders "drop off" afterwards or are incomplete Other experiences with network filesystems (CIFS/NFS) on multiple platforms sometimes have things getting hairy if the connection to the server drops while the filesystem is mounted or in use
- Randor 6y agoHi, A quick glance at https://github.com/johang/btfs/blob/master/src/btfs.cc https://github.com/johang/btfs/blob/master/src/btfs.cc on line 190 reveals that no seeders would result in an infinite wait state: pthread_cond_wait(&signal_cond, &lock); The author could mitigate this by automatically failing any reads where there are zero seeders and/or if the peer piece map does not contain the block.
- diericx 6y agoI would be way more interested in this project if it downloaded normally in the background immediately after the torrent is added, and then downloaded specific bits as programs requested them. I haven't looked into the code base for this exactly but it says nothing about it in the Readme. Otherwise, like you're describing, you could add a file and then go to read it a week later and be met with no seeders and thus no file when it could have been grabbed somewhere in between. But maybe that extends beyond the capabilities of a file system?
- diericx 6y agoThis is freaking awesome!!
- Giorgi 6y agoCool but can't really see any practical use of this...