6 ms·

Whenever I read about such issues I always wonder why we all don’t make more use of BitTorrent. Why is it not the underlying protocol for much more stuff? Like

Whenever I read about such issues I always wonder why we all don’t make more use of BitTorrent. Why is it not the underlying protocol for much more stuff? Like container registries? Package repos, etc.

vaylian 1y ago
> Like container registries? Package repos, etc. I had the same thoughts for some time now. It would be really nice to distribute software and containers this way. A lot of people have the same data locally and we could just share it.
maeln 1y ago
I can imagine a few things : 1. BitTorrent has a bad rep. Most people still associate it with just illegal download. 2. It requires slightly more complex firewall rules, and asking the network admin to put them in place might raise some eyebrow for reason 1. On very restrictive network, they might not want to allow them at all due to the fact that it opens the door for, well, BitTorrent. 3. A BitTorrent client is more complicated than an HTTP client, and not installed on most company computer / ci pipeline (for lack of need, and again reason 1.). A lot of people just want to `curl` and be done with it. 4. A lot of people think they are required to seed, and for some reason that scare the hell of them. Overall, I think it is mostly 1 and the fact that you can just simply `curl` stuff and have everything working. I do sadden me that people do not understand how good of a file transfer protocol BT is and how it is underused. I do remember some video game client using BT for updates under the hood, and peertube use webtorrent, but BT is sadly not very popular.
- simonmales 1y ago
  At least the planet download offers BitTorrent. https://planet.openstreetmap.org/ https://planet.openstreetmap.org/
  - lostmsu 1y ago
    So does Wikipedia https://meta.m.wikimedia.org/wiki/Data_dump_torrents https://meta.m.wikimedia.org/wiki/Data_dump_torrents Truly the last two open web titans.
- _def 1y ago
  > A lot of people think they are required to seed, and for some reason that scare the hell of them. Some of the reasons consists of lawyers sending put costly cease and desist letters even to "legitimate" users
  - skrebbel 1y ago
    For seeding map data?
    - swiftcoder 1y ago
      Particularly if your torrent traffic is encrypted, they don't always bother to check what you are torrenting
      - skrebbel 1y ago
        Are you just theorizing, or is there precedent of this? I mean of lawyers sending cease and desist letters to people torrenting random encrypted streams of data?
        swiftcoder 1y ago
        When I was working in p2p research, we used to get regular C&Ds just for scraping the tracker (not even downloading anything!). IP holders be wilding
        skrebbel 1y ago
        Appreciate your response, madness indeed!
  - squigz 1y ago
    Do companies send out C&Ds for your average user torrenting? I've gotten thousands of DMCA letters but never a C&D, and I've only ever heard of 1 person getting one, and they were silly enough to be hosting a collection of paid content that they scraped themselves, from their home. DMCA demands are, as far as I'm aware, completely automated and couldn't really cost much.
- Fokamul 1y ago
  Lol, bad rep? Interesting, in my country everybody is using it to download movies :D Even more so now, after this botched streaming war. (EU)
  - maeln 1y ago
    Which is exactly why it has a bad rep. In most people mind BitTorrent = illegal download.
    - _zoltan_ 1y ago
      downloading movies for personal use is legal in many countries.
      - ahofmann 1y ago
        This is a useless discussion. Imagine how the firewall-guy/network-team in your company will react to that argument.
        loa_in_ 1y ago
        This is not a useless discussion just because it'll inconvenience someone who is at work anyway.
      - _flux 1y ago
        How about the uploading part of it, which is behind the magic of Bittorrent and default mode of operation?
      - lobochrome 1y ago
        Really?? Which countries allow copyright infringement by individuals?
        Pooge 1y ago
        None. Because you projected your country's laws in the discussion, you failed to see that the countries that allow copyrighted material to be downloaded for personal usage do not qualify that download as "copyright infringement" in the first place. To answer your question with the only answer I know: Switzerland.
- joao 1y ago
  At a previous job, I was downloading daily legal torrent data when IT flagged me. The IT admin, eager to catch me doing something wrong, burst into the room shouting with management in tow. I had to calmly explain the situation, as management assumed all torrenting was illegal and there had been previous legal issues with an intern pirating movies. Fortunately, other colleagues backed me up.
  - Yoric 1y ago
    Hey, ages ago, as an intern, I have been flagged for BitTorrent downloads. As it turned out, I was downloading/sharing Ubuntu isos, so things didn't escalate too far, but it was a scary moment. So, I'm not using BT at work anymore.
    - Symbiote 1y ago
      I left a Linux ISO (possibly Ubuntu) seeding on a lab computer at university, and forgot about it after I'd burned the DVD. You can see this was a while ago. A month later an IT admin came to ask what I might be doing with port 6881. Once I remembered, we went to the tracker's website and saw "imperial.ac.uk" had the top position for seeding, by far. The admin said to leave running.
      - account42 1y ago
        > The admin said to leave running. This can be read in two wildly different ways.
        Symbiote 1y ago
        Fortunately, it was the nice way — that university is one of the backbone nodes of the UK academic network, so the bandwidth use was pretty much irrelevant.
  - belter 1y ago
    S3 had BitTorrent support for a long time... "S3 quietly deprecates BitTorrent support" - https://news.ycombinator.com/item?id=27524549 https://news.ycombinator.com/item?id=27524549
- em-bee 1y ago
  seeding is uploading after you are done downloading. but you are already uploading while you are still downloading. and that can't be turned off. if seeding scares someone, then uploading should scare them too. so they are right, because they are required to upload.
  - foobarbecue 1y ago
    Some BitTorrent clients make it easier to set than others, but if it's a healthy torrent I often limit upload rate to so slow that it doesn't transfer anything up. Ratio is 0.00 and I still get 100s of mb/s.
  - ProtoAES256 1y ago
    If you are in the scene long enough, you should have known that there are some uncooperative clients that always send 0% (Xunlei being one of the more notorious example with their VIP schemes, and later on they would straight up spoof their client string when people started blocking them). Being a leecher nowadays is almost a norm for a lot of users, and I don't blame them since they are afraid of consequences in a more regulated jurisdiction. But a must seed when leech requirement? Hoho no, that's more like a suggestion.
    - account42 1y ago
      > Hoho no, that's more like a suggestion. For public trackers maybe.
  - maeln 1y ago
    > but you are already uploading while you are still downloading. and that can't be turned off Almost every client let you set uploading limit, which you can set at 0. The only thing that generate upload bp usage that cannot be deactivated would be protocol stuff (but you can deactivate part of bt like using the DHT).
  - alternatetwo 1y ago
    Transmission allows turning this off by setting upload to 0. It's simply a client setting, but most clients don't offer it.
- out_of_protocol 1y ago
  > It requires slightly more complex firewall rules, and asking the network admin to put them in place might raise some eyebrow for reason 1 Well, in many such situations data is provided for free, putting huge burden on the other side. Even it it's a little bit less convenient it makes service a lot more sustainable. I imagine torrent for free tier and direct download as a premium option would work perfectly
  - thomastjeffery 1y ago
    I wish... I overpay more than double market value for my connection, and am not allowed to configure my router. This is the norm for most apartment dwellers in the US as far as I'm aware.
    - out_of_protocol 1y ago
      VPN is a thing
      - thomastjeffery 1y ago
        Sure, but I would also like to be able to forward ports. Right now, my best workaround is tailnet, which requires a client.
- amelius 1y ago
  Bad rep ... You know what has a bad rep? Big companies that use and trade my personal information like they own it. I'll start caring about copyrights when governments force these big companies to care about my information.
- jauntywundrkind 1y ago
  Webtorrent exists. It uses webrtc to let users connect to each other. There's support in popular trackers. This basically handles every problem stated. There's nothing to install on computers: it's just js running on the page. There's no firewall rules or port forwarding to setup, all handled by the stun/turn in webrtc. Users wouldn't necessarily even be aware they are uploading.
  - account42 1y ago
    STUN is not always possible and TURN means proxying the connection through a server which would be counter-productive for the purpose of using bit-torrent as an alternative to direct HTTP downloads as you are now paying for the bandwidth in both directions. This is very much not a problem with magic solutions.
    - jauntywundrkind 1y ago
      Agreed! But STUN's success rate is pretty good! As the number of peers goes up it should be less likely that one would need to use TURN to connect to a peer, but I am skeptical webrtc is built to fall back like this, to try other peers first. The advantage is that at least it's all builtin. It's not a magic solution, but it's a pretty good solution, with fallbacks builtin for when the networking gets in the way of the magic.
- account42 1y ago
  5. Most residential internet connections are woefully underprovisioned for upload so anything that uses it more (and yes you need people to seed for bittorrent to make sense) can slow down the entire connection. 6. Service providers have little control over the service level of seeders and thus the user experience. And that's before you get malicious users.
nativeit 1y ago
I assume it’s simply the lack of the inbuilt “universal client” that http enjoys, or that devs tend to have with ssh/scp. Not that such a client (even an automated/scripted CLI client) would be so difficult to setup, but then trackers are also necessary, and then the tooling for maintaining it all. Intuitively, none of this sounds impossible, or even necessarily that difficult apart from a few tricky spots. I think it’s more a matter of how large the demand is for frequent downloads of very large files/sets, which leads to a questions of reliability and seeding volume, all versus the effort involved to develop the tooling and integrate it with various RCS and file syncing services. Would something like Git LFS help here? I’m at the limit of my understanding for this.
- nativeit 1y ago
  I certainly take advantage of BitTorrent mirrors for downloading Debian ISOs, as they are generally MUCH faster.
  - nopurpose 1y ago
    All Linux ISOs collectors in the world wholeheartedly agree.
  - Sesse__ 1y ago
    Are you serious? Most Debian ISO mirrors I've used have 10gig connectivity and usually push a gigabit or two fairly easily. BitTorrent is generally a lot slower than that (it's a pretty terrible protocol for connecting you to actually fast peers and getting stuff quickly from them).
    - queenkjuul 1y ago
      I've definitely seen higher speeds with BitTorrent, pretty easily maxing out my gbe nics, but I'm not downloading Debian images specifically with much frequency.
- mschuster91 1y ago
  Trackers haven't been necessary for well over a decade now thanks to DHT.
zaphodias 1y ago
I remember seeing the concept of "torrents with dynamic content" a few years ago, but apparently never became a thing[1]. I kind of wish it did, but I don't know if there are critical problems (i.e. security?). [1]: https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html
charcircuit 1y ago
AFAIK Bittorrent doesn't allow for updating the files for a torrent.
- out_of_protocol 1y ago
  This is a feature, not a bug. Torrent file/magnet link contains hash of a data which is immutable. Just publish new link (you should anyway, even with http)
  - opan 1y ago
    Agreed, it's cool that you can reverify a torrent from 3 years ago and make sure the data on your disk isn't lost or damaged.
    - account42 1y ago
      That's useful if you did not prepare for that eventuality. But if you do want to store data long-term it's better to generate error-correcting codes like PAR2 so that you can actually recover partial errors without having to rely on seeders being available - and when you do that you no longer need the torrent for verification.
  - charcircuit 1y ago
    Imagine if for websites you had to get a brand new domain if you ever wanted to change the contents of your web page. You can't just go to google.com because it would be immutable. You would have to somehow know you have to go to google10393928.com and any links to google on the internet would be linking to some old version. The ability to have a link always refer to the latest version is useful. The same applies to torrents. It's possible for the magnet link to the latest version to get lost and then a bunch of people are accidently downloading old, worse versions of the file with no way to find the newest version.
    - jacquesm 1y ago
      What a dumb strawman. Domains are not hashes.
deleted 1y ago
[deleted]
trenchpilgrim 1y ago
> Like container registries? https://github.com/uber/kraken https://github.com/uber/kraken exists, using a modified BT protocol, but unless you are distributing quite large images to a very large number of nodes, a centralized registry is probably faster, simpler and cheaper
marklit 1y ago
Amazon, Esri, Grab, Hyundai, Meta, Microsoft, Precisely, Tripadvisor and TomTom, along with 10s of other businesses got together and offer OSM data in Parquet on S3 free of charge. You can query it surgically and run analytics on it needing only MBs of bandwidth on what is a multi-TB dataset at this point. https://tech.marksblogg.com/overture-dec-2024-update.html https://tech.marksblogg.com/overture-dec-2024-update.html If you're using ArcGIS Pro, use this plugin: https://tech.marksblogg.com/overture-maps-esri-arcgis-pro.html https://tech.marksblogg.com/overture-maps-esri-arcgis-pro.ht...
- willtemperley 1y ago
  It's just great that bounding box queries can be translated into HTTP range requests.
- n4r9 1y ago
  As someone who works with mapping data for HGV routing, I've been keeping an eye on Overture. I wonder do you know if anyone has measured the data coverage and quality between this and proprietary datasets like HEREmaps? Does Overture supplement OSM road attributes (such as max height restrictions) where they can find better data from other sources?
  - marklit 1y ago
    I haven't done any deep dives into their road data but there was ~80 GB of it, mostly from TomTom, in the August release. I think the big question would be how much overlap there is with HERE and how would the metadata compare. TomTom did a few write ups on their contribution, this one is from 2023: https://www.tomtom.com/newsroom/behind-the-map/how-tomtom-makes-overture-transportation-layer/ https://www.tomtom.com/newsroom/behind-the-map/how-tomtom-ma... If you have QGIS running, I did a walkthrough using the GeoParquet Downloader Plugin with the 2.75B Building dataset TUM released a few weeks ago. It can take any bounding box you have your workspace centred on and download the latest transport layers for Overture. No need for a custom URL as its one of the default data sources the plugin ships with. https://tech.marksblogg.com/building-footprints-gba.html https://tech.marksblogg.com/building-footprints-gba.html
dotwaffle 1y ago
From a network point of view, BitTorrent is horrendous. It has no way of knowing network topology which frequently means traffic flows from eyeball network to eyeball network for which there is no "cheap" path available (potentially causing congestion of transit ports affecting everyone) and no reliable way of forecasting where the traffic will come from making capacity planning a nightmare. Additionally, as anyone who has tried to share an internet connection with someone heavily torrenting, the excessive number of connections means overall quality of non-torrent traffic on networks goes down. Not to mention, of course, that BitTorrent has a significant stigma attached to it. The answer would have been a squid cache box before, but https makes that very difficult as you would have to install mitm certs on all devices. For container images, yes you have pull through registries etc, but not only are these non-trivial to setup (as a service and for each client) the cloud providers charge quite a lot for storage making it difficult to justify when not having a check "works just fine". The Linux distros (and CPAN and texlive etc) have had mirror networks for years that partially addresses these problems, and there was an OpenCaching project running that could have helped, but it is not really sustainable for the wide variety of content that would be cached outside of video media or packages that only appear on caches hours after publishing. BitTorrent might seem seductive, but it just moves the problem, it doesn't solve it.
- rlpb 1y ago
  > From a network point of view, BitTorrent is horrendous. It has no way of knowing network topology which frequently means traffic flows from eyeball network to eyeball network for which there is no "cheap" path available... As a consumer, I pay the same for my data transfer regardless of the location of the endpoint though, and ISPs arrange peering accordingly. If this topology is common then I expect ISPs to adjust their arrangements to cater for it, just the same as any other topology.
  - dotwaffle 1y ago
    > ISPs arrange peering accordingly Two eyeball networks (consumer/business ISPs) are unlikely to have large PNIs with each other across wide geographical areas to cover sudden bursts of traffic between them. They will, however, have substantial capacity to content networks (not just CDNs, but AWS/Google etc) which is what they will have built out. BitTorrent turns fairly predictable "North/South" traffic where capacity can be planned in advance and handed off "hot potato" as quickly as possible, into what is essentially "East/West" with no clear consistency which would cause massive amounts of congestion and/or unused capacity as they have to carry it potentially over long distances they have not been used to, with no guarantee that this large flow will exist in a few weeks time. If BitTorrent knew network topology, it could act smarter -- CDNs accept BGP feeds from carriers and ISPs so that they can steer the traffic, this isn't practical for BitTorrent!
RobotToaster 1y ago
Or IPFS/IPNS
rwmj 1y ago
With some fairly minimal changes to HTTP it would be possible to get much of the benefit of bittorrent while keeping the general principals of HTTP: https://rwmj.wordpress.com/2013/09/09/half-baked-idea-content-addressable-web-proxy/ https://rwmj.wordpress.com/2013/09/09/half-baked-idea-conten... But yes I do think Bittorrent would be ideal for OSM here.
krautsauer 1y ago
I agree with the sentiment but I need those files behind a corporate firewall. :(
7952 1y ago
A lot of people will be using this data at work where BitTorrent is a non-starter.
eleveriven 1y ago
I think a big part of why it's not more widely used comes down to a mix of UX friction, NAT/firewall issues, and lack of incentives
kmfrk 1y ago
IPFS looked like a fun middle ground, but it didn't take off. Probably didn't help that it caught the attention of some Web 3.0 people. https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System
- kevincox 1y ago
  In my experience the official software was very buggy and unreliable. Which isn't great for something about making immutable data live forever. I had bugs with silent data truncation, GC deleting live paths and the server itself just locking up and not providing anything it had to the network. The developers always seemed focus on making new versions of the protocols with very minor changes (no more protocol buffers, move everything to CBOR) rather than actually adding new features like encryption support or making it more suitable for hosting static sites (which seems to have been on of its main niches). It also would have been a great too for package repositories and other open source software archives. Large distros tend to have extensive mirror lists but you need to configure them, find out which ones have good performance for you and you can still only download from one mirror at a time. Decentralizing that would be very cool. Even if the average system doesn't seed any of the content the fact that anyone can just mirror the repo and downloads automatically start pulling from them was very nice. It also makes the download resilient to any official mirror going down or changing URL. The fact that there is strong content verification built in is also great. Typically software mirrors need to use additional levels of verification (like PGP signatures) to avoid trusting the mirror. I really like the idea, and the protocol is pretty good overall. But the implementation and evolution really didn't work well in my opinion. I tried using it for a long time, offering many of my sites over it and mirroring various data. But eventually I gave up. And maybe controversially it provided no capabilities for network separation and statistics tracking. This isn't critical for success but on entrypoint to this market is private file sharing sites. Having the option to use these things could give it a foot in the door and get a lot more people interested in development. Hopefully the next similar protocol will come at some point, maybe it will catch on where IPFS didn't.
- opan 1y ago
  I used IPFS several years ago to get some rather large files from a friend, who had recently been interested in IPFS. From what I recall it took a full week or so to start actually transferring the files. It was so slow and finicky to connect. Bittorrent is dramatically easier to use, faster, and more reliable. It was hard to take IPFS seriously after that. I also recall an IRC bot that was supposed to post links to memes at IPFS links and they were all dead, even though it's supposed to be more resilient. I don't have the backstory on that one to know how/why the links didn't work.
bombcar 1y ago
World of Warcraft used a BitTorrent-like protocol for patches for awhile, as a default option if I remember right. https://www.bluetracker.gg/wow/topic/us-en/10043224047-need-to-disable-all-peer-to-peer-for-wow/ https://www.bluetracker.gg/wow/topic/us-en/10043224047-need-... As an example mentioning it. It became disliked because of various problems and complaints, but mainly disappeared because Blozzard got the bright idea of preloading the patchset, especially to new expansions, in the weeks before. You can send down a ten gig patch a month before release, and then patch that patch a week before release, and a final small patch on the day before release, and everything is preloaded. The great Carboniferous explosion of CDNs inspired by Netflix and friends has also greatly simplified the market, too.
craftkiller 1y ago
I used to work at a company that had to deliver huge files to every developer every week. At some point they switch from a thundering herd of rsyncs to using BitTorrent. The speed gains were massive.
- Symbiote 1y ago
  Our previous cluster management software used Bittorrent for distributing application images. It took maybe 10 seconds longer for the downloads to start, but they then ran almost as fast as the master could upload one copy.
amelius 1y ago
What I wonder about is why we don't use the XOR principle more. If A is a copyrighted work, and B is pure noise, then C=A^B is also pure noise. Distribute B and C. Both of these files have nothing to do with A, because they are both pure noise. However, B^C gives you back A.
- bmn__ 1y ago
  What is the point of this? If you think you can mount an adequate defense based on xor in a court of law, then you are sorely mistaken. Any state attorney will say infringement with an additional step of obfuscation is still infringement, and any judge will follow that assessment.
  - amelius 1y ago
    The point is that you don't end in court in the first place because you were just downloading noise.
    - bmn__ 1y ago
      That's a delusion.
- lioeters 1y ago
  > C=A^B is also pure noise Is C really "pure noise" if you can get A back out of it? It's like an encoding format or primitive encryption, where A is merely transformed into unrecognizable data, meaningful noise, which still retains the entirety of the information.
  - LegionMammal978 1y ago
    > Is C really "pure noise" if you can get A back out of it? If you throw out B, then there's no possible way to get A out of C (short of blindly guessing what A is): that's one of the properties of a one-time pad. But distributing both B and C is no different than distributing A in two parts, and I'd have a hard time imagining it would be treated any differently on a legal level.
nine_k 1y ago
To use bittorrent, your machine has to listen, and otherwise be somehow reachable. In many cases, it's not a given, and sometimes not even desirable. It sticks out. I think a variant of bittorrent which may be successful in corporate and generally non-geek environments should have the following qualities: - Work via WebSockets. - Run in browser, no installation. - Have no rebel reputation. It's so obvious that it must have been implemented, likely multiple times. It would not be well-known because the very purpose of such an implementation would be to not draw attention.
- baobun 1y ago
  https://github.com/webtorrent/webtorrent https://github.com/webtorrent/webtorrent WebTorrent is ubiquitous by now and also supported by Brave and many torrent clients. There is still much room to build, though. Get in, the water is warm! https://en.wikipedia.org/wiki/WebTorrent#Adoption https://en.wikipedia.org/wiki/WebTorrent#Adoption
foobarian 1y ago
I think the reason is mainly that modern pipes are big enough that there is no need to bother with a protocol as complex as BT.
thomastjeffery 1y ago
I have a more direct answer for you: moderation. It's not all about how you distribute content. We must also decide which content do distribute, and that is a hard problem. The most successful strategy so far has been moderation. Moderation requires hierarchical authority: a moderator who arbitrarily determines which data is and is not allowed to flow. Even bittorrent traffic is moderated in most cases. For data to flow over bittorrent, two things must happen: 1. There must be one or more seeders ready to connect when the leecher starts their download. 2. There must be a way for a prospective leecher to find the torrent. The best way to meet both of these needs is with a popular tracker. So here are the pertinent questions: 1. Is your content fit for a popular tracker? Will it get buried behind all the Disney movies and porn? Does it even belong to an explicit category? If not, then you are probably going to end up running your own tracker. Does that just mean hosting a CDN with extra steps? Cloud storage is quite cheap, and the corporate consolidation of the internet by Cloudflare, Amazon, etc. has resulted in a network infrastructure that is optimized for that kind of traffic, not for bittorrent. 2. Is a popular tracker a good fit for your content? Will your prospective downloaders even think to look there? Will they be offended by the other content on that tracker, and leave? Again, a no will lead to you making your own tracker. Even in the simplest case, will users even bother to click your magnet link, or will they just use the regular CDN download that they are used to? So what about package repos? Personally, I think this would be a great fit, particularly for Nix, but it's important to be explicit about participation. Seeding is a bad default for many reasons, which means you still need a relatively reliable CDN/seed anyway. --- The internet has grown into an incredibly hierarchical network, with incredibly powerful and authoritative participants. I would love to see a revolution in decentralized computing. All of the technical needs are met, but the sociopolitical needs need serious attention. Every attempt at decentralized content distribution I have seen has met the same fate: drowned in offensive and shallow content by those who are most immediately excited to be liberated from authority. Even if it technically works, it just smells too foul to use. I propose a new strategy to replace moderation: curation. Instead of relying on authority to block out undesirable content, we should use attested curation to filter in desirable content. Want to give people the option to browse an internet without porn? Clearly and publicly attest which content is porn. Don't light the shit on fire, just open the windows and let it air out.

vaylian 1y ago

> Like container registries? Package repos, etc. I had the same thoughts for some time now. It would be really nice to distribute software and containers this way. A lot of people have the same data locally and we could just share it.

maeln 1y ago

I can imagine a few things : 1. BitTorrent has a bad rep. Most people still associate it with just illegal download. 2. It requires slightly more complex firewall rules, and asking the network admin to put them in place might raise some eyebrow for reason 1. On very restrictive network, they might not want to allow them at all due to the fact that it opens the door for, well, BitTorrent. 3. A BitTorrent client is more complicated than an HTTP client, and not installed on most company computer / ci pipeline (for lack of need, and again reason 1.). A lot of people just want to `curl` and be done with it. 4. A lot of people think they are required to seed, and for some reason that scare the hell of them. Overall, I think it is mostly 1 and the fact that you can just simply `curl` stuff and have everything working. I do sadden me that people do not understand how good of a file transfer protocol BT is and how it is underused. I do remember some video game client using BT for updates under the hood, and peertube use webtorrent, but BT is sadly not very popular.

simonmales 1y ago
At least the planet download offers BitTorrent. https://planet.openstreetmap.org/ https://planet.openstreetmap.org/
- lostmsu 1y ago
  So does Wikipedia https://meta.m.wikimedia.org/wiki/Data_dump_torrents https://meta.m.wikimedia.org/wiki/Data_dump_torrents Truly the last two open web titans.
_def 1y ago
> A lot of people think they are required to seed, and for some reason that scare the hell of them. Some of the reasons consists of lawyers sending put costly cease and desist letters even to "legitimate" users
- skrebbel 1y ago
  For seeding map data?
  - swiftcoder 1y ago
    Particularly if your torrent traffic is encrypted, they don't always bother to check what you are torrenting
    - skrebbel 1y ago
      Are you just theorizing, or is there precedent of this? I mean of lawyers sending cease and desist letters to people torrenting random encrypted streams of data?
      - swiftcoder 1y ago
        When I was working in p2p research, we used to get regular C&Ds just for scraping the tracker (not even downloading anything!). IP holders be wilding
        skrebbel 1y ago
        Appreciate your response, madness indeed!
- squigz 1y ago
  Do companies send out C&Ds for your average user torrenting? I've gotten thousands of DMCA letters but never a C&D, and I've only ever heard of 1 person getting one, and they were silly enough to be hosting a collection of paid content that they scraped themselves, from their home. DMCA demands are, as far as I'm aware, completely automated and couldn't really cost much.
Fokamul 1y ago
Lol, bad rep? Interesting, in my country everybody is using it to download movies :D Even more so now, after this botched streaming war. (EU)
- maeln 1y ago
  Which is exactly why it has a bad rep. In most people mind BitTorrent = illegal download.
  - _zoltan_ 1y ago
    downloading movies for personal use is legal in many countries.
    - ahofmann 1y ago
      This is a useless discussion. Imagine how the firewall-guy/network-team in your company will react to that argument.
      - loa_in_ 1y ago
        This is not a useless discussion just because it'll inconvenience someone who is at work anyway.
    - _flux 1y ago
      How about the uploading part of it, which is behind the magic of Bittorrent and default mode of operation?
    - lobochrome 1y ago
      Really?? Which countries allow copyright infringement by individuals?
      - Pooge 1y ago
        None. Because you projected your country's laws in the discussion, you failed to see that the countries that allow copyrighted material to be downloaded for personal usage do not qualify that download as "copyright infringement" in the first place. To answer your question with the only answer I know: Switzerland.
joao 1y ago
At a previous job, I was downloading daily legal torrent data when IT flagged me. The IT admin, eager to catch me doing something wrong, burst into the room shouting with management in tow. I had to calmly explain the situation, as management assumed all torrenting was illegal and there had been previous legal issues with an intern pirating movies. Fortunately, other colleagues backed me up.
- Yoric 1y ago
  Hey, ages ago, as an intern, I have been flagged for BitTorrent downloads. As it turned out, I was downloading/sharing Ubuntu isos, so things didn't escalate too far, but it was a scary moment. So, I'm not using BT at work anymore.
  - Symbiote 1y ago
    I left a Linux ISO (possibly Ubuntu) seeding on a lab computer at university, and forgot about it after I'd burned the DVD. You can see this was a while ago. A month later an IT admin came to ask what I might be doing with port 6881. Once I remembered, we went to the tracker's website and saw "imperial.ac.uk" had the top position for seeding, by far. The admin said to leave running.
    - account42 1y ago
      > The admin said to leave running. This can be read in two wildly different ways.
      - Symbiote 1y ago
        Fortunately, it was the nice way — that university is one of the backbone nodes of the UK academic network, so the bandwidth use was pretty much irrelevant.
- belter 1y ago
  S3 had BitTorrent support for a long time... "S3 quietly deprecates BitTorrent support" - https://news.ycombinator.com/item?id=27524549 https://news.ycombinator.com/item?id=27524549
em-bee 1y ago
seeding is uploading after you are done downloading. but you are already uploading while you are still downloading. and that can't be turned off. if seeding scares someone, then uploading should scare them too. so they are right, because they are required to upload.
- foobarbecue 1y ago
  Some BitTorrent clients make it easier to set than others, but if it's a healthy torrent I often limit upload rate to so slow that it doesn't transfer anything up. Ratio is 0.00 and I still get 100s of mb/s.
- ProtoAES256 1y ago
  If you are in the scene long enough, you should have known that there are some uncooperative clients that always send 0% (Xunlei being one of the more notorious example with their VIP schemes, and later on they would straight up spoof their client string when people started blocking them). Being a leecher nowadays is almost a norm for a lot of users, and I don't blame them since they are afraid of consequences in a more regulated jurisdiction. But a must seed when leech requirement? Hoho no, that's more like a suggestion.
  - account42 1y ago
    > Hoho no, that's more like a suggestion. For public trackers maybe.
- maeln 1y ago
  > but you are already uploading while you are still downloading. and that can't be turned off Almost every client let you set uploading limit, which you can set at 0. The only thing that generate upload bp usage that cannot be deactivated would be protocol stuff (but you can deactivate part of bt like using the DHT).
- alternatetwo 1y ago
  Transmission allows turning this off by setting upload to 0. It's simply a client setting, but most clients don't offer it.
out_of_protocol 1y ago
> It requires slightly more complex firewall rules, and asking the network admin to put them in place might raise some eyebrow for reason 1 Well, in many such situations data is provided for free, putting huge burden on the other side. Even it it's a little bit less convenient it makes service a lot more sustainable. I imagine torrent for free tier and direct download as a premium option would work perfectly
- thomastjeffery 1y ago
  I wish... I overpay more than double market value for my connection, and am not allowed to configure my router. This is the norm for most apartment dwellers in the US as far as I'm aware.
  - out_of_protocol 1y ago
    VPN is a thing
    - thomastjeffery 1y ago
      Sure, but I would also like to be able to forward ports. Right now, my best workaround is tailnet, which requires a client.
amelius 1y ago
Bad rep ... You know what has a bad rep? Big companies that use and trade my personal information like they own it. I'll start caring about copyrights when governments force these big companies to care about my information.
jauntywundrkind 1y ago
Webtorrent exists. It uses webrtc to let users connect to each other. There's support in popular trackers. This basically handles every problem stated. There's nothing to install on computers: it's just js running on the page. There's no firewall rules or port forwarding to setup, all handled by the stun/turn in webrtc. Users wouldn't necessarily even be aware they are uploading.
- account42 1y ago
  STUN is not always possible and TURN means proxying the connection through a server which would be counter-productive for the purpose of using bit-torrent as an alternative to direct HTTP downloads as you are now paying for the bandwidth in both directions. This is very much not a problem with magic solutions.
  - jauntywundrkind 1y ago
    Agreed! But STUN's success rate is pretty good! As the number of peers goes up it should be less likely that one would need to use TURN to connect to a peer, but I am skeptical webrtc is built to fall back like this, to try other peers first. The advantage is that at least it's all builtin. It's not a magic solution, but it's a pretty good solution, with fallbacks builtin for when the networking gets in the way of the magic.
account42 1y ago
5. Most residential internet connections are woefully underprovisioned for upload so anything that uses it more (and yes you need people to seed for bittorrent to make sense) can slow down the entire connection. 6. Service providers have little control over the service level of seeders and thus the user experience. And that's before you get malicious users.

nativeit 1y ago

I assume it’s simply the lack of the inbuilt “universal client” that http enjoys, or that devs tend to have with ssh/scp. Not that such a client (even an automated/scripted CLI client) would be so difficult to setup, but then trackers are also necessary, and then the tooling for maintaining it all. Intuitively, none of this sounds impossible, or even necessarily that difficult apart from a few tricky spots. I think it’s more a matter of how large the demand is for frequent downloads of very large files/sets, which leads to a questions of reliability and seeding volume, all versus the effort involved to develop the tooling and integrate it with various RCS and file syncing services. Would something like Git LFS help here? I’m at the limit of my understanding for this.

nativeit 1y ago
I certainly take advantage of BitTorrent mirrors for downloading Debian ISOs, as they are generally MUCH faster.
- nopurpose 1y ago
  All Linux ISOs collectors in the world wholeheartedly agree.
- Sesse__ 1y ago
  Are you serious? Most Debian ISO mirrors I've used have 10gig connectivity and usually push a gigabit or two fairly easily. BitTorrent is generally a lot slower than that (it's a pretty terrible protocol for connecting you to actually fast peers and getting stuff quickly from them).
  - queenkjuul 1y ago
    I've definitely seen higher speeds with BitTorrent, pretty easily maxing out my gbe nics, but I'm not downloading Debian images specifically with much frequency.
mschuster91 1y ago
Trackers haven't been necessary for well over a decade now thanks to DHT.

zaphodias 1y ago

I remember seeing the concept of "torrents with dynamic content" a few years ago, but apparently never became a thing[1]. I kind of wish it did, but I don't know if there are critical problems (i.e. security?). [1]: https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html

charcircuit 1y ago

AFAIK Bittorrent doesn't allow for updating the files for a torrent.

out_of_protocol 1y ago
This is a feature, not a bug. Torrent file/magnet link contains hash of a data which is immutable. Just publish new link (you should anyway, even with http)
- opan 1y ago
  Agreed, it's cool that you can reverify a torrent from 3 years ago and make sure the data on your disk isn't lost or damaged.
  - account42 1y ago
    That's useful if you did not prepare for that eventuality. But if you do want to store data long-term it's better to generate error-correcting codes like PAR2 so that you can actually recover partial errors without having to rely on seeders being available - and when you do that you no longer need the torrent for verification.
- charcircuit 1y ago
  Imagine if for websites you had to get a brand new domain if you ever wanted to change the contents of your web page. You can't just go to google.com because it would be immutable. You would have to somehow know you have to go to google10393928.com and any links to google on the internet would be linking to some old version. The ability to have a link always refer to the latest version is useful. The same applies to torrents. It's possible for the magnet link to the latest version to get lost and then a bunch of people are accidently downloading old, worse versions of the file with no way to find the newest version.
  - jacquesm 1y ago
    What a dumb strawman. Domains are not hashes.

deleted 1y ago

[deleted]

trenchpilgrim 1y ago

> Like container registries? https://github.com/uber/kraken https://github.com/uber/kraken exists, using a modified BT protocol, but unless you are distributing quite large images to a very large number of nodes, a centralized registry is probably faster, simpler and cheaper

marklit 1y ago

Amazon, Esri, Grab, Hyundai, Meta, Microsoft, Precisely, Tripadvisor and TomTom, along with 10s of other businesses got together and offer OSM data in Parquet on S3 free of charge. You can query it surgically and run analytics on it needing only MBs of bandwidth on what is a multi-TB dataset at this point. https://tech.marksblogg.com/overture-dec-2024-update.html https://tech.marksblogg.com/overture-dec-2024-update.html If you're using ArcGIS Pro, use this plugin: https://tech.marksblogg.com/overture-maps-esri-arcgis-pro.html https://tech.marksblogg.com/overture-maps-esri-arcgis-pro.ht...

willtemperley 1y ago
It's just great that bounding box queries can be translated into HTTP range requests.
n4r9 1y ago
As someone who works with mapping data for HGV routing, I've been keeping an eye on Overture. I wonder do you know if anyone has measured the data coverage and quality between this and proprietary datasets like HEREmaps? Does Overture supplement OSM road attributes (such as max height restrictions) where they can find better data from other sources?
- marklit 1y ago
  I haven't done any deep dives into their road data but there was ~80 GB of it, mostly from TomTom, in the August release. I think the big question would be how much overlap there is with HERE and how would the metadata compare. TomTom did a few write ups on their contribution, this one is from 2023: https://www.tomtom.com/newsroom/behind-the-map/how-tomtom-makes-overture-transportation-layer/ https://www.tomtom.com/newsroom/behind-the-map/how-tomtom-ma... If you have QGIS running, I did a walkthrough using the GeoParquet Downloader Plugin with the 2.75B Building dataset TUM released a few weeks ago. It can take any bounding box you have your workspace centred on and download the latest transport layers for Overture. No need for a custom URL as its one of the default data sources the plugin ships with. https://tech.marksblogg.com/building-footprints-gba.html https://tech.marksblogg.com/building-footprints-gba.html

dotwaffle 1y ago

From a network point of view, BitTorrent is horrendous. It has no way of knowing network topology which frequently means traffic flows from eyeball network to eyeball network for which there is no "cheap" path available (potentially causing congestion of transit ports affecting everyone) and no reliable way of forecasting where the traffic will come from making capacity planning a nightmare. Additionally, as anyone who has tried to share an internet connection with someone heavily torrenting, the excessive number of connections means overall quality of non-torrent traffic on networks goes down. Not to mention, of course, that BitTorrent has a significant stigma attached to it. The answer would have been a squid cache box before, but https makes that very difficult as you would have to install mitm certs on all devices. For container images, yes you have pull through registries etc, but not only are these non-trivial to setup (as a service and for each client) the cloud providers charge quite a lot for storage making it difficult to justify when not having a check "works just fine". The Linux distros (and CPAN and texlive etc) have had mirror networks for years that partially addresses these problems, and there was an OpenCaching project running that could have helped, but it is not really sustainable for the wide variety of content that would be cached outside of video media or packages that only appear on caches hours after publishing. BitTorrent might seem seductive, but it just moves the problem, it doesn't solve it.

rlpb 1y ago
> From a network point of view, BitTorrent is horrendous. It has no way of knowing network topology which frequently means traffic flows from eyeball network to eyeball network for which there is no "cheap" path available... As a consumer, I pay the same for my data transfer regardless of the location of the endpoint though, and ISPs arrange peering accordingly. If this topology is common then I expect ISPs to adjust their arrangements to cater for it, just the same as any other topology.
- dotwaffle 1y ago
  > ISPs arrange peering accordingly Two eyeball networks (consumer/business ISPs) are unlikely to have large PNIs with each other across wide geographical areas to cover sudden bursts of traffic between them. They will, however, have substantial capacity to content networks (not just CDNs, but AWS/Google etc) which is what they will have built out. BitTorrent turns fairly predictable "North/South" traffic where capacity can be planned in advance and handed off "hot potato" as quickly as possible, into what is essentially "East/West" with no clear consistency which would cause massive amounts of congestion and/or unused capacity as they have to carry it potentially over long distances they have not been used to, with no guarantee that this large flow will exist in a few weeks time. If BitTorrent knew network topology, it could act smarter -- CDNs accept BGP feeds from carriers and ISPs so that they can steer the traffic, this isn't practical for BitTorrent!

RobotToaster 1y ago

Or IPFS/IPNS

rwmj 1y ago

With some fairly minimal changes to HTTP it would be possible to get much of the benefit of bittorrent while keeping the general principals of HTTP: https://rwmj.wordpress.com/2013/09/09/half-baked-idea-content-addressable-web-proxy/ https://rwmj.wordpress.com/2013/09/09/half-baked-idea-conten... But yes I do think Bittorrent would be ideal for OSM here.

krautsauer 1y ago

I agree with the sentiment but I need those files behind a corporate firewall. :(

7952 1y ago

A lot of people will be using this data at work where BitTorrent is a non-starter.

eleveriven 1y ago

I think a big part of why it's not more widely used comes down to a mix of UX friction, NAT/firewall issues, and lack of incentives

kmfrk 1y ago

IPFS looked like a fun middle ground, but it didn't take off. Probably didn't help that it caught the attention of some Web 3.0 people. https://en.wikipedia.org/wiki/InterPlanetary_File_System https://en.wikipedia.org/wiki/InterPlanetary_File_System

kevincox 1y ago
In my experience the official software was very buggy and unreliable. Which isn't great for something about making immutable data live forever. I had bugs with silent data truncation, GC deleting live paths and the server itself just locking up and not providing anything it had to the network. The developers always seemed focus on making new versions of the protocols with very minor changes (no more protocol buffers, move everything to CBOR) rather than actually adding new features like encryption support or making it more suitable for hosting static sites (which seems to have been on of its main niches). It also would have been a great too for package repositories and other open source software archives. Large distros tend to have extensive mirror lists but you need to configure them, find out which ones have good performance for you and you can still only download from one mirror at a time. Decentralizing that would be very cool. Even if the average system doesn't seed any of the content the fact that anyone can just mirror the repo and downloads automatically start pulling from them was very nice. It also makes the download resilient to any official mirror going down or changing URL. The fact that there is strong content verification built in is also great. Typically software mirrors need to use additional levels of verification (like PGP signatures) to avoid trusting the mirror. I really like the idea, and the protocol is pretty good overall. But the implementation and evolution really didn't work well in my opinion. I tried using it for a long time, offering many of my sites over it and mirroring various data. But eventually I gave up. And maybe controversially it provided no capabilities for network separation and statistics tracking. This isn't critical for success but on entrypoint to this market is private file sharing sites. Having the option to use these things could give it a foot in the door and get a lot more people interested in development. Hopefully the next similar protocol will come at some point, maybe it will catch on where IPFS didn't.
opan 1y ago
I used IPFS several years ago to get some rather large files from a friend, who had recently been interested in IPFS. From what I recall it took a full week or so to start actually transferring the files. It was so slow and finicky to connect. Bittorrent is dramatically easier to use, faster, and more reliable. It was hard to take IPFS seriously after that. I also recall an IRC bot that was supposed to post links to memes at IPFS links and they were all dead, even though it's supposed to be more resilient. I don't have the backstory on that one to know how/why the links didn't work.

bombcar 1y ago

World of Warcraft used a BitTorrent-like protocol for patches for awhile, as a default option if I remember right. https://www.bluetracker.gg/wow/topic/us-en/10043224047-need-to-disable-all-peer-to-peer-for-wow/ https://www.bluetracker.gg/wow/topic/us-en/10043224047-need-... As an example mentioning it. It became disliked because of various problems and complaints, but mainly disappeared because Blozzard got the bright idea of preloading the patchset, especially to new expansions, in the weeks before. You can send down a ten gig patch a month before release, and then patch that patch a week before release, and a final small patch on the day before release, and everything is preloaded. The great Carboniferous explosion of CDNs inspired by Netflix and friends has also greatly simplified the market, too.

craftkiller 1y ago

I used to work at a company that had to deliver huge files to every developer every week. At some point they switch from a thundering herd of rsyncs to using BitTorrent. The speed gains were massive.

Symbiote 1y ago
Our previous cluster management software used Bittorrent for distributing application images. It took maybe 10 seconds longer for the downloads to start, but they then ran almost as fast as the master could upload one copy.

amelius 1y ago

What I wonder about is why we don't use the XOR principle more. If A is a copyrighted work, and B is pure noise, then C=A^B is also pure noise. Distribute B and C. Both of these files have nothing to do with A, because they are both pure noise. However, B^C gives you back A.

bmn__ 1y ago
What is the point of this? If you think you can mount an adequate defense based on xor in a court of law, then you are sorely mistaken. Any state attorney will say infringement with an additional step of obfuscation is still infringement, and any judge will follow that assessment.
- amelius 1y ago
  The point is that you don't end in court in the first place because you were just downloading noise.
  - bmn__ 1y ago
    That's a delusion.
lioeters 1y ago
> C=A^B is also pure noise Is C really "pure noise" if you can get A back out of it? It's like an encoding format or primitive encryption, where A is merely transformed into unrecognizable data, meaningful noise, which still retains the entirety of the information.
- LegionMammal978 1y ago
  > Is C really "pure noise" if you can get A back out of it? If you throw out B, then there's no possible way to get A out of C (short of blindly guessing what A is): that's one of the properties of a one-time pad. But distributing both B and C is no different than distributing A in two parts, and I'd have a hard time imagining it would be treated any differently on a legal level.

nine_k 1y ago

To use bittorrent, your machine has to listen, and otherwise be somehow reachable. In many cases, it's not a given, and sometimes not even desirable. It sticks out. I think a variant of bittorrent which may be successful in corporate and generally non-geek environments should have the following qualities: - Work via WebSockets. - Run in browser, no installation. - Have no rebel reputation. It's so obvious that it must have been implemented, likely multiple times. It would not be well-known because the very purpose of such an implementation would be to not draw attention.

baobun 1y ago
https://github.com/webtorrent/webtorrent https://github.com/webtorrent/webtorrent WebTorrent is ubiquitous by now and also supported by Brave and many torrent clients. There is still much room to build, though. Get in, the water is warm! https://en.wikipedia.org/wiki/WebTorrent#Adoption https://en.wikipedia.org/wiki/WebTorrent#Adoption

foobarian 1y ago

I think the reason is mainly that modern pipes are big enough that there is no need to bother with a protocol as complex as BT.

thomastjeffery 1y ago

I have a more direct answer for you: moderation. It's not all about how you distribute content. We must also decide which content do distribute, and that is a hard problem. The most successful strategy so far has been moderation. Moderation requires hierarchical authority: a moderator who arbitrarily determines which data is and is not allowed to flow. Even bittorrent traffic is moderated in most cases. For data to flow over bittorrent, two things must happen: 1. There must be one or more seeders ready to connect when the leecher starts their download. 2. There must be a way for a prospective leecher to find the torrent. The best way to meet both of these needs is with a popular tracker. So here are the pertinent questions: 1. Is your content fit for a popular tracker? Will it get buried behind all the Disney movies and porn? Does it even belong to an explicit category? If not, then you are probably going to end up running your own tracker. Does that just mean hosting a CDN with extra steps? Cloud storage is quite cheap, and the corporate consolidation of the internet by Cloudflare, Amazon, etc. has resulted in a network infrastructure that is optimized for that kind of traffic, not for bittorrent. 2. Is a popular tracker a good fit for your content? Will your prospective downloaders even think to look there? Will they be offended by the other content on that tracker, and leave? Again, a no will lead to you making your own tracker. Even in the simplest case, will users even bother to click your magnet link, or will they just use the regular CDN download that they are used to? So what about package repos? Personally, I think this would be a great fit, particularly for Nix, but it's important to be explicit about participation. Seeding is a bad default for many reasons, which means you still need a relatively reliable CDN/seed anyway. --- The internet has grown into an incredibly hierarchical network, with incredibly powerful and authoritative participants. I would love to see a revolution in decentralized computing. All of the technical needs are met, but the sociopolitical needs need serious attention. Every attempt at decentralized content distribution I have seen has met the same fate: drowned in offensive and shallow content by those who are most immediately excited to be liberated from authority. Even if it technically works, it just smells too foul to use. I propose a new strategy to replace moderation: curation. Instead of relying on authority to block out undesirable content, we should use attested curation to filter in desirable content. Want to give people the option to browse an internet without porn? Clearly and publicly attest which content is porn. Don't light the shit on fire, just open the windows and let it air out.