6 ms·

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The

Unpopular opinion: Severely rate limit retrieving the files from the website / HTTP endpoint, and loudly point towards downloading the files via torrents. The torrent protocol was meant to relieve this level of server load in mind.

userbinator 3y ago
and loudly point towards downloading the files via torrents. By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?
- x-complexity 3y ago
  ...that can work.
- remram 3y ago
  A Link header would be more appropriate (maybe with rel="alternate"?)
dan_wood 3y ago
[flagged]
- legends2k 3y ago
  Torrents aren't illegal inherently; the content downloaded are often copyright violations, yes, but using torrenting protocol itself isn't illegal. Most Linux distributions provide a torrent option to download their OS image file.
- chaxor 3y ago
  Torrents are the backbone of a lot of academic research downloads. They're the absolute best way to distribute neural network weights, or large models used in physical science studies. That or IPFS. If you use huggingface or dockerhub for anything like this, seriously consider switching, as it will be extremely useful when those centralized systems inevitably fail.
maksimur 3y ago
Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.
- 5e92cb50239222b 3y ago
  How is that worse than serving files via HTTP? With HTTP archive.org is the only peer that can serve files to you, with BitTorrent it will be one of the (hopefully) many peers, and will degrade to the same level of redundancy as HTTP if all other peers leave. BitTorrent also supports web seeds and they don't even really have to keep a full client running, just embed an HTTP link into the .torrent file.
- chronogram 3y ago
  Have you tried an Archive.org torrent yet? It's backed by the servers, but has the advantages of selecting which parts of the archive you want and being verifiable and being able to have more bandwidth on popular archives. The Archive.org servers show up on the "HTTP sources" tab next to the "Peers" tab for me.
- happywolf 3y ago
  Seeding peers or machines are meant to mitigate this issue.
- nextaccountic 3y ago
  The very same machines that serve http can also seed torrents. If those machines go offline, they also take down the http endpoint
- shepherdjerred 3y ago
  That's no worse than just relying on Arhcive.org to be sole provider
- maksimur 3y ago
  Thanks everyone for your reply. It made me reconsider my opinion. Though the downvotes are unnecessary. Use them when they're actually necessary. Not as gate keeping or as a "don't agree" button or whatever frivolous thing is passing through your mind.
ranger_danger 3y ago
Depending on the type of content you download, the chances of anyone else already seeding that might actually be zero.
- andruby 3y ago
  Archive.org would also act as a seed, so at worst it’s similar to regular download traffic. At best there’s a few more people downloading and seeding.

userbinator 3y ago

and loudly point towards downloading the files via torrents. By "loudly", perhaps actually redirect to the .torrent instead, for those who trip the rate limit?

x-complexity 3y ago
...that can work.
remram 3y ago
A Link header would be more appropriate (maybe with rel="alternate"?)

dan_wood 3y ago

[flagged]

legends2k 3y ago
Torrents aren't illegal inherently; the content downloaded are often copyright violations, yes, but using torrenting protocol itself isn't illegal. Most Linux distributions provide a torrent option to download their OS image file.
chaxor 3y ago
Torrents are the backbone of a lot of academic research downloads. They're the absolute best way to distribute neural network weights, or large models used in physical science studies. That or IPFS. If you use huggingface or dockerhub for anything like this, seriously consider switching, as it will be extremely useful when those centralized systems inevitably fail.

maksimur 3y ago

Torrents have the habit of disappearing when no users keep them alive. It happened to me enough times to be wary of such solution. If there's a way to keep them alive regardless of interest I'm all for it.

5e92cb50239222b 3y ago
How is that worse than serving files via HTTP? With HTTP archive.org is the only peer that can serve files to you, with BitTorrent it will be one of the (hopefully) many peers, and will degrade to the same level of redundancy as HTTP if all other peers leave. BitTorrent also supports web seeds and they don't even really have to keep a full client running, just embed an HTTP link into the .torrent file.
chronogram 3y ago
Have you tried an Archive.org torrent yet? It's backed by the servers, but has the advantages of selecting which parts of the archive you want and being verifiable and being able to have more bandwidth on popular archives. The Archive.org servers show up on the "HTTP sources" tab next to the "Peers" tab for me.
happywolf 3y ago
Seeding peers or machines are meant to mitigate this issue.
nextaccountic 3y ago
The very same machines that serve http can also seed torrents. If those machines go offline, they also take down the http endpoint
shepherdjerred 3y ago
That's no worse than just relying on Arhcive.org to be sole provider
maksimur 3y ago
Thanks everyone for your reply. It made me reconsider my opinion. Though the downvotes are unnecessary. Use them when they're actually necessary. Not as gate keeping or as a "don't agree" button or whatever frivolous thing is passing through your mind.

ranger_danger 3y ago

Depending on the type of content you download, the chances of anyone else already seeding that might actually be zero.

andruby 3y ago
Archive.org would also act as a seed, so at worst it’s similar to regular download traffic. At best there’s a few more people downloading and seeding.