25 ms·

Ask HN: What is your favorite method of sending large files?

319 points by mettamage 6y ago

I just opened up a simple HTTP server to send someone a large file. Then I figured, I never gave this question proper thought. But some of you have, and I figured they make for fun and interesting stories ;-) So what's your favorite method to send large files, of at least 5GB or bigger? Though, I'm also curious on how you'd send 10TB or more.

erezsh 6y ago
I'm partial to https://mega.nz/ https://mega.nz/
hosteur 6y ago
I use http://ifile.dk http://ifile.dk Simple. Works.
kvn_95 6y ago
Magic Wormhole (https://magic-wormhole.readthedocs.io/en/latest/ https://magic-wormhole.readthedocs.io/en/latest/) if it's between OSes. Between OSX, AirDrop works very well. I have sent >10GB files between Macs, quite quick as well. I have never send a 10TB file so I wouldn't know. None of my drives are that large yet :)
- mettamage 6y ago
  I use the magic of AirDrop as well :) Unfortunately, sometimes I'm on my Linux machine (in which case python3 -m http.server works just fine), or need to send a file to someone else and then I'd like to have at least some form of monitoring/progress. Edit: magic wormhole is awesome! Thanks! :D
- theobeers 6y ago
  Magic Wormhole is good. These days I use croc,[0] which I find even better. [0]: https://github.com/schollz/croc https://github.com/schollz/croc
  - kvn_95 6y ago
    Oh this looks great! Thanks for this, checking it out.
  - psanford 6y ago
    I never understood why croc chose to copy the features of Magic Wormhole but implement them in an incompatible way. Partially out of my annoyance with this I implemented a port of Magic Wormhole in Go that fully interoperates with the official python version: https://github.com/psanford/wormhole-william https://github.com/psanford/wormhole-william
    - qrv3w 6y ago
      I initiated the croc project. Others have asked me the same thing, so I'll bite: I chose not to copy the features of magic wormhole because it seemed to me (at the time) to be very complicated and lacking in features crucial to myself (namely restarting transfers [1], sending folders without zipping them). This has been to my benefit, I think, because I can iterate on croc quickly without having to conform to the wormhole spec and I've since added features still not available in wormhole (like restarting transfers [1], and ipv6 support [2]). That being said, I really like magic wormhole and consider it my inspiration. And I'm glad that you ported it to Go! [1]: https://github.com/warner/magic-wormhole/issues/88 https://github.com/warner/magic-wormhole/issues/88 [2]: https://github.com/warner/magic-wormhole/issues/354 https://github.com/warner/magic-wormhole/issues/354
      - clankyclanker 6y ago
        Would it be possible to sniff the protocol on the wire and provide some sort of cross-compatible magic wormcroc service?
- kilroy123 6y ago
  I once sent a huge file over airdrop and was shocked how much faster it was then sending through the local LAN.
  - mistrial9 6y ago
    some copy methods verify and check during the transfer operation, others do not, so the final speeds are very different between them
    - icebraining 6y ago
      The major difference is that Airdrop establishes a direct WiFi connection, bypassing the router completely.
- emit_time 6y ago
  Just had a 41GB set of files sent to me in one transfer using magic-wormhole. Worked first try without a hitch. Pretty awesome from the receiving end.
- Zenbit_UX 6y ago
  This is local lan only right? Or are you port forwarding to expose it?
britmob 6y ago
Resilio Sync (formerly bittorent sync) is my go-to for any file larger than a few hundred megabytes.
- misterbwong 6y ago
  Same here. I use this to deploy large files to my VPS
  - willcipriano 6y ago
    I third this suggestion, if you have to send the file to multiple parties they can all share among themselves (it's BitTorrent after all) speeding up the process even further.
thedanbob 6y ago
I send public links from the Seafile instance running on my home server. For 10+ TB, I think the only practical option would be to mail a hard drive or two.
makeworld 6y ago
I like gofile.io, as it's private and has no limits. But for something like 10 TB or more, I'd see a torrent as the only way. My uploads speeds are too slow for anything else, the connection would be reset. The torrent also helps prevent corruption.
hotwire 6y ago
A station wagon full of tapes hurtling down the highway...
- auxym 6y ago
  No one makes station wagons anymore, except luxury european makers :(
  - BenjiWiebe 6y ago
    There's plenty of used ones available. Nice and cheap.
  - celim307 6y ago
    Subaru Outback’s
  - myself248 6y ago
    Aside from the already-mentioned Subaru Outback, there's the Buick Regal TourX, the Mini Cooper Clubman, and I'll contend that the Dodge Journey is more stationwagon than SUV; it's on the Avenger sedan platform.
- parliament32 6y ago
  We laugh, but even Amazon has realized sometimes literally driving drives is the cheapest and fastest way to move serious amounts of data: https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
  - lmkg 6y ago
    Google Cloud also has an offering here, although it doesn't literally come with its own truck. https://cloud.google.com/transfer-appliance/ https://cloud.google.com/transfer-appliance/ There's an amusing infographic towards the bottom. Only 42 days!
- maest 6y ago
  Some exchanges still sell historical tick data by posting over large amounts of HDD/SSD with the data preloaded.
villgax 6y ago
Backblaze/Torrent
cpach 6y ago
Related: If you need to transfer sensitive data over Bittorent, Age is a good tool for encrypting it before transmission. https://github.com/FiloSottile/age https://github.com/FiloSottile/age
- parliament32 6y ago
  Why a new tool? How is this better than gpg symmetric encryption, considering gpg is installed/available effectively everywhere? Encrypt: gpg --symmetric file.dat (enter a password) Decrypt: gpg --decrypt file.dat.gpg > file.dat (enter the password)
  - cpach 6y ago
    IMO, GPG is simply not a good tool and it should be replaced. Others have said it better than I can. See e.g. https://latacora.micro.blog/2019/07/16/the-pgp-problem.html https://latacora.micro.blog/2019/07/16/the-pgp-problem.html
    - forgotmypw17 6y ago
      That's said often about any tool which has been around long enough. People without experience come around and think they can replace an old tool with a better one, but it's usually only ignorance of either the complexity of the task, knowledge of using the tool properly, or both.
      - cpach 6y ago
        GPG is just not a very good tool. I think 'tptacek explained it quite well in the article I linked. With that said, Magic Wormhole is also a very good tool for transferring files. It will encrypt in transit. So for many files, using a separate encryption tool is not necessary. (So far I haven’t tried it for large files.)
        parliament32 6y ago
LinuxBender 6y ago
SFTP Chroot server. lftp client using mirror subsystem + sftp. It is multi-threaded, even for a single file and supports rsync like behavior even in sftp chroot. I can max out any internet link using lftp (client) + sftp (protocol).
wilsonnb3 6y ago
USPS
parliament32 6y ago
I self-host a Nextcloud instance. If it's too big for a browser download, Bittorrent is the way to go.
- anderspitman 6y ago
  This is interesting. What conditions would make a file too big for a browser download? If the server supports range requests (I'm almost certain Nextcloud does), then the browser can download in chunks and just retry any failed chunks.
  - parliament32 6y ago
    It's technically possible, but most browsers don't auto-resume, and most webservers will timeout a request after a certain amount of time has elapsed. You can get around all this with browser extensions, but it's honestly easier to use a more robust protocol (this is in the context of, say, a 10TB file that'll take you a few days to download).
    - anderspitman 6y ago
      I don't see how anything would be much more robust than HTTP. Any network based protocol can fail at any time, so you're going to have to have some concept of tracking your progress and retrying failed requests. Both are easy to do with HTTP. Can you give a concrete example of something that would work better than HTTP?
      - parliament32 6y ago
        Here's an exercise for you: In your browser, start downloading a test file, like [1]. Turn off your network card for a few seconds, then turn it back on. See what your browser does. Next, start downloading a large file over Bittorrent, like [2]. Turn off your network card for a few seconds, then turn it back on. See what the client does. [1] http://speedtest.tele2.net/100GB.zip http://speedtest.tele2.net/100GB.zip [2] https://cdimage.debian.org/debian-cd/current/amd64/bt-dvd/debian-10.5.0-amd64-DVD-1.iso.torrent https://cdimage.debian.org/debian-cd/current/amd64/bt-dvd/de...
t312227 6y ago
as a game of thrones fan: a raven with a sufficient large micro-sd card ;)
BenjiWiebe 6y ago
Sending a 10TB file on my internet connection would take 2.5 years of constant uploading. Shipping a hard drive is cheaper and quicker.
- njsubedi 6y ago
  8x10,240,000Mbits /2.5/365/86400 = 1.03Mbs?
  - rsa25519 6y ago
    That's slightly faster than my Silicon Valley upload speed.
simon_acca 6y ago
Google cloud storage. Ui is actually nice, permissions are flexible and has both a robust web uploader as well as a cli one. Not to mention it’s cheap and only billed by the GB/hour or something.
zepearl 6y ago
Personally I would use my own Nextcloud instance for up to 20-30GBs. Not sure about TBs. What about using "Firefox Send"? (I never used it so far) https://support.mozilla.org/en-US/kb/send-files-anyone-securely-firefox-send https://support.mozilla.org/en-US/kb/send-files-anyone-secur... I read that the limit is 1-2.5GBs => maybe you could break down the file and upload it in multiple pieces... . EDIT: oopps, Firefox Send doesn seem to be available anymore - https://support.mozilla.org/en-US/kb/what-happened-firefox-send https://support.mozilla.org/en-US/kb/what-happened-firefox-s...
- Apreche 6y ago
  People were using Firefox Send to "send" malware and other bad things to people. That's what happens if you let anyone anonymously host files on your domain for free.
  - butz 6y ago
    And how other platforms, like wetransfer, are dealing with that?
cheeze 6y ago
python -m http.server
emerged 6y ago
upload to s3 bucket, make public, send link
badrabbit 6y ago
ngrok.io and similar reverse tunnels
akerro 6y ago
>I'm also curious on how you'd send 10TB or more. For my 12TiB of data I use Syncthing when I need to sync them more often, rsync. I used rsync several times for billions of smaller files totalling to 300GiB, but really all depending on how I connect nodes. I prefer syncthing, but when only ssh is available, then rsync is good too. Currently largest synced directory by syncthing (that shares usage stats) is over 61384 GiB :)) https://data.syncthing.net/ https://data.syncthing.net/
- temp007007 6y ago
  Is there an iOS client for Syncthing?
  - KindOne 6y ago
    No. https://docs.syncthing.net/users/faq.html#why-is-there-no-ios-client https://docs.syncthing.net/users/faq.html#why-is-there-no-io...
hprotagonist 6y ago
magic-wormhole: https://magic-wormhole.readthedocs.io/en/latest/ https://magic-wormhole.readthedocs.io/en/latest/ pipx install magic-wormhole wormhole send /some/big/stupid/thing.tar.gz
- anaganisk 6y ago
  Also install microsoft visual c++ build tools which is like 1.2GB :(
codethief 6y ago
Syncthing works pretty well in my experience, even across NAT. I recently shared 10GB worth of photos with a family member and ~30min later it was done.
Apreche 6y ago
Upload to cloud storage then have the other party download it. If it's too big for cloud storage, ship a hard drive in the mail.
maliker 6y ago
https://transfer.pcloud.com https://transfer.pcloud.com
jve 6y ago
Private Nextcloud. Or some free tier OneDrive/alternative is enought for me. However no-one mentioned a super simple service: https://wetransfer.com/ https://wetransfer.com/ - Simple as drag & drop, enter recipient address, SEND. Pretty simple if you want non techie to send you something.
- meigwilym 6y ago
  +1 for WeTransfer. A dead simple UI and a link is nicely emailed to the receiver.
  - spodek 6y ago
    Also emails the sender when the receiver downloads it.
- _Microft 6y ago
  I can recommend WeTransfer as well. Super simple interface, no sign-up required, free.
- dash2 6y ago
  I work with geneticists and they all seem to use wetransfer for big data files.
- epanchin 6y ago
  Wetransfer is brilliant. I’ve used their white label option in the past to transfer files to customers and allow them to send in files. It’s ace.
AnIdiotOnTheNet 6y ago
Locally: SMB. Over the internet: SendGB. Ridiculously sized: HDDs in a car.
esaym 6y ago
Usually a local HTTP server. But I've wondered before if binary data could be encoded in to a video file and then uploaded to youtube or some other video service...
- RealStickman_ 6y ago
  Any compression would ruin that. Though it should be possible to do what you described with 16MP pictures and upload them to google drive. Those won't be compressed according to someone under that LTT video where they abuse 5 business accounts for their backup.
  - esaym 6y ago
    >Any compression would ruin that. Well not really. You could have an entire frame as a solid color red to represent a '1' and a solid white frame to represent a '0'. At 24 fps, that is 3 bytes per second and would certainly survive compression. Just keep drilling down, half a frame that is red and another half that is white would represent a '10' pair, now you've doubled your throughput. You could keep cutting these blocks down until you get to a limit, you certainly couldn't get down to a single pixel, but some factor of that. Plus you could use an encoding with error correction instead of just raw binary data.
    - ta95846893 6y ago
      Sounds like you'll end up with a couple of qr codes per frame in the end, which would probably survive compression
- Budabellly 6y ago
  example of that! https://news.ycombinator.com/item?id=19916205 https://news.ycombinator.com/item?id=19916205
mullen 6y ago
If the data is below 50G in size and on my personal computer, then I just drop it into the Google Drive folder and it syncs to Google over night. When it is done, I export it and then send a link to the person(s). I pay $2 a month for the 100G account and I usually have about 50G disk space unused, so this is not an issue for me. If it is above 50G and on my personal computer, I encrypt the data and then physically mail a USB stick with the data on it. Trying to arrange the downloading/uploading of 50G of data from my personal computer to another personal computer is a real pain. The people that I would send/receive that much data to/from, that is stored on my personal computer are usually people who don't know much about ftp, scp or easily sharing files over the Internet. Sending a USB stick is just so much easier and in many cases, faster. I make sure the recipients know how to decrypt the data before sending the data. If it is on a server (For example, a side project instance in AWS), then I drop it into S3 bucket, export it and send the URL to the recipient. I just eat the cost of them downloading it. Usually I am making money off the work anyways, so it is the cost of doing business.
- lowwave 6y ago
  >If it is on a server (For example, a side project instance in AWS), then I drop it into S3 bucket, export it and send the URL to the recipient. I just eat the cost of them downloading it. Usually I am making money off the work anyways, so it is the cost of doing business. Consider BunnyCDN way cheaper than s3.
  - StavrosK 6y ago
    BunnyCDN is a CloudFront alternative, not an S3 alternative.
    - farmdawgnation 6y ago
      They also offer an object storage product, it seems: https://bunnycdn.com/solutions/cdn-cloud-storage https://bunnycdn.com/solutions/cdn-cloud-storage
  - incredimike 6y ago
GRBurst 6y ago
It depends in whether I want to send it to someone in the local network or through the internet. For local network: I use miniserve ( https://github.com/svenstaro/miniserve https://github.com/svenstaro/miniserve ) which is just a simple http server. There are similar tools for when I want to share it from the smartphone. Through the internet it really varies: Sometimes it is Firefox send ( https://send.firefox.com/ https://send.firefox.com/ ) For photos, I use a self hosted photo solution piwigo ( https://github.com/Piwigo/Piwigo https://github.com/Piwigo/Piwigo ) In earlier days it has been a self hosted nextcloud ( https://github.com/nextcloud/server https://github.com/nextcloud/server ) instance. I still use it when the files are getting too large for Firefox send. I also tried the already mentioned wormhole but this works only with tec ppl.
- lowwave 6y ago
  for local network. You can just do python3 -m http.server in the dir, or python -m SimpleHTTPServer (python2) Or npm install -g http-server for nodejs, then http-server Local network is easy.
  - domh 6y ago
    If you're using a recent version of node/npm, you can just do: npx http-server . To remove the global install step. This will install and run all in one command.
- cprecioso 6y ago
  Firefox Send doesn’t work anymore
  - mkl 6y ago
    That's unfortunate. That was my go-to method. From https://send.firefox.com/ https://send.firefox.com/: "Firefox Send is temporarily unavailable while we work on product improvements. We appreciate your patience while we make the Firefox Send experience better."
    - 6y ago
usmannk 6y ago
For local networks I use the builtin python http server `python3 -m http.server` (or `python -m SimpleHTTPServer` in 2.x). By default it binds to all so anyone on your network can access it.
- tandav 6y ago
  When send large files I often get broken pipe error using this method
notatoad 6y ago
if i think the recipient is competent enough, i'll ask for or provide ssh access and copy it with rsync otherwise, i've mailed a flash drive before, and for <20GB i'll just put it on a public S3 bucket.
ashton314 6y ago
Netcat: $ nc -l 4242 > dest And then on the sending end: $ nc hostname 4242 < file_to_send This works great when you just need to get a file of any size from one machine to another and you’re both on the same network. Are used this a lot at one of my offices to schlep the files around between a few of the machines we had.
- unilynx 6y ago
  This is especially fun when combined with piping through tar, and adding pv in the mix for a transfer speed "progress" bar. Fastest way to transfer a collection of files on a local network and doesn't require temporary storage for the archive
  - enriquto 6y ago
    piping through tar is my "power move" when I want to steer a new recruit. I just do it casually in front of them... depending on whether they are awed or horrified by that, they'll get more hacky or more formal work.
    - parliament32 6y ago
      You can also pipe through gpg if you want encryption along the way!
      - ahsima1 6y ago
        And through gzip/zstd/etc if you want compression!
        enriquto 6y ago
        that's what the -C option to ssh is for
        cellularmitosis 6y ago
        lzop if you gotta go fast :)
Faaak 6y ago
Plain and simple: I use swisstransfer.com which is free (50Gb) and swiss-based. I did it like you "in the past" (on my own fqdn / public-data) but I've given up on maintaining my website. A torrent works really good if you've got multi-Tb files and many recipients.
geocrasher 6y ago
5GB on my local network: Windows File Share, Rsync, or HTTP depending on source/destination 5GB on Internet: Upload to my OVH VPS and HTTP or Rsync it to its destination 10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! https://www.tidbitsfortechs.com/2013/09/never-underestimate-the-bandwidth-of-a-station-wagon-filled-with-backup-tapes/ https://www.tidbitsfortechs.com/2013/09/never-underestimate-... And since you now buy 1TB micro SD cards, so perhaps I'd split the file 11 ways (no way it'll fit exactly) and send them via carrier pigeon. Or heck, I could just tape them to a drone and hope they aren't damaged in the crash. There's lots of ways to move data around. Maybe you want to UUENCODE it and submit bad URL's to a servers log so that it can be exfiltrated later? It would probably take a very, very long time, but could be done. I call it "CURLyTP" https://miscdotgeek.com/curlytp-every-web-server-is-a-dead-drop/ https://miscdotgeek.com/curlytp-every-web-server-is-a-dead-d...
- terramex 6y ago
  10TB, local or Internet: Physical Hard Drive. Never underestimate the bandwidth of a station wagon full of backup tapes! In Poland there is an option to send package via train for 8$. Just go to station and give package to conductor/train guard, few hours later receiver can get it at destination station. Sending HDDs full of raw video footage this way is very popular among video editors here.
  - jpxw 6y ago
    That’s a really cool idea actually
    - StavrosK 6y ago
      Same thing happens in Greece with buses. It's a very popular way for moms to send home-made food to their single/university student sons.
      - immigrantsheep 6y ago
banana_giraffe 6y ago
For things smaller than around 10gb, I tend to use S3. Either with pre-signed links behind a little bespoke portal that people can log into and see what files I've "shared" with them, or just direct S3 links for those that what to automate it all. For bigger things, there are two basic paths: If it's to someone that's not primary a tech person, then the data goes on a portable hard drive, and I either drive it to them, or mail it and walk them through accessing the data. In both cases I encrypt the data, generally with Veracrypt. If it's to someone that won't mind using AWS's toolchain, and has decent connectivity, I'll use Snowcone or Snowball to get the data to S3 and give them S3 URLs. I tend to get the data to S3 sooner or later so I can migrate it to Glacier. More than once I've had to recover GBs or TBs of data because the customer lost it, so I'm prepared now.
summm 6y ago
Retroshare
- Qision 6y ago
  I didn't know this software had users...
- johnisgood 6y ago
  Neat! Glad someone knows about it, heh. :D
cellularmitosis 6y ago
socat on both ends :)
rootsudo 6y ago
Onedrive/sharrepoint online. Technically, unlimited storage on OneDrive, can force the recipient to verify 2fa via their email address, force expiration link, etc.
renewiltord 6y ago
S3. Sometimes I do that even when I'm transferring on my local network just because I know the flow so well. After all, my speed to the Internet is roughly the same as the speed on my LAN. They're both gigabit duplex.
- _yhdy 6y ago
  You config IAM perms every time?
  - renewiltord 6y ago
    I actually just use https://docs.aws.amazon.com/cli/latest/reference/s3/presign.html https://docs.aws.amazon.com/cli/latest/reference/s3/presign.... Then I copy-paste that into a self-Slack and `wget` it on the other side. It's almost equivalent ergonomically to a `python3 -m http.server` and because it's the same whether I'm giving to someone else or to myself, I spend less thinking!
m463 6y ago
"Never underestimate the bandwidth of a 747 fully loaded with backup tapes"
carlreid 6y ago
I use justbeamit if it's a direct send to someone. https://justbeamit.com/ https://justbeamit.com/
e12e 6y ago
Apparently not explicitly mentioned: rsync over ssh. Windows now comes with real sshd - Mac has it, and linux/bsd of course has it. For small files (less than 10gb?) generally scp - but trying to get in the habit of using rsync. Generally scp isn't a very good idea. For larger filesystems zfs send over ssh. For streaming: dlna/upnp over zerotier vpn. Shame Firefox send service imploded - for smaller files and where ssh isn't an option - it was a nice service. But a little too much hassle to self-host.
cmckn 6y ago
I run FileBrowser [1] to share media with friends. I would suggest trying IPFS if you don't want to forward a port from the internet. You'll get similar download performance to FileBrowser once your node integrates with the network (this takes 30 minutes or so). Check it out! [1]: https://github.com/filebrowser/filebrowser https://github.com/filebrowser/filebrowser
Qision 6y ago
Someone proposed WeTransfer but there is also Tresorit Send (https://send.tresorit.com/ https://send.tresorit.com/) who does the exact same job. Bonus point: they say they encrypt the data and their servers are hosted in Switzerland.
thehappypm 6y ago
Google Drive! Send a public link.
livueta 6y ago
Bittorrent. No, really. Lots of nice behaviors when transferring large amounts of data between arbitrary endpoints. Transferring the torrent metadata is pretty trivial and can be done via a wide range of methods, and having that flexibility can be nice. Unlike HTTP, you get reasonable retry behavior on network hiccups. Also, more robust data integrity guarantees, though a manual hash test is probably a good idea either way. At least among people I'm throwing TBs of data around with, torrent infra is common and it's nice to not have to deal with some special-purpose tool that, in practice, is probably a pain in the ass to get compatible versions deployed across a range of OSes. Basically every platform known to man can run a torrent client of some sort. And obviously, no dependency on an intermediary. This is good if you're trying to avoid Google et al. That does, however, bring a potential con: if my side is fast and your side is slow, I'm seeing until you're done. If I'm uploading something to gdrive or whatever, I can disconnect one r the upload is done. If you control an intermediary like a seedbox, that's less of a problem. In general, though, torrents are pretty great for this sort of thing. Just encrypt your stuff beforehand.
- mderazon 6y ago
  Can you recommend easy software to do the creation? Don't you need a tracker as well?
  - deleted 6y ago
    [deleted]
  - jedimastert 6y ago
    +1
  - parliament32 6y ago
    Almost all BT clients can create torrents. You can use a public tracker if you want, or you can use a magnet link (DHT magic will find the peer eventually), or worst-case scenario you can manually add the peer IP on either end.
  - tick_tock_tick 6y ago
    qBittorrent has a built in tracker for private torrents https://www.ghacks.net/2019/09/20/how-to-create-a-private-torrent-using-qbittorrent/ https://www.ghacks.net/2019/09/20/how-to-create-a-private-to... this method will also not leak information into the DHT though 99% of the time you don't care about that.
raverbashing 6y ago
I think the question for 10TB is more complicated than it seems at first Is it one 10TB file? Multiple files? How do you handle file integrity? Error correction?
Groxx 6y ago
Since 5GB would take quite a long time to upload on DSL: almost universally dropbox, simply because it resumes + it's accessible to people with a url. If the recipient is ssh-friendly and there's a shared machine we can both access, rsync is quite a lot faster and more controllable and doesn't make my CPU angry for hours. Otherwise I liked Firefox Send while it was running since I mostly trust them to not be full of intrusive nonsense.
zacksinclair 6y ago
Git LFS ;)
nakodari 6y ago
I am running Jumpshare (https://jumpshare.com https://jumpshare.com) and our users regularly send large files anywhere from 2GB to 100GB and in some cases even more. My recommendation is to use cloud providers for sharing files that are maximum 50GB in size. Please note that although you can share even bigger files you have to consider the reliability of the internet provider of both the sender and receiver. For files bigger than 50GB in size, I would recommend a P2P solution.
searchableguy 6y ago
I have a raspberry pi with a hard drive. I throw the files in that and then use dat/ipfs.
justinweiss 6y ago
I was asking myself exactly this a few days ago, to share ~500mb worth of files with a few people on a message board. https://gofile.io/ https://gofile.io/ worked fine.
darkblackcorner 6y ago
https://www.file.io/ https://www.file.io/
Paul-ish 6y ago
I use Dropbox to send and receive files that over the email/slack/etc attachment limit. Dropbox has a cool "request files" feature that let's people upload files directly to your Dropbox. I've never needed to send TBs of data.
jedimastert 6y ago
I asked the same question on HN about 3 years ago[0]. I'm curious to see how the answers have changed and what might have stayed the same. [0]: https://news.ycombinator.com/item?id=15440571 https://news.ycombinator.com/item?id=15440571
rocky1138 6y ago
scp
really3452 6y ago
Uploading to Sia's Skynet, then emailing or texting the link.
- Covalent 6y ago
  Or you can use https://defy.chat https://defy.chat It's still under development, but it's working towards being a decentralized discord alternative.
mcculley 6y ago
Keybase
itake 6y ago
You can create a simple python http server to expose files in a directory to the local web: $ python -m http.server 8000 and then you can start ngrok to expose the file $ ngrok http 8000 that will give you an URL to share with whoever wants it.
- anderspitman 6y ago
  Does ngrok limit data at all or just connections?
  - itake 6y ago
    I’ve sent a few GB over this method using their free service. Doesn’t seem to have a transfer limit https://ngrok.com/pricing https://ngrok.com/pricing
- adzm 6y ago
  Note that the built-in web server does not support resuming downloads, unless that changed recently. twistd works well and just as simple. ngrok is an amazing utility though!
noxer 6y ago
Telegram. Up to 2GB per file so just 7zip and split the file(s). Never needed to send anyone 10TB nor do I have that much storage but if you do you probably have some kind of NAS/server where you can enable FTP access.
- gvjddbnvdrbv 6y ago
  10TB equals mail a HDD.
holychiz 6y ago
for non-technical folks to send me big files (5GB limit), I send them to https://transfer.pcloud.com/ https://transfer.pcloud.com/. no registration needed.
skmurphy 6y ago
I have used Hightail for years, from when it was called YouSendIt. Here is info from their website on plans and pricing https://www.hightail.com/file-sharing https://www.hightail.com/file-sharing Hightail offers four different subscription levels for file sharing, with varying restrictions on file upload limits and storage. With our free Lite plan, you can upload files up to 100MB each, and have a storage limit of 2GB. Pro plans, which start at $15/month, allow you to upload individual files up to 25GB each. Teams plans, which start at $30/user/month, allow you to upload individual files up to 50GB each. Business plans, which start at $45/user/month, allow you to upload individual files up to 500GB each. Pro, Teams and Business plans come with unlimited storage.
jeffbee 6y ago
Google Drive seems a lot more practical than pretty much every other answer in this thread.
ha-ckernews 6y ago
Fram's Fast File Exchange (https://fex.rus.uni-stuttgart.de/ https://fex.rus.uni-stuttgart.de/) works well. Free/open source
aaron695 6y ago
Free unused Google Drive Some times I'll have to open the file in a hex editor and change the hash if it sets off copyright restrictions. This is a 10 gig limit.
gnufx 6y ago
Not something I've used, and it's difficult to countenance anything called that, but what Globus (globus.org) has become seems to be quite popular for transferring large research datasets between academic sites which subscribe. For ssh, there's https://www.psc.edu/index.php/hpn-ssh https://www.psc.edu/index.php/hpn-ssh for improving wide-area transfer speeds, and something else, I think.
tobyhinloopen 6y ago
Either via Google Drive or iCloud Drive. If it doesn’t fit on these, use physical media
wheels 6y ago
I have Syncthing syncing to a "/shared" directory on my private web server. If the file is under the 20-ish GB I have free there, that's good enough.
gpm 6y ago
5G from a kubernetes pod: netcat Send help... no seriously... how do I upload a file from a server I have a shell on but can't easily open ports to or install software on (and where the built in cp tool fails on big files...)?
- deleted 6y ago
  [deleted]
- anderspitman 6y ago
  One of the main uses I made https://patchbay.pub https://patchbay.pub for was transferring files to servers/VMs where I don't have easy rsync access. You just need curl or another HTTP client installed.
phendrenad2 6y ago
S3
superkuh 6y ago
I have run a internet reachable whatever.com webserver from my home desktop computer for 20 years. I just copy or softlink the file to a place in ~/www/ and give out the corresponding web link. I have a couple nginx locations prepared with pre-existing bandwith throttles so it's a matter of soft linking to the appropriate ~/www/dirwithassociatedrate/. If it's for more than 1 person I upload it to a VPS if it's small (<20 GB) or make a torrent if it's not.
beginrescueend 6y ago
Inside a company, I like transfer.sh, which is like an open source version of file.io - * https://github.com/dutchcoders/transfer.sh https://github.com/dutchcoders/transfer.sh That's good for the 5G +/- file transfer.
simonw 6y ago
I use Transmit on macOS to upload to an S3 bucket and then send them a link.
alfg 6y ago
S3 and signed URLs.
anderspitman 6y ago
I've been working on solutions in this space for a couple years now. IMO making data on your local device available to others via HTTP range requests is the sweet spot between storing your data in the cloud and going full p2p. Here's a couple of my projects: https://patchbay.pub/ https://patchbay.pub/ code[0] Sender: curl https://patchbay.pub/random-channel/filename.bin --data-binary @filename.bin Receiver: curl -O https://patchbay.pub/random-channel/filename.bin https://fbrg.xyz https://fbrg.xyz code[1] This one works in the browser. You select a file and it gives you a link you can share with others. The underlying tech[2] is more complicated (relies on WebSockets) than patchbay, but the server is currently more reliable. I'm in the process of improving the patchbay server to be more robust, and I think it will eventually replace fibridge completely. My current project is building something along the lines of Google Drive, but much simpler and designed to be self-hosted from within your home. [0]: https://github.com/patchbay-pub/ https://github.com/patchbay-pub/ [1]: https://github.com/anderspitman/fibridge-proxy-rs https://github.com/anderspitman/fibridge-proxy-rs [2]: https://github.com/omnistreams/omnistreams-spec https://github.com/omnistreams/omnistreams-spec
WarOnPrivacy 6y ago
I and my customers all have symmetric FTTH. When I'm copying a file from me to me (working remotely), I use a VPN. These are usually .vhdx files, ranging from 6GB to 300GB. I copy them to here for troubleshooting, then I send them back. When I want to make something large available to someone else, I post it on my local webserver. It points to an NAS on my LAN; anyone in the house can just drop whatever on it and hand out a link.
onelastjob 6y ago
For transfers on the smaller side, I'd have a look at https://massive.io/ https://massive.io/ I've done a fair bit of looking into options for sending files online and I like their pricing model a lot because it is purely usage based and has no caps. It's $0.25/GB for downloads. For larger transfers, I'd look at File Catalyst Spaces. If you have a bigger budget, you could look at IBM Aspera or Media Shuttle.
BrandonM 6y ago
I remember using AIM (AOL (America Online) Instant Messenger) and other instant messaging applications for direct peer-to-peer file transfer back in 2002. On fast university connections, you could transfer movies in a few minutes. It's crazy to me that direct, fast file sharing was easier and more ubiquitous almost 20 years ago than it seems to be now. More context for my oft-misunderstood early feedback to dhouston about Dropbox.
heavyset_go 6y ago
Physical media for anything over a few gigs. It's just easier, especially if the recipient isn't technologically inclined. A lot of the media industry ships hard drives around to one another because content can be so massive.
werber 6y ago
Maybe I’m old, but physically. Most of my large transfers are photographs, video and audio recordings and I like flash drives.
arthurcolle 6y ago
Magic Wormhole
laluser 6y ago
Dropbox transfer supports up to 250GB files now. That's the way to go for most of my needs.
tbronchain 6y ago
It doesn't have to be large files, file sharing sucks. Even copy/past across devices, even your own devices, is a pain. How many times did I email myself some links to share them across devices. And I'm not even speaking about sharing pictures with people not very accustomed to technology (in other words, don't try to get them install a cloud storage app). I gave that question a few tries but I feel part of the problem is the market being saturated by giants doing half the job. Another part of it is the lack of interoperability regarding file sharing between operating systems. I mean - a native right click>send to>user - native notification to the user with a solution to direct p2p download from that person. No software needed, seemless integration. Why is that so hard? I really wish the big boys would give that a try rather than giving us toy-like features and price bump.
- StavrosK 6y ago
  You're gonna love magic-wormhole.
  - tbronchain 6y ago
    It's great, but we're very far from the os-agnostic seemless integrated files sharing. You see, we have protocols like HTTP and the web relies on it, everyone agrees this is how the web works and all OSes have one or several browser able to access it. Something similar works with emails. Yet, we don't have anything to share files between computers, imagine something like bittorrent directly integrated to all OSes, letting you right click > send to.
    - nicolas314 6y ago
      Having a generic "Send to" would only solve a part of it though. There is currently no standard way of connecting the two ends of the same cable to two computers and expect them to exchange files without heavy configuration on both sides. Wireless goes the same way.
sumnole 6y ago
OneDrive has served well enough for my purposes. I've only uploaded files smaller than 1tb though.
vbezhenar 6y ago
Up to a few GBs I'll host them on my server and send a link. More than that: I'll host a torrent.
colabiblen 6y ago
WDT from Facebook - https://github.com/facebook/wdt/ https://github.com/facebook/wdt/ We use it for copying hundreds of TiB's regularly, occasionally over a PiB.
m0xte 6y ago
>10Gb - Stick it on a bitlocker encrypted USB stick and chuck it in the post. Next day download if you send it 1st class here. <10Gb - chuck it on S3 and send a link to the other person.
robert_foss 6y ago
Magic wormhole - it traverses NATs, is encrypted, requires configuration of the source or destination. https://github.com/warner/magic-wormhole https://github.com/warner/magic-wormhole
- kvn_95 6y ago
  Do you mean requires no configuration on the source or destination? :)
  - anaganisk 6y ago
    Installing it on windows required me to install visual c++ build tools, which is a beast of 1.2GB. So yeah it requires a lot. I rather recommend croc, just dropped it on my path it Just works.
    - robert_foss 6y ago
      Being on windows is its own reward. Snark aside, Croc does look a lot like magic wormhole. I'll definitely have a closer look at it.
    - psanford 6y ago
      https://github.com/psanford/wormhole-william/releases/ https://github.com/psanford/wormhole-william/releases/
  - robert_foss 6y ago
    :p little configuration
ikeboy 6y ago
I like mega, it's encrypted, there's command line tools (megatools), it's very fast to upload from a server in my experience.
- jokethrowaway 6y ago
  Not to mention the historical value: https://youtube.com/watch?v=o0Wvn-9BXVc https://youtube.com/watch?v=o0Wvn-9BXVc
ryankrage77 6y ago
For anything 5GB plus - external drive & cycle over The only person I'm sharing with lives near me, so sneakernet is the most convenient. I once hit a transfer speed of 30TBs/hour carrying a box of hard drives home from work.
- Natfan 6y ago
  "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." – Andrew Tanenbaum, 1981
orionblastar 6y ago
When I worked as a federal contractor in 1996-1997 we used password protected FTP servers to store giant text files for import to a database. Later on they moved it to a website you login and download the text file. Since it was so long it took almost all day and if it aborted had to start all over again. I had to keep Netscape open because it showed the status of the file download. People asked why my web browser was open all day, it was for the giant download that is part of my job.
sillysaurusx 6y ago
Syncthing. https://syncthing.net/ https://syncthing.net/ It's like a private dropbox. For files on the order of <= 10GB, magic wormhole is lovely: https://techcrunch.com/2017/06/27/magic-wormhole-is-a-clever-way-to-send-files-easily-and-securely/ https://techcrunch.com/2017/06/27/magic-wormhole-is-a-clever... `sudo pip3 install magic-wormhole` is an automatic "must run this command" on all my servers. Simple, secure way of sending files from A to B, with no bullshit. No account creation, even.
Arkanosis 6y ago
If the network bandwidth on both end allows for a transfer in less than 48 hours, most likely rsync with the --partial-dir=.rsync-partial flag. Otherwise SD card / USB key / hard drive in the mail.
talove 6y ago
- Airdrop for immediate stuff. Gets dicey around 30-50GB tho. - iCloud drive for personal stuff I need to share between devices. I trust this to sync anything up to a TB reliably. - Google Drive when I need to share to someone else. - Occasionally external drives when I need to move data fast locally. - Some combination of S3 / AWS CLI / EC2 when things go beyond personal computer capacity depending on where the data is coming from and going to.
Seb-C 6y ago
I have been using mega.nz to synchronize and backup my files for years, and so far I am very happy with it. Everything works as expected, there is a lot of storage space and end-to-end encryption. There are clients for all platforms and it just works. I can easily create secure share links whenever I want. The only downsides are that the browser-based app is very slow to start once you have lots of files, and the android client does not have the same synchronization capabilities.
egypturnash 6y ago
Apple Mail has a 30-day storage zone for large email attachments that it automatically offers when you attach big stuff, and I usually use that. When I don't I usually just zip it and upload it to the media section of my Wordpress-managed website. I've never generated files in the terabyte range, so I don't worry about that.
1ark 6y ago
10TB or more, not sure. Anecdotally I did have to send GBs of home directory locally recently. Fastest way I found out the hard way was mounting SMBFS and `tar` it there. But of course it is unreliable without resume etc. For the thing you did, these could have worked. https://file.pizza https://file.pizza https://instant.io https://instant.io https://www.sharedrop.io https://www.sharedrop.io
motohagiography 6y ago
Bike courier with checkered past, simultaneously on the run from organized crime clans and mercenaries working for retired spies, probably has mirror shade implants, lives in squat, punk haircut, etc. Didn't realize there was another way.
natch 6y ago
There was some tool for doing this securely on the command line. You run it with the file name and it gives you back a token. You give the token to your friend, they run the same tool with the token, and it downloads the file. Can’t for the life of me find it or remember what it was called. Wormhole? Edit: I guess it was magic wormhole, discussed elsewhere in this thread.
dheera 6y ago
I used to do the HTTP server approach when I was a student at MIT and had a static IPv4 and symmetric gigabit in my dorm room and therefore the recipient could download at usually their full downlink speed. Now I live in the heart of Silicon Valley, a couple km from Google's Headquarters, and have a crappy 15 mbps uplink with no other options available to me, so I typically throw the file on Amazon S3, Google Drive, or Dropbox before sending a link to the other person so that they don't have to put up with a 15 mbps download.
Sean-Der 6y ago
https://webwormhole.io/ https://webwormhole.io/ has web and native clients. You can build for Windows/Mac/Linux/FreeBSD/Mobile. It uses WebRTC so if you are in the same network it will establish the best path possible. I see a lot of people mentioning magic-wormhole and NAT traversal. I can't find any docs that confirm this. I think it always runs through a relay server?
danbmil99 6y ago
Worth noting that there's an expensive but very popular product from IBM called Aspera I've never used it myself but I've heard but it's actually quite impressive but the business model makes it inaccessible to individuals Apparently they do some sort of complicated Trace routing and open a ton of simultaneous routes. Probably also uses something other than TCP 4 correction. Seems like a good place for some disruption
kyleee 6y ago
Firefox send works great for this sort of thing though there is an upper limit on file size (can't recall what it is right now)
ghthor 6y ago
IPFS
gulerc 6y ago
https://Sendgb.com https://Sendgb.com is good way over the internet. Possible to send large files up to 20 gb. No needed registration or sign in.
askvictor 6y ago
Given the quantity of data sent from the LHC to universities across the Atlantic, I'm curious how they (CERN et al) handle their data transfers.
- jerven 6y ago
  Mostly gridftp https://en.wikipedia.org/wiki/GridFTP https://en.wikipedia.org/wiki/GridFTP
jerven 6y ago
For 500Gb or more, aspera soft/ascp. Used by quite a few large bioinformatics institutes for allowing data uploads. [1] https://www.ibm.com/products/aspera https://www.ibm.com/products/aspera
11235813213455 6y ago
streaming json-lines (for large json array files)
retouchup 6y ago
There are many ways of sending big files over the Internet, so I will go into the top three, save time. The first one is the shortest, when using Gmail. Click on the drive button instead, rather than just the paper clip button you will be using with regular attachments. It provides an outline of your drive files, from which you can pick the file to import. Just as easy. You should go to another cloud computing provider if you don't like Google Drive. My third alternative is even more easy, but it's a little sluggish because it cuts you at two gigs either way. It is a service called WeTransfer, which was almost your only choice 15 years ago if you had to submit a huge file.
Fornax96 6y ago
If it's a file less than 10 GB I will use my own website: https://pixeldrain.com https://pixeldrain.com. For things slightly larger I will split it up using 7zip and use pixeldrain too. For much larger files I always fall back to BitTorrent. I have also kept an eye on WebWormhole, but haven't been in a situation yet where I needed it.
eFishCent 6y ago
I have used linux's rsync to send up to 100GiB at a time if the connection is reasonably stable. There are a couple challenges with this method: 1) the way it resumes takes a while to recheck the file before staring again; the larger the file, the longer this takes 2) you need to know a bit about writing a simple script to loop the rsync until the transfer is completed.
anonymoushn 6y ago
I upload it to my VPS and send someone a link. I think google drive is a decent solution as well.
Foivos 6y ago
At 10 TB I guess your best option is to physically mail an HDD.
ohthehugemanate 6y ago
I have a home nextcloud with a few extra TB of storage. I would put it there and use a share link. But HTTP is not a great protocol for really large transfers like 10TB. Ideally you'd want something that parallelizes and does hash checks. Even then, at 10TB you need a 2 gigabit connection to be even competitive with SSD-and-overnight-shipping (about 12hrs).
Ace__ 6y ago
Large files, and I mean from hundreds of GB to terrabytes I use https://www.filemail.com/ https://www.filemail.com/. Their desktop app uses UDP, so it is a lot faster than TCP based protocols like FTP and HTTP. Yeah you can still use their website to send stuff, and if small in size, free anyway for up to 50 GB.
- stantechman 6y ago
  UDP really makes the difference in speed! <3
rawoke083600 6y ago
scp to one one of my unrelated websites and send http-link. Remove file afterwards... #notperfect.
reportgunner 6y ago
In person. If it's not possible to share in person I point them to the place where I myself got it.
nikeee 6y ago
When I'm the sender, I use an http server in the current directory using http-Server (node) or http.server (python 3). When I'm the receiver, I'm using a self-written CLI tool "send me a file" (smaf): https://github.com/nikeee/send-me-a-file https://github.com/nikeee/send-me-a-file It basically opens up an http server on my machine and let's a user upload a file via curl or a web interface. It also computes a sha256 hash while receiving. These methods only work on a local network. I use my VPS and sftp/http for transfers descending the local network.
kenneth 6y ago
In the enterprise space, there is dedicated software for this function. It uses an UDP protocol designed directly for efficiency (less overhead, vs. TCP), and builds in resilience such that it'll work over connections with lots of packet drops, or connections with high latency or low reliability. https://gojetstream.io/ https://gojetstream.io/
- mongol 6y ago
  Interesting. Is this implemented similiar to QUIC as some kind of connection oriented protocol on top of UDP?
ohazi 6y ago
rsync over ssh with retries
manexploitsman 6y ago
Have a look at the "File Sharing and Synchronization" section https://github.com/awesome-selfhosted/awesome-selfhosted#file-sharing-and-synchronization https://github.com/awesome-selfhosted/awesome-selfhosted#fil...
daniel_iversen 6y ago
Dropbox was literally built for that sort of thing! And with 5GB you can probably even do it with the free account by getting yourself a tiny bit more bonus space.
known 6y ago
Interesting information; Thanks to all commentators;
kn100 6y ago
For files less than 1GB I tend to use Telegram. It obviously has the downside of uploading to server X and then downloading from server X, but usually if I am sending a file to one person, I'll likely be sending it to others too, and therefore the ability to forward the file to others arbitrarily after the fact actually proves to be a pretty useful quality. If I care about the datas security, an encrypted 7z container or something will do. For files I need more control over that are less than say 5gb, I tend to scp them to a web server I control, so that I can delete them afterwards. For files larger than that, I'll use a private Bittorrent file. It's very rare I need to transfer files this large, but I really like this solution.
whalesalad 6y ago
S3
pcvarmint 6y ago
USPS Priority Mail. You can send a 10TB hard drive faster by USPS than it would take to transmit across the internet.
bethecloud 6y ago
I like to use transfer.sh
guar47 6y ago
I am actually transferring 400Gb of data across the world right now. I decided to use Apple iCloud and it works pretty good. Too bad the laptop I am transferring from in a bad condition and with Windows so sometimes iCloud service is freezing and I need to restart it. I tried to create torrent but it seems it takes forever on such a weak machine.

I want to comment on that article you keep referring to, but I don't want to clutter up the top of the thread so I'll do it here. The author really wants to dislike PGP, but the reason everyone trusts PGP is because it's been around forever. Yeah there've been deficiencies, just like there've been deficiencies in OpenSSL, but that doesn't make it a bad tool. I could go on but this xkcd sums it up: https://xkcd.com/2347/ https://xkcd.com/2347/ >Absurd Complexity / Swiss Army Knife Design Git is complex, yet effectively every project ever uses it. The reason is you're fine to avoid the edge-cases and just focus on the main functionality, but that one time you need to do something ridiculously hacky, there's a tool to do it, instead of having to roll your own solution. >Backwards Compatibility Would you rather your software not have backwards compat? GPG has sane defaults, and everyone you talk to using modern versions will be secure by default. Not sure what the author is going on about with weak default password encryption: $ gpg -vv --symmetric test.txt ... gpg: using cipher AES256 gpg: writing to 'test.txt.gpg' >Obnoxious UX I don't really find GPG any harder to use than Git. http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/ http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/ >Long-Term Secrets By default, GPG keys expire in a year. >Broken Authentication I've never heard of any of this. Sign and encrypt, by default you get AES256 encryption and a SHA512 digest: $ gpg --sign --encrypt test.txt ... $ gpg -vv -o /dev/null --decrypt test.txt.gpg ... gpg: encrypted with 3072-bit RSA key, ID 74588E74DDD483BC, created 2020-09-02 "test" gpg: AES256 encrypted data gpg: binary signature, digest algorithm SHA512, key algorithm rsa3072 >Incoherent Identity Have an identity. Have other people verify it. Trust based on that. It's the same way the PKI works, you know, that thing that runs the entire internet. Except you don't need to trust CAs anymore. >Leaks Metadata He's not wrong about this one, normally you can see who's ID a message is encrypted for. If you're trying to be sneaky just use symmetric encryption I guess, it feels like a different use case. >No Forward Secrecy Definitely a different use case. There's no case where I want to decrypt a packet from the middle of a TLS conversation a few years later. But an encrypted attachment in an old email? >Clumsy Keys How are GPG keys harder to handle than SSH keys? Both are just blocks of base64 (gpg --export-secret-keys -a)... one is 80 lines while the other is 50, but does it really matter? >Negotiation Same argument as Backwards Compatibility, I think. >Janky Code The page he linked has 27 CVEs. Over the last 15 years. For comparison, OpenSSL has over 200.

dane-pgp 6y ago
I wish the downvoters would explain what's wrong with the parent comment, as the counter-arguments raised by parliament32 seem completely reasonable to me. Perhaps it's just the length of the comment, and they would prefer it written up as a blog post somewhere.
- parliament32 6y ago
  Don't worry, this is normal for HN. As a commenter higher up said, "People without experience come around and think they can replace an old tool with a better one". People tend to like thinking along the lines of "surely old things must be bad" and you'll get met with disagree-downvotes anytime you try to explain why you want old software in fields like crypto or internet routing. Try defending BGP in any of the DDOS or hijack threads and you'll be met with the same fate.
  - DarthGhandi 6y ago
    > People without experience come around and think they can replace an old tool with a better one Just for posterity, the tool linked above that everyone is discussing here was written by the Go Lead Dev for cryptography and security at Google.
- loup-vaillant 6y ago
  > I wish the downvoters would explain what's wrong with the parent comment, One obvious guess is contradicting Latacora. 'tptacek is well known here, his name alone give significant weight to anything he writes. In any case, they have valid points. PGP was written at a time we didn't understand cryptography as well as we do now. We can do better. Have done better, if half of what I've heard about Age is true. Absurd complexity: We can definitely do simpler than PGP, at no loss of functionality. Swiss Army Knife design: I think I disagree with Latacora there. Doing many things doesn't mean you have to do them poorly. There's no material difference between having 3 programs, and having one programs with 3 options, at least on the command line. If PGP does anything poorly, it's for other reasons. Mired In Backwards Compatibility: well that depends. It makes sense that PGP can decrypt old obsolete ciphers & formats. Ability to encrypt to those same old stuff wouldn't. For instance, PGP longer be able to generate RSA keys at all. Then, one year later, once all RSA keys have expired, new PGP versions should no longer be able to encrypt to RSA keys at all. (In an ideal world. More realistically, we should wait a couple more years.) Only the ability to decrypt old messages should be kept until the end of times. Obnoxious UX: I don't know enough to have an opinion. Long term secrets: Sure they're bad, but I don't think we can avoid them. People need your public key to send you anything, so it can't be too short lived. My guess here is that Latacora is attacking the whole file encryption + web of trust thing, not PGP in particular. Broken Authentication: If attackers can trick PGP decoders into decrypting forged messages, that's fairly critical, and should be fixed even if it breaks backwards compatibility (we could have an optional `-legacy` flag or something to compensate). Now if you go sign and encrypt… well there are two possibilities: if you sign then encrypt, you run into the cryptographic doom principle: the decoder will decrypt then verify, which creates the temptation to process unauthenticated data. Many vulnerabilities have been caused by such errors. If you encrypt then sign, you reveal to the entire world that you signed this particular ciphertext. Not the kind of data most people would like to leak. In my opinion what we really want is authenticated key exchange followed by AEAD. With the Noise X pattern, you'd even hide your identity from snoopers. Incoherent Identity: Okay, they're clearly attacking the very notion of web of trust, not PGP specifically. They say it doesn't work, but I'd like to know why. First, I'm not sure I want to take their word for it, and second, the causes might be fixable. Leaks Metadata: that one is clearly avoidable. Noise X for instance uses an ephemeral key to encrypt the transmitted public keys, and the recipient's key is implicit. Can't know who the message is for (nor from) without the recipient's private key. No Forward Secrecy: Different use case indeed. Again, Latacora is attacking the very notion of file encryption, not PGP specifically. Clumsy Keys: I'm with Latacora on this one. The 50 lines SSH keys are clearly RSA based, and as such obsolete. Modern keys use elliptic curves, and those take one line, which is more easily copy & pasted in various contexts. Arguably a detail, though. Negotiation: It's more than just backwards compatibility. Backwards compatibility can be achieved with a simple version number. If instead we have a range of algorithms to chose from, things get more complicated. Now, you can't avoid the need for different kinds of encryption: you can send against a public key, or you can encrypt with a password. Possibly both. Beyond that however it's simpler to have a version number that hints at a single public key encryption and a single password based encryption. Janky Code: Can't judge for myself. I can guess however that much of it is caused by the (ever evolving) PGP specifications themselves. Probably more a consequence of all the other issues than a separate problem. Still, I think we can do much better. I mean, I've written an entire crypto library, and people give me hell for a single vulnerability in over 3 years. 27 CVE in comparison would be worth burning in Crypto Hell for a long time.
bscphil 6y ago
> By default, GPG keys expire in a year. This speaks to the article's complaint that GPG is usually the wrong tool for the job. For example if you just need to transfer a file securely (and have a fast, reliable internet connection on both ends and don't need to worry about active tracking of metadata), you can use Magic Wormhole (or a similar PAKE system) to do it. Imagine two scenarios: one with GPG and one with PAKE, and in both cases an adversary captures a ciphertext. With GPG if they can get your private key 6 months later, you're screwed. With PAKE the keys used to exchange the data are ephemeral, and so this isn't even a possibility. > > Broken Authentication > I've never heard of any of this. I believe this is referring to authenticated encryption (AEAD), which is definitely valuable and GPG does not provide. AGE does. Most of the other stuff you mention also falls under "wrong tool for the job". If you want a better argument, I'd talk about GPG having a web of trust system built in. On the other hand I think it's an open question whether this has ever brought real value to anyone. We have enough other secure messaging systems that it's no longer necessary for a single program to get there on its own. Usually you can count on some other mechanism to confirm your contact's identity.
- parliament32 6y ago
  >With PAKE the keys used to exchange the data are ephemeral, and so this isn't even a possibility. AFAIK this isn't really true. If the adversary captures the initial key exchange plus all the data (ie the full transaction), then later discovers your PSK, they'll be able to decrypt. The only case where this helps you is if they capture some packets out of the middle without the initial handshake. >authenticated encryption It doesn't matter if your ciphertext is authenticated if it's both signed and encrypted, which is what I was getting at. In a normal TLS-like encrypted conversation, yes AEAD is very useful. But it's not applicable here.
  - ta95846893 6y ago
    > AFAIK this isn't really true. If the adversary captures the initial key exchange plus all the data (ie the full transaction), then later discovers your PSK, they'll be able to decrypt. The only case where this helps you is if they capture some packets out of the middle without the initial handshake. Someone can correct me if I'm wrong, but I believe the idea behind a PAKE is that the password only authenticates the key exchange and doesn't contribute to it. So if you record all transmitted data you still need to break the key exchange which should have used a bunch of random bytes from both parties that are thrown away after use. The password is only there to prevent MITM, not to derive keys. I believe magic wormhole uses SPAKE2, which has perfect forward secrecy. When using passwords to secure transmitted files it's really important to have forward secrecy otherwise you risk the transmission being recorded and the password being attacked offline which depending on your password strength might lead to trivially decrypting the data.

libertine 6y ago

Sometimes I wonder if the bad connotation due to piracy made this protocol not standard (in a sense, that's not widely adopted, and people need dedicated clients, instead of it working for example in browsers). I don't know enough, maybe it has security issues, or some limits... but the fact is that it's still widely used, by tech literate people, so it still stands the test of time and it must be doing something right.

aargh_aargh 6y ago
> people need dedicated clients, instead of it working for example in browsers "Version 9.0 [...], released on June 20, 2006" "Opera introduced [...] a built-in BitTorrent client" [1] That's history now, of course, courtesy of today's browser monoculture. [1] https://en.wikipedia.org/wiki/History_of_the_Opera_web_browser#Version_9 https://en.wikipedia.org/wiki/History_of_the_Opera_web_brows...
- roenxi 6y ago
  Does Firefox have an extension to download torernts? I know it used to have one, but that now that I'm aware of the v57+ extension issue, I notice that the one linked to first in a search seems to be dead.
- rglullis 6y ago
  Brave has a built-in torrent downloader.
ogisan 6y ago
WebTorrent [1] can be used to run BitTorrent on the web. It's actually integrated into the Brave browser which is a nice feature. Other browsers might catch on soon. [1] https://webtorrent.io/ https://webtorrent.io/
- mc10 6y ago
  WebTorrent is pretty limited though: > ... a browser-based WebTorrent client or "web peer" can only connect to other clients that support WebTorrent/WebRTC. (from https://webtorrent.io/faq https://webtorrent.io/faq) This essentially splinters the world into the bittorrent and webtorrent halves, which makes it far less useful.
  - 0-_-0 6y ago
    WebTorrent is part of libtorrent now, so should be part of all torrent clients based on it, e.g. qBittorrent: https://github.com/arvidn/libtorrent/pull/4123 https://github.com/arvidn/libtorrent/pull/4123
    - capableweb 6y ago
      Not quite there yet. Latest release of libtorrent was 1.2.9, seems it's scheduled for "some branch" after 2.0 (https://github.com/arvidn/libtorrent/pull/4123#issuecomment-676074545 https://github.com/arvidn/libtorrent/pull/4123#issuecomment-...). Add in some time for clients to actually start using it as well, which is not guaranteed either. The DHT remains split for now.
  - vbezhenar 6y ago
    For some reason browser vendors do not allow JavaScript to connect to any port. So there's no other way.
  - Sean-Der 6y ago
    If you want a CLI/Desktop WebTorrent client https://github.com/anacrolix/torrent https://github.com/anacrolix/torrent works great.
- aeyes 6y ago
  Caution: Don't open this page on a metered connection, immediately starts downloading a 130MB movie file.
  - clemParis 6y ago
    Ouch, I understand that it demoes their solution, but with the autoplay that's very insensitive.
johnisgood 6y ago
Probably, but it is quite a neat technology. As people in this industry, we know what else it is being used for, and it serves those purposes pretty well. I love torrents. I love the technologies that it uses. Many other projects use it, for example take a look at https://docs.ipfs.io/concepts/dht https://docs.ipfs.io/concepts/dht and https://github.com/savoirfairelinux/opendht https://github.com/savoirfairelinux/opendht just for DHT alone. There is also https://en.wikipedia.org/wiki/Magnet_URI_scheme https://en.wikipedia.org/wiki/Magnet_URI_scheme. There are many more, just too lazy to find them at the moment, but believe me, the things you will encounter once you go down this road, is truly amazing. Worth it!
Polylactic_acid 6y ago
It was probably phones and datacaps that really did it. Torrents are really only ok on laptops/desktops and the unexpected data usage for customers is much worse than the costs of distributing data. Steam could probably benefit loads from torrents but they do have extremely good distribution servers already.
paxys 6y ago
BitTorrent didn't catch on simply because there aren't very many actual use cases for it. The majority of P2P content distribution will always be illegal (due to the nature of copyright), and so the client-server model is much more representative of the real world. It is technologically pretty neat though, and some publishers do actually use the protocol for things like software updates.
- Sysosmaster 6y ago
  Not very many use cases?? Except for: - updates of software. - distribution of public video files. - distribution of install images. - distribution of raw public measurements (ea science data). Basically BitTorrent is the protocol for any usecase with 1:n (or n:n) with public data (ea. No data access controle is needed). Copyright has nothing to do with it and the nature of BitTorrent does make it also a good fit for some pirates. Never forget that the following is shared over BitTorrent: - World of Warcraft updates. - Linux distribution install images. - haveibeenpowned sha lists. And that’s just what I personally know. I know there is more.
Thlom 6y ago
In my previous job we used BitTorrent to distribute movies to cinemas. :-)
- tintor 6y ago
  What an irony! :)
m0nty 6y ago
> the bad connotation due to piracy made this protocol not standard Back when Resilio Sync was known as Bittorrent Sync, I proposed it as a file-sharing solution between multiple locations for the company I was working for. Immediately returned with an emphatic refusal to use any kind of software that had "Bittorrent" in the name, regardless of what it was for. This is the same company where they picked up I had "Bittorrent Software" installed on my work laptop, and the people who ran the auditing software freaked out because, you know, it's Bittorrent! I told them it was a single icon in my PortableApps folder but they just kept saying "There's a Bittorrent client on your PC, you could be fired!" until I deleted the icon. So yes, the fact it's associated with piracy has made a huge difference in some places.
- jgimenez 6y ago
  I was kicked out of the public library once for running Syncthing, I guess because the firewall detected it as some sort of piracy tool

arsome 6y ago

Just be careful - unless you mark that torrent private it'll get posted to the DHT and crawlers like BtDig will pick it up and list it publicly. For this reason I prefer using something like Syncthing which is designed more with this purpose in mind.

Polylactic_acid 6y ago
You could also gpg encrypt the file first which is a little more work but bulletproof. Maybe someone should make a tool that creates a torrent and encrypts it for you and presents a password box on the other end.
- iaml 6y ago
  Like magic warmhole?
  - edjw 6y ago
    Magic Wormhole https://magic-wormhole.readthedocs.io/en/latest https://magic-wormhole.readthedocs.io/en/latest
voltagex_ 6y ago
How are BtDig crawling the DHT? I thought this was made impossible or at least more difficult a while ago.
- chid 6y ago
  I don't see how it was made impossible. It's definitely possible, you can do it yourself there's many examples on github (even btdig has an old version on it)
3np 6y ago
Its trivial to disable PEX and DHT in any reasonable client
orionblastar 6y ago
Sort of like Limewire sharing your documents directory? They have MuWire now: https://muwire.com/ https://muwire.com/ Not private but public sharing.
jivank 6y ago
I am working on a tool that will help with this. It is built with AlpineJS, Nim, Aria2 and Webview resulting in a 5MB download which doesn't include the torrent file as that varies. The idea is that it is a one click solution as the torrent is embedded with the binary. The inspiration for this tool is for assist with LAN parties. The one thing I have against the private flag is that it also disables LAN peer discovery, which would be okay if you use a tracker behind the LAN (though the torrent would need to be modified if the hostname/IP changes). Since you can't configure other people's clients I found it simpler to use Aria2 that is preconfigured to disable PEX and DHT. Combine this with Metalink or Web Seeds, so that you can have a initial seeder based on HTTP. I think using IPFS would be a great web seed as long as the their gateway and Cloudfare's continues to stay up. IPFS creates a permanent URL if you will, so no need to worry about dynamic IPs or domains. It would be great if IPFS would have a smaller binary (right now it is around 20MB compressed) and had a way to get a file and "seed" it like a torrent. But for now I think torrent is mature enough.
- brink 6y ago
  Why not use Syncthing or Resilio Sync for LAN parties?
  - jivank 6y ago
    I have considered it. The one thing about Syncthing is that there is no true readonly way to sync. For example, someone may accidentally extract an archive in the Syncthing folder and it will sync everywhere. If Syncthing gets this feature I think I would be 100% onboard with it for LAN parties. Resilio I believe supports it but I would prefer an opensource alternative. https://github.com/syncthing/syncthing/issues/62 https://github.com/syncthing/syncthing/issues/62
    - csdreamer7 6y ago
      You can use the send only feature in Syncthing to make your folder read only. https://docs.syncthing.net/users/foldertypes.html#folder-sendonly https://docs.syncthing.net/users/foldertypes.html#folder-sen...
cgb223 6y ago
Technically speaking, what occurs when you mark a torrent private? Does it actually block trackers from picking it up or just like tell them not to if they want to obey it?
- arsome 6y ago
  I'm not an expert on this, but as far as I'm aware, the main difference is that your client won't go out and announce the infohash to the DHT (used for trackerless torrents, magnet links, that kind of thing) and won't provide it to other clients for download. It'll only ever send requests relating to it to the trackers specifically listed within the torrent itself. This means that anyone who wants to download it basically needs to posses the torrent file. Trackers don't really crawl in this way, however, some searching sites like the one I mentioned there do, they'll go out and look at the entire DHT, trying to find entries that they don't know yet and when they find one - go download the torrent and post it to their index.
AnonHP 6y ago
For some reason I haven’t understood, syncthing would often miss transferring files but mark the entire transfer/sync as complete. No errors in the logs either. I tried it for a few months earlier this year, then gave up and switched to plain rsync in a shell. Based on that sample of one personal experience, I wouldn’t recommend syncthing to anyone. Note: I was using syncthing for a one way sync all the time, with send-only and receive-only settings on the endpoints.
- JeremyNT 6y ago
  I've been using Syncthing extensively for years and never observed this behavior - it always either completes the transfer or fails to complete the transfer (which is obvious in the UI). A silent failure is certainly not typical. This sounds like a pretty severe bug, so if you still have the systems you were able to produce the behavior on it might be helpful to reproduce it and submit an issue.
  - nonbirithm 6y ago
    Yeah, I gave up an Syncthing when using it between my Android device and Linux server. Too many hours were lost trying to debug why the server no longer showed as connected on the phone. At some point I went back to emailing myself files, because that actually worked, and I realized that I didn't actually need an entire program with a synchronization algorithm to send a few large files on the occasion.
bserge 6y ago
That's why you encrypt it or at least make it a password protected archive. That way no one will see the file names in the content list. And without seeders, it's highly unlikely anyone will download it. Torrents work well with 1 seeder, 1 downloader.