4 ms·

I have a few questions about curl: -- Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how

I have a few questions about curl: -- Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? -- Are there any examples of a fully functional high-level data motion tool that is based on Curl? -- Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers.

Cthulhu_ 4y ago
> for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? That's out of scope for cURL, use `scp` for that. > Are there any examples of a fully functional high-level data motion tool that is based on Curl? What's data motion? > Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers. There's a few posts on Stack Overflow about that, but they go up to gigabytes. cURL is not the best solution for transfering terabytes of data. http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html seems to be a good overview of the challenges (and tools) to use to securely and reliably transfer terabytes of data, referencing rsync, bbcp, bittorrent, etc.
rjh29 4y ago
> Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? You could try it yourself! The man page is pretty good.
charonn0 4y ago
> Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? SSH file transfers use either SCP:// or SFTP:// urls, not SSH:// urls. >> curl sftp://public.example.com/file.txt

Cthulhu_ 4y ago

> for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? That's out of scope for cURL, use `scp` for that. > Are there any examples of a fully functional high-level data motion tool that is based on Curl? What's data motion? > Has anyone used Curl to perform data transfers at large scale such as terabytes of data? I would love to read more about it if there are any pointers describing such transfers. There's a few posts on Stack Overflow about that, but they go up to gigabytes. cURL is not the best solution for transfering terabytes of data. http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html seems to be a good overview of the challenges (and tools) to use to securely and reliably transfer terabytes of data, referencing rsync, bbcp, bittorrent, etc.

rjh29 4y ago

> Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? You could try it yourself! The man page is pretty good.

charonn0 4y ago

> Curl claims that it supports so many protocols but I do not see many examples where they are in action, for instance, how to use ssh protocol to perform transfer between two machines? Where can I find such examples? SSH file transfers use either SCP:// or SFTP:// urls, not SSH:// urls. >> curl sftp://public.example.com/file.txt