7 ms·
> curl will do the same (up to a point) if the output stream is slower than the download itself That "up to a point" is crucial. Storing chunks in memory up to
by mort96 11mo ago
> curl will do the same (up to a point) if the output stream is slower than the download itself
That "up to a point" is crucial. Storing chunks in memory up to some max size as you wait for them to be written to disk makes complete sense. Buffering the entire download in memory before writing to disk at the end doesn't make sense at all.
- jeroenhd 11mo agocurl's approach will lead to partial and failed downloads. When a client stops accepting new data, servers tend to close the connection after a while. There are smoother ways to deal with this (i.e. reduce download rate by faking dropped packets to output speed), but if you just want a simple download command, I think both simple solutions are fine. If the download doesn't fit in RAM, it'll end up swapped out and effectively cached to disk anyway.
- mort96 11mo agoThe standard solution to this is to write the download to a temporary hidden file on the same volume and then rename it into place once the download succeeds (or delete it on failure).
- jeroenhd 11mo agoThat's true when downloading to a file, but Invoke-WebRequest is more curl-like than wget-like. It's designed to return an object/struct rather than simply download a file. If you want to download many/large files, you're probably better off with Start-BitsTransfer.