5 ms·

Am I alone in thinking that this behaviour is expected and intuitive? To the extent that I had to verify cURL didn't have the same behaviour.

by nilved 10y ago

andrewstuart2 10y ago
Not at all. Though following redirects and not using the original "file" name by default may be a bad idea. The whole purpose of wget is to get a file from the web and store it locally, by default using the remote filename in the working directory.
- raimue 10y ago
  It is important to make a difference between a remote filename supplied in a Content-Disposition header or the substring that can be extracted from the given URL by splitting at slashes. wget by default uses the latter, appending the QSA (?foo=bar) even when not intended. Same behavior for `curl -O`. You have to use `wget --content-disposition` or `curl -O -J` to explicitly trust the filename send by the server.
pierrec 10y ago
It's kind of a gray area, but looking closely I find it leans heavily towards the buggy, "unexpected behavior" side of things. Here's why: when encountering HTTP->HTTP redirects, wget already ignores the redirected file name and writes to the original file name that was provided in the URL. It was strangely inconsistent that it specifically didn't do this for HTTP->FTP redirects, and at the very least the behavior should be consistent by default. This is clarified in the linked release notes, and also explains why they had no hesitation in considering it a wget bug and fixing it immediately. As for the question of whether using redirected file names is more expected and intuitive, IMO providing a URL without specifying an output filename is basically a shorthand that merges the two arguments into one.
peterwwillis 10y ago
If you specifically issue the command as trying to download a file called "safe_file.txt", and then the actual file that gets downloaded is not only a different file but even a different file name, this is definitely unexpected and un-intuitive.
- RubyPinch 10y ago
  eh, the path the url contains isn't a file system path, its a string of characters that a server can treat in a million different ways, so its not exactly practical to rely on that e.g. if the url was along the lines of http://example.com/dl/f3ct4t3c4c3/ what do you expect the file name to be?
  - peterwwillis 10y ago
    "index.html", as Wget creates a default name (defined by the "default-page" option) when you download a directory path using HTTP. The path in a URL is defined as the following (https://en.wikipedia.org/wiki/Uniform_Resource_Identifier https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) : A path, which contains data, usually organized in hierarchical form, that appears as a sequence of segments separated by slashes. Such a sequence may resemble or map exactly to a file system path, but does not always imply a relation to one.[9] The path must begin with a single slash (/) if an authority part was present, and may also if one was not, but must not begin with a double slash. You can rely on the path to be made up of parts that will not begin with a double slash, and will be separated by slashes, and be defined by a specific character set. URLs also contain a query and a fragment after the path, and the client can choose whether to keep those in the file name or not. Wget has many options which define how file names are created, mainly to better serve in mirroring large complex websites. But in general, whatever the "path" was, the final part of the path will be the file name, or a default one is provided. There are options for enforcing suffixes, creating directory structures recursively, changing accepted character sets for file names, and hundreds more options.
    - RubyPinch 10y ago
      You are taking my comment a bit too literally (especially since the only thing I was trying to convey was: url files != filesystem files). I don't mean defined behavior, I mean what is expected, since that is what you were talking about. If we go with defined behavior, then wget using the server-provided name just like a web browser does (resolving the redirects, and then explicitly noting that the file is being saved with that new file name), is completely expected, no?
Puts 10y ago
It's not a security hole and it's not a bug. Wget even prints out all the things that are happening. Everybody seems to be grasping at straws right now just to be able to make the next branded vulnerability.
- deleted 10y ago
  [deleted]
- icebraining 10y ago
  The wget developers disagree, they call it a security vulnerability. I'm inclined to agree because knowing that HTTP → HTTP redirections don't have this behaviour, it's dangerous for HTTP → FTP to have it. Whether wget prints it out is irrelevant, as it's often used in scripts. http://lists.gnu.org/archive/html/info-gnu/2016-06/msg00004.html http://lists.gnu.org/archive/html/info-gnu/2016-06/msg00004....

Am I alone in thinking that this behaviour is expected and intuitive? To the extent that I had to verify cURL didn't have the same behaviour.

by nilved 10y ago

Am I alone in thinking that this behaviour is expected and intuitive? To the extent that I had to verify cURL didn't have the same behaviour.

andrewstuart2 10y ago

Not at all. Though following redirects and not using the original "file" name by default may be a bad idea. The whole purpose of wget is to get a file from the web and store it locally, by default using the remote filename in the working directory.

raimue 10y ago
It is important to make a difference between a remote filename supplied in a Content-Disposition header or the substring that can be extracted from the given URL by splitting at slashes. wget by default uses the latter, appending the QSA (?foo=bar) even when not intended. Same behavior for `curl -O`. You have to use `wget --content-disposition` or `curl -O -J` to explicitly trust the filename send by the server.

pierrec 10y ago

It's kind of a gray area, but looking closely I find it leans heavily towards the buggy, "unexpected behavior" side of things. Here's why: when encountering HTTP->HTTP redirects, wget already ignores the redirected file name and writes to the original file name that was provided in the URL. It was strangely inconsistent that it specifically didn't do this for HTTP->FTP redirects, and at the very least the behavior should be consistent by default. This is clarified in the linked release notes, and also explains why they had no hesitation in considering it a wget bug and fixing it immediately. As for the question of whether using redirected file names is more expected and intuitive, IMO providing a URL without specifying an output filename is basically a shorthand that merges the two arguments into one.

peterwwillis 10y ago

If you specifically issue the command as trying to download a file called "safe_file.txt", and then the actual file that gets downloaded is not only a different file but even a different file name, this is definitely unexpected and un-intuitive.

RubyPinch 10y ago
eh, the path the url contains isn't a file system path, its a string of characters that a server can treat in a million different ways, so its not exactly practical to rely on that e.g. if the url was along the lines of http://example.com/dl/f3ct4t3c4c3/ what do you expect the file name to be?
- peterwwillis 10y ago
  "index.html", as Wget creates a default name (defined by the "default-page" option) when you download a directory path using HTTP. The path in a URL is defined as the following (https://en.wikipedia.org/wiki/Uniform_Resource_Identifier https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) : A path, which contains data, usually organized in hierarchical form, that appears as a sequence of segments separated by slashes. Such a sequence may resemble or map exactly to a file system path, but does not always imply a relation to one.[9] The path must begin with a single slash (/) if an authority part was present, and may also if one was not, but must not begin with a double slash. You can rely on the path to be made up of parts that will not begin with a double slash, and will be separated by slashes, and be defined by a specific character set. URLs also contain a query and a fragment after the path, and the client can choose whether to keep those in the file name or not. Wget has many options which define how file names are created, mainly to better serve in mirroring large complex websites. But in general, whatever the "path" was, the final part of the path will be the file name, or a default one is provided. There are options for enforcing suffixes, creating directory structures recursively, changing accepted character sets for file names, and hundreds more options.
  - RubyPinch 10y ago
    You are taking my comment a bit too literally (especially since the only thing I was trying to convey was: url files != filesystem files). I don't mean defined behavior, I mean what is expected, since that is what you were talking about. If we go with defined behavior, then wget using the server-provided name just like a web browser does (resolving the redirects, and then explicitly noting that the file is being saved with that new file name), is completely expected, no?

Puts 10y ago

It's not a security hole and it's not a bug. Wget even prints out all the things that are happening. Everybody seems to be grasping at straws right now just to be able to make the next branded vulnerability.

deleted 10y ago
[deleted]
icebraining 10y ago
The wget developers disagree, they call it a security vulnerability. I'm inclined to agree because knowing that HTTP → HTTP redirections don't have this behaviour, it's dangerous for HTTP → FTP to have it. Whether wget prints it out is irrelevant, as it's often used in scripts. http://lists.gnu.org/archive/html/info-gnu/2016-06/msg00004.html http://lists.gnu.org/archive/html/info-gnu/2016-06/msg00004....