6 ms·

I don't think storing the mime type separately is better: That requires every system which you might use to transfer or store your documents to support storing

by shawnz 6y ago

I don't think storing the mime type separately is better: That requires every system which you might use to transfer or store your documents to support storing the mime type separately as well.

klodolph 6y ago
This is already how HTTP works. Storing the MIME type separately is definitely a good thing. This isn't the only way you do it. You start by checking the MIME type and then fall back to one of our crude detection mechanisms, like guessing what the file type is based on extension. We already store plenty of metadata with files--permissions and timestamps for starters. The metadata is sometimes lost when you transfer to another system. That's OK... you can repair the metadata. Given that this metadata is already present in the HTTP protocol, it's present in files embedded in email messages, and given that extended attributes are supported by many different filesystems, I think this solution is completely workable. You could make the argument that since we spend so much time interacting with files over HTTP, that having a MIME type in the metadata is actually the common way to do things.
- lmm 6y ago
  What is it that makes the MIME type good and reliable and the file extension crude and unreliable? In my experience it's the opposite - the file extension is much more user-visible, much more likely to be preserved when the file is transferred, and so more reliable than any other way of storing the file type. HTTP and email were overly influenced by the unix tradition in which file extensions are seen as less important, but nowadays the file extension is the most widely supported "extended attribute" that a file might have - why use a less reliable version of the same thing?
  - klodolph 6y ago
    One huge problem is that the space of available prefixes is just too damn small. It's tiny. This may be fine for you if you only deal with a few mundane filetypes like plain text, PNG, etc. However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. So you need something to disambiguate. I can understand that if you've never personally run into this problem, it may seem unimportant. It is extremely frustrating to work with two different files with the same extension on a regular basis.
    - iforgotpassword 6y ago
      > However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. Care to share some examples? I've very rarely had this problem when some specific program chose to use something like .dat as the extension, but I doubt someone making this decision would have picked a sensible mime type either. Then you mention http already solves this problem. Great. Now you only need to fix the remaining five trillion protocols used to transfer files that don't. If you're old enough you might know the unix haters handbook. It also made fun of this problem and smarty-pants explained how the next version of macos will strictly store any metadata, such as mime type, separately in the file system meta data and not in the file(name) anymore. I think they even gave the size of an image as an example. Thirty years later and this is still what macos does; Jpeg files still end in jpg and contain exif data. Metadata goes in the file, file type is the same. That's the only sane way to get this to work in an interoperable way. It's the least common denominator, since you always have these two, no matter the environment. For anything else, the ship has sailed.
      - nyanpasu64 6y ago
        File extensions are flawed. The .spc file extension is used for both cryptographic certificates and SPC700 audio files. Every time I setup a Windows machine and want to play .spc music, I have to reassociate .spc with a music player. Every time I setup a Linux KDE machine and want to play .spc music, I associate .spc to a new MIME type in my KDE settings, and file picks up this new file type, but xdg-open and KDE insists on treating it as application/pkcs7-mime. MIME types are flawed in a world that only stores file extensions. On Android, you can only register apps as handling specific file MIME types, not file extensions. And for a non-mainstream file extension like .spc, every file browser app synthesizes a different MIME type (which may be blank in some cases).
        shawnz 6y ago
        Not to diminish the argument about file extensions having a too-small namespace, but... why not just save your SNES music in wav or flac files or something? Wouldn't that be a simpler solution to manage in many ways (not just because of file extensions)?
- icedchai 6y ago
  And where does the MIME type originate? With many web servers, its based on file extensions or magic numbers...
  - klodolph 6y ago
    Sure, static web servers serving files from a standard filesystem. Does it apply to dynamically generated content? No. Does it apply to key-value stores like S3 and GCS? No.
    - icedchai 6y ago
      Seems simpler just to use extensions or magic #s. Most normal users don't understand metadata.
      - klodolph 6y ago
        Most users don’t understand file extensions.
- gorgoiler 6y ago
  MIME type feels as if it is as intrinsic a property as something like file size. It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. Could you give an example where one would want to choose the MIME type of some data, from a selection of options?
  - klodolph 6y ago
    "It is a property of the data itself," meaning, it's metadata. I'm in favor of preserving this metadata. You will sometimes need to change the MIME type of something. For example, if you use a text editor to create an HTML file, you'll want to change the MIME type to text/html. It's also nice to be able to mark *.h files as containing C or C++ code.
    - gorgoiler 6y ago
      It seems like you’ve misunderstood me? MIME-type is an intrinsic property which can be derived from the data itself. I have a cat. I can derive that it is a cat when the situation requires it. I don’t need to have a metadata label attached to it, in my own handwriting, saying “cat”.
      - hibbelig 6y ago
        I argue that it’s not possible to determine the mime type of a file from its contents. For example the content could be malformed. An editor or IDE should still highlight it so the user can find the syntax error. Of course, for some files it is possible. But we need the metadata for the other files.
      - kaba0 6y ago
        Well, they are bytes, not a cat. It can be interpreted as an image (in multiple formats) or text or whatever, so I don’t think this metaphor is apt.
  - jcelerier 6y ago
    > It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. I entirely disagree. If I have an empty .json or empty .bin or empty .txt I expect a different behaviour in the software that I am opening the file with, even if in all cases the actual data is zero bytes. More generally file formats can be polyglots: https://github.com/Polydet/polyglot-database https://github.com/Polydet/polyglot-database
deathanatos 6y ago
It does, but that seems like something those system must do, if you want a robust solution. If the systems in question are only capable of transferring untyped blobs of bytes, then you get untyped blobs of bytes out the other end of that pipe. And, if it happens that the file goes through such a system that doesn't understand mimetypes, you're only back to the status quo of having to sniff the type from hints like extension or content.
- shawnz 6y ago
  > ... that seems like something those system must do, if you want a robust solution. Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? > ... you're only back to the status quo of having to sniff the type from hints like extension or content. Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? Also, I still think there are some issues. For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). And what do you do if the internal mime type disagrees with the extension? Aside from just being confusing, that could be purposely abused by malware.
  - deathanatos 6y ago
    > Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? I mean, yeah. They all guess at the mimetype, all the time. They all fall on their face fairly often in that regard. > Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? No? A system that just stores a two-tuple of (type, data) and doesn't have to guess — when it knows the type — is strictly better than a system that always guesses. Where it integrates with other systems that understand how to type the incoming data, it would work flawlessly, every time. Where it integrates with systems that send untyped bytes, there is again no choice but to guess: the data simply isn't there. > For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). Yes, I agree. But people are only going to hit trouble where the mimetype isn't known and the sniffing fails, which is the same issue they'd hit today. I'd argue setting a mimetype has a better shot at being a good UX than trying to get them to set the file extension ever will though. E.g., Right click → Set file type → prompt with different types (use friendly names, if at all possible). E.g., This file looks like it is probably one of the following: JPEG image (can be opened with GIMP, Photoshop.) PDF document (can be opened with Adobe Acrobat.) > See all options (accordion dropdown to show all options) Yes, that still requires the user to know the differences between a JPEG & and PNG and to a layman, that's considerably sub-optimal. But we're at the point where the system didn't record the mimetype, and can't guess it correctly, so there's not much left, really: some human has to make a call. A form of this exists today in most systems, with an "Open with" context menu, and generally with an option to "always open files of this type with this application". (But that's more about the binding between the mimetype — still determined currently through sniffing or extension — and the handler for that mimetype.) But ideally, if the interfaces were there to allow the process to be deterministic & obvious when the data is known, things could or would grow to adopt those interfaces. (Though it'll likely be a looong time.) Unix screwed up, in that regard, in that it set us down the path of "everything is untyped bytes", vs. "everything is strongly typed bytes". With the latter, the system can start figuring out correct or incorrect actions. With the former, it simply lacks the information to make a decision. > Aside from just being confusing, that could be purposely abused by malware. OS X essentially just marks, in the metadata of the file, that it came from the Internet. It could keep doing that. Whatever process "this file could harm your machine" happens with in the browser today could keep on happening that same way. Granted, there is the possibility of "foo.jpg" being sent with a "application/executable" mimetype. That's a real concern. I think this comes down to the system being clear about what you're dealing with, and the consequences of actions. This problem already exists today: people have crafted ".pdf" files that are valid executables. Having better security controls on apps (not having desktop apps run with the same privilege as the user) would help (limits the damage). We've learned, repeatedly, that strong typing results in most robust systems. I don't think the answer is any difference with the bag of bytes a file comes with: knowing the type is better than guessing it. We've also learned, I think, that sniffing almost always leads to loopholes… (I'm not a fan of the "or guess" bit of the proposed idea in my comment; I think a simple "the file carries the mime and that's that" would be better, but I suspect that the roughness of integrating with legacy code that can't communicate what type of data it is reading/writing would hamper that.)

by shawnz 6y ago

I don't think storing the mime type separately is better: That requires every system which you might use to transfer or store your documents to support storing the mime type separately as well.

klodolph 6y ago

This is already how HTTP works. Storing the MIME type separately is definitely a good thing. This isn't the only way you do it. You start by checking the MIME type and then fall back to one of our crude detection mechanisms, like guessing what the file type is based on extension. We already store plenty of metadata with files--permissions and timestamps for starters. The metadata is sometimes lost when you transfer to another system. That's OK... you can repair the metadata. Given that this metadata is already present in the HTTP protocol, it's present in files embedded in email messages, and given that extended attributes are supported by many different filesystems, I think this solution is completely workable. You could make the argument that since we spend so much time interacting with files over HTTP, that having a MIME type in the metadata is actually the common way to do things.

lmm 6y ago
What is it that makes the MIME type good and reliable and the file extension crude and unreliable? In my experience it's the opposite - the file extension is much more user-visible, much more likely to be preserved when the file is transferred, and so more reliable than any other way of storing the file type. HTTP and email were overly influenced by the unix tradition in which file extensions are seen as less important, but nowadays the file extension is the most widely supported "extended attribute" that a file might have - why use a less reliable version of the same thing?
- klodolph 6y ago
  One huge problem is that the space of available prefixes is just too damn small. It's tiny. This may be fine for you if you only deal with a few mundane filetypes like plain text, PNG, etc. However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. So you need something to disambiguate. I can understand that if you've never personally run into this problem, it may seem unimportant. It is extremely frustrating to work with two different files with the same extension on a regular basis.
  - iforgotpassword 6y ago
    > However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. Care to share some examples? I've very rarely had this problem when some specific program chose to use something like .dat as the extension, but I doubt someone making this decision would have picked a sensible mime type either. Then you mention http already solves this problem. Great. Now you only need to fix the remaining five trillion protocols used to transfer files that don't. If you're old enough you might know the unix haters handbook. It also made fun of this problem and smarty-pants explained how the next version of macos will strictly store any metadata, such as mime type, separately in the file system meta data and not in the file(name) anymore. I think they even gave the size of an image as an example. Thirty years later and this is still what macos does; Jpeg files still end in jpg and contain exif data. Metadata goes in the file, file type is the same. That's the only sane way to get this to work in an interoperable way. It's the least common denominator, since you always have these two, no matter the environment. For anything else, the ship has sailed.
    - nyanpasu64 6y ago
      File extensions are flawed. The .spc file extension is used for both cryptographic certificates and SPC700 audio files. Every time I setup a Windows machine and want to play .spc music, I have to reassociate .spc with a music player. Every time I setup a Linux KDE machine and want to play .spc music, I associate .spc to a new MIME type in my KDE settings, and file picks up this new file type, but xdg-open and KDE insists on treating it as application/pkcs7-mime. MIME types are flawed in a world that only stores file extensions. On Android, you can only register apps as handling specific file MIME types, not file extensions. And for a non-mainstream file extension like .spc, every file browser app synthesizes a different MIME type (which may be blank in some cases).
      - shawnz 6y ago
        Not to diminish the argument about file extensions having a too-small namespace, but... why not just save your SNES music in wav or flac files or something? Wouldn't that be a simpler solution to manage in many ways (not just because of file extensions)?
icedchai 6y ago
And where does the MIME type originate? With many web servers, its based on file extensions or magic numbers...
- klodolph 6y ago
  Sure, static web servers serving files from a standard filesystem. Does it apply to dynamically generated content? No. Does it apply to key-value stores like S3 and GCS? No.
  - icedchai 6y ago
    Seems simpler just to use extensions or magic #s. Most normal users don't understand metadata.
    - klodolph 6y ago
      Most users don’t understand file extensions.
gorgoiler 6y ago
MIME type feels as if it is as intrinsic a property as something like file size. It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. Could you give an example where one would want to choose the MIME type of some data, from a selection of options?
- klodolph 6y ago
  "It is a property of the data itself," meaning, it's metadata. I'm in favor of preserving this metadata. You will sometimes need to change the MIME type of something. For example, if you use a text editor to create an HTML file, you'll want to change the MIME type to text/html. It's also nice to be able to mark *.h files as containing C or C++ code.
  - gorgoiler 6y ago
    It seems like you’ve misunderstood me? MIME-type is an intrinsic property which can be derived from the data itself. I have a cat. I can derive that it is a cat when the situation requires it. I don’t need to have a metadata label attached to it, in my own handwriting, saying “cat”.
    - hibbelig 6y ago
      I argue that it’s not possible to determine the mime type of a file from its contents. For example the content could be malformed. An editor or IDE should still highlight it so the user can find the syntax error. Of course, for some files it is possible. But we need the metadata for the other files.
    - kaba0 6y ago
      Well, they are bytes, not a cat. It can be interpreted as an image (in multiple formats) or text or whatever, so I don’t think this metaphor is apt.
- jcelerier 6y ago
  > It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. I entirely disagree. If I have an empty .json or empty .bin or empty .txt I expect a different behaviour in the software that I am opening the file with, even if in all cases the actual data is zero bytes. More generally file formats can be polyglots: https://github.com/Polydet/polyglot-database https://github.com/Polydet/polyglot-database

deathanatos 6y ago

It does, but that seems like something those system must do, if you want a robust solution. If the systems in question are only capable of transferring untyped blobs of bytes, then you get untyped blobs of bytes out the other end of that pipe. And, if it happens that the file goes through such a system that doesn't understand mimetypes, you're only back to the status quo of having to sniff the type from hints like extension or content.

shawnz 6y ago
> ... that seems like something those system must do, if you want a robust solution. Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? > ... you're only back to the status quo of having to sniff the type from hints like extension or content. Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? Also, I still think there are some issues. For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). And what do you do if the internal mime type disagrees with the extension? Aside from just being confusing, that could be purposely abused by malware.
- deathanatos 6y ago
  > Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? I mean, yeah. They all guess at the mimetype, all the time. They all fall on their face fairly often in that regard. > Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? No? A system that just stores a two-tuple of (type, data) and doesn't have to guess — when it knows the type — is strictly better than a system that always guesses. Where it integrates with other systems that understand how to type the incoming data, it would work flawlessly, every time. Where it integrates with systems that send untyped bytes, there is again no choice but to guess: the data simply isn't there. > For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). Yes, I agree. But people are only going to hit trouble where the mimetype isn't known and the sniffing fails, which is the same issue they'd hit today. I'd argue setting a mimetype has a better shot at being a good UX than trying to get them to set the file extension ever will though. E.g., Right click → Set file type → prompt with different types (use friendly names, if at all possible). E.g., This file looks like it is probably one of the following: JPEG image (can be opened with GIMP, Photoshop.) PDF document (can be opened with Adobe Acrobat.) > See all options (accordion dropdown to show all options) Yes, that still requires the user to know the differences between a JPEG & and PNG and to a layman, that's considerably sub-optimal. But we're at the point where the system didn't record the mimetype, and can't guess it correctly, so there's not much left, really: some human has to make a call. A form of this exists today in most systems, with an "Open with" context menu, and generally with an option to "always open files of this type with this application". (But that's more about the binding between the mimetype — still determined currently through sniffing or extension — and the handler for that mimetype.) But ideally, if the interfaces were there to allow the process to be deterministic & obvious when the data is known, things could or would grow to adopt those interfaces. (Though it'll likely be a looong time.) Unix screwed up, in that regard, in that it set us down the path of "everything is untyped bytes", vs. "everything is strongly typed bytes". With the latter, the system can start figuring out correct or incorrect actions. With the former, it simply lacks the information to make a decision. > Aside from just being confusing, that could be purposely abused by malware. OS X essentially just marks, in the metadata of the file, that it came from the Internet. It could keep doing that. Whatever process "this file could harm your machine" happens with in the browser today could keep on happening that same way. Granted, there is the possibility of "foo.jpg" being sent with a "application/executable" mimetype. That's a real concern. I think this comes down to the system being clear about what you're dealing with, and the consequences of actions. This problem already exists today: people have crafted ".pdf" files that are valid executables. Having better security controls on apps (not having desktop apps run with the same privilege as the user) would help (limits the damage). We've learned, repeatedly, that strong typing results in most robust systems. I don't think the answer is any difference with the bag of bytes a file comes with: knowing the type is better than guessing it. We've also learned, I think, that sniffing almost always leads to loopholes… (I'm not a fan of the "or guess" bit of the proposed idea in my comment; I think a simple "the file carries the mime and that's that" would be better, but I suspect that the roughness of integrating with legacy code that can't communicate what type of data it is reading/writing would hamper that.)