12 ms·

File associations are certainly an area where I feel desktop systems in general have failed a bit. Both magic sniffing and and file extensions are very crude wa

by zokier 6y ago

File associations are certainly an area where I feel desktop systems in general have failed a bit. Both magic sniffing and and file extensions are very crude ways of managing the associations. Mime type in xattrs (or similar) is slightly better, but does anyone actually use that? And of course they are very platform specific and well hidden in general.

causality0 6y ago
That may be true in terms of possibility but not relatively. Compare it to the situation on mobile, where you're lucky if you get any choice at all about what programs open which files.
shawnz 6y ago
I don't think storing the mime type separately is better: That requires every system which you might use to transfer or store your documents to support storing the mime type separately as well.
- klodolph 6y ago
  This is already how HTTP works. Storing the MIME type separately is definitely a good thing. This isn't the only way you do it. You start by checking the MIME type and then fall back to one of our crude detection mechanisms, like guessing what the file type is based on extension. We already store plenty of metadata with files--permissions and timestamps for starters. The metadata is sometimes lost when you transfer to another system. That's OK... you can repair the metadata. Given that this metadata is already present in the HTTP protocol, it's present in files embedded in email messages, and given that extended attributes are supported by many different filesystems, I think this solution is completely workable. You could make the argument that since we spend so much time interacting with files over HTTP, that having a MIME type in the metadata is actually the common way to do things.
  - lmm 6y ago
    What is it that makes the MIME type good and reliable and the file extension crude and unreliable? In my experience it's the opposite - the file extension is much more user-visible, much more likely to be preserved when the file is transferred, and so more reliable than any other way of storing the file type. HTTP and email were overly influenced by the unix tradition in which file extensions are seen as less important, but nowadays the file extension is the most widely supported "extended attribute" that a file might have - why use a less reliable version of the same thing?
    - klodolph 6y ago
      One huge problem is that the space of available prefixes is just too damn small. It's tiny. This may be fine for you if you only deal with a few mundane filetypes like plain text, PNG, etc. However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. So you need something to disambiguate. I can understand that if you've never personally run into this problem, it may seem unimportant. It is extremely frustrating to work with two different files with the same extension on a regular basis.
      - iforgotpassword 6y ago
        > However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. Care to share some examples? I've very rarely had this problem when some specific program chose to use something like .dat as the extension, but I doubt someone making this decision would have picked a sensible mime type either. Then you mention http already solves this problem. Great. Now you only need to fix the remaining five trillion protocols used to transfer files that don't. If you're old enough you might know the unix haters handbook. It also made fun of this problem and smarty-pants explained how the next version of macos will strictly store any metadata, such as mime type, separately in the file system meta data and not in the file(name) anymore. I think they even gave the size of an image as an example. Thirty years later and this is still what macos does; Jpeg files still end in jpg and contain exif data. Metadata goes in the file, file type is the same. That's the only sane way to get this to work in an interoperable way. It's the least common denominator, since you always have these two, no matter the environment. For anything else, the ship has sailed.
  - icedchai 6y ago
    And where does the MIME type originate? With many web servers, its based on file extensions or magic numbers...
    - klodolph 6y ago
      Sure, static web servers serving files from a standard filesystem. Does it apply to dynamically generated content? No. Does it apply to key-value stores like S3 and GCS? No.
      - icedchai 6y ago
        Seems simpler just to use extensions or magic #s. Most normal users don't understand metadata.
        klodolph 6y ago
        Most users don’t understand file extensions.
  - gorgoiler 6y ago
    MIME type feels as if it is as intrinsic a property as something like file size. It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. Could you give an example where one would want to choose the MIME type of some data, from a selection of options?
    - klodolph 6y ago
      "It is a property of the data itself," meaning, it's metadata. I'm in favor of preserving this metadata. You will sometimes need to change the MIME type of something. For example, if you use a text editor to create an HTML file, you'll want to change the MIME type to text/html. It's also nice to be able to mark *.h files as containing C or C++ code.
      - gorgoiler 6y ago
        It seems like you’ve misunderstood me? MIME-type is an intrinsic property which can be derived from the data itself. I have a cat. I can derive that it is a cat when the situation requires it. I don’t need to have a metadata label attached to it, in my own handwriting, saying “cat”.
        hibbelig 6y ago
        I argue that it’s not possible to determine the mime type of a file from its contents. For example the content could be malformed. An editor or IDE should still highlight it so the user can find the syntax error. Of course, for some files it is possible. But we need the metadata for the other files.
        kaba0 6y ago
        Well, they are bytes, not a cat. It can be interpreted as an image (in multiple formats) or text or whatever, so I don’t think this metaphor is apt.
    - jcelerier 6y ago
      > It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. I entirely disagree. If I have an empty .json or empty .bin or empty .txt I expect a different behaviour in the software that I am opening the file with, even if in all cases the actual data is zero bytes. More generally file formats can be polyglots: https://github.com/Polydet/polyglot-database https://github.com/Polydet/polyglot-database
- deathanatos 6y ago
  It does, but that seems like something those system must do, if you want a robust solution. If the systems in question are only capable of transferring untyped blobs of bytes, then you get untyped blobs of bytes out the other end of that pipe. And, if it happens that the file goes through such a system that doesn't understand mimetypes, you're only back to the status quo of having to sniff the type from hints like extension or content.
  - shawnz 6y ago
    > ... that seems like something those system must do, if you want a robust solution. Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? > ... you're only back to the status quo of having to sniff the type from hints like extension or content. Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? Also, I still think there are some issues. For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). And what do you do if the internal mime type disagrees with the extension? Aside from just being confusing, that could be purposely abused by malware.
    - deathanatos 6y ago
      > Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? I mean, yeah. They all guess at the mimetype, all the time. They all fall on their face fairly often in that regard. > Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? No? A system that just stores a two-tuple of (type, data) and doesn't have to guess — when it knows the type — is strictly better than a system that always guesses. Where it integrates with other systems that understand how to type the incoming data, it would work flawlessly, every time. Where it integrates with systems that send untyped bytes, there is again no choice but to guess: the data simply isn't there. > For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). Yes, I agree. But people are only going to hit trouble where the mimetype isn't known and the sniffing fails, which is the same issue they'd hit today. I'd argue setting a mimetype has a better shot at being a good UX than trying to get them to set the file extension ever will though. E.g., Right click → Set file type → prompt with different types (use friendly names, if at all possible). E.g., This file looks like it is probably one of the following: JPEG image (can be opened with GIMP, Photoshop.) PDF document (can be opened with Adobe Acrobat.) > See all options (accordion dropdown to show all options) Yes, that still requires the user to know the differences between a JPEG & and PNG and to a layman, that's considerably sub-optimal. But we're at the point where the system didn't record the mimetype, and can't guess it correctly, so there's not much left, really: some human has to make a call. A form of this exists today in most systems, with an "Open with" context menu, and generally with an option to "always open files of this type with this application". (But that's more about the binding between the mimetype — still determined currently through sniffing or extension — and the handler for that mimetype.) But ideally, if the interfaces were there to allow the process to be deterministic & obvious when the data is known, things could or would grow to adopt those interfaces. (Though it'll likely be a looong time.) Unix screwed up, in that regard, in that it set us down the path of "everything is untyped bytes", vs. "everything is strongly typed bytes". With the latter, the system can start figuring out correct or incorrect actions. With the former, it simply lacks the information to make a decision. > Aside from just being confusing, that could be purposely abused by malware. OS X essentially just marks, in the metadata of the file, that it came from the Internet. It could keep doing that. Whatever process "this file could harm your machine" happens with in the browser today could keep on happening that same way. Granted, there is the possibility of "foo.jpg" being sent with a "application/executable" mimetype. That's a real concern. I think this comes down to the system being clear about what you're dealing with, and the consequences of actions. This problem already exists today: people have crafted ".pdf" files that are valid executables. Having better security controls on apps (not having desktop apps run with the same privilege as the user) would help (limits the damage). We've learned, repeatedly, that strong typing results in most robust systems. I don't think the answer is any difference with the bag of bytes a file comes with: knowing the type is better than guessing it. We've also learned, I think, that sniffing almost always leads to loopholes… (I'm not a fan of the "or guess" bit of the proposed idea in my comment; I think a simple "the file carries the mime and that's that" would be better, but I suspect that the roughness of integrating with legacy code that can't communicate what type of data it is reading/writing would hamper that.)
symlinkk 6y ago
I honestly don't see a problem with using file extensions to detect file type. Windows gets a lot of shit from Linux users for this but I don't know why.
- deleted 6y ago
  [deleted]
jandrese 6y ago
I miss the old Mac system of explicit file type and associated application fields in every file metadata.
- klodolph 6y ago
  The system is still there, but it's a bit vestigial.
- asveikau 6y ago
  I seem to recall that system was a bit annoying if the file type matched but the creator didn't.
  - yrro 6y ago
    AFAIR the creator code determined which program would be used when you opened the file, as well as the file's icon. The file type determine which applications could be used to open a file (by drag-and-drop onto the application, or via the application's Open dialog box).
yrro 6y ago
The problem with mime type in xattr is that you lose it as soon as you copy it to another filesystem. It also assumes that every system names its MIME types identically.

by zokier 6y ago

causality0 6y ago

That may be true in terms of possibility but not relatively. Compare it to the situation on mobile, where you're lucky if you get any choice at all about what programs open which files.

shawnz 6y ago

I don't think storing the mime type separately is better: That requires every system which you might use to transfer or store your documents to support storing the mime type separately as well.

klodolph 6y ago
This is already how HTTP works. Storing the MIME type separately is definitely a good thing. This isn't the only way you do it. You start by checking the MIME type and then fall back to one of our crude detection mechanisms, like guessing what the file type is based on extension. We already store plenty of metadata with files--permissions and timestamps for starters. The metadata is sometimes lost when you transfer to another system. That's OK... you can repair the metadata. Given that this metadata is already present in the HTTP protocol, it's present in files embedded in email messages, and given that extended attributes are supported by many different filesystems, I think this solution is completely workable. You could make the argument that since we spend so much time interacting with files over HTTP, that having a MIME type in the metadata is actually the common way to do things.
- lmm 6y ago
  What is it that makes the MIME type good and reliable and the file extension crude and unreliable? In my experience it's the opposite - the file extension is much more user-visible, much more likely to be preserved when the file is transferred, and so more reliable than any other way of storing the file type. HTTP and email were overly influenced by the unix tradition in which file extensions are seen as less important, but nowadays the file extension is the most widely supported "extended attribute" that a file might have - why use a less reliable version of the same thing?
  - klodolph 6y ago
    One huge problem is that the space of available prefixes is just too damn small. It's tiny. This may be fine for you if you only deal with a few mundane filetypes like plain text, PNG, etc. However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. So you need something to disambiguate. I can understand that if you've never personally run into this problem, it may seem unimportant. It is extremely frustrating to work with two different files with the same extension on a regular basis.
    - iforgotpassword 6y ago
      > However, it's very easy to end up in a situation where multiple applications share the file extension. I've run into it all too often. Care to share some examples? I've very rarely had this problem when some specific program chose to use something like .dat as the extension, but I doubt someone making this decision would have picked a sensible mime type either. Then you mention http already solves this problem. Great. Now you only need to fix the remaining five trillion protocols used to transfer files that don't. If you're old enough you might know the unix haters handbook. It also made fun of this problem and smarty-pants explained how the next version of macos will strictly store any metadata, such as mime type, separately in the file system meta data and not in the file(name) anymore. I think they even gave the size of an image as an example. Thirty years later and this is still what macos does; Jpeg files still end in jpg and contain exif data. Metadata goes in the file, file type is the same. That's the only sane way to get this to work in an interoperable way. It's the least common denominator, since you always have these two, no matter the environment. For anything else, the ship has sailed.
- icedchai 6y ago
  And where does the MIME type originate? With many web servers, its based on file extensions or magic numbers...
  - klodolph 6y ago
    Sure, static web servers serving files from a standard filesystem. Does it apply to dynamically generated content? No. Does it apply to key-value stores like S3 and GCS? No.
    - icedchai 6y ago
      Seems simpler just to use extensions or magic #s. Most normal users don't understand metadata.
      - klodolph 6y ago
        Most users don’t understand file extensions.
- gorgoiler 6y ago
  MIME type feels as if it is as intrinsic a property as something like file size. It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. Could you give an example where one would want to choose the MIME type of some data, from a selection of options?
  - klodolph 6y ago
    "It is a property of the data itself," meaning, it's metadata. I'm in favor of preserving this metadata. You will sometimes need to change the MIME type of something. For example, if you use a text editor to create an HTML file, you'll want to change the MIME type to text/html. It's also nice to be able to mark *.h files as containing C or C++ code.
    - gorgoiler 6y ago
      It seems like you’ve misunderstood me? MIME-type is an intrinsic property which can be derived from the data itself. I have a cat. I can derive that it is a cat when the situation requires it. I don’t need to have a metadata label attached to it, in my own handwriting, saying “cat”.
      - hibbelig 6y ago
        I argue that it’s not possible to determine the mime type of a file from its contents. For example the content could be malformed. An editor or IDE should still highlight it so the user can find the syntax error. Of course, for some files it is possible. But we need the metadata for the other files.
      - kaba0 6y ago
        Well, they are bytes, not a cat. It can be interpreted as an image (in multiple formats) or text or whatever, so I don’t think this metaphor is apt.
  - jcelerier 6y ago
    > It is a property of the data itself, not a property that a person chooses for the file like, say, the file name. I entirely disagree. If I have an empty .json or empty .bin or empty .txt I expect a different behaviour in the software that I am opening the file with, even if in all cases the actual data is zero bytes. More generally file formats can be polyglots: https://github.com/Polydet/polyglot-database https://github.com/Polydet/polyglot-database
deathanatos 6y ago
It does, but that seems like something those system must do, if you want a robust solution. If the systems in question are only capable of transferring untyped blobs of bytes, then you get untyped blobs of bytes out the other end of that pipe. And, if it happens that the file goes through such a system that doesn't understand mimetypes, you're only back to the status quo of having to sniff the type from hints like extension or content.
- shawnz 6y ago
  > ... that seems like something those system must do, if you want a robust solution. Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? > ... you're only back to the status quo of having to sniff the type from hints like extension or content. Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? Also, I still think there are some issues. For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). And what do you do if the internal mime type disagrees with the extension? Aside from just being confusing, that could be purposely abused by malware.
  - deathanatos 6y ago
    > Would you say that all the common desktop filesystems, flash memory filesystems, optical filesystems, archive formats, etc. today all aren't robust since they don't do this? I mean, yeah. They all guess at the mimetype, all the time. They all fall on their face fairly often in that regard. > Combining multiple methods does solve that problem, but doesn't it kind of negate the benefits too? No? A system that just stores a two-tuple of (type, data) and doesn't have to guess — when it knows the type — is strictly better than a system that always guesses. Where it integrates with other systems that understand how to type the incoming data, it would work flawlessly, every time. Where it integrates with systems that send untyped bytes, there is again no choice but to guess: the data simply isn't there. > For example, it would need the creation of new mime type changing UI, which might be a hard thing to teach laypeople (they already barely understand extensions). Yes, I agree. But people are only going to hit trouble where the mimetype isn't known and the sniffing fails, which is the same issue they'd hit today. I'd argue setting a mimetype has a better shot at being a good UX than trying to get them to set the file extension ever will though. E.g., Right click → Set file type → prompt with different types (use friendly names, if at all possible). E.g., This file looks like it is probably one of the following: JPEG image (can be opened with GIMP, Photoshop.) PDF document (can be opened with Adobe Acrobat.) > See all options (accordion dropdown to show all options) Yes, that still requires the user to know the differences between a JPEG & and PNG and to a layman, that's considerably sub-optimal. But we're at the point where the system didn't record the mimetype, and can't guess it correctly, so there's not much left, really: some human has to make a call. A form of this exists today in most systems, with an "Open with" context menu, and generally with an option to "always open files of this type with this application". (But that's more about the binding between the mimetype — still determined currently through sniffing or extension — and the handler for that mimetype.) But ideally, if the interfaces were there to allow the process to be deterministic & obvious when the data is known, things could or would grow to adopt those interfaces. (Though it'll likely be a looong time.) Unix screwed up, in that regard, in that it set us down the path of "everything is untyped bytes", vs. "everything is strongly typed bytes". With the latter, the system can start figuring out correct or incorrect actions. With the former, it simply lacks the information to make a decision. > Aside from just being confusing, that could be purposely abused by malware. OS X essentially just marks, in the metadata of the file, that it came from the Internet. It could keep doing that. Whatever process "this file could harm your machine" happens with in the browser today could keep on happening that same way. Granted, there is the possibility of "foo.jpg" being sent with a "application/executable" mimetype. That's a real concern. I think this comes down to the system being clear about what you're dealing with, and the consequences of actions. This problem already exists today: people have crafted ".pdf" files that are valid executables. Having better security controls on apps (not having desktop apps run with the same privilege as the user) would help (limits the damage). We've learned, repeatedly, that strong typing results in most robust systems. I don't think the answer is any difference with the bag of bytes a file comes with: knowing the type is better than guessing it. We've also learned, I think, that sniffing almost always leads to loopholes… (I'm not a fan of the "or guess" bit of the proposed idea in my comment; I think a simple "the file carries the mime and that's that" would be better, but I suspect that the roughness of integrating with legacy code that can't communicate what type of data it is reading/writing would hamper that.)

symlinkk 6y ago

I honestly don't see a problem with using file extensions to detect file type. Windows gets a lot of shit from Linux users for this but I don't know why.

deleted 6y ago
[deleted]

jandrese 6y ago

I miss the old Mac system of explicit file type and associated application fields in every file metadata.

klodolph 6y ago
The system is still there, but it's a bit vestigial.
asveikau 6y ago
I seem to recall that system was a bit annoying if the file type matched but the creator didn't.
- yrro 6y ago
  AFAIR the creator code determined which program would be used when you opened the file, as well as the file's icon. The file type determine which applications could be used to open a file (by drag-and-drop onto the application, or via the application's Open dialog box).

yrro 6y ago

The problem with mime type in xattr is that you lose it as soon as you copy it to another filesystem. It also assumes that every system names its MIME types identically.