7 ms·

This is also sort of a flame war topic for some. Divided primarily by those who believe the file’s content should contain the type (or be sampled to infer it) a

This is also sort of a flame war topic for some. Divided primarily by those who believe the file’s content should contain the type (or be sampled to infer it) and those who prefer that the file’s name contain the type. More or less.

netsharc 3y ago
I'm already tempted to throw in my arguments. Is it a flame war or a religious debate? E.g. vi vs emacs
- gustavus 3y ago
  For those embroiled in this holy war there has come a mediator that has quelled the discord and brough harmony and his name is Doom, Doom Emacs (Seriously if you've been a vimmer for a while and wanted to give emacs a go give Doom Emacs a try and drop hlissner a couple of bucks it's definitly worth it)
- alexvitkov 3y ago
  Yeah, except one side is obviously correct. Not knowing what program will open up when they click a file is insane.
  - __MatrixMan__ 3y ago
    Having files double as a launcher for some program was a mistake, the result of which is that we not how have to sit through those awful trainings about not clicking untrusted files. Just open the program first. People should fear untrusted programs, but should be at ease viewing data through programs that they trust.
    - gmac 3y ago
      (1) I would find this very annoying. (2) I suspect for lots of files, lots of users wouldn't know which program to open.
      - __MatrixMan__ 3y ago
        I guess I'm not in a good position to judge how annoying it would be for others. I can't remember the last time I let the OS decide how to handle a file type. As for (2)... good. Asking your computer to perform actions that you understand so poorly that you can't figure out what program is going to run is a recipe for disaster. That's the behavior we should be training people to avoid, not merely engaging with suspicious data but doing so recklessly.
        mminer237 3y ago
        I understand computers plenty well, but I can't envision any scenario where not knowing the default program for a file type would bring disaster. No one should have to memorize what program opens every file, and there's no good way to even learn that information easily. I want to open a photo. Why would I care whether this system uses Shotwell or GNOME Photos or what have you? How would I even learn what's possible? Especially with Linux, there's so much fragmentation that just trying to open the file manager feels like a guessing game of going through all the ones I've heard of to see which works until I finally break down and Google what fork this distro uses. That change would make computers completely unusable beyond Chrome and maybe Office for 99% of people.
        alexvitkov 3y ago
        You certainly will care if you double click a .png and suddenly WinRAR opens up, because that .png is actually an ACE archive (application/x-ace-compressed), and guess what ACE stands for - arbitrary code execution! Probably not, but they've had one for 20-ish years: [1] I find that insane, and that's basically what happens on Linux today. [1] https://www.theverge.com/2019/2/21/18234448/winrar-winace-19-year-old-vulnerability-patched-version-5-70-beta-1 https://www.theverge.com/2019/2/21/18234448/winrar-winace-19...
        __MatrixMan__ 3y ago
        What happened to me today on Linux was `$ geeqie image.png`. It might be possible to trick geeqie into doing something bad, but it's a heck of a lot harder.
    - alexvitkov 3y ago
      All programs become untrusted after they reach a certain size. It's highly unlikely that a 1,000,000 line monstrosity with a file format so complex no single human can understand has no exploitable flaws in its file handling. With "open by extension" the .pdf file can only target vulnerabilities in my PDF reader, but with "open by mimetype" they could be trying to exploit any program that is configured to open files. If you're doing "open by extension" and know that Adobe's exquisite PDF reader will open when you double click a PDF, doing so is not more unsafe than opening up Reader and navigating to the file from within. That's without mentioning the main obvious advantage of "open by extension" - the fact that I can configure programs that open file formats that haven't been blessed by whoever wrote your `file` utility.
    - pbhjpbhj 3y ago
      IIRC my KDE has little icon overlays, so a traffic cone on the mp4 preview image or mp4 icon. That seems like a nice way to give that information. But, it's using magic numbers.
  - michaelcampbell 3y ago
    You don't REALLY know NOW. You rely on the assumption that the extension->application maps in a given machine are sane.
    - alexvitkov 3y ago
      A string->string table is not that difficult to get right, I've never been surprised by what happens when I open a file in Windows. I could write 10 paragraphs on the thought process which my Linux install goes through as it decides that when I click a .zip file the correct behavior is to open it in Chromium and immediately download it to ~/Downloads.
      - mauvia 3y ago
        I've had this exact kind of problem where windows decides to open picture files in edge because the image preview program doesn't understand it.
      - michaelcampbell 3y ago
        > A string->string table is not that difficult to get right You're correct, which makes it doubly confusing as to why Windows doesn't do it.
veidr 3y ago
No, just an explicitly-defined metadata field. Inference would be worse than filename extensions (which are awful). I think the original Mac OS fucked us all by implementing this pretty much perfectly in the 1980s, but using their weird HFS "resource forks" to do it so that it not only couldn't be implemented on other platforms, but also lent credence to the "ugh fuck it lets just mangle the file name... and then hide it~!!! wooo000!~!" argument.. ¯\_(ಠ_ಠ)_/¯ UPDATE: My memory was faulty; the "what kind of file is this?" metadata was actually stored as per-file metadata, the "type" and "creator" fields in the HFS(+) filesystem.
- pavlov 3y ago
  Frustratingly NTFS has had proper metadata support for 30 years but FAT32 was a bag of rotten hacks, so Microsoft couldn’t fix this either even though their forward-looking OS had everything needed.
  - veidr 3y ago
    I know. I'm getting close to 50 years old, and I have lots of *COMPLAINTS!!!* about how various tech (Atari 2600 ~ iPhone 15 Pro Max) worked out. But filesystems seem like the biggest species-level fuckup. Windows had their shit, Mac had HFS then "HFS+" until... tbh, quite breathtakingly, like five minutes ago, When they finally introduced APFS. But like... no checksumming, just Mötley Crüe data integrity, it's all good, iCloud backups FTW amirite!?! ZFS is good; I use it. But it didn't pan out, because a humanity-saving filesystem has to work on the systems humans actually use... (T_T)
    - babypuncher 3y ago
      Apple considered using ZFS but decided to build their own because Oracle. I do not blame them one bit.
      - veidr 3y ago
        I don't, either; that Oracle submarine pervert is the worst-remaining tech villain. Nevertheless, Apple built something objectively worse (T_T)
gus_massa 3y ago
For extra confussion, most files have the info in a header at the begining, but zip files have the info at the end, so someone made a file that is at the same time a valid .pdf and a valivalid .zip. (I can't find the link now.)
- frabert 3y ago
  https://www.sultanik.com/cv https://www.sultanik.com/cv
- capableweb 3y ago
  POC||GTFO does this with their releases, where the file released usually doubles as multiple file formats, and contains a ton of hidden messages all over the place. https://pocorgtfo.hacke.rs/ https://pocorgtfo.hacke.rs/ From the latest edition (#21): > Technical Note: The electronic edition of this magazine is valid as both PDF and ZIP. Thanks to Ange Albertini, it is also a PCAP-NG packet capture of an experiment by Yannay Livneh. See page 7. Edit: Ah, and of course, Cosmopolitan (by @jart) that produces an amalgamation of formats bundled into one file (including ZIP) that runs across a bunch of OSes https://news.ycombinator.com/item?id=38101613 https://news.ycombinator.com/item?id=38101613
- BobaFloutist 3y ago
  That's actually a very fun idea.
- saghm 3y ago
  There also used to be an exploit where you could make a file both a valid .jar Java executable and a valid .gif, which back in the days of Java applets would potentially let you execute code when someone loaded your image (e.g. if you uploaded it as your profile picture): https://en.wikipedia.org/wiki/Polyglot_%28computing%29#GIFAR_attack https://en.wikipedia.org/wiki/Polyglot_%28computing%29#GIFAR...
- 13of40 3y ago
  My favorite is storing a batch file uncompressed as the first entry in the ZIP archive, because you can rename it with .BAT or .CMD and run it. Sprinkle in some PowerShell and you can make a self extracting archive/installer.
deleted 3y ago
[deleted]
dylan604 3y ago
Filenames are the worst place to store metadata. Even a file's specific metadata fields are not always trust worthy. In the realm of media assets, relying on the file's metadata is a giant footgun. I'm not even talking about anything malicious, as that's designed to attempt to fool people. I'm referring to simple mistakes which would cause incorrect processing down the line if solely depending on metadata. For example, marking the video field dominance as progressive when the content is truly interlaced or vice versa. Even in the world of image uploading via browser, just checking extensions is discouraged. If the feature is that humans can be allowed to manipulate it, it must be sanitized/verified before accepting. This isn't just for text heading for a database.
- pbhjpbhj 3y ago
  >Filenames are the worst place to store metadata. Where's the best place? It seems like being able to send a file handle to an app without having to open the file nor read a sidecar/db could be quite convenient?
  - dylan604 3y ago
    the problem with depending on metadata alone is just an error prone concept. this could matter more or less depending on the use case. telling a text editor that a plain text file is rich text is much less problematic than telling a video app that a file is 16x9 when it is actually 4x3. saving a tab delimited file as CSV is also going to cause issues if only the extension is used. i'm not saying throw away extensions, as they are great hints, but trust without verifying is not just for politics.
babypuncher 3y ago
The problem with declaring a file's type in the content itself is that you end up rendering all prior file formats obsolete because they lack this information. The better solution I've seen proposed is to store file type information as metadata in the filesystem itself, but this would lead to compatibility problems with archiving tools and any other scenario where a file is moved between differing file systems. File extensions are the best way to make determining a file type easy without parsing the whole thing while also maintaining cross and backwards compatibility.
morsch 3y ago
It's getting less relevant as regular users stop using the filesystem directly and instead rely on proprietary services for their data. Google Drive doesn't rely on file extensions. Though for all I know, Microsoft's web stuff does... What's the web equivalent of using a floppy disk to move data between Mac, OS and Windows, I wonder. Downloading to the local computer and uploading, presumably. I guess that is pretty equivalent. Though you can't even download the native representation of a Google (text, spreadsheet, ...) document.

netsharc 3y ago

I'm already tempted to throw in my arguments. Is it a flame war or a religious debate? E.g. vi vs emacs

gustavus 3y ago
For those embroiled in this holy war there has come a mediator that has quelled the discord and brough harmony and his name is Doom, Doom Emacs (Seriously if you've been a vimmer for a while and wanted to give emacs a go give Doom Emacs a try and drop hlissner a couple of bucks it's definitly worth it)
alexvitkov 3y ago
Yeah, except one side is obviously correct. Not knowing what program will open up when they click a file is insane.
- __MatrixMan__ 3y ago
  Having files double as a launcher for some program was a mistake, the result of which is that we not how have to sit through those awful trainings about not clicking untrusted files. Just open the program first. People should fear untrusted programs, but should be at ease viewing data through programs that they trust.
  - gmac 3y ago
    (1) I would find this very annoying. (2) I suspect for lots of files, lots of users wouldn't know which program to open.
    - __MatrixMan__ 3y ago
      I guess I'm not in a good position to judge how annoying it would be for others. I can't remember the last time I let the OS decide how to handle a file type. As for (2)... good. Asking your computer to perform actions that you understand so poorly that you can't figure out what program is going to run is a recipe for disaster. That's the behavior we should be training people to avoid, not merely engaging with suspicious data but doing so recklessly.
      - mminer237 3y ago
        I understand computers plenty well, but I can't envision any scenario where not knowing the default program for a file type would bring disaster. No one should have to memorize what program opens every file, and there's no good way to even learn that information easily. I want to open a photo. Why would I care whether this system uses Shotwell or GNOME Photos or what have you? How would I even learn what's possible? Especially with Linux, there's so much fragmentation that just trying to open the file manager feels like a guessing game of going through all the ones I've heard of to see which works until I finally break down and Google what fork this distro uses. That change would make computers completely unusable beyond Chrome and maybe Office for 99% of people.
        alexvitkov 3y ago
        You certainly will care if you double click a .png and suddenly WinRAR opens up, because that .png is actually an ACE archive (application/x-ace-compressed), and guess what ACE stands for - arbitrary code execution! Probably not, but they've had one for 20-ish years: [1] I find that insane, and that's basically what happens on Linux today. [1] https://www.theverge.com/2019/2/21/18234448/winrar-winace-19-year-old-vulnerability-patched-version-5-70-beta-1 https://www.theverge.com/2019/2/21/18234448/winrar-winace-19...
        __MatrixMan__ 3y ago
        What happened to me today on Linux was `$ geeqie image.png`. It might be possible to trick geeqie into doing something bad, but it's a heck of a lot harder.
  - alexvitkov 3y ago
    All programs become untrusted after they reach a certain size. It's highly unlikely that a 1,000,000 line monstrosity with a file format so complex no single human can understand has no exploitable flaws in its file handling. With "open by extension" the .pdf file can only target vulnerabilities in my PDF reader, but with "open by mimetype" they could be trying to exploit any program that is configured to open files. If you're doing "open by extension" and know that Adobe's exquisite PDF reader will open when you double click a PDF, doing so is not more unsafe than opening up Reader and navigating to the file from within. That's without mentioning the main obvious advantage of "open by extension" - the fact that I can configure programs that open file formats that haven't been blessed by whoever wrote your `file` utility.
  - pbhjpbhj 3y ago
    IIRC my KDE has little icon overlays, so a traffic cone on the mp4 preview image or mp4 icon. That seems like a nice way to give that information. But, it's using magic numbers.
- michaelcampbell 3y ago
  You don't REALLY know NOW. You rely on the assumption that the extension->application maps in a given machine are sane.
  - alexvitkov 3y ago
    A string->string table is not that difficult to get right, I've never been surprised by what happens when I open a file in Windows. I could write 10 paragraphs on the thought process which my Linux install goes through as it decides that when I click a .zip file the correct behavior is to open it in Chromium and immediately download it to ~/Downloads.
    - mauvia 3y ago
      I've had this exact kind of problem where windows decides to open picture files in edge because the image preview program doesn't understand it.
    - michaelcampbell 3y ago
      > A string->string table is not that difficult to get right You're correct, which makes it doubly confusing as to why Windows doesn't do it.

veidr 3y ago

No, just an explicitly-defined metadata field. Inference would be worse than filename extensions (which are awful). I think the original Mac OS fucked us all by implementing this pretty much perfectly in the 1980s, but using their weird HFS "resource forks" to do it so that it not only couldn't be implemented on other platforms, but also lent credence to the "ugh fuck it lets just mangle the file name... and then hide it~!!! wooo000!~!" argument.. ¯\_(ಠ_ಠ)_/¯ UPDATE: My memory was faulty; the "what kind of file is this?" metadata was actually stored as per-file metadata, the "type" and "creator" fields in the HFS(+) filesystem.

pavlov 3y ago
Frustratingly NTFS has had proper metadata support for 30 years but FAT32 was a bag of rotten hacks, so Microsoft couldn’t fix this either even though their forward-looking OS had everything needed.
- veidr 3y ago
  I know. I'm getting close to 50 years old, and I have lots of *COMPLAINTS!!!* about how various tech (Atari 2600 ~ iPhone 15 Pro Max) worked out. But filesystems seem like the biggest species-level fuckup. Windows had their shit, Mac had HFS then "HFS+" until... tbh, quite breathtakingly, like five minutes ago, When they finally introduced APFS. But like... no checksumming, just Mötley Crüe data integrity, it's all good, iCloud backups FTW amirite!?! ZFS is good; I use it. But it didn't pan out, because a humanity-saving filesystem has to work on the systems humans actually use... (T_T)
  - babypuncher 3y ago
    Apple considered using ZFS but decided to build their own because Oracle. I do not blame them one bit.
    - veidr 3y ago
      I don't, either; that Oracle submarine pervert is the worst-remaining tech villain. Nevertheless, Apple built something objectively worse (T_T)

gus_massa 3y ago

For extra confussion, most files have the info in a header at the begining, but zip files have the info at the end, so someone made a file that is at the same time a valid .pdf and a valivalid .zip. (I can't find the link now.)

frabert 3y ago
https://www.sultanik.com/cv https://www.sultanik.com/cv
capableweb 3y ago
POC||GTFO does this with their releases, where the file released usually doubles as multiple file formats, and contains a ton of hidden messages all over the place. https://pocorgtfo.hacke.rs/ https://pocorgtfo.hacke.rs/ From the latest edition (#21): > Technical Note: The electronic edition of this magazine is valid as both PDF and ZIP. Thanks to Ange Albertini, it is also a PCAP-NG packet capture of an experiment by Yannay Livneh. See page 7. Edit: Ah, and of course, Cosmopolitan (by @jart) that produces an amalgamation of formats bundled into one file (including ZIP) that runs across a bunch of OSes https://news.ycombinator.com/item?id=38101613 https://news.ycombinator.com/item?id=38101613
BobaFloutist 3y ago
That's actually a very fun idea.
saghm 3y ago
There also used to be an exploit where you could make a file both a valid .jar Java executable and a valid .gif, which back in the days of Java applets would potentially let you execute code when someone loaded your image (e.g. if you uploaded it as your profile picture): https://en.wikipedia.org/wiki/Polyglot_%28computing%29#GIFAR_attack https://en.wikipedia.org/wiki/Polyglot_%28computing%29#GIFAR...
13of40 3y ago
My favorite is storing a batch file uncompressed as the first entry in the ZIP archive, because you can rename it with .BAT or .CMD and run it. Sprinkle in some PowerShell and you can make a self extracting archive/installer.

deleted 3y ago

[deleted]

dylan604 3y ago

Filenames are the worst place to store metadata. Even a file's specific metadata fields are not always trust worthy. In the realm of media assets, relying on the file's metadata is a giant footgun. I'm not even talking about anything malicious, as that's designed to attempt to fool people. I'm referring to simple mistakes which would cause incorrect processing down the line if solely depending on metadata. For example, marking the video field dominance as progressive when the content is truly interlaced or vice versa. Even in the world of image uploading via browser, just checking extensions is discouraged. If the feature is that humans can be allowed to manipulate it, it must be sanitized/verified before accepting. This isn't just for text heading for a database.

pbhjpbhj 3y ago
>Filenames are the worst place to store metadata. Where's the best place? It seems like being able to send a file handle to an app without having to open the file nor read a sidecar/db could be quite convenient?
- dylan604 3y ago
  the problem with depending on metadata alone is just an error prone concept. this could matter more or less depending on the use case. telling a text editor that a plain text file is rich text is much less problematic than telling a video app that a file is 16x9 when it is actually 4x3. saving a tab delimited file as CSV is also going to cause issues if only the extension is used. i'm not saying throw away extensions, as they are great hints, but trust without verifying is not just for politics.

babypuncher 3y ago

The problem with declaring a file's type in the content itself is that you end up rendering all prior file formats obsolete because they lack this information. The better solution I've seen proposed is to store file type information as metadata in the filesystem itself, but this would lead to compatibility problems with archiving tools and any other scenario where a file is moved between differing file systems. File extensions are the best way to make determining a file type easy without parsing the whole thing while also maintaining cross and backwards compatibility.

morsch 3y ago

It's getting less relevant as regular users stop using the filesystem directly and instead rely on proprietary services for their data. Google Drive doesn't rely on file extensions. Though for all I know, Microsoft's web stuff does... What's the web equivalent of using a floppy disk to move data between Mac, OS and Windows, I wonder. Downloading to the local computer and uploading, presumably. I guess that is pretty equivalent. Though you can't even download the native representation of a Google (text, spreadsheet, ...) document.