5 ms·

I'm gonna argue that the bug is not in "file". The bug is that the workflow uses file, which by it's very nature must be based on heuristics and will get things

by im3w1l 7y ago

I'm gonna argue that the bug is not in "file". The bug is that the workflow uses file, which by it's very nature must be based on heuristics and will get things wrong occasionally. EDIT: Well file really had a bug in this case, where they didn't detect what they thought they were, but the point stands.

jakobegger 7y ago
I agree that you should use care when using file. My app relied on "file" to identify file types, and in some cases it mis-identified some Microsoft Office files. The problem was that Apple ships a very old magic file with macOS, and when I reported the bug they said they don't care.
- gdfasfklshg4 7y ago
  Not sure why people think they can rely on the pre-installed Unix utilities on macOS. EDIT: Apple routinely don't update the Unix utilities for a decade...
  - EliRivers 7y ago
    Yeah, those idiots! They buy something from Apple, and then find it doesn't work. What did they expect. This, of course, is passive-aggressive snark. Suffice to say that I disagree with you; why shouldn't people expect software they buy from Apple to work? People buy from Apple because they don't want to have to deal with all this sort of crap; they just want it to work.
    - giancarlostoro 7y ago
      I am more surprised Apple doesnt invest in their own Unix utils packages honestly. Its not like they dont have the money to do so.
      - wtracy 7y ago
        Steve Jobs originally wanted to strip out the shell entirely and only support POSIX at the level of the C API. To this day, Apple acts like they've been strong-armed into providing support for the Unix utilities. I can understand the attitude that 99% of their userbase will never touch the command line, but that last 1% represents the developers that keep the whole ecosystem going.
    - deleted 7y ago
      [deleted]
    - gdfasfklshg4 7y ago
      We actually broadly agree that it is a shame that Apple don't support their Unix utilities.
- DonHopkins 7y ago
  Excel mis-identifies CSV files whose first column is named "ID" as SYLK files. But who would EVER think of naming the first column of a spreadsheet "ID"? https://www.optibrium.com/community/forums/6-general-how-do-i/21-unable-to-open-csv-files-in-excel https://www.optibrium.com/community/forums/6-general-how-do-... >Q: I have saved the result table from admensa into .csv format. However I am not able to open in excel. It is said that SYLK: file format is not valid. Do you have any experience on it? Thanks. >A: This is an Excel 'feature.' I would guess that the first column of your data is entitled "ID". For some reason, Excel interprets any CSV file beginning with "ID" as an SYLK format file (not really sure what that is!). If you edit the file in, say, Notepad and change the name of the first column in the header row, you should find that it will load without a problem. >This will happen if you save a CSV file from any application with the first characters "ID". So, unfortunately, there's not much we can do about it, other than to recommend that users don't call their first column "ID"!
  - dfox 7y ago
    Excel actually mostly ignores the extension and relies on heuristics to detect the file type on load. Many web applications used to implement "XLS export" by writing out CSV file with .xls extension. This magic detection of file type has one unfortunate side effect: it is not exactly obvious how to convince Excel to load plain text data that is not CSV with delimiters from the current locale. (which is major PITA with most European locales where Excel uses "Semicolon"-SV and not "Comma"-SV but still calls it CSV)
    - wmoser 7y ago
      You’ll have to excuse me because I’m not in front of a windows computer right now, but if you open excel first, there is a data tab on the ribbon and an import data wizard or something like that and you can tell it what the delimiter is. You can also apply types to columns and whatnot before importing. Not the quickest but does help extract the data.
      - Symbiote 7y ago
        This is a fairly new feature in Excel. The lack of this feature pushed a lot of scientists to OpenOffice/LibreOffice Calc. At first, just to open the file and "Save as Excel", but of course, then they start using Calc. (At least that's my experience writing scientific software in Europe. Excel localized not just display, but the input and output. With Calc you could just open a file as "pipe separated UTF-8" and be done with it.)
- cannam 7y ago
  > Apple ships a very old magic file with macOS I particularly relish the way it identifies macOS disk image files as VAX executables. I hope they never change that.
- andreareina 7y ago
  Oh yeah, I got bitten by that. Turns out that MS Office files are zip-compressed xml files. Well I didn't know that, and the files were named something like data.csv, so that left me confused until a coworker told me what's up.
wazari972 7y ago
I'm not sure to agree: if/as *nix systems do not use the extension to know the filetype, then `file`-like heuristics is the only other way to know the file nature, or do I miss someting?
- OskarS 7y ago
  Presumably OpenOffice could pass along a thing that says "please print this PostScript file" so that the printer software knows which type of file it is printing and doesn't have to guess with "file". OpenOffice knows what type of file it's outputting, after all. But your point arguably stands: if a user says "print file X", and you want different files printed using different logic (e.g. PDFs and PostScript files should be printed differently than text files), what other option is there other than using something like "file" to try and guess the file type? It would be very annoying for the user to have to specify each time from a massive list of possible optins.
  - angrygoat 7y ago
    I think the issue here is the question that should be being asked is: "is this a postscript file?". `file` answers a different question, "what is the most likely type for this file?"
    - TazeTSchnitzel 7y ago
      There is no perfect heuristic here, because what if you, for example, want to print a plain text file that happens to be PostScript?
      - zaarn 7y ago
        In most cases, you can't print plaintext easily to a printer, it needs some format like PS, so your PS file would be converted into a PS script that outputs your PS file.
        TazeTSchnitzel 7y ago
IshKebab 7y ago
Yes, this is why you should engineer things properly. Not duck tape them together with shell scripts!
- hiccuphippo 7y ago
  Actually the user was able to solve it because it was in a shell script. If the program had used libfile instead, there would be no easy way to edit it to change the date format other than recompiling it. They would have needed to wait until the maintainers fixed file itself.
  - silon42 7y ago
    using file is a bad idea anyway... you can't rely on a command that performs guesswork (aka "smart"). It's been a cause of security holes in similar contexts (browsers sniffing mime type from file content/extensions)
- generationP 7y ago
  > duck tape That's a fascinatingly apt mixed metaphor: duct taping meets duck typing.
  - IshKebab 7y ago
    Ha I just looked up duck tape on wikipedia and it says "not to be confused with duck typing"!
- rconti 7y ago
  *duct, ironically.
  - wtracy 7y ago
    Duck Tape is the brand name of the original product. "Duct tape" is the generic name that the industry settled on. ("Duck" refers to the water-resistant nature of the tape. Duct doesn't actually mean anything other than sounding like "duck". Ironically, duct tape is the wrong type of tape to use on heating ducts.) If you're going to be pedantic, then get it right. ;-)
blablabla123 7y ago
I wish filesystems would commonly support to store the Mime type. File extensions seem to me still a residue from MS DOS times. That would also make working with static (file) webservers much easier. (It's such a breeze to work with blob stores that allow to save the mime type...)
- AnIdiotOnTheNet 7y ago
  Well one advantage of the "file extension" mechanism is that it is the stupidest thing that will work [0]. It's just like having a type code only it is embedded in the file name so it doesn't require any special features from any filesystem. [0] Actually, UNIX's "everybody just guess" mechanism is probably the stupidest, but it is so stupid I didn't even consider it.
  - blablabla123 7y ago
    Yes sure, it can always be attached somehow. But even if used everywhere it has it's limits and ambiguities, e.g. when dealing with .jpg vs. .jpeg, .yml vs. .yaml and so on... It gets even more interesting when Editors create a temporary file by adding a prefix or suffix, or git/a package manager creates an additional file for conflict resolution.
- D-Coder 7y ago
  File extensions go back a lot farther than MS-DOS, kid. :-)
- protanopia 7y ago
  BeFS does use MIME types. https://arstechnica.com/gadgets/2001/08/metadata/9/ https://arstechnica.com/gadgets/2001/08/metadata/9/
lazyjones 7y ago
It's both really. A workflow using "file" only (and without -k or so) is silly because, as everybody knows, some files are valid for more than one file format. "file" was obviously wrong in the described circumstances...

by im3w1l 7y ago

jakobegger 7y ago

I agree that you should use care when using file. My app relied on "file" to identify file types, and in some cases it mis-identified some Microsoft Office files. The problem was that Apple ships a very old magic file with macOS, and when I reported the bug they said they don't care.

gdfasfklshg4 7y ago
Not sure why people think they can rely on the pre-installed Unix utilities on macOS. EDIT: Apple routinely don't update the Unix utilities for a decade...
- EliRivers 7y ago
  Yeah, those idiots! They buy something from Apple, and then find it doesn't work. What did they expect. This, of course, is passive-aggressive snark. Suffice to say that I disagree with you; why shouldn't people expect software they buy from Apple to work? People buy from Apple because they don't want to have to deal with all this sort of crap; they just want it to work.
  - giancarlostoro 7y ago
    I am more surprised Apple doesnt invest in their own Unix utils packages honestly. Its not like they dont have the money to do so.
    - wtracy 7y ago
      Steve Jobs originally wanted to strip out the shell entirely and only support POSIX at the level of the C API. To this day, Apple acts like they've been strong-armed into providing support for the Unix utilities. I can understand the attitude that 99% of their userbase will never touch the command line, but that last 1% represents the developers that keep the whole ecosystem going.
  - deleted 7y ago
    [deleted]
  - gdfasfklshg4 7y ago
    We actually broadly agree that it is a shame that Apple don't support their Unix utilities.
DonHopkins 7y ago
Excel mis-identifies CSV files whose first column is named "ID" as SYLK files. But who would EVER think of naming the first column of a spreadsheet "ID"? https://www.optibrium.com/community/forums/6-general-how-do-i/21-unable-to-open-csv-files-in-excel https://www.optibrium.com/community/forums/6-general-how-do-... >Q: I have saved the result table from admensa into .csv format. However I am not able to open in excel. It is said that SYLK: file format is not valid. Do you have any experience on it? Thanks. >A: This is an Excel 'feature.' I would guess that the first column of your data is entitled "ID". For some reason, Excel interprets any CSV file beginning with "ID" as an SYLK format file (not really sure what that is!). If you edit the file in, say, Notepad and change the name of the first column in the header row, you should find that it will load without a problem. >This will happen if you save a CSV file from any application with the first characters "ID". So, unfortunately, there's not much we can do about it, other than to recommend that users don't call their first column "ID"!
- dfox 7y ago
  Excel actually mostly ignores the extension and relies on heuristics to detect the file type on load. Many web applications used to implement "XLS export" by writing out CSV file with .xls extension. This magic detection of file type has one unfortunate side effect: it is not exactly obvious how to convince Excel to load plain text data that is not CSV with delimiters from the current locale. (which is major PITA with most European locales where Excel uses "Semicolon"-SV and not "Comma"-SV but still calls it CSV)
  - wmoser 7y ago
    You’ll have to excuse me because I’m not in front of a windows computer right now, but if you open excel first, there is a data tab on the ribbon and an import data wizard or something like that and you can tell it what the delimiter is. You can also apply types to columns and whatnot before importing. Not the quickest but does help extract the data.
    - Symbiote 7y ago
      This is a fairly new feature in Excel. The lack of this feature pushed a lot of scientists to OpenOffice/LibreOffice Calc. At first, just to open the file and "Save as Excel", but of course, then they start using Calc. (At least that's my experience writing scientific software in Europe. Excel localized not just display, but the input and output. With Calc you could just open a file as "pipe separated UTF-8" and be done with it.)
cannam 7y ago
> Apple ships a very old magic file with macOS I particularly relish the way it identifies macOS disk image files as VAX executables. I hope they never change that.
andreareina 7y ago
Oh yeah, I got bitten by that. Turns out that MS Office files are zip-compressed xml files. Well I didn't know that, and the files were named something like data.csv, so that left me confused until a coworker told me what's up.

wazari972 7y ago

I'm not sure to agree: if/as *nix systems do not use the extension to know the filetype, then `file`-like heuristics is the only other way to know the file nature, or do I miss someting?

OskarS 7y ago
Presumably OpenOffice could pass along a thing that says "please print this PostScript file" so that the printer software knows which type of file it is printing and doesn't have to guess with "file". OpenOffice knows what type of file it's outputting, after all. But your point arguably stands: if a user says "print file X", and you want different files printed using different logic (e.g. PDFs and PostScript files should be printed differently than text files), what other option is there other than using something like "file" to try and guess the file type? It would be very annoying for the user to have to specify each time from a massive list of possible optins.
- angrygoat 7y ago
  I think the issue here is the question that should be being asked is: "is this a postscript file?". `file` answers a different question, "what is the most likely type for this file?"
  - TazeTSchnitzel 7y ago
    There is no perfect heuristic here, because what if you, for example, want to print a plain text file that happens to be PostScript?
    - zaarn 7y ago
      In most cases, you can't print plaintext easily to a printer, it needs some format like PS, so your PS file would be converted into a PS script that outputs your PS file.
      - TazeTSchnitzel 7y ago

IshKebab 7y ago

Yes, this is why you should engineer things properly. Not duck tape them together with shell scripts!

hiccuphippo 7y ago
Actually the user was able to solve it because it was in a shell script. If the program had used libfile instead, there would be no easy way to edit it to change the date format other than recompiling it. They would have needed to wait until the maintainers fixed file itself.
- silon42 7y ago
  using file is a bad idea anyway... you can't rely on a command that performs guesswork (aka "smart"). It's been a cause of security holes in similar contexts (browsers sniffing mime type from file content/extensions)
generationP 7y ago
> duck tape That's a fascinatingly apt mixed metaphor: duct taping meets duck typing.
- IshKebab 7y ago
  Ha I just looked up duck tape on wikipedia and it says "not to be confused with duck typing"!
rconti 7y ago
*duct, ironically.
- wtracy 7y ago
  Duck Tape is the brand name of the original product. "Duct tape" is the generic name that the industry settled on. ("Duck" refers to the water-resistant nature of the tape. Duct doesn't actually mean anything other than sounding like "duck". Ironically, duct tape is the wrong type of tape to use on heating ducts.) If you're going to be pedantic, then get it right. ;-)

blablabla123 7y ago

I wish filesystems would commonly support to store the Mime type. File extensions seem to me still a residue from MS DOS times. That would also make working with static (file) webservers much easier. (It's such a breeze to work with blob stores that allow to save the mime type...)

AnIdiotOnTheNet 7y ago
Well one advantage of the "file extension" mechanism is that it is the stupidest thing that will work [0]. It's just like having a type code only it is embedded in the file name so it doesn't require any special features from any filesystem. [0] Actually, UNIX's "everybody just guess" mechanism is probably the stupidest, but it is so stupid I didn't even consider it.
- blablabla123 7y ago
  Yes sure, it can always be attached somehow. But even if used everywhere it has it's limits and ambiguities, e.g. when dealing with .jpg vs. .jpeg, .yml vs. .yaml and so on... It gets even more interesting when Editors create a temporary file by adding a prefix or suffix, or git/a package manager creates an additional file for conflict resolution.
D-Coder 7y ago
File extensions go back a lot farther than MS-DOS, kid. :-)
protanopia 7y ago
BeFS does use MIME types. https://arstechnica.com/gadgets/2001/08/metadata/9/ https://arstechnica.com/gadgets/2001/08/metadata/9/

lazyjones 7y ago

It's both really. A workflow using "file" only (and without -k or so) is silly because, as everybody knows, some files are valid for more than one file format. "file" was obviously wrong in the described circumstances...