4 ms·

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first fe

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.

potatoman22 3y ago
This also works for formats like Python, HTML, and JSON.
- LiamPowell 3y ago
  file (https://www.darwinsys.com/file/ https://www.darwinsys.com/file/) already detects all these formats.
  - ebursztein 3y ago
    Indeed but as pointed out in the blog post -- file is significantly less accurate that Magika. There are also some file type that we support and file doesn't as reported in the table.
    - LiamPowell 3y ago
      I can't immediately find the dataset used for benchmarking. Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?
      - schleck8 3y ago
        > Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see? That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.
- amelius 3y ago
  Yes, but shouldn't the file type be part of the file, or (better) of the metadata of the file? Knowing is better than guessing.
- lopkeny12ko 3y ago
  I still don't see how this is useful. The only time I want to answer the question "what type of file is this" is if it is an opaque blob of binary data. If it's a plain text file like Python, HTML, or JSON, I can figure that out by just catting the file.
vintermann 3y ago
Consider, it's perfectly possible for a file to fit two or more file formats - polyglot files are a hobby for some people. And there are also a billion formats that are not uniquely determined by magic bytes. You don't have to go further than text files.
- KOLANICH 3y ago
  This tool doesn't work this way.
TacticalCoder 3y ago
> What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design. I use the file command all the time. The value is when you get this: ... $ file somefile.xyz somefile.xyz: data AIUI from reading TFA, magika can determine more filetypes than what the file command can detect. It'd actually be very easy to determine if there's any value in magika: run file on every file on your filesystem and then for every file where the file command returns "data", run magika and see if magika is right. If it's right, there's your value. P.S: it may also be easier to run on Windows than the file command? But then I can't do much to help people who are on Windows.
- Eiim 3y ago
  From elsewhere in this thread, it appears that Magika detects far fewer file types than file (116 vs ~1600), which makes sense. For file, you just need to drop in a few rules to add a new, somewhat obscure type. An AI approach like Magika will need lots of training and test data for each new file type. Where Magika might have a leg up is with distinguishing different textual data files (i.e., source code), but I don't see that as a particularly big use case honestly.
cle 3y ago
It's not always deterministic, sometimes it's fuzzy depending on the file type. Example of this is a one-line CSV file. I tested one case of that, libmagic detects it as a text file while magika correctly detects it as a CSV (and gives a confidence score, which is killer).
alkonaut 3y ago
But even with determinism, it's not always right. It's not too rare to find a text file with a byte order mark indicating UTF16 (0xFE 0xFF) but then actually containing utf-8. But what "format" does it have then? Is it UTF-8 or UTF-16? Same with e.g. a jar file missing a manifest. That's just a zip, even though I'm sure some runtime might eat it. But the question is when you have the issue of having to guess the format of a file? Is it when reverse engineering? Last time I did something like this was in the 90's when trying to pick apart some texture from a directory of files called asset0001.k and it turns out it was a bitmap or whatever. Fun times.

potatoman22 3y ago

This also works for formats like Python, HTML, and JSON.

LiamPowell 3y ago
file (https://www.darwinsys.com/file/ https://www.darwinsys.com/file/) already detects all these formats.
- ebursztein 3y ago
  Indeed but as pointed out in the blog post -- file is significantly less accurate that Magika. There are also some file type that we support and file doesn't as reported in the table.
  - LiamPowell 3y ago
    I can't immediately find the dataset used for benchmarking. Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?
    - schleck8 3y ago
      > Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see? That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.
amelius 3y ago
Yes, but shouldn't the file type be part of the file, or (better) of the metadata of the file? Knowing is better than guessing.
lopkeny12ko 3y ago
I still don't see how this is useful. The only time I want to answer the question "what type of file is this" is if it is an opaque blob of binary data. If it's a plain text file like Python, HTML, or JSON, I can figure that out by just catting the file.

vintermann 3y ago

Consider, it's perfectly possible for a file to fit two or more file formats - polyglot files are a hobby for some people. And there are also a billion formats that are not uniquely determined by magic bytes. You don't have to go further than text files.

KOLANICH 3y ago
This tool doesn't work this way.

TacticalCoder 3y ago

> What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design. I use the file command all the time. The value is when you get this: ... $ file somefile.xyz somefile.xyz: data AIUI from reading TFA, magika can determine more filetypes than what the file command can detect. It'd actually be very easy to determine if there's any value in magika: run file on every file on your filesystem and then for every file where the file command returns "data", run magika and see if magika is right. If it's right, there's your value. P.S: it may also be easier to run on Windows than the file command? But then I can't do much to help people who are on Windows.

Eiim 3y ago
From elsewhere in this thread, it appears that Magika detects far fewer file types than file (116 vs ~1600), which makes sense. For file, you just need to drop in a few rules to add a new, somewhat obscure type. An AI approach like Magika will need lots of training and test data for each new file type. Where Magika might have a leg up is with distinguishing different textual data files (i.e., source code), but I don't see that as a particularly big use case honestly.

cle 3y ago

It's not always deterministic, sometimes it's fuzzy depending on the file type. Example of this is a one-line CSV file. I tested one case of that, libmagic detects it as a text file while magika correctly detects it as a CSV (and gives a confidence score, which is killer).

alkonaut 3y ago

But even with determinism, it's not always right. It's not too rare to find a text file with a byte order mark indicating UTF16 (0xFE 0xFF) but then actually containing utf-8. But what "format" does it have then? Is it UTF-8 or UTF-16? Same with e.g. a jar file missing a manifest. That's just a zip, even though I'm sure some runtime might eat it. But the question is when you have the issue of having to guess the format of a file? Is it when reverse engineering? Last time I did something like this was in the 90's when trying to pick apart some texture from a directory of files called asset0001.k and it turns out it was a bitmap or whatever. Fun times.