19 ms·

Magika: AI powered fast and efficient file type identification

695 points by alphabetting 3y ago

kushie 3y ago
this couldnt have been released at a better time for me! really needed a library like this.
- petesergeant 3y ago
  Tell us why!
- ebursztein 3y ago
  Thanks :)
NiloCK 3y ago
A somewhat surprising and genuinely useful application of the family of techniques. I wonder how susceptible it is to adversarial binaries or, hah, prompt-injected binaries.
- dghlsakjg 3y ago
  “These aren’t the binaries you are looking for…”
- jamesdwilson 3y ago
  For the extremely limited number of file types supported, I question the utility of this compared to `magic`
- star4040 3y ago
  It gets a lot of binary file formats wrong for me out-of-the-box. I think it needs to be a bit more effective before we can truly determine the effectiveness of such exploits.
  - queuebert 3y ago
    But they reported >99% accuracy on their cherry-picked dataset! /s
- nicklecompte 3y ago
  Elsewhere in the thread kevincox[1] points out that it's extremely susceptible to adversarial binaries: > Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data". Seems like this is genuinely useless for anybody but AI researchers. [1] https://news.ycombinator.com/item?id=39395677 https://news.ycombinator.com/item?id=39395677
thorum 3y ago
Supported file types: https://github.com/google/magika/blob/main/docs/supported-content-types-list.md https://github.com/google/magika/blob/main/docs/supported-co...
- s1mon 3y ago
  It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.
  - _3u10 3y ago
    No tracker / .mod files either, just use file.
    - ebursztein 3y ago
      Thanks for the list, we will probably try to extend the list of format supported in future revision.
  - photoGrant 3y ago
    Yeah this quickly went from 'additional helpful tool in the kit' to 'probably should use something else first'
  - vintermann 3y ago
    Well, what they used this for at Google was apparently scanning their user's files for things they shouldn't store in the cloud. Probably they don't care much about MIDI.
  - kevincox 3y ago
    Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data".
vunderba 3y ago
As somebody who's dealt with the ambiguity of attempting to use file signatures in order to identify file type, this seems like a pretty useful library. Especially since it seems to be able to distinguish between different types of text files based on their format/content e.g. CSV, markdown, etc.
semitones 3y ago
Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?
- hiddencost 3y ago
  malware can intentionally obfuscate itself
- callalex 3y ago
  Nothing is ever simple. Even for the most basic .txt files it’s still useful to know what the character encoding is (utf? 8/16? Latin-whatever? etc.) and what the line format is (\n,\cr\lf,\n\lf) as well as determining if some maniac removed all the indentation characters and replaced them with a mystery number of spaces. Then there are all the container formats that have different kinds of formats embedded in them (mov,mkv,pdf etc.)
  - cole-k 3y ago
    A fun read in service of your first point: https://en.wikipedia.org/wiki/Bush_hid_the_facts https://en.wikipedia.org/wiki/Bush_hid_the_facts
- SnowflakeOnIce 3y ago
  Yes! Sometimes a file has no extension. Other times the extension is a lie. Still other times, you may be dealing with an unnamed bytestring and wish to know what kind of content it is. This last case happens quite a lot in Nosey Parker [1], a detector of secrets in textual data. There, it is possible to come across unnamed files in Git history, and it would be useful to the user to still indicate what type of file it seems to be. I added file type detection based on libmagic to Nosey Parker a while back, but it's not compiled in by default because libmagic is slow and complicates the build process. Also, libmagic is implemented as a large C library whose primary job is parsing, which makes the security side of me jittery. I will likely add enabled-by-default filetype detection to Nosey Parker using Magika's ONNX model. [1] https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker
userbinator 3y ago
Today web browsers, code editors, and countless other software rely on file-type detection to decide how to properly render a file. "web browsers"? Odd to see this coming from Google itself. https://en.wikipedia.org/wiki/Content_sniffing https://en.wikipedia.org/wiki/Content_sniffing was widely criticised for being problematic for security.
- rafram 3y ago
  Content sniffing can be disabled by the server (X-Content-Type-Options: nosniff), but it’s still used by default. Web browsers have to assume that servers are stupid, and that for relatively harmless cases, it’s fine to e.g. render a PNG loaded by an <img> even if it’s served as text/plain.
stevepike 3y ago
Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 https://bugs.freedesktop.org/show_bug.cgi?id=78797. My patch got rejected for needing to look at the first 3KB bytes of the file to figure out the type. They had a hard limit that they wouldn't see past the first 256 bytes. Now in 2024 we're doing this with deep learning! It'd be cool if google released some speed performance benchmarks here against the old-fashioned implementations. Obviously it'd be slower, but is it 1000x or 10^6x?
- renonce 3y ago
  From the first paragraph: > enabling precise file identification within milliseconds, even when running on a CPU. Maybe your old-fashioned implementations were detecting in microseconds?
  - deleted 3y ago
    [deleted]
  - stevepike 3y ago
    Yeah I saw that, but that could cover a pretty wide range and it's not clear to me whether that relies on preloading a model.
    - ryanjshaw 3y ago
      > At inference time Magika uses Onnx as an inference engine to ensure files are identified in a matter of milliseconds, almost as fast as a non-AI tool even on CPU.
- ebursztein 3y ago
  Co-author of Magika here (Elie) so we didn't include the measurements in the blog post to avoid making it too long but we did those measurements. Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples. So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeated detection. This is why we said it is not that much slower. We will have more performance measurements in the upcoming research paper. Hope that answer the question
rfl890 3y ago
We have had file(1) for years
- Am4TIfIsER0ppos 3y ago
  [flagged]
  - samtheprogram 3y ago
    This a 1mb Keras ML model that’s open source. I passionately dislike Surveillance Capitalism but bringing it up when it’s completely irrelevant only weakens the argument. RTFA.
    - floobertoober 3y ago
      > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". From the HN guidelines
      - samtheprogram 3y ago
        > can be shortened to "The article mentions that". I shortened it to RTFA.
- samtheprogram 3y ago
  This is beyond what file is capable of. It’s also mentioned in the third paragraph. RTFA.
  - wruza 3y ago
    Some HN readers may not know about file(1) even. It's fine to mention that $subj enhances that, but the rtfa part seems pretty unnecessary.
  - Vogtinator 3y ago
    FWICT file is more capable, predictable and also faster while being more energy-efficient at the same time.
lifthrasiir 3y ago
I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on precision or recall.
- charcircuit 3y ago
  apks are also zipaligned so it's not like random users are going to be making them either
- HtmlProgrammer 3y ago
  Minecraft mods 14 years ago used to tell you to open the JAR and delete the META-INF when installing them so can’t rely on that one…
- supriyo-biswas 3y ago
  The `file` command checks only the first few bytes, and doesn’t parse the structure of the file. APK files are indeed reported as Zip archives by the latest version of `file`.
  - m0shen 3y ago
    This is false in every sense for https://www.darwinsys.com/file/ https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version. FILE_45: ./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk ../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing Block
    - supriyo-biswas 3y ago
      Interesting! I checked with file 5.44 from Ubuntu 23.10 and 5.45 on macOS using homebrew, and in both cases, I got “Zip archive data, at least v2.0 to extract” for the file here[1]. I don’t have an Android phone to check and I’m also not familiar with Android tooling, so is this a corrupt APK? [1] https://download.apkpure.net/custom/com.apkpure.aegon-3197817.apk?_fn=QVBLUHVyZV92My4xOS43OF9hcGtwdXJlLmNvbS5hcGs&as=952379a4c2cf139483a198906d30724d65cef794&ai=-711161207&at=1708062492&_sa=ai%2Cat&k=af2a27ab4a6fcef520110d64b356610d65d19a1c&r=https%3A%2F%2Fapkpure.net%2Famp&_p=Y29tLmFwa3B1cmUuYWVnb24&c=1%7CTOOLS%7CZGV2PUFQS1B1cmUmdD1hcGsmcz0xMDQzMzgyNiZ2bj0zLjE5Ljc4JnZjPTMxOTc4MTc&hot=1&arg=apkpure%3A%2F%2Fcampaign%2F%3Futm_medium%3Dapkpure%26utm_source%3Dpre-register%26report_context%3D%7B%22channel_id%22%3A2007%2C%22source_package_name%22%3A%22com.levelinfinite.sgameGlobal%22%7D&arg=apkpure%3A%2F%2Fcustom%3Ftype%3Dopen_detail%26package_name%3Dcom.levelinfinite.sgameGlobal%26auto_pre_register%3D1 https://download.apkpure.net/custom/com.apkpure.aegon-319781...
Vt71fcAqt7 3y ago
This feels like old school google. I like that it's just a static webpage that basically can't be shut down or sunsetted. It reminds of when Google just made useful stuff and gave them away for free on a webpage like translate and google books. Obviously less life changing than the above but still a great option to have when I need this.
vrnvu 3y ago
At $job we have been using Apache Tika for years. Works but occasionally having bugs and weird collisions when working with billions of files. Happy to see new contributions in the space.
johnea 3y ago
The results of which you'll never be 100% sure are correct...
- wruza 3y ago
  They missed such an opportunity to name it "fail". It's like "file" but with "ai" in it.
  - tamrix 3y ago
    What about faile?
- rfoo 3y ago
  But file(2) is already like that - my data files without headers are reported randomly as disk images, compressed archives or even executables for never-heard-of machines.
- _vufv 3y ago
  Other methods use heuristics to guess many filetypes and in the benchmark they show worse performance (in terms of precision). Assuming benchmarks are not biased, the fact that this approach uses AI heuristics instead of hard-coded heuristics shouldn't make it strictly worse.
Imnimo 3y ago
I wonder how big of a deal it is that you'd have to retrain the model to support a new or changed file type? It doesn't seem like the repo contains training code, but I could be missing it...
m0shen 3y ago
As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/google/magika/blob/120205323e260dad4e58778093fb220aa1991c2b/js/README.md#L43-L44 https://github.com/google/magika/blob/120205323e260dad4e5877... It loads the model an runtime: https://github.com/google/magika/blob/120205323e260dad4e58778093fb220aa1991c2b/js/magika.js#L74-L75 https://github.com/google/magika/blob/120205323e260dad4e5877... They mark it as Experimental in the documentation, but it seems like it was just made for the web demo. Also as others have mentioned. The model appears to only detect 116 file types: https://github.com/google/magika/blob/120205323e260dad4e58778093fb220aa1991c2b/docs/supported-content-types-list.md https://github.com/google/magika/blob/120205323e260dad4e5877... Where libmagic detects... a lot. Over 1600 last time I checked: https://github.com/file/file/tree/4cbd5c8f0851201d203755b76cb66ba991ffd8be/magic/Magdir https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c... I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.
- lebean 3y ago
  It's for researchers, probably.
  - m0shen 3y ago
    Yeah, there is this line: By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale. Which implies a production-ready release for general usage, as well as usage by security researchers.
- m0shen 3y ago
  Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9e13 https://gist.github.com/moshen/784ee4a38439f00b17855233617e9... hyperfine ./magika.bash ./file.bash Benchmark 1: ./magika.bash Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms] Range (min … max): 684.0 ms … 738.9 ms 10 runs Benchmark 2: ./file.bash Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms] Range (min … max): 22.4 ms … 29.0 ms 111 runs Summary './file.bash' ran 29.88 ± 1.65 times faster than './magika.bash'
kazinator 3y ago
> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format. This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand. Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".
- deleted 3y ago
  [deleted]
- cAtte_ 3y ago
  the rules aren't based on the formats, but on a small portion of them (their magic numbers). this makes them inaccurate (think docx vs zip) and heuristic.
- cle 3y ago
  Besides compound file types, not all formats are well-specified either. Example is CSV.
- summerlight 3y ago
  1. Not all file formats are well specified 2. Not all files are precisely following the specification 3. Not all file formats are mutually exclusive Those facts are clearly reflected in the table.
Nullabillity 3y ago
It seems to detect my Android build.gradle.kts as Scala, which I suppose is a kind of hilarious confusion but not exactly useful.
krick 3y ago
What are use-cases for this? I mean, obviously detecting the filetype is useful, but we kinda already have plenty of tools to do that, and I cannot imagine, why we need some "smart" way of doing this. If you are not a human, and you are not sure what is it (like, an unknown file being uploaded to a server) you would be better off just rejecting it completely, right? After all, there's absolutely no way an "AI powered" tool can be more reliable than some dumb, err-on-safer-side heuristic, and you wouldn't want to trust that thing to protect you from malicious payloads.
- n2d4 3y ago
  Virus detection is mentioned in the article. Code editors need to find the programming language for syntax highlighting of code before you give it a name. Your desktop OS needs to know which program to open files with. Or, recovering files from a corrupted drive. Etc It's easy to distinguish, say, a PNG from a JPG file (or anything else that has well-defined magic bytes). But some files look virtually identical (eg. .jar files are really just .zip files). Also see polyglot files [1]. If you allow an `unknown` label or human intervention, then yes, magic bytes might be enough, but sometimes you'd rather have a 99% chance to be right about 95% of files vs. a 100% chance to be right about 50% of files. [1] https://en.wikipedia.org/wiki/Polyglot_(computing) https://en.wikipedia.org/wiki/Polyglot_(computing)
- nindalf 3y ago
  > no way an "AI powered" tool can be more reliable The article provides accuracy benchmarks. > you would be better off just rejecting it completely They mention using it in gmail and Drive, neither of which have the luxury of rejecting files willy-nilly.
  - fuzztester 3y ago
    I have not tried it recently, but IIRC, Gmail does reject attachments which are zip files, for security reasons.
    - wildrhythms 3y ago
star4040 3y ago
It seems like it defeats the purpose of such a tool that this initial version doesn’t have polyglot files. I hope they’re quick to work on that.
VikingCoder 3y ago
What does it do with an Actually Portable Executable compiled by Cosmopolitan libc compiler?
- supriyo-biswas 3y ago
  It’s reported as a PE executable, `file` on the other hand reports it as a “DOS/MBR boot sector.”
lopkeny12ko 3y ago
I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.
- potatoman22 3y ago
  This also works for formats like Python, HTML, and JSON.
  - LiamPowell 3y ago
    file (https://www.darwinsys.com/file/ https://www.darwinsys.com/file/) already detects all these formats.
    - ebursztein 3y ago
      Indeed but as pointed out in the blog post -- file is significantly less accurate that Magika. There are also some file type that we support and file doesn't as reported in the table.
      - LiamPowell 3y ago
        I can't immediately find the dataset used for benchmarking. Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?
        schleck8 3y ago
        > Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see? That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.
hansonpeter 3y ago
[dead]
a-dub 3y ago
probably a lot of interesting work going on that looks like this for the virustotal db itself.
account-5 3y ago
Assuming that I've not misunderstood, how does this compare to things like: TrID [0]?? Apart from being open source. [0] https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html
- JacobThreeThree 3y ago
  The bulk of the short article is a set of performance benchmarks comparing Magika to TrID and others.
  - account-5 3y ago
    Argh, the risks of browsing the web without JavaScript and/or third party scripts enabled, you miss content, because rendering text and images on the modern web can't be done without them, apparently. (Sarcasm). You are of course correct. I can see the images showing the comparison. Apologies.
earth2mars 3y ago
how do i pronounce this? Myajika or MaGika? anyhow, its super cool.
thangalin 3y ago
My FOSS desktop text editor performs a subset of file type identification using the first 12 bytes, detecting the type quite quickly: * https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/src/main/java/com/keenwrite/io/MediaTypeSniffer.java https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/src/main... There's a much larger list of file signatures at: * https://github.com/veniware/Space-Maker/blob/master/FileSignatures.cs https://github.com/veniware/Space-Maker/blob/master/FileSign...
Labo333 3y ago
I wonder what the output will be on polyglot files like run-anywhere binaries produced by cosmopolitan [1] [1]: https://justine.lol/cosmopolitan/ https://justine.lol/cosmopolitan/
awaythrow999 3y ago
Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf Edit: just tested, and it does only identify the zip layer
- rvnx 3y ago
  You can try it here: https://google.github.io/magika/ https://google.github.io/magika/ It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope. I guess it's more for detecting common file formats then with high recall. However, where is the actual source of the model ? Let's say I want to add a new file format myself. Apparently only the source of the interpreter is here, not the source of the model nor the training set, which is the most important thing.
  - alexandreyc 3y ago
    Yes, I totally agree; it's not what I would qualify as open source. Do you plan to release the training code along the research paper? What about the dataset? In any case, it's very neat to have ML-based technique and lightweight model for such tasks!
  - tempay 3y ago
    Is there anything about the performance on unknown files? I've tried a few that aren't "basic" but are widely used enough to be well supported in libmagic and it thinks they're zip files. I know enough about the underlying formats to know they're not using zip as a container under-the-hood.
  - kevincox 3y ago
    Apparenty the Super Mario Bros. 3 ROM is 100% a SWF file. Cool that you can use it online though. Might end up using it like that. Although it seems like it may focus on common formats.
diimdeep 3y ago
> Magika: AI powered fast and efficient file type identification of 116 file types with proprietary puny model with no training code and no dataset. > We are releasing a paper later this year detailing how the Magika model was trained and its performance on large datasets. And ? How do you advance industry by this googleblog post and source code that is useless without closed source model ? All I see here is loud marketing name, loud promises, but actually barely anything useful. Hooly rooftop characters sideproject?
secondary_op 3y ago
Why is this piece of code being sold as open source, when in reality it just calls into proprietary ML blob that is tiny and useless, and actual source code of model is closed while properly useful large model is non existing ?
- KOLANICH 3y ago
  Not into proprietary, the blob is within an Apache-licensed repo. Though there was no code to train it, but the repo contains some info allowing to recreate the code training it. Basically a JSON-based configs containing graph architecture. Even if you didn't have them, the repo contains an ONNX model, from which one can devise the architecture.
flohofwoe 3y ago
I wonder how it performs with detecting C vs C++ vs ObjC vs ObjC++ and for bonus points: the common C/C++ subset (which is an incompatible C fork), also extra bonus points for detecting language version compatibility (e.g. C89 vs C99 vs C11...). Separating C from C++ and ObjC is where the file type detection on Github traditionally had problems with (but has been getting dramatically better over time), from an "AI-powered" solution which has been trained on the entire internet I would expect to do better right from the start. The list here doesn't even mention any of those languages except C though: https://github.com/google/magika/blob/main/docs/supported-content-types-list.md https://github.com/google/magika/blob/main/docs/supported-co...
andrewstuart 3y ago
Very useful. I wrote an editor that needed file type detection but the results of traditional approaches were flaky.
20after4 3y ago
I just want to say thank you for the release. There are quite a lot of complaints in the comments but I think this is a useful and worthwhile contribution and I appreciate the authors for going through the effort to get it approved for open source release. It would be great if the model training data was included (or at lease documentation about how to reproduce it.) but that doesn’t preclude this being useful. Thanks!
Someone 3y ago
If their “Exif Tool” is https://exiftool.org/ https://exiftool.org/ (what else could it be?), I don’t understand why they included it in their tests. Also, how does ExifTool recognize Python and html files?
Andugal 3y ago
I have a question: Is something like Magika enough to check if a file is malicious or not? Example: users can upload PNG file (and only PNG is accepted). If Malika detects that the file is a PNG, does this mean the file is clean?
- cjg 3y ago
  > does this mean the file is clean? No.
- TacticalCoder 3y ago
  If that PNG of yours is not just an example note that you can detect easily if the PNG as any extra data (which may or may not indicate an attempt as mischief) and reject the (rare) PNGs with extra data. I ran a script checking the thousands of PNGs on my system and found three with extra data, all three probably due to the "PNG acropalypse" bug (but mischief cannot be ruled out). P.S: btw I'm not implying using extra data that shouldn't be there in a PNG is the only way to have a malicious PNG.
- nicklecompte 3y ago
  This comment from kevincox[1] says the answer is a hard "no": > Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data". There are other comments in this thread that make me think Google contaminated their test data with training data and the 99% results should not be taken at face value. OTOH I am not particularly surprised that Magika would be better than the other tools at distinguishing semi-unstructured plain text e.g. Java source vs. C++ source or YAMLs versus INIs. But that's a very different use case than many security applications. The comments here suggest Magika is especially susceptible to binary obfuscation. [1] https://news.ycombinator.com/item?id=39395677 https://news.ycombinator.com/item?id=39395677
- kevincox 3y ago
  The only way to do this reliably is to render the PNG to pixels then render it back to an PNG with a trusted encoder. Of course now you are taking the risk of vulnerabilities in the "render to pixels" step. But the result will be clean. AKA parse, don't validate.
TacticalCoder 3y ago
To me the obvious use case is to first use the file command but then, when file returns "DATA" (meaning it couldn't guess the file type), call magika. I guess I'll be writing a wrapper (only for when using my shell in interactive mode) around file doing just that when I come back from vacation. I hate it when file cannot do its thing. Put it this way: I use file a lot and I know at times it cannot detect a filetype. But is file often wrong when it does have a match? I don't think so... So in most of the cases I'd have file correctly give me the filetype, very quickly but then in those rare cases where file cannot find anything, I'd then use the slower but apparently more capable magika.
- SnowflakeOnIce 3y ago
  I have seen 'file' misclassify many things when running it at large scale (millions of files) from a hodgepodge of sources. Unrelated types getting called 'GPG Private Keys', for example. For textual data types, 'file' gets confused often, or doesn't give a precise type. GitHub's 'linguist' [1] tool does much better here, but is structured in such a way that it is difficult to call it on an arbitrary file or bytestring that doesn't reside in a git repo. I'd love to have a classification tool that can more granularly classify textual files! It may not be Magika _today_ since it only supports 116-something types. For this use case, an ML-based approach will be more successful than an approach based solely on handwritten heuristic rules. I'm excited to see where this goes.
lakomen 3y ago
Why? Just check the damn headers. Why do you need a power hungry and complicated AI model to do it? Why?
TomNomNom 3y ago
This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff and woff2 files it identified as "TrueType Font Data", others are "Unknown binary data (unknown)" with low confidence guesses ranging from FLAC audio to ISO 9660. Again, the `file` utility correctly identifies these files as "Web Open Font Format". I like the idea, but the current implementation can't be relied on IMO; especially not for automation. A minor pet peeve also: it doesn't seem to detect when its output is a pipe and strip the shell colour escapes resulting in `^[[1;37` and `^[[0;39m` wrapping every line if you pipe the output into a vim buffer or similar.
- michaelmior 3y ago
  > the current implementation can't be relied on IMO What's your reasoning for not relying on this? (It seems to me that this would be application-dependent at the very least.)
  - jdiff 3y ago
    I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into question the claims made about its performance. What reasoning is there to trust it?
    - michaelmior 3y ago
      > It whiffed multiple common softballs I must have missed this in the article. Where was this?
      - deleted 3y ago
        [deleted]
      - 3y ago
nayuki 3y ago
The name sounds like the Pokémon Magikarp or the anime series Madoka Magica.
pier25 3y ago
I use FFMPEG to detect if uploaded files are valid audio files. Would this be much faster?
omni 3y ago
Can someone please help me understand why this is useful? The article mentions malware scanning applications, but if I'm sending you a malicious PDF, won't I want to clearly mark it with a .pdf extension so that you open it in your PDF app? Their examples are all very obvious based on file extensions.
chromaton 3y ago
It can't correctly identify a DXF file in my testing. It categorizes it as plain text.
Eiim 3y ago
I ran a quick test on 100 semi-random files I had laying around. Of those, 81 were detected correctly, 6 were detected as the wrong file type, and 12 were detected with an unspecific file type (unknown binary/generic text) when a more specific type existed. In 4 of the unspecific cases, a low-confidence guess was provided, which was wrong in each case. However, almost all of the files which were detected wrong/unspecific are of types not supported by Magika, with one exception of a JSON file containing a lot of JS code as text, which was detected as JS code. For comparison, file 5.45 (the version I happened to have installed) got 83 correct, 6 wrong, and 10 not specific. It detected the weird JSON correctly, but also had its own strange issues, such as detecting a CSV as just "data". The "wrong" here was somewhat skewed by the 4 GLSL shader code files that were in the dataset for some reason, all of which it detected as C code (Magika called them unknown). The other two "wrong" detections were also code formats that it seems it doesn't support. It was also able to output a lot more information about the media files. Not sure what to make of these tests but perhaps they're useful to somebody.
- pizzalife 3y ago
  > The "wrong" here was somewhat skewed by the 4 GLSL shader code files that were in the dataset for some reason, all of which it detected as C code To be fair though, a snippet of GLSL shader code can be perfectly valid C.
  - Eiim 3y ago
    Indeed, which is why I felt the need to call it out here. I'm not certain if the files on question actually happened to be valid C but whether that's a meaningful mistake regardless is left to the reader to decide.
Delumine 3y ago
Voidtools - Everything.. looking at you to implement this
init0 3y ago
Why not detect it by checking the magic number of the buffer?
- dagmx 3y ago
  Not every file has one for starters and many can be incorrect. Especially in the context of use as a virus scanner, you don’t trust what the file says it is
YoshiRulz 3y ago
So instead of spending some of their human resources to improve libmagic, they used some of their computing power to create an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes), and which is much less effective in an adversarial context, and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. Thanks guys.
- 12_throw_away 3y ago
  Come on, can't you help but be impressed by this amazing AI tech? That gives us sci-fi tools like ... a less-accurate, incomplete, stochastic, un-debuggable, slower, electricity-guzzling version of `file`.
- famouswaffles 3y ago
  >So instead of spending some of their human resources to improve libmagic A large megacorp can work on multiple things at once. >an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes) You say that like it's a contradiction but it's not. >and which is much less effective in an adversarial context, Is it? This seems like an assumption. >and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. Being introspectable or not has no bearing on the accuracy of a system.
  - YoshiRulz 3y ago
    > > an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes) > You say that like it's a contradiction but it's not. > > and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. > Being introspectable or not has no bearing on the accuracy of a system. "Open source" and "neural net" is the contradiction, as I went on to write. Even if magika were a more accurate version of file, the implication that it could "help [libmagic] improve" isn't really true, because how do you distill the knowledge from it into a patch for libmagic? My point re: their "error-prone" claim is that their comparison was disingenuous due to the functionality difference between the tools. (Also with their implication that AIs work perfectly, though this one sounds pretty good by the numbers. I of course accept that there's likely to be some bugs in code written by humans.) > > and which is much less effective in an adversarial context, > Is it? This seems like an assumption. It is, one based on what I've heard about AI classifiers over the years. Other commenters here are interested in this point, but while I don't see anyone experimenting on magika (it's new after all), the fact it's not mentioned in the article leads me to believe they didn't try to attack themselves. (Or did, but with bad results, and so decided not to include that. Funnily enough they did mention adversarial attacks on manually-written classifiers...)
breather 3y ago
Can we please god stop using AI like it's a meaningful word? This is really interesting technology; it's hamstrung by association with a predatory marketing term.
goshx 3y ago
I used an HTML file and added JPEG magic bytes to its header: magika file.jpg file.jpg: JPEG image data (image)
runxel 3y ago
Took a .dxf file and fed it to Magika. It says with confidence of 97% that that must be a PowerShell file. A classic .dwg could be "mscompress" (whatever that is), 81%, or a GIF. Both couldn't be further from the truth. Common files are categorized successfully – but well, yeah that's not really an achievement. Pretty much nothing more than a toy right now.
- queuebert 3y ago
  The real problem with deep learning approaches is hallucination and edge case failures. When someone finally fixes this, I hope it makes the HN front page.
lqcfcjx 3y ago
After reading thru all the comments, honestly I still don't get the point of this system. What is potential practical value or applications of this model?
jwithington 3y ago
I guess I'm kind of a dummy on this, but why is it impressive to identify that a .js file is Javascript, a .md file is Markdown, etc?
- playingalong 3y ago
  Because it's done by inspecting the content, not the name of the file.
woliveirajr 3y ago
Reminds me when someone asked (at StackOverflow) on how to recognize binaries for different architetures, like x86 or ARM-something or Apple M1 and so on. I gave the idea to use the technique of NCD (Normalized compression distance), based on Kolmogorov complexity. Celibrasi, R. was one great researcher in this area, and I think he worked at Google at some point. Using AI seems to follow the same path: "learn" what represents some specific file and then compare the unknown file to those references (AI:all the parameters, NCD:compression against a known type).
aidenn0 3y ago
But will it let you print on Tuesday[1]? 1: https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28/+index https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
- queuebert 3y ago
  For a subscription fee.
jjsimpso 3y ago
I wrote an implementation of libmagic in Racket a few years ago(https://github.com/jjsimpso/magic https://github.com/jjsimpso/magic). File type identification is a pretty interesting topic. As others have noted, libmagic detects many more file types than Magika, but I can see Magika being useful for text files in particular, because anything written by humans doesn't have a rigid format.
notso411 3y ago
[dead]
runald 3y ago
This is useful for detecting file types of unknown blobs with custom file extension, when the file command just returns data. Though it doesn't correctly identify lua code for some reason, it guesses with low probability that it's either ruby or javascript, or anything but lua.
delijati 3y ago
Nice aka perfect timing.I just restored "some" files (40GB) with [1] But the filetype detection of photorec set some wrong file types. Edit: It would be super helpful if the "suffix" could be added as output so i can move the files to the right directory [2] ;) [1] https://www.cgsecurity.org/wiki/PhotoRec https://www.cgsecurity.org/wiki/PhotoRec [2] https://github.com/google/magika/issues/63 https://github.com/google/magika/issues/63
alexott 3y ago
Mime type detection is very interesting thing. I wrote media type detection for McAfee Web Gateway 7.x and because it was a high performance proxy, the detection speed was a major focus, but also the precision, especially for "container types, like, MS Office, OLE-based files, etc. The base of it was a simple Lisp-like language that allowed to write signatures very fast, and everything was combined with very aggressive caching of the data, so we avoided to read data again and again, and used internal caches a lot. In tests, the detection was ~10x faster than file, and with more flexible language we got more file types recognized precisely. Although there were challenges with some formats, like, OLE-based files had FAT directory structure at the end of the file, and you were need to walk the tree to find the top-level structure to distinguish Excel file from Excel file embedded into Word. Streams detection was also quite funny task...
- alexott 3y ago
  Ah, I remember, the self-extracted .msi file was one of the quite challenging files - it's executable, it's a .cab file, and OLE2-container
glassonion999 3y ago
I created a demo site for Magika. https://9revolution9.com/tools/security/file_scanner/ https://9revolution9.com/tools/security/file_scanner/

695 points by alphabetting 3y ago

kushie 3y ago

this couldnt have been released at a better time for me! really needed a library like this.

petesergeant 3y ago
Tell us why!
ebursztein 3y ago
Thanks :)

NiloCK 3y ago

A somewhat surprising and genuinely useful application of the family of techniques. I wonder how susceptible it is to adversarial binaries or, hah, prompt-injected binaries.

dghlsakjg 3y ago
“These aren’t the binaries you are looking for…”
jamesdwilson 3y ago
For the extremely limited number of file types supported, I question the utility of this compared to `magic`
star4040 3y ago
It gets a lot of binary file formats wrong for me out-of-the-box. I think it needs to be a bit more effective before we can truly determine the effectiveness of such exploits.
- queuebert 3y ago
  But they reported >99% accuracy on their cherry-picked dataset! /s
nicklecompte 3y ago
Elsewhere in the thread kevincox[1] points out that it's extremely susceptible to adversarial binaries: > Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data". Seems like this is genuinely useless for anybody but AI researchers. [1] https://news.ycombinator.com/item?id=39395677 https://news.ycombinator.com/item?id=39395677

thorum 3y ago

Supported file types: https://github.com/google/magika/blob/main/docs/supported-content-types-list.md https://github.com/google/magika/blob/main/docs/supported-co...

s1mon 3y ago
It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.
- _3u10 3y ago
  No tracker / .mod files either, just use file.
  - ebursztein 3y ago
    Thanks for the list, we will probably try to extend the list of format supported in future revision.
- photoGrant 3y ago
  Yeah this quickly went from 'additional helpful tool in the kit' to 'probably should use something else first'
- vintermann 3y ago
  Well, what they used this for at Google was apparently scanning their user's files for things they shouldn't store in the cloud. Probably they don't care much about MIDI.
- kevincox 3y ago
  Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data".

vunderba 3y ago

As somebody who's dealt with the ambiguity of attempting to use file signatures in order to identify file type, this seems like a pretty useful library. Especially since it seems to be able to distinguish between different types of text files based on their format/content e.g. CSV, markdown, etc.

semitones 3y ago

Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?

hiddencost 3y ago
malware can intentionally obfuscate itself
callalex 3y ago
Nothing is ever simple. Even for the most basic .txt files it’s still useful to know what the character encoding is (utf? 8/16? Latin-whatever? etc.) and what the line format is (\n,\cr\lf,\n\lf) as well as determining if some maniac removed all the indentation characters and replaced them with a mystery number of spaces. Then there are all the container formats that have different kinds of formats embedded in them (mov,mkv,pdf etc.)
- cole-k 3y ago
  A fun read in service of your first point: https://en.wikipedia.org/wiki/Bush_hid_the_facts https://en.wikipedia.org/wiki/Bush_hid_the_facts
SnowflakeOnIce 3y ago
Yes! Sometimes a file has no extension. Other times the extension is a lie. Still other times, you may be dealing with an unnamed bytestring and wish to know what kind of content it is. This last case happens quite a lot in Nosey Parker [1], a detector of secrets in textual data. There, it is possible to come across unnamed files in Git history, and it would be useful to the user to still indicate what type of file it seems to be. I added file type detection based on libmagic to Nosey Parker a while back, but it's not compiled in by default because libmagic is slow and complicates the build process. Also, libmagic is implemented as a large C library whose primary job is parsing, which makes the security side of me jittery. I will likely add enabled-by-default filetype detection to Nosey Parker using Magika's ONNX model. [1] https://github.com/praetorian-inc/noseyparker https://github.com/praetorian-inc/noseyparker

userbinator 3y ago

Today web browsers, code editors, and countless other software rely on file-type detection to decide how to properly render a file. "web browsers"? Odd to see this coming from Google itself. https://en.wikipedia.org/wiki/Content_sniffing https://en.wikipedia.org/wiki/Content_sniffing was widely criticised for being problematic for security.

rafram 3y ago
Content sniffing can be disabled by the server (X-Content-Type-Options: nosniff), but it’s still used by default. Web browsers have to assume that servers are stupid, and that for relatively harmless cases, it’s fine to e.g. render a PNG loaded by an <img> even if it’s served as text/plain.

stevepike 3y ago

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 https://bugs.freedesktop.org/show_bug.cgi?id=78797. My patch got rejected for needing to look at the first 3KB bytes of the file to figure out the type. They had a hard limit that they wouldn't see past the first 256 bytes. Now in 2024 we're doing this with deep learning! It'd be cool if google released some speed performance benchmarks here against the old-fashioned implementations. Obviously it'd be slower, but is it 1000x or 10^6x?

renonce 3y ago
From the first paragraph: > enabling precise file identification within milliseconds, even when running on a CPU. Maybe your old-fashioned implementations were detecting in microseconds?
- deleted 3y ago
  [deleted]
- stevepike 3y ago
  Yeah I saw that, but that could cover a pretty wide range and it's not clear to me whether that relies on preloading a model.
  - ryanjshaw 3y ago
    > At inference time Magika uses Onnx as an inference engine to ensure files are identified in a matter of milliseconds, almost as fast as a non-AI tool even on CPU.
ebursztein 3y ago
Co-author of Magika here (Elie) so we didn't include the measurements in the blog post to avoid making it too long but we did those measurements. Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples. So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeated detection. This is why we said it is not that much slower. We will have more performance measurements in the upcoming research paper. Hope that answer the question

rfl890 3y ago

We have had file(1) for years

Am4TIfIsER0ppos 3y ago
[flagged]
- samtheprogram 3y ago
  This a 1mb Keras ML model that’s open source. I passionately dislike Surveillance Capitalism but bringing it up when it’s completely irrelevant only weakens the argument. RTFA.
  - floobertoober 3y ago
    > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that". From the HN guidelines
    - samtheprogram 3y ago
      > can be shortened to "The article mentions that". I shortened it to RTFA.
samtheprogram 3y ago
This is beyond what file is capable of. It’s also mentioned in the third paragraph. RTFA.
- wruza 3y ago
  Some HN readers may not know about file(1) even. It's fine to mention that $subj enhances that, but the rtfa part seems pretty unnecessary.
- Vogtinator 3y ago
  FWICT file is more capable, predictable and also faster while being more energy-efficient at the same time.

lifthrasiir 3y ago

I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on precision or recall.

charcircuit 3y ago
apks are also zipaligned so it's not like random users are going to be making them either
HtmlProgrammer 3y ago
Minecraft mods 14 years ago used to tell you to open the JAR and delete the META-INF when installing them so can’t rely on that one…
supriyo-biswas 3y ago
The `file` command checks only the first few bytes, and doesn’t parse the structure of the file. APK files are indeed reported as Zip archives by the latest version of `file`.
- m0shen 3y ago
  This is false in every sense for https://www.darwinsys.com/file/ https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version. FILE_45: ./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk ../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing Block
  - supriyo-biswas 3y ago
    Interesting! I checked with file 5.44 from Ubuntu 23.10 and 5.45 on macOS using homebrew, and in both cases, I got “Zip archive data, at least v2.0 to extract” for the file here[1]. I don’t have an Android phone to check and I’m also not familiar with Android tooling, so is this a corrupt APK? [1] https://download.apkpure.net/custom/com.apkpure.aegon-3197817.apk?_fn=QVBLUHVyZV92My4xOS43OF9hcGtwdXJlLmNvbS5hcGs&as=952379a4c2cf139483a198906d30724d65cef794&ai=-711161207&at=1708062492&_sa=ai%2Cat&k=af2a27ab4a6fcef520110d64b356610d65d19a1c&r=https%3A%2F%2Fapkpure.net%2Famp&_p=Y29tLmFwa3B1cmUuYWVnb24&c=1%7CTOOLS%7CZGV2PUFQS1B1cmUmdD1hcGsmcz0xMDQzMzgyNiZ2bj0zLjE5Ljc4JnZjPTMxOTc4MTc&hot=1&arg=apkpure%3A%2F%2Fcampaign%2F%3Futm_medium%3Dapkpure%26utm_source%3Dpre-register%26report_context%3D%7B%22channel_id%22%3A2007%2C%22source_package_name%22%3A%22com.levelinfinite.sgameGlobal%22%7D&arg=apkpure%3A%2F%2Fcustom%3Ftype%3Dopen_detail%26package_name%3Dcom.levelinfinite.sgameGlobal%26auto_pre_register%3D1 https://download.apkpure.net/custom/com.apkpure.aegon-319781...

Vt71fcAqt7 3y ago

This feels like old school google. I like that it's just a static webpage that basically can't be shut down or sunsetted. It reminds of when Google just made useful stuff and gave them away for free on a webpage like translate and google books. Obviously less life changing than the above but still a great option to have when I need this.

vrnvu 3y ago

At $job we have been using Apache Tika for years. Works but occasionally having bugs and weird collisions when working with billions of files. Happy to see new contributions in the space.

johnea 3y ago

The results of which you'll never be 100% sure are correct...

wruza 3y ago
They missed such an opportunity to name it "fail". It's like "file" but with "ai" in it.
- tamrix 3y ago
  What about faile?
rfoo 3y ago
But file(2) is already like that - my data files without headers are reported randomly as disk images, compressed archives or even executables for never-heard-of machines.
_vufv 3y ago
Other methods use heuristics to guess many filetypes and in the benchmark they show worse performance (in terms of precision). Assuming benchmarks are not biased, the fact that this approach uses AI heuristics instead of hard-coded heuristics shouldn't make it strictly worse.

Imnimo 3y ago

I wonder how big of a deal it is that you'd have to retrain the model to support a new or changed file type? It doesn't seem like the repo contains training code, but I could be missing it...

m0shen 3y ago

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/google/magika/blob/120205323e260dad4e58778093fb220aa1991c2b/js/README.md#L43-L44 https://github.com/google/magika/blob/120205323e260dad4e5877... It loads the model an runtime: https://github.com/google/magika/blob/120205323e260dad4e58778093fb220aa1991c2b/js/magika.js#L74-L75 https://github.com/google/magika/blob/120205323e260dad4e5877... They mark it as Experimental in the documentation, but it seems like it was just made for the web demo. Also as others have mentioned. The model appears to only detect 116 file types: https://github.com/google/magika/blob/120205323e260dad4e58778093fb220aa1991c2b/docs/supported-content-types-list.md https://github.com/google/magika/blob/120205323e260dad4e5877... Where libmagic detects... a lot. Over 1600 last time I checked: https://github.com/file/file/tree/4cbd5c8f0851201d203755b76cb66ba991ffd8be/magic/Magdir https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c... I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.

lebean 3y ago
It's for researchers, probably.
- m0shen 3y ago
  Yeah, there is this line: By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale. Which implies a production-ready release for general usage, as well as usage by security researchers.
m0shen 3y ago
Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9e13 https://gist.github.com/moshen/784ee4a38439f00b17855233617e9... hyperfine ./magika.bash ./file.bash Benchmark 1: ./magika.bash Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms] Range (min … max): 684.0 ms … 738.9 ms 10 runs Benchmark 2: ./file.bash Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms] Range (min … max): 22.4 ms … 29.0 ms 111 runs Summary './file.bash' ran 29.88 ± 1.65 times faster than './magika.bash'

kazinator 3y ago

> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format. This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand. Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".

deleted 3y ago
[deleted]
cAtte_ 3y ago
the rules aren't based on the formats, but on a small portion of them (their magic numbers). this makes them inaccurate (think docx vs zip) and heuristic.
cle 3y ago
Besides compound file types, not all formats are well-specified either. Example is CSV.
summerlight 3y ago
1. Not all file formats are well specified 2. Not all files are precisely following the specification 3. Not all file formats are mutually exclusive Those facts are clearly reflected in the table.

Nullabillity 3y ago

It seems to detect my Android build.gradle.kts as Scala, which I suppose is a kind of hilarious confusion but not exactly useful.

krick 3y ago

What are use-cases for this? I mean, obviously detecting the filetype is useful, but we kinda already have plenty of tools to do that, and I cannot imagine, why we need some "smart" way of doing this. If you are not a human, and you are not sure what is it (like, an unknown file being uploaded to a server) you would be better off just rejecting it completely, right? After all, there's absolutely no way an "AI powered" tool can be more reliable than some dumb, err-on-safer-side heuristic, and you wouldn't want to trust that thing to protect you from malicious payloads.

n2d4 3y ago
Virus detection is mentioned in the article. Code editors need to find the programming language for syntax highlighting of code before you give it a name. Your desktop OS needs to know which program to open files with. Or, recovering files from a corrupted drive. Etc It's easy to distinguish, say, a PNG from a JPG file (or anything else that has well-defined magic bytes). But some files look virtually identical (eg. .jar files are really just .zip files). Also see polyglot files [1]. If you allow an `unknown` label or human intervention, then yes, magic bytes might be enough, but sometimes you'd rather have a 99% chance to be right about 95% of files vs. a 100% chance to be right about 50% of files. [1] https://en.wikipedia.org/wiki/Polyglot_(computing) https://en.wikipedia.org/wiki/Polyglot_(computing)
nindalf 3y ago
> no way an "AI powered" tool can be more reliable The article provides accuracy benchmarks. > you would be better off just rejecting it completely They mention using it in gmail and Drive, neither of which have the luxury of rejecting files willy-nilly.
- fuzztester 3y ago
  I have not tried it recently, but IIRC, Gmail does reject attachments which are zip files, for security reasons.
  - wildrhythms 3y ago

star4040 3y ago

It seems like it defeats the purpose of such a tool that this initial version doesn’t have polyglot files. I hope they’re quick to work on that.

VikingCoder 3y ago

What does it do with an Actually Portable Executable compiled by Cosmopolitan libc compiler?

supriyo-biswas 3y ago
It’s reported as a PE executable, `file` on the other hand reports it as a “DOS/MBR boot sector.”

lopkeny12ko 3y ago

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.

potatoman22 3y ago
This also works for formats like Python, HTML, and JSON.
- LiamPowell 3y ago
  file (https://www.darwinsys.com/file/ https://www.darwinsys.com/file/) already detects all these formats.
  - ebursztein 3y ago
    Indeed but as pointed out in the blog post -- file is significantly less accurate that Magika. There are also some file type that we support and file doesn't as reported in the table.
    - LiamPowell 3y ago
      I can't immediately find the dataset used for benchmarking. Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?
      - schleck8 3y ago
        > Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see? That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.

hansonpeter 3y ago

[dead]

a-dub 3y ago

probably a lot of interesting work going on that looks like this for the virustotal db itself.

account-5 3y ago

Assuming that I've not misunderstood, how does this compare to things like: TrID [0]?? Apart from being open source. [0] https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html

JacobThreeThree 3y ago
The bulk of the short article is a set of performance benchmarks comparing Magika to TrID and others.
- account-5 3y ago
  Argh, the risks of browsing the web without JavaScript and/or third party scripts enabled, you miss content, because rendering text and images on the modern web can't be done without them, apparently. (Sarcasm). You are of course correct. I can see the images showing the comparison. Apologies.

earth2mars 3y ago

how do i pronounce this? Myajika or MaGika? anyhow, its super cool.

thangalin 3y ago

My FOSS desktop text editor performs a subset of file type identification using the first 12 bytes, detecting the type quite quickly: * https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/src/main/java/com/keenwrite/io/MediaTypeSniffer.java https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/src/main... There's a much larger list of file signatures at: * https://github.com/veniware/Space-Maker/blob/master/FileSignatures.cs https://github.com/veniware/Space-Maker/blob/master/FileSign...

Labo333 3y ago

I wonder what the output will be on polyglot files like run-anywhere binaries produced by cosmopolitan [1] [1]: https://justine.lol/cosmopolitan/ https://justine.lol/cosmopolitan/

awaythrow999 3y ago

Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf Edit: just tested, and it does only identify the zip layer

rvnx 3y ago
You can try it here: https://google.github.io/magika/ https://google.github.io/magika/ It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope. I guess it's more for detecting common file formats then with high recall. However, where is the actual source of the model ? Let's say I want to add a new file format myself. Apparently only the source of the interpreter is here, not the source of the model nor the training set, which is the most important thing.
- alexandreyc 3y ago
  Yes, I totally agree; it's not what I would qualify as open source. Do you plan to release the training code along the research paper? What about the dataset? In any case, it's very neat to have ML-based technique and lightweight model for such tasks!
- tempay 3y ago
  Is there anything about the performance on unknown files? I've tried a few that aren't "basic" but are widely used enough to be well supported in libmagic and it thinks they're zip files. I know enough about the underlying formats to know they're not using zip as a container under-the-hood.
- kevincox 3y ago
  Apparenty the Super Mario Bros. 3 ROM is 100% a SWF file. Cool that you can use it online though. Might end up using it like that. Although it seems like it may focus on common formats.

diimdeep 3y ago

> Magika: AI powered fast and efficient file type identification of 116 file types with proprietary puny model with no training code and no dataset. > We are releasing a paper later this year detailing how the Magika model was trained and its performance on large datasets. And ? How do you advance industry by this googleblog post and source code that is useless without closed source model ? All I see here is loud marketing name, loud promises, but actually barely anything useful. Hooly rooftop characters sideproject?

secondary_op 3y ago

Why is this piece of code being sold as open source, when in reality it just calls into proprietary ML blob that is tiny and useless, and actual source code of model is closed while properly useful large model is non existing ?

KOLANICH 3y ago
Not into proprietary, the blob is within an Apache-licensed repo. Though there was no code to train it, but the repo contains some info allowing to recreate the code training it. Basically a JSON-based configs containing graph architecture. Even if you didn't have them, the repo contains an ONNX model, from which one can devise the architecture.

flohofwoe 3y ago

I wonder how it performs with detecting C vs C++ vs ObjC vs ObjC++ and for bonus points: the common C/C++ subset (which is an incompatible C fork), also extra bonus points for detecting language version compatibility (e.g. C89 vs C99 vs C11...). Separating C from C++ and ObjC is where the file type detection on Github traditionally had problems with (but has been getting dramatically better over time), from an "AI-powered" solution which has been trained on the entire internet I would expect to do better right from the start. The list here doesn't even mention any of those languages except C though: https://github.com/google/magika/blob/main/docs/supported-content-types-list.md https://github.com/google/magika/blob/main/docs/supported-co...

andrewstuart 3y ago

Very useful. I wrote an editor that needed file type detection but the results of traditional approaches were flaky.

20after4 3y ago

I just want to say thank you for the release. There are quite a lot of complaints in the comments but I think this is a useful and worthwhile contribution and I appreciate the authors for going through the effort to get it approved for open source release. It would be great if the model training data was included (or at lease documentation about how to reproduce it.) but that doesn’t preclude this being useful. Thanks!

Someone 3y ago

If their “Exif Tool” is https://exiftool.org/ https://exiftool.org/ (what else could it be?), I don’t understand why they included it in their tests. Also, how does ExifTool recognize Python and html files?

Andugal 3y ago

I have a question: Is something like Magika enough to check if a file is malicious or not? Example: users can upload PNG file (and only PNG is accepted). If Malika detects that the file is a PNG, does this mean the file is clean?

cjg 3y ago
> does this mean the file is clean? No.
TacticalCoder 3y ago
If that PNG of yours is not just an example note that you can detect easily if the PNG as any extra data (which may or may not indicate an attempt as mischief) and reject the (rare) PNGs with extra data. I ran a script checking the thousands of PNGs on my system and found three with extra data, all three probably due to the "PNG acropalypse" bug (but mischief cannot be ruled out). P.S: btw I'm not implying using extra data that shouldn't be there in a PNG is the only way to have a malicious PNG.
nicklecompte 3y ago
This comment from kevincox[1] says the answer is a hard "no": > Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data". There are other comments in this thread that make me think Google contaminated their test data with training data and the 99% results should not be taken at face value. OTOH I am not particularly surprised that Magika would be better than the other tools at distinguishing semi-unstructured plain text e.g. Java source vs. C++ source or YAMLs versus INIs. But that's a very different use case than many security applications. The comments here suggest Magika is especially susceptible to binary obfuscation. [1] https://news.ycombinator.com/item?id=39395677 https://news.ycombinator.com/item?id=39395677
kevincox 3y ago
The only way to do this reliably is to render the PNG to pixels then render it back to an PNG with a trusted encoder. Of course now you are taking the risk of vulnerabilities in the "render to pixels" step. But the result will be clean. AKA parse, don't validate.

TacticalCoder 3y ago

To me the obvious use case is to first use the file command but then, when file returns "DATA" (meaning it couldn't guess the file type), call magika. I guess I'll be writing a wrapper (only for when using my shell in interactive mode) around file doing just that when I come back from vacation. I hate it when file cannot do its thing. Put it this way: I use file a lot and I know at times it cannot detect a filetype. But is file often wrong when it does have a match? I don't think so... So in most of the cases I'd have file correctly give me the filetype, very quickly but then in those rare cases where file cannot find anything, I'd then use the slower but apparently more capable magika.

SnowflakeOnIce 3y ago
I have seen 'file' misclassify many things when running it at large scale (millions of files) from a hodgepodge of sources. Unrelated types getting called 'GPG Private Keys', for example. For textual data types, 'file' gets confused often, or doesn't give a precise type. GitHub's 'linguist' [1] tool does much better here, but is structured in such a way that it is difficult to call it on an arbitrary file or bytestring that doesn't reside in a git repo. I'd love to have a classification tool that can more granularly classify textual files! It may not be Magika _today_ since it only supports 116-something types. For this use case, an ML-based approach will be more successful than an approach based solely on handwritten heuristic rules. I'm excited to see where this goes.

lakomen 3y ago

Why? Just check the damn headers. Why do you need a power hungry and complicated AI model to do it? Why?

TomNomNom 3y ago

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff and woff2 files it identified as "TrueType Font Data", others are "Unknown binary data (unknown)" with low confidence guesses ranging from FLAC audio to ISO 9660. Again, the `file` utility correctly identifies these files as "Web Open Font Format". I like the idea, but the current implementation can't be relied on IMO; especially not for automation. A minor pet peeve also: it doesn't seem to detect when its output is a pipe and strip the shell colour escapes resulting in `^[[1;37` and `^[[0;39m` wrapping every line if you pipe the output into a vim buffer or similar.

michaelmior 3y ago
> the current implementation can't be relied on IMO What's your reasoning for not relying on this? (It seems to me that this would be application-dependent at the very least.)
- jdiff 3y ago
  I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into question the claims made about its performance. What reasoning is there to trust it?
  - michaelmior 3y ago
    > It whiffed multiple common softballs I must have missed this in the article. Where was this?
    - deleted 3y ago
      [deleted]
    - 3y ago

nayuki 3y ago

The name sounds like the Pokémon Magikarp or the anime series Madoka Magica.

pier25 3y ago

I use FFMPEG to detect if uploaded files are valid audio files. Would this be much faster?

omni 3y ago

Can someone please help me understand why this is useful? The article mentions malware scanning applications, but if I'm sending you a malicious PDF, won't I want to clearly mark it with a .pdf extension so that you open it in your PDF app? Their examples are all very obvious based on file extensions.

chromaton 3y ago

It can't correctly identify a DXF file in my testing. It categorizes it as plain text.

Eiim 3y ago

I ran a quick test on 100 semi-random files I had laying around. Of those, 81 were detected correctly, 6 were detected as the wrong file type, and 12 were detected with an unspecific file type (unknown binary/generic text) when a more specific type existed. In 4 of the unspecific cases, a low-confidence guess was provided, which was wrong in each case. However, almost all of the files which were detected wrong/unspecific are of types not supported by Magika, with one exception of a JSON file containing a lot of JS code as text, which was detected as JS code. For comparison, file 5.45 (the version I happened to have installed) got 83 correct, 6 wrong, and 10 not specific. It detected the weird JSON correctly, but also had its own strange issues, such as detecting a CSV as just "data". The "wrong" here was somewhat skewed by the 4 GLSL shader code files that were in the dataset for some reason, all of which it detected as C code (Magika called them unknown). The other two "wrong" detections were also code formats that it seems it doesn't support. It was also able to output a lot more information about the media files. Not sure what to make of these tests but perhaps they're useful to somebody.

pizzalife 3y ago
> The "wrong" here was somewhat skewed by the 4 GLSL shader code files that were in the dataset for some reason, all of which it detected as C code To be fair though, a snippet of GLSL shader code can be perfectly valid C.
- Eiim 3y ago
  Indeed, which is why I felt the need to call it out here. I'm not certain if the files on question actually happened to be valid C but whether that's a meaningful mistake regardless is left to the reader to decide.

Delumine 3y ago

Voidtools - Everything.. looking at you to implement this

init0 3y ago

Why not detect it by checking the magic number of the buffer?

dagmx 3y ago
Not every file has one for starters and many can be incorrect. Especially in the context of use as a virus scanner, you don’t trust what the file says it is

YoshiRulz 3y ago

So instead of spending some of their human resources to improve libmagic, they used some of their computing power to create an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes), and which is much less effective in an adversarial context, and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. Thanks guys.

12_throw_away 3y ago
Come on, can't you help but be impressed by this amazing AI tech? That gives us sci-fi tools like ... a less-accurate, incomplete, stochastic, un-debuggable, slower, electricity-guzzling version of `file`.
famouswaffles 3y ago
>So instead of spending some of their human resources to improve libmagic A large megacorp can work on multiple things at once. >an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes) You say that like it's a contradiction but it's not. >and which is much less effective in an adversarial context, Is it? This seems like an assumption. >and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. Being introspectable or not has no bearing on the accuracy of a system.
- YoshiRulz 3y ago
  > > an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes) > You say that like it's a contradiction but it's not. > > and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. > Being introspectable or not has no bearing on the accuracy of a system. "Open source" and "neural net" is the contradiction, as I went on to write. Even if magika were a more accurate version of file, the implication that it could "help [libmagic] improve" isn't really true, because how do you distill the knowledge from it into a patch for libmagic? My point re: their "error-prone" claim is that their comparison was disingenuous due to the functionality difference between the tools. (Also with their implication that AIs work perfectly, though this one sounds pretty good by the numbers. I of course accept that there's likely to be some bugs in code written by humans.) > > and which is much less effective in an adversarial context, > Is it? This seems like an assumption. It is, one based on what I've heard about AI classifiers over the years. Other commenters here are interested in this point, but while I don't see anyone experimenting on magika (it's new after all), the fact it's not mentioned in the article leads me to believe they didn't try to attack themselves. (Or did, but with bad results, and so decided not to include that. Funnily enough they did mention adversarial attacks on manually-written classifiers...)

breather 3y ago

Can we please god stop using AI like it's a meaningful word? This is really interesting technology; it's hamstrung by association with a predatory marketing term.

goshx 3y ago

I used an HTML file and added JPEG magic bytes to its header: magika file.jpg file.jpg: JPEG image data (image)

runxel 3y ago

Took a .dxf file and fed it to Magika. It says with confidence of 97% that that must be a PowerShell file. A classic .dwg could be "mscompress" (whatever that is), 81%, or a GIF. Both couldn't be further from the truth. Common files are categorized successfully – but well, yeah that's not really an achievement. Pretty much nothing more than a toy right now.

queuebert 3y ago
The real problem with deep learning approaches is hallucination and edge case failures. When someone finally fixes this, I hope it makes the HN front page.

lqcfcjx 3y ago

After reading thru all the comments, honestly I still don't get the point of this system. What is potential practical value or applications of this model?

jwithington 3y ago

I guess I'm kind of a dummy on this, but why is it impressive to identify that a .js file is Javascript, a .md file is Markdown, etc?

playingalong 3y ago
Because it's done by inspecting the content, not the name of the file.

woliveirajr 3y ago

Reminds me when someone asked (at StackOverflow) on how to recognize binaries for different architetures, like x86 or ARM-something or Apple M1 and so on. I gave the idea to use the technique of NCD (Normalized compression distance), based on Kolmogorov complexity. Celibrasi, R. was one great researcher in this area, and I think he worked at Google at some point. Using AI seems to follow the same path: "learn" what represents some specific file and then compare the unknown file to those references (AI:all the parameters, NCD:compression against a known type).

aidenn0 3y ago

But will it let you print on Tuesday[1]? 1: https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28/+index https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...

queuebert 3y ago
For a subscription fee.

jjsimpso 3y ago

I wrote an implementation of libmagic in Racket a few years ago(https://github.com/jjsimpso/magic https://github.com/jjsimpso/magic). File type identification is a pretty interesting topic. As others have noted, libmagic detects many more file types than Magika, but I can see Magika being useful for text files in particular, because anything written by humans doesn't have a rigid format.

notso411 3y ago

[dead]

runald 3y ago

This is useful for detecting file types of unknown blobs with custom file extension, when the file command just returns data. Though it doesn't correctly identify lua code for some reason, it guesses with low probability that it's either ruby or javascript, or anything but lua.

delijati 3y ago

Nice aka perfect timing.I just restored "some" files (40GB) with [1] But the filetype detection of photorec set some wrong file types. Edit: It would be super helpful if the "suffix" could be added as output so i can move the files to the right directory [2] ;) [1] https://www.cgsecurity.org/wiki/PhotoRec https://www.cgsecurity.org/wiki/PhotoRec [2] https://github.com/google/magika/issues/63 https://github.com/google/magika/issues/63

alexott 3y ago

Mime type detection is very interesting thing. I wrote media type detection for McAfee Web Gateway 7.x and because it was a high performance proxy, the detection speed was a major focus, but also the precision, especially for "container types, like, MS Office, OLE-based files, etc. The base of it was a simple Lisp-like language that allowed to write signatures very fast, and everything was combined with very aggressive caching of the data, so we avoided to read data again and again, and used internal caches a lot. In tests, the detection was ~10x faster than file, and with more flexible language we got more file types recognized precisely. Although there were challenges with some formats, like, OLE-based files had FAT directory structure at the end of the file, and you were need to walk the tree to find the top-level structure to distinguish Excel file from Excel file embedded into Word. Streams detection was also quite funny task...

alexott 3y ago
Ah, I remember, the self-extracted .msi file was one of the quite challenging files - it's executable, it's a .cab file, and OLE2-container

glassonion999 3y ago

I created a demo site for Magika. https://9revolution9.com/tools/security/file_scanner/ https://9revolution9.com/tools/security/file_scanner/