3 ms·

Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts K

Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf Edit: just tested, and it does only identify the zip layer

rvnx 3y ago
You can try it here: https://google.github.io/magika/ https://google.github.io/magika/ It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope. I guess it's more for detecting common file formats then with high recall. However, where is the actual source of the model ? Let's say I want to add a new file format myself. Apparently only the source of the interpreter is here, not the source of the model nor the training set, which is the most important thing.
- alexandreyc 3y ago
  Yes, I totally agree; it's not what I would qualify as open source. Do you plan to release the training code along the research paper? What about the dataset? In any case, it's very neat to have ML-based technique and lightweight model for such tasks!
- tempay 3y ago
  Is there anything about the performance on unknown files? I've tried a few that aren't "basic" but are widely used enough to be well supported in libmagic and it thinks they're zip files. I know enough about the underlying formats to know they're not using zip as a container under-the-hood.
- kevincox 3y ago
  Apparenty the Super Mario Bros. 3 ROM is 100% a SWF file. Cool that you can use it online though. Might end up using it like that. Although it seems like it may focus on common formats.

rvnx 3y ago

You can try it here: https://google.github.io/magika/ https://google.github.io/magika/ It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope. I guess it's more for detecting common file formats then with high recall. However, where is the actual source of the model ? Let's say I want to add a new file format myself. Apparently only the source of the interpreter is here, not the source of the model nor the training set, which is the most important thing.

alexandreyc 3y ago
Yes, I totally agree; it's not what I would qualify as open source. Do you plan to release the training code along the research paper? What about the dataset? In any case, it's very neat to have ML-based technique and lightweight model for such tasks!
tempay 3y ago
Is there anything about the performance on unknown files? I've tried a few that aren't "basic" but are widely used enough to be well supported in libmagic and it thinks they're zip files. I know enough about the underlying formats to know they're not using zip as a container under-the-hood.
kevincox 3y ago
Apparenty the Super Mario Bros. 3 ROM is 100% a SWF file. Cool that you can use it online though. Might end up using it like that. Although it seems like it may focus on common formats.