4 ms·

From what I can gather they are trying to compose a database of information on what application generates/can read a specific file, primarily by file extension.

by mike_d 6y ago

From what I can gather they are trying to compose a database of information on what application generates/can read a specific file, primarily by file extension. This already exists in libmagic (https://github.com/file/file https://github.com/file/file) and can be used on any BSD/Linux system by typing `file [filename]`

pabs3 6y ago
The goal is a bit more expansive than libmagic, it is a wiki of file format resources, including extensions, software, format specifications, related formats and more.
- brownbat 6y ago
  Yeah, it includes mechanical specifications of physical media so, say, alien archaeologists can get data off a VHS. People are reading the title thinking it's about filesystems. It's about cataloging all human information storage methods, from phonographs to zip drives to word perfect files.
TAForObvReasons 6y ago
libmagic focuses on identifying file type using magic numbers and other properties. The wiki has links to resources for programmatically extracting image size or color palette, for example.
- pabs3 6y ago
  libmagic can also extract metadata for images.
jaclaz 6y ago
And - in case of need - on Windows there is Trid: https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html which has also a couple online options and a database of signatures/magic numbers: https://mark0.net/soft-trid-deflist.html https://mark0.net/soft-trid-deflist.html
- chaz6 6y ago
  There is a Linux version too! But sadly, there is not a library which would be incredibly useful for analysing, for example, blobs in a database, or categorising random files uploaded by a user of a website.
alexott 6y ago
libmagic is quite primitive and can't reliably handle the complex data types, like, MS Office documents with embedded objects (especially the versions before 2007), files with common container types (jar/war files are detected just as zip), etc. - the signature language is quite primitive from my point of view. Plus it's quite slow... I've spent a long time (> 10 years) developing code for detection of file types (without relying onto the file extension), and extraction of the data... And everything was need to be done very fast & reliably. The last project was the content detection for the filtering web proxy, where every millisecond counts. And for signatures, there was a lisp-like language that allowed to describe very complex detection rules, although it's sometime was still necessary to go to the C++, for things like, OLE2 parsing, XML parsing, or listing files in Zip files, etc. From the open source alternatives, it makes sense to look to the Apache Tika that has both content detection & extraction capabilities, although it's written in Java...

by mike_d 6y ago

pabs3 6y ago

The goal is a bit more expansive than libmagic, it is a wiki of file format resources, including extensions, software, format specifications, related formats and more.

brownbat 6y ago
Yeah, it includes mechanical specifications of physical media so, say, alien archaeologists can get data off a VHS. People are reading the title thinking it's about filesystems. It's about cataloging all human information storage methods, from phonographs to zip drives to word perfect files.

TAForObvReasons 6y ago

libmagic focuses on identifying file type using magic numbers and other properties. The wiki has links to resources for programmatically extracting image size or color palette, for example.

pabs3 6y ago
libmagic can also extract metadata for images.

jaclaz 6y ago

And - in case of need - on Windows there is Trid: https://mark0.net/soft-trid-e.html https://mark0.net/soft-trid-e.html which has also a couple online options and a database of signatures/magic numbers: https://mark0.net/soft-trid-deflist.html https://mark0.net/soft-trid-deflist.html

chaz6 6y ago
There is a Linux version too! But sadly, there is not a library which would be incredibly useful for analysing, for example, blobs in a database, or categorising random files uploaded by a user of a website.

alexott 6y ago

libmagic is quite primitive and can't reliably handle the complex data types, like, MS Office documents with embedded objects (especially the versions before 2007), files with common container types (jar/war files are detected just as zip), etc. - the signature language is quite primitive from my point of view. Plus it's quite slow... I've spent a long time (> 10 years) developing code for detection of file types (without relying onto the file extension), and extraction of the data... And everything was need to be done very fast & reliably. The last project was the content detection for the filtering web proxy, where every millisecond counts. And for signatures, there was a lisp-like language that allowed to describe very complex detection rules, although it's sometime was still necessary to go to the C++, for things like, OLE2 parsing, XML parsing, or listing files in Zip files, etc. From the open source alternatives, it makes sense to look to the Apache Tika that has both content detection & extraction capabilities, although it's written in Java...