7 ms·
Somewhat related, the question of "what IS a file's 'type' really" is pretty messy and AFAIK doesn't have any satisfying solution or any satisfying single answe
by xkcd-sucks 3y ago
Somewhat related, the question of "what IS a file's 'type' really" is pretty messy and AFAIK doesn't have any satisfying solution or any satisfying single answer [0]. A satisfying solution might look like a (long) feature vector. Libmagic goes in the right direction at least although the binary-fingerprint-at-the-beginning approach doesn't work well for everything / past some level of detail
[0] where different use cases might be satisfied by e.g.
"a text file"
"a file exclusively comprised of alphanumeric UTF16 characters"
"a CSV file"
"a file with tabular data and key-value metadata that calls itself CSV but is not spec-compliant so internally we'll call it CSVprime"
"A file with tabular data written as ASCII text that represents $this_kind_of_measurement"
... etc
If you have any resources on this topic off the top of your head I'd appreciate it if you shared them
- duped 3y agoGNU file doesn't just check magic numbers against libmagic, it also defines a scripting language and a series of tests/printers written in it. That's what allows it to do complex things, eg identify all the flavors of ELF objects even though they share a magic number or determine if something is JSON or CSV without one.
- prmph 3y agoThe point is that, fundamentally, the concept of a file type is undecidable or not well defined. Think about it: A JSON file can also be considered a text file. It could also be some higher level type of file, depending on whether it conforms to some application-specific JSON schema. Thus the kind of file it is has more to do with what you want to do with it; it's not some intrinsic property of the file.
- duped 3y agoOk? I was replying to a comment asking about how such systems can work. GNU file is example of a program that makes a best effort to classify file type in a useful way.
- pests 3y agoA JSON file is also valid YAML.
- justsomehnguy 3y ago> A JSON file can also be considered A Curly braces Separated file filled with Values
- doubloon 3y agoso much of our mathematical theories are based on the idea that we have objects which are certainly of one type or another. what would happen to type theory if, say, the type of an object has a probability attached to it?
- clort 3y agoI'm pretty sure the file command is not GNU btw, and I can't find anything about a GNU version.. do you have a reference? https://en.wikipedia.org/wiki/File_(command) https://en.wikipedia.org/wiki/File_(command)
- duped 3y agoMy bad, I thought it was a part of coreutils and didn't check. I've only ever dug into the Linux utility and assumed. Too late to edit!
- giancarlostoro 3y agohttps://www.darwinsys.com/file/ https://www.darwinsys.com/file/ This is the authors website. Apparently yeah its not part of GNU utils, I had no idea, I knew it came with most Linux systems so I looked for the Debian package and found the site linked above. https://packages.debian.org/bookworm/file https://packages.debian.org/bookworm/file
- spullara 3y agoNot sure why it would think a module javascript file is java. Maybe just not updated very often. % file crawl.mjs crawl.mjs: Java source, ASCII text
- vpzom 3y agoprobably looks for "import" at the start Edit: yep, the test for Java source is just /^import.*;$/
- giancarlostoro 3y agoI just think of every novice who uses Java when referring to JavaScript, even software does it! Heh
- amelius 3y agoWell, you can define a file's type in the same way as you define in-memory data structures, with a type system.
- prmph 3y agoSee my comment: https://news.ycombinator.com/item?id=38117891 https://news.ycombinator.com/item?id=38117891
- amelius 3y agoA C struct can also be addressed as bytes if one really wants to. The structured way of looking at the data is still more valuable, most of the time. Another example: you can easily grep through a JSON file sometimes. But what if that JSON doesn't contain newlines? In that case it would be great if the system knew how to convert the data to a multiline string that makes sense for grep. This cannot be done if the data is "just bytes". Further, JSON data can be represented with less bytes if you know the structure.
- deleted 3y ago[deleted]
- nomel 3y agoAre you suggesting putting the (I assumed versioned) type system into the file's metadata, so that others can make sense of it? Or, some type system UUID, in the metadata, that can be used? I think anything like this would require many new concepts, like the move away from files being a collection of arbitrary stored bytes.
- anonymfus 3y agoWell in theory if we are only doing an inheritance, we can just combine file extensions hierarchically in the name. For example, for CSV files we can use .csv.txt extension, so if shell has no software for CSV files, it will treat them as a plain text, and if it has a software for viewing CSV files, but not for editing or printing them, it will add these verbs from records for txt files. This is backwards compatible and would require minimal changes to file managers and graphic shells.
- Lammy 3y ago> If you have any resources on this topic off the top of your head I'd appreciate it if you shared them I wrote a Ruby library that attempts to be good at this https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSATION/CHECKING%20YOU%20OUT#feature-matrix https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSA...
- fsckboy 3y agoA file is "of a type" if there exists some agent (program X or human H) that creates/writes such files, and whereby there exists some agent (program Y or could be Y==X or human I or could be I==H) that reads/modifies such files, and whereby the entire contents of such files can be made use of or understood in a useful/meaningful way; and other files not of the type's contents either can't be made use of meaningfully nor updated. Some file types may be sub- or super-sets of other file types (ex: .C source files are .TXT files, but .TXT source files are not necessarily .C files, they could be .CSV files) And files could be streams. might wish to discuss files containing type information, and data/value information both, thus numerous but different valued files can all be of the same type. you can get fancier if you want and talk about valid C source files or C source files which contain bugs, etc. you could consider a file "weakly or adhoc typed" if there is only one or a pair of agents that use that type, and "strongly or standard typed" if there are numerous "different/distinguished" agents that "rely" on that file type. this was off the top of my head