6 ms·

> That said very few operating systems actually store MIME types, [...] MIME types are almost always derived from file extensions or magic numbers. OpenDocumen

by patrec 6y ago

> That said very few operating systems actually store MIME types, [...] MIME types are almost always derived from file extensions or magic numbers. OpenDocument is generally a horrible format, but there is one clever aspect about it which I wish more file formats would adopt, but which is I think not widely known. Since it ought to be, here is a quick rundown: All OpenDocument subformats like ODT are zipfiles where the first entry is an uncompressed file with the name "mimetype" that has the mimetype (e.g. application/vnd.oasis.opendocument.text) as content. The way zip files are laid out, this means the string "mimeetype$ACTUALMIMETYPE" ends up next to the pkzip magic numbers right at the beginning at the file and can be robustly detected by sniffing just a few bytes at the start of the file. http://docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-os-part3.html#MIME_type_stream http://docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-... Sadly, few other zip based file formats seem to do that, in particular docx and family do not. If you ever find yourself in the situation that you design a zip based file format, please adopt it.

ancarda 6y ago
Seems the dollar is missing - I only see "mimetypeapplication/vnd.oasis.opendocument.text". Nevertheless, I really like this approach. I will remember it in future. $ hexdump -C CV.odt | head -n 5 00000000 50 4b 03 04 14 00 00 08 00 00 9c ad 7e 51 5e c6 |PK..........~Q^.| 00000010 32 0c 27 00 00 00 27 00 00 00 08 00 00 00 6d 69 |2.'...'.......mi| 00000020 6d 65 74 79 70 65 61 70 70 6c 69 63 61 74 69 6f |metypeapplicatio| 00000030 6e 2f 76 6e 64 2e 6f 61 73 69 73 2e 6f 70 65 6e |n/vnd.oasis.open| 00000040 64 6f 63 75 6d 65 6e 74 2e 74 65 78 74 50 4b 03 |document.textPK.|
- layer8 6y ago
  The GP intended $ACTUALMIMETYPE to be read as a variable — there is no $ after replacement.
  - ancarda 6y ago
    Oh silly me, of course. I thought perhaps $ denoted the end of a filename and the start of the contents in Zip. I don't know much about that format.
kergonath 6y ago
That is actually brilliant. I had no idea, thanks for the information.
epitactic 6y ago
> zipfiles where the first entry is an uncompressed file with the name "mimetype" that has the mimetype The EPUB format also adopted this convention: https://www.w3.org/publishing/epub3/epub-spec.html#sec-intro-spec-org https://www.w3.org/publishing/epub3/epub-spec.html#sec-intro... "The EPUB Publication's resources are bundled for distribution in a ZIP-based archive with the file extension .epub. As conformant ZIP archives, EPUB Publications can be unzipped by many software programs, simplifying both their production and consumption. The container format not only provides a means of determining that the zipped content represents an EPUB Publication (the mimetype file), "
PeterisP 6y ago
Given your description, it seems that any software writing a docx file could do just that without breaking compatibility.
greggman3 6y ago
Assuming that the data at the start of a zip file is valid is an invalid way to interpret a zip file. Zipfiles start at the end ALWAYS. The format was designed to be able to just jump to the end of the file, read the table of contents, append new data, write a new table of contents. That new table of contents might not even reference the data at the front of the file. It was designed this way because it comes from the era of floppy disks. If you had a zip file spanning multiple discs and you wanted to update `firstfile.txt` at the front of the zip file you didn't want to have to insert all 10 disks and read and re-write all the data. You just wanted to insert the last disk, read the table of contents, append your new `firstfile.txt` and write a new table of contents. The invalid `firstfile.txt` still exists at the front but it's ignored by the table of contents. If you're going do stupid tricks by scanning from the front then you'd be better off choosing a format for which that's actually valid like .tar.gz or something
- alisonkisk 6y ago
  Since editing a document doesn't change its type, this objection doesn't apply to zip-based filetypes.
- JeremyBanks 6y ago
  It's valid if you're not reading a generic zip file but instead a zip derived format like this.
alexvoda 6y ago
I believe Uniform Type Identifiers and Bundles are a much nicer solution. A real shame only Mac OS X supports them. I would love to see them implemented in other OSs.
- patrec 6y ago
  UTIs are nicer than mime types in that they allow a simple way to make up a unique name you control (java style com.myurl.myformat) and in that there is an inheritance mechanism (you can say com.myurl.myformat also is a sqlite database, for example). But unless I misunderstand something the fundamental problem of figuring out what particular logical type some collection of bytes on your disk has is in fact not solved at all, because the UTI is not stored together with the file in the filesystem (unless I'm misunderstanding something). As far as I can tell macOS, apart from a few places like the clipboard, basically just takes a guess based on registered file extensions (or, for rest calls, the mime type) -- is that not so?
btschaegg 6y ago
TIL. Thanks for sharing that!
jez 6y ago
Another little known file type feature is that if you’re using a sqlite database file as an application file format, there are a handful of bytes reserved for applications to register that “this is a SQLite file for use specifically with application X” Application ID docs: https://www.sqlite.org/draft/fileformat2.html#application_id https://www.sqlite.org/draft/fileformat2.html#application_id
- yencabulator 6y ago
  Using SQLite as a transport format is not a good idea: https://research.checkpoint.com/select-code_execution-from-using-sqlite/ https://research.checkpoint.com/select-code_execution-from-u...

by patrec 6y ago

ancarda 6y ago

Seems the dollar is missing - I only see "mimetypeapplication/vnd.oasis.opendocument.text". Nevertheless, I really like this approach. I will remember it in future. $ hexdump -C CV.odt | head -n 5 00000000 50 4b 03 04 14 00 00 08 00 00 9c ad 7e 51 5e c6 |PK..........~Q^.| 00000010 32 0c 27 00 00 00 27 00 00 00 08 00 00 00 6d 69 |2.'...'.......mi| 00000020 6d 65 74 79 70 65 61 70 70 6c 69 63 61 74 69 6f |metypeapplicatio| 00000030 6e 2f 76 6e 64 2e 6f 61 73 69 73 2e 6f 70 65 6e |n/vnd.oasis.open| 00000040 64 6f 63 75 6d 65 6e 74 2e 74 65 78 74 50 4b 03 |document.textPK.|

layer8 6y ago
The GP intended $ACTUALMIMETYPE to be read as a variable — there is no $ after replacement.
- ancarda 6y ago
  Oh silly me, of course. I thought perhaps $ denoted the end of a filename and the start of the contents in Zip. I don't know much about that format.

kergonath 6y ago

That is actually brilliant. I had no idea, thanks for the information.

epitactic 6y ago

> zipfiles where the first entry is an uncompressed file with the name "mimetype" that has the mimetype The EPUB format also adopted this convention: https://www.w3.org/publishing/epub3/epub-spec.html#sec-intro-spec-org https://www.w3.org/publishing/epub3/epub-spec.html#sec-intro... "The EPUB Publication's resources are bundled for distribution in a ZIP-based archive with the file extension .epub. As conformant ZIP archives, EPUB Publications can be unzipped by many software programs, simplifying both their production and consumption. The container format not only provides a means of determining that the zipped content represents an EPUB Publication (the mimetype file), "

PeterisP 6y ago

Given your description, it seems that any software writing a docx file could do just that without breaking compatibility.

greggman3 6y ago

Assuming that the data at the start of a zip file is valid is an invalid way to interpret a zip file. Zipfiles start at the end ALWAYS. The format was designed to be able to just jump to the end of the file, read the table of contents, append new data, write a new table of contents. That new table of contents might not even reference the data at the front of the file. It was designed this way because it comes from the era of floppy disks. If you had a zip file spanning multiple discs and you wanted to update `firstfile.txt` at the front of the zip file you didn't want to have to insert all 10 disks and read and re-write all the data. You just wanted to insert the last disk, read the table of contents, append your new `firstfile.txt` and write a new table of contents. The invalid `firstfile.txt` still exists at the front but it's ignored by the table of contents. If you're going do stupid tricks by scanning from the front then you'd be better off choosing a format for which that's actually valid like .tar.gz or something

alisonkisk 6y ago
Since editing a document doesn't change its type, this objection doesn't apply to zip-based filetypes.
JeremyBanks 6y ago
It's valid if you're not reading a generic zip file but instead a zip derived format like this.

alexvoda 6y ago

I believe Uniform Type Identifiers and Bundles are a much nicer solution. A real shame only Mac OS X supports them. I would love to see them implemented in other OSs.

patrec 6y ago
UTIs are nicer than mime types in that they allow a simple way to make up a unique name you control (java style com.myurl.myformat) and in that there is an inheritance mechanism (you can say com.myurl.myformat also is a sqlite database, for example). But unless I misunderstand something the fundamental problem of figuring out what particular logical type some collection of bytes on your disk has is in fact not solved at all, because the UTI is not stored together with the file in the filesystem (unless I'm misunderstanding something). As far as I can tell macOS, apart from a few places like the clipboard, basically just takes a guess based on registered file extensions (or, for rest calls, the mime type) -- is that not so?

btschaegg 6y ago

TIL. Thanks for sharing that!

jez 6y ago

Another little known file type feature is that if you’re using a sqlite database file as an application file format, there are a handful of bytes reserved for applications to register that “this is a SQLite file for use specifically with application X” Application ID docs: https://www.sqlite.org/draft/fileformat2.html#application_id https://www.sqlite.org/draft/fileformat2.html#application_id

yencabulator 6y ago
Using SQLite as a transport format is not a good idea: https://research.checkpoint.com/select-code_execution-from-using-sqlite/ https://research.checkpoint.com/select-code_execution-from-u...