5 ms·

In addition to this resource and UK's equivalent (PRONOM/DROID, also mentioned in the linked post), I've found ArchiveTeam's wiki to be very useful for obscure

by Lammy 5y ago

In addition to this resource and UK's equivalent (PRONOM/DROID, also mentioned in the linked post), I've found ArchiveTeam's wiki to be very useful for obscure file format details: http://fileformats.archiveteam.org/ http://fileformats.archiveteam.org/ The `shared-mime-info` database from freedesktop-dot-org is probably more worthy of contribution than these government-backed databases, at least in terms of number-of-end-users. New type definitions in their database will improve the entire Linux/BSD ecosystem (both desktop and server!) because it's consumed not only by fd.o's own `update-mime-database` utility but by many language-specific type-identification libraries too https://gitlab.freedesktop.org/xdg/shared-mime-info/-/blob/master/data/freedesktop.org.xml.in https://gitlab.freedesktop.org/xdg/shared-mime-info/-/blob/m... …including (shameless plug) the new Ractor-based Ruby type library I've been working on in the wake of the `mimemagic` drama earlier this year: https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSATION/CHECKING%20YOU%20OUT https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSA...

pabs3 5y ago
Also the magic number database for guessing the format of a file: https://www.darwinsys.com/file/ https://www.darwinsys.com/file/
- alexott 5y ago
  magic database has its own problems based on the language itself
  - pabs3 5y ago
    What kind of problems?
    - alexott 5y ago
      Primarily because it’s very hard to express complex conditions
alexott 5y ago
There is also Apache Tika (https://tika.apache.org/ https://tika.apache.org/) - file format detection & content extraction library
- Lammy 5y ago
  Tika also stores its type definitions in the freedesktop format internally: https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml;h=e65b2691fbe66aa37c4eafc39cbb646f3e04195d;hb=HEAD https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika... However their data is mostly focused on text/document types for obvious reasons :)
cxr 5y ago
> I've found ArchiveTeam's wiki to be very useful for obscure file format details: http://fileformats.archiveteam.org/ http://fileformats.archiveteam.org/ I took a serious look at trying to contribute to the Archive Team's "just fix the file format problem" effort a while ago. I ran into the problem that they require contributions be public domain/CC0. This prohibits material derived from the vast corpus of FOSS (even the most permissively licensed stuff—e.g. MIT, BSD, etc.) Considering how rare public domain/CC0 is, essentially everything would need to be done from scratch.
- kortex 5y ago
  You should reach out to them and let them know. I don't know what the alternative would be - are there better CC licenses out there? Dual licensing CC+MIT/BSD? Pragmatically (but not legally speaking, INAL) you could probably get by with summarizing and linking (especially to web archive).
  - TAForObvReasons 5y ago
    There are two problems that reasonably explain the decision of only accepting public domain info: licensing and provenance. "Licensing" is hard. The "Open Specifications Promise" [1], which covers a bunch of Microsoft-designed file formats, is merely a covenant not to sue. "Provenance" is tricky. For example, much of the knowledge of the Apple iWork formats were derived by reverse-engineering the source programs and extracting protobuf definitions. Many open source projects have freely copied from each other, making detailed analysis tricky [2]. [1] https://en.wikipedia.org/wiki/Microsoft_Open_Specification_Promise https://en.wikipedia.org/wiki/Microsoft_Open_Specification_P... [2] https://github.com/jazzband/tablib/issues/114 https://github.com/jazzband/tablib/issues/114
    - cxr 5y ago
      > There are two problems that reasonably explain the decision Boy do I hate these kinds of after-the-fact, speculative "explanations"—anachronistic rationalizations that come out of a mental exercise to justify what is by retroactively (and non-authoritatively) trying to invent a plausible why. And if you know the personality and values of the progenitor of the whole thing, then you know that this better-safe-than-sorry deference to hair-splittingly nuanced IP hurdles (that in practice don't really exist) is not the MO. Besides that, requiring public domain/CC0 doesn't solve either of those problems.

by Lammy 5y ago

pabs3 5y ago

Also the magic number database for guessing the format of a file: https://www.darwinsys.com/file/ https://www.darwinsys.com/file/

alexott 5y ago
magic database has its own problems based on the language itself
- pabs3 5y ago
  What kind of problems?
  - alexott 5y ago
    Primarily because it’s very hard to express complex conditions

alexott 5y ago

There is also Apache Tika (https://tika.apache.org/ https://tika.apache.org/) - file format detection & content extraction library

Lammy 5y ago
Tika also stores its type definitions in the freedesktop format internally: https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml;h=e65b2691fbe66aa37c4eafc39cbb646f3e04195d;hb=HEAD https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika... However their data is mostly focused on text/document types for obvious reasons :)

cxr 5y ago

> I've found ArchiveTeam's wiki to be very useful for obscure file format details: http://fileformats.archiveteam.org/ http://fileformats.archiveteam.org/ I took a serious look at trying to contribute to the Archive Team's "just fix the file format problem" effort a while ago. I ran into the problem that they require contributions be public domain/CC0. This prohibits material derived from the vast corpus of FOSS (even the most permissively licensed stuff—e.g. MIT, BSD, etc.) Considering how rare public domain/CC0 is, essentially everything would need to be done from scratch.

kortex 5y ago
You should reach out to them and let them know. I don't know what the alternative would be - are there better CC licenses out there? Dual licensing CC+MIT/BSD? Pragmatically (but not legally speaking, INAL) you could probably get by with summarizing and linking (especially to web archive).
- TAForObvReasons 5y ago
  There are two problems that reasonably explain the decision of only accepting public domain info: licensing and provenance. "Licensing" is hard. The "Open Specifications Promise" [1], which covers a bunch of Microsoft-designed file formats, is merely a covenant not to sue. "Provenance" is tricky. For example, much of the knowledge of the Apple iWork formats were derived by reverse-engineering the source programs and extracting protobuf definitions. Many open source projects have freely copied from each other, making detailed analysis tricky [2]. [1] https://en.wikipedia.org/wiki/Microsoft_Open_Specification_Promise https://en.wikipedia.org/wiki/Microsoft_Open_Specification_P... [2] https://github.com/jazzband/tablib/issues/114 https://github.com/jazzband/tablib/issues/114
  - cxr 5y ago
    > There are two problems that reasonably explain the decision Boy do I hate these kinds of after-the-fact, speculative "explanations"—anachronistic rationalizations that come out of a mental exercise to justify what is by retroactively (and non-authoritatively) trying to invent a plausible why. And if you know the personality and values of the progenitor of the whole thing, then you know that this better-safe-than-sorry deference to hair-splittingly nuanced IP hurdles (that in practice don't really exist) is not the MO. Besides that, requiring public domain/CC0 doesn't solve either of those problems.