3 ms·

Consider DER format. Partial parsing is possible; you can easily ignore any part of the file that you do not care about, since the framing is consistent. Additi

Consider DER format. Partial parsing is possible; you can easily ignore any part of the file that you do not care about, since the framing is consistent. Additionally, it works like the "chunked" formats mentioned in the article, and one of the bits of the header indicates whether it includes other chunks or includes data. (Furthermore, I made up a text-based format called TER which is intended to be converted to DER. TER is not intended to be used directly; it is only intended to be converted to DER for then use in other programs. I had also made up some additional data types, and one of these (called ASN1_IDENTIFIED_DATA) can be used for identifying the format of a file (which might conform to multiple formats, and it allows this too).) I dislike JSON and some other modern formats (even binary formats); they often are just not as good in my opinion. One problem is they tend to insist on using Unicode, and/or on other things (e.g. 32-bit integers where you might need 64-bits). When using a text-based format where binary would do better, it can also be inefficient especially if binary data is included within the text as well, especially if the format does not indicate that it is meant to represent binary data. However, even if you use an existing format, you should avoid using the existing format badly; using existing formats badly seems to be common. There is also the issue of if the existing format is actually good or not; many formats are not good, for various reasons (some of which I mentioned above, but there are others, depending on the application). About target hardware, not all software is intended for a specific target hardware, although some is. For compression, another consideration is: there are general compression schemes as well as being able to make up a compression scheme that is specific for the kind of data that is being compressed. They also mention file names. However, this can also depend on the target system; e.g. for DOS files you will need to be limited to three characters after the dot. Also, some programs would not need to care about file names in some or all cases (many programs I write don't care about file names).

aidenn0 1y ago
Maybe it's just because I've never needed the complexity, but ASN.1 seems a bit much for any of the formats I've created.
- zzo38computer 1y ago
  For me too, although you only need to use (and implement) the parts which are relevant for your application and not all of them, so it is not really the problem. (I also never needed to write ASN.1 schemas, and a full implementation of ASN.1 is not necessary for my purpose.) (This is also a reason I use DER instead of BER, even if canonical form is not required; DER is simpler to handle than all of the possibilities of BER.)
- jeroenhd 1y ago
  The ASN.1 format itself is pretty well-suited for generic file types. Unfortunately, there are very few good, open source/free ASN.1 (de)serializers out there. In theory you could use ASN.1 DER files the same way you would JSON for human-readable formats. In practice, you're better off picking a different format. Modern evolutions of ASN.1 like ProtoBuf or Cap'n Proto designed for transmitting data across the network might fit this purpose pretty well, too. On the other hand, using ASN.1 may be a good way to make people trying to reverse engineer your format give up in despair, especially if you start using the quirks ASN.1 DER comes with and change the identifiers.
  - zzo38computer 1y ago
    > Unfortunately, there are very few good, open source/free ASN.1 (de)serializers out there. I wrote a library to read/write DER, which I have found suitable for my uses. (Although, I might change or add some things later, and possibly also some things might be removed too if I think they are unnecessary or cause problems.) https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.c https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.c https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.h https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.h (You can complain about it if there is something that you don't like.) > In theory you could use ASN.1 DER files the same way you would JSON for human-readable formats. In practice, you're better off picking a different format. I do use ASN.1 DER for some things, because, in my opinion it is (generally) better than JSON, XML, etc. > Modern evolutions of ASN.1 like ProtoBuf or Cap'n Proto designed for transmitting data across the network might fit this purpose pretty well, too. I have found them to be unsuitable, with many problems, and that ASN.1 does them better in my experience. > On the other hand, using ASN.1 may be a good way to make people trying to reverse engineer your format give up in despair, especially if you start using the quirks ASN.1 DER comes with and change the identifiers. I am not so sure of this.

aidenn0 1y ago

Maybe it's just because I've never needed the complexity, but ASN.1 seems a bit much for any of the formats I've created.

zzo38computer 1y ago
For me too, although you only need to use (and implement) the parts which are relevant for your application and not all of them, so it is not really the problem. (I also never needed to write ASN.1 schemas, and a full implementation of ASN.1 is not necessary for my purpose.) (This is also a reason I use DER instead of BER, even if canonical form is not required; DER is simpler to handle than all of the possibilities of BER.)
jeroenhd 1y ago
The ASN.1 format itself is pretty well-suited for generic file types. Unfortunately, there are very few good, open source/free ASN.1 (de)serializers out there. In theory you could use ASN.1 DER files the same way you would JSON for human-readable formats. In practice, you're better off picking a different format. Modern evolutions of ASN.1 like ProtoBuf or Cap'n Proto designed for transmitting data across the network might fit this purpose pretty well, too. On the other hand, using ASN.1 may be a good way to make people trying to reverse engineer your format give up in despair, especially if you start using the quirks ASN.1 DER comes with and change the identifiers.
- zzo38computer 1y ago
  > Unfortunately, there are very few good, open source/free ASN.1 (de)serializers out there. I wrote a library to read/write DER, which I have found suitable for my uses. (Although, I might change or add some things later, and possibly also some things might be removed too if I think they are unnecessary or cause problems.) https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.c https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.c https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.h https://github.com/zzo38/scorpion/blob/trunk/asn1/asn1.h (You can complain about it if there is something that you don't like.) > In theory you could use ASN.1 DER files the same way you would JSON for human-readable formats. In practice, you're better off picking a different format. I do use ASN.1 DER for some things, because, in my opinion it is (generally) better than JSON, XML, etc. > Modern evolutions of ASN.1 like ProtoBuf or Cap'n Proto designed for transmitting data across the network might fit this purpose pretty well, too. I have found them to be unsuitable, with many problems, and that ASN.1 does them better in my experience. > On the other hand, using ASN.1 may be a good way to make people trying to reverse engineer your format give up in despair, especially if you start using the quirks ASN.1 DER comes with and change the identifiers. I am not so sure of this.