8 ms·

I once foolishly thought, I'll write a tar parser because, "how hard can it be" [1]. I simply tried to follow the tar(5) man page[2], and got a reference test

I once foolishly thought, I'll write a tar parser because, "how hard can it be" [1]. I simply tried to follow the tar(5) man page[2], and got a reference test set from another website posted previously on HN[3]. Along the way I discovered that NetBSD pax apparently cannot handle the PAX format[3] and my parser inadvertently uncovered that git-archive was storing the checksums wrong, but nobody noticed because other tar implementations were more lax about it[4]. As the article describes (as does the man page), tar is actually a really simple format, but there are just so many variants to choose from. Turns out, if you strive for maximum compatibility, it's easiest to stick to what GNU tar does and favor GNU extensions over PAX header fields. If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix. [1] https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/tar https://github.com/AgentD/squashfs-tools-ng/tree/master/lib/... [2] https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5 https://www.freebsd.org/cgi/man.cgi?query=tar&sektion=5 [3] https://mgorny.pl/articles/portability-of-tar-features.html https://mgorny.pl/articles/portability-of-tar-features.html [4] https://www.spinics.net/lists/git/msg363049.html https://www.spinics.net/lists/git/msg363049.html

slaymaker1907 6y ago
There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.
- lstamour 6y ago
  Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identification http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif... In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example. I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.
- brazzy 6y ago
  How many, though? Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character. If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Woe be unto you if you have to process input data where those are not used consistently, of course.
  - roberto 6y ago
    First, there's no specification. Second, this: http://georgemauer.net/2017/10/07/csv-injection.html http://georgemauer.net/2017/10/07/csv-injection.html
    - brazzy 6y ago
      Quote from the site you linked to: "So who’s fault is all of this anyways? Well it’s not the CSV format’s." And what I wrote before is basically all you need for a specification. Yes, it sucks that there is no fixed one that everyone uses.
  - probably_wrong 6y ago
    > If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Well, yes, but I feel that that's the point at which CSV definitely earns its "ill-specified format" hat. Given that there is no way to extract the file specification from the file itself, you can only follow the specification if someone tells you the specification beforehand. You can even use the "wrong" specification and don't notice it because your specification overlaps with the "true" specification in the types of data you're typically seeing (looking at you, optional quotes).
  - tinus_hn 6y ago
    All you need to do is specify all these unspecified things and it’ll be fine. You forgot the localized decimal separator symbol.
    - 6y ago
vanviegen 6y ago
> If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix. The "extend" phase from the Microsoft playbook is always something that competitors cannot duplicate. Either for legal reasons (patents) or technical reasons (having the "spec" depend on all of the Windows ecosystem). I don't think that's what GNU did.
- stonogo 6y ago
  No, this was pretty explicitly the reason GCC was not permitted to have independent front- and back-ends like llvm does today. Stallman was concerned someone would write proprietary extensions that way. So, if you wanted GCC, you got the whole GCC, GNU extensions to C and all.
  - josefx 6y ago
    And you only got GCC. There was someone who tried to implement refactoring support for emacs based on gcc, since there already was a llvm based implementation and that obviously wasn't GNU enough. Stallman left him hanging for a year before telling him to stop because it would expose too many gcc internals. He ended up killing the discussion when too many people disagreed with his assertion that text base search and replace was good enough.
- st_goliath 6y ago
  GNU extensions are all over the place. The C library has all kind of GNU functions that are more convenient/practical than what Unix had, the command line programs (e.g. coreutils) have feature flags and extensions that the originals didn't have but that made them more convenient to use, and I already mentioned GNU extensions in the tar format. POSIX aside, Unix-like systems that are still around nowadays and not directly use GNU user space, have copied many of the GNU extensions out of necessity. People have become accustomed to those extensions and written programs & scripts depending on them, so you now need to implement many of the GNU extensions simply to stay compatible. Even if not consciously done and without malicious intents, this is IMO still fairly similar to how Microsoft played it: add extensions that are convenient, so people use them, eventually rely on them and become a de facto standard.

slaymaker1907 6y ago

There are similar problems with csv. A deceptively simple format with many variants to handle various edge cases.

lstamour 6y ago
Yes, except it's easy to debug a csv problem because it's generally text in some encoding you can hopefully identify without looking at specific byte offsets in a hex editor: http://fileformats.archiveteam.org/wiki/Tape_Archive#Identification http://fileformats.archiveteam.org/wiki/Tape_Archive#Identif... In fact, one of the biggest questions I've still yet to answer is whether Docker or OCI have ever clarified just which TAR format must be followed? As far as I can tell, it's a mix of "whatever library X supports"? https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for example. I'm not sure I want a more complicated format, but I do wish somebody would pick a new file extension for their next use of tar so we could clearly differentiate one tar format from another the next time someone builds on top of simple tar files for their binary distribution, etc.
brazzy 6y ago
How many, though? Off the top of my head, you need to specify text encoding, line endings, a quoting mechanism in case a field contains the separator, and an escaping mechanism in case a field contains multiple lines or the quote character. If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Woe be unto you if you have to process input data where those are not used consistently, of course.
- roberto 6y ago
  First, there's no specification. Second, this: http://georgemauer.net/2017/10/07/csv-injection.html http://georgemauer.net/2017/10/07/csv-injection.html
  - brazzy 6y ago
    Quote from the site you linked to: "So who’s fault is all of this anyways? Well it’s not the CSV format’s." And what I wrote before is basically all you need for a specification. Yes, it sucks that there is no fixed one that everyone uses.
- probably_wrong 6y ago
  > If you have those specified and everyone follows the specification, there shouldn't be any problems, right? Well, yes, but I feel that that's the point at which CSV definitely earns its "ill-specified format" hat. Given that there is no way to extract the file specification from the file itself, you can only follow the specification if someone tells you the specification beforehand. You can even use the "wrong" specification and don't notice it because your specification overlaps with the "true" specification in the types of data you're typically seeing (looking at you, optional quotes).
- tinus_hn 6y ago
  All you need to do is specify all these unspecified things and it’ll be fine. You forgot the localized decimal separator symbol.
  - 6y ago

vanviegen 6y ago

> If you think about it, in many ways the GNU project IMO ended up doing "embrace, extend, extinguish" with Unix. The "extend" phase from the Microsoft playbook is always something that competitors cannot duplicate. Either for legal reasons (patents) or technical reasons (having the "spec" depend on all of the Windows ecosystem). I don't think that's what GNU did.

stonogo 6y ago
No, this was pretty explicitly the reason GCC was not permitted to have independent front- and back-ends like llvm does today. Stallman was concerned someone would write proprietary extensions that way. So, if you wanted GCC, you got the whole GCC, GNU extensions to C and all.
- josefx 6y ago
  And you only got GCC. There was someone who tried to implement refactoring support for emacs based on gcc, since there already was a llvm based implementation and that obviously wasn't GNU enough. Stallman left him hanging for a year before telling him to stop because it would expose too many gcc internals. He ended up killing the discussion when too many people disagreed with his assertion that text base search and replace was good enough.
st_goliath 6y ago
GNU extensions are all over the place. The C library has all kind of GNU functions that are more convenient/practical than what Unix had, the command line programs (e.g. coreutils) have feature flags and extensions that the originals didn't have but that made them more convenient to use, and I already mentioned GNU extensions in the tar format. POSIX aside, Unix-like systems that are still around nowadays and not directly use GNU user space, have copied many of the GNU extensions out of necessity. People have become accustomed to those extensions and written programs & scripts depending on them, so you now need to implement many of the GNU extensions simply to stay compatible. Even if not consciously done and without malicious intents, this is IMO still fairly similar to how Microsoft played it: add extensions that are convenient, so people use them, eventually rely on them and become a de facto standard.