6 ms·
How I wound up finding a bug in GNU Tar
- oculusthrift 8y agothat’s cool! thanks for the write up. i learned a bit
- gumby 8y agofun trivia: gnu tar predates the GNU project. What? It started out as an implementation of tar written by John Gilmore whose uname is 'gnu'. gnu has written a lot of free software as well, but the fact that his name and the GNU project's name are the same is a complete coincidence.
- jchw 8y agoFascinating but is that actually true? As far as I have been able to search their implementation seems to have been called pdtar.
- w7 8y agoThe FSF references this in a news article [1]. Stating that pdtar is what GNU tar was based on. [1] https://www.fsf.org/news/2009-free-software-awards https://www.fsf.org/news/2009-free-software-awards
- jchw 8y agoYes, that is definitely true. I was simply referring to the claim that pdtar was known as gnu tar.
- gumby 8y agopdtar was another name for gnu tar as can be seen on john's home page or wikipedia page if you can't access it.
- lambda 8y agoI see references to John Gilmore being the original author of pdtar, and John Gilmore using the gnu handle, but I don't see any references to his usage of the name predating the GNU project or pdtar being referred to as gnutar before it became part of the GNU project. I'm not saying that I disbelieve you, I just haven't been able to find any references for your claim. ChangeLog.1 and NEWS only go back to after it was migrated to the GNU project (http://git.savannah.gnu.org/gitweb/?p=tar.git;a=tree http://git.savannah.gnu.org/gitweb/?p=tar.git;a=tree), his personal site, http://www.toad.com/gnu/ http://www.toad.com/gnu/ seems to be down but an archive (https://web.archive.org/web/20180623140434/http://www.toad.com/gnu/ https://web.archive.org/web/20180623140434/http://www.toad.c...) just says 'In 1985 I wrote the "pdtar" program, which eventually became GNU Tar.' He clearly uses the name "gnu", but did that usage predate the GNU project? I haven't found a reference for that.
- gumby 8y agoIt sure did. We have been friends for longer then that.
- lambda 8y agoAre you sure you're not confusing GNU tar with gnuplot? http://www.gnuplot.info/faq/faq.html#x1-70001.2 http://www.gnuplot.info/faq/faq.html#x1-70001.2
- gumby 8y agoYep, see another comment by me down thread.
- ezoe 8y ago> if you run GNU Tar with --sparse and a file shrinks while tar is reading it, tar fails to properly handle the resulting earlier than expected end of file. Is that a bug? Yes. It's different than changing the data of file but still.
- fanf2 8y agoClearly it’s a bug because tar goes into a mad loop reading nothing and outputting empty blocks forever, or until the file changes size again. It needs to check for eof to avoid this mistake, even if the eof is unexpected because the file size was different.
- the8472 8y agoI think most gnu utils are written so that they treat input files as a stream of data so you can also use pipes instead of regular files. This also means they can somewhat handle size changes since they will keep reading until they encounter an EOF without relying on an earlier snapshot of the file size or sparse chunks. Whether the results are useful depends on the file format, if it's just temporary padding as in this case it works.
- newnewpdro 8y agoYes, it's expected for a general system utility like tar to handle such simultaneous file operations gracefully.
- hollerith 8y agoAgreed. OTOH, it is not a bug if the fact that tar is not instantaneous causes the tarball to contain combinations of files unlikely to occur during normal operation of the system. e.g., if a file is renamed while tar is running it is not a bug for the tarball to contain zero or two copies of that file.
- newnewpdro 8y agoIt is a bug for tar to enter an infinite loop writing an ever expanding file of trailing zeroes because the current file was truncated to a size smaller than its stat returned. It's basically a classic TOCTOU race, tar needs to honor the EOF returned from read() as authoritative. This is actually a very common trap for junior *nix programmers; treating stat() and a subsequent mmap() or read() loop as if they were atomic with regards to stat.st_size. The size returned by stat() can be used as an estimate, but otherwise can't be applied to subsequent operations.
- noja 8y agoI think he should have reported this but writing it up as a fairly detailed blog entry.
- Dylan16807 8y agoGive him a couple days! You're responding to the PPS so take the entire PPS into account.
- crazysim 8y agoI think the guy just wants to enjoy the break. Talking and responding to an email/bug thread is bit more involved than just shouting text to your blog.
- smnscu 8y agohttps://twitter.com/thatcks/status/1076166645708668928 https://twitter.com/thatcks/status/1076166645708668928
- DavidNielsen 8y agoI rather enjoyed reading the adventure of hitting a problem in a large system, and then working towards narrowing down the specific bug or set of bugs. Such posts have in the past been super helpful for me personally (and to others I imagine), in going from plain weeping that stuff just randomly breaks, to learning to enjoy examining the possible causes and understanding how to explain the problem concisely and with enough detail to make it useful to a developer. He’ll be able to reuse much of his blog post in an great bug report with easy steps to reproduce the problem, excepted and actual outcomes of following those steps, and a scenario where it will happen in real deployments as well as a providing a workaround.
- Twirrim 8y agoThe blog post author is one who routinely submits bug reports upstream, and even code fixes where he can.
- ac_4763267 8y agoA relevant question about tar and backups: https://unix.stackexchange.com/questions/333975/is-using-tar-while-source-directory-is-being-updated-safe https://unix.stackexchange.com/questions/333975/is-using-tar...
- cesarb 8y agoWhile the bug is in tar, the API for the read(2) system call doesn't help. Its return value can be positive, which indicates the number of bytes read; negative, which indicates an error, unless errno is EINTR, in which case you should retry; or zero, which indicates either end of file, or the number of bytes read when the buffer length is zero (and in this case you can't distinguish end of file from a normal read). It's easy to forget to handle one of these five cases.
- saagarjha 8y agoThe issue with EINTR is pretty standard between most syscall; if you get one you should retry (or, depending on what flags you have set, this can be done for you automatically in some cases). A return value of zero is an EOF, there is no such thing as an “empty read”. Why would you pass in an empty buffer?
- jandrese 8y agoYou can get an empty read if you have setup the FD to be nonblocking.
- cesarb 8y agoThat would be a sixth case I had forgotten above (negative return value with errno EAGAIN and/or EWOULDBLOCK - these two might or might not be the same number).
- thatcks 8y agoApparently the use of an empty read is to probe to see if you get various errors or an immediate 0-size return. This is merely a 'may' requirement in the Single Unix Specification, so I'm not sure how portable it is in practice.
- davemp 8y agoIssues like this are almost entirely solved with rust-esque enum pattern matching. The compiler with fail if you don’t have every match case handled. There is ‘_’ to abuse the system but doing so should be fairly obvious. —— Rust in general is pretty exciting for systems work. I just wish there were more shops using it professionally.
- heinrichhartman 8y agoFTR: This gdb script will print full stack traces for every read syscall made: catch syscall read commmands backtrace continue end Put that into a file `trace-read.gdb` and attach to a running process like so: gdb -x trace-read.gdb -p $(pgrep -n tar) This assumes you are running an executable with built-in debugging symbols (gcc -g). It should be possible to side-load debugging symbols provided in an external package, though I don't have a command at hand. (Anyone?) This works well enough on Linux, to quickly debug situations like the one above. However, it can make the attached process painfully slow, and occasionally bring it down all-together. The right tool for this kind of in situations where slowing down or crashing the process is not acceptable, is DTrace: dtrace -p $(pgrep -n tar) -n 'syscall::read:entry { ustack(); }' - Tutorial: https://wiki.freebsd.org/DTrace/Tutorial https://wiki.freebsd.org/DTrace/Tutorial - ustack: http://dtrace.org/guide/chp-user.html#chp-user-4 http://dtrace.org/guide/chp-user.html#chp-user-4 Ironically, DTrace is one of the main selling points for Solaris/OmniOS (or FreeBSD) over Linux. The situation has gotten better recently, with bpftrace becoming available: - http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux... - https://github.com/iovisor/bpftrace https://github.com/iovisor/bpftrace Until you have a 4.x Kernel with the right configuration options running, I am afraid the above gdb scripts is your best option on Linux.
- dima55 8y agoYou can do this on Linux using perf, if you want less overhead and less impact on the application-under-test: $ perf record -e syscalls:sys_enter_read -g -- application arg1 arg2 ... [ application runs while perf writes out a log, recording every read() syscall, and keeping track of the backtrace each time] $ perf report -g --stdio [ perf reads the log, writing out the backtraces ] This is the basic usage. Lots more is available, obviously. This has been available for a LONG time.
- heinrichhartman 8y agoGood point. I did not know perf could print backtraces.
- deleted 8y ago[deleted]
- Annatar 8y agoFinding a bug in GNU tar is not hard... just ask Jörg Schilling. One of the first things I do when I download source code is decompress it and then de-archive it with GNU tar, then promptly re-archive it wit a real, AT&T UNIX System V 4.0 tar, because it's a known fact among us old UNIX folks that GNU tar is buggy as hell and only GNU tar can correctly read GNU tar (that wasn't the point of a tape archiver, but GNU people didn't get that memo).
- giancarlostoro 8y agoKind of curious as to where you get that version of tar from.
- Aloha 8y agohttp://heirloom.sourceforge.net/tools.html http://heirloom.sourceforge.net/tools.html
- Annatar 8y agoEither from Heirloom tools, or by running any freeware, open source operating system which is built from the illumos source code, like SmartOS. illumos code base is the most direct, canonical descendant of AT&T UNIX System V 4.0. If you run anything based on illumos not only are you running a true UNIX, but a reference implementation of it. Additionally, any true System V UNIX like IRIX and HP-UX will have it.
- marbu 8y agoJust to provide some context, Jörg Schilling is author of star[1], which is listed among key implementations of tar on wikipedia[2]. One can check the problems he noticed in gnu tar implementation in the readme file of his start project: https://sourceforge.net/projects/s-tar/files/README.otherbugs/download https://sourceforge.net/projects/s-tar/files/README.otherbug... But note that the last version of star was released in 2008 and README.otherbugs file is actually from 2001. [1] https://sourceforge.net/projects/s-tar https://sourceforge.net/projects/s-tar [2] https://en.wikipedia.org/wiki/Tar_(computing)#Key_implementations https://en.wikipedia.org/wiki/Tar_(computing)#Key_implementa...