4 ms·
Mtime Comparison Considered Harmful (2018)
- ChrisSD 6y ago> More trivia: if you mount your filesystem readonly, it is technically not POSIX compliant any more because the atimes won't be updated Isn't POSIX fun?
- mprovost 6y agoatime doesn't quite work because it makes a read operation into a write. Mostly that's ok, because you're just overwriting the same timestamp field over and over again. But it gets strange with filesystem snapshots or anything else that might be copy on write. Now your reads end up using disk space as it has to make a copy of each directory to update the atime - if it is in fact doing that. Traditionally, reading a snapshot just didn't update the atime (but then does the snapshot keep the atime as it was at the time of the snapshot, and reading files in the live filesystem means making a copy?). This means, as was pointed out, snapshots aren't POSIX compliant.
- formerly_proven 6y agoatime isn't updated on every access. That "relatime" mount option that's almost universally used? It updates atime when the file is written (so mtime and ctime are updated anyway, no extra cost to bumping atime), or after a day if no writes happen but reads.
- jsrcout 6y ago> atime doesn't quite work because it makes a read operation into a write. I remember discovering this little fact a lot of years ago. It really shook my confidence in the existence of a rational world. Then I learned about noatime and never looked back.
- Spivak 6y agoThis really isn't that weird. Imagine you had auditing requirements for a SQL database that you must know when the last time each row was accessed. But in this universe the SQL database is your only persistent datastore (because machines might only have one partition and your system must still function when there is only one partition) so you have to store access times in the rows. But your SQL server only has two modes, the whole database is read-only or the whole database is read-write. You really wish there was a soft read-only where only the auditing data is allowed to be updated but it doesn't exit. So now you're stuck with read-only replicas of your database being noncomplaint. It's a problem that really has no good solution outside of the filesystem driver supporting this mode.
- JdeBP 6y agoErroneous descriptions of it are, certainly. The Single Unix Specification explicitly requires this effect of read-only mounts. * https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_09 https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
- paulddraper 6y agoatime is the biggest misfeature ever. Turning every read into a write......there are very few cases that would be preferrable for.
- TonyTrapp 6y agoOff-topic: The font of this blog looks distractingly weird on Firefox + Ubuntu. For some reason the e and o letters are slightly bigger than the rest: https://i.imgur.com/W1JaiWQ.png https://i.imgur.com/W1JaiWQ.png Does anyone have an idea what's going on there?
- ChrisSD 6y agoMy suspicion is that the font rendering is slightly off for whatever font is being selected. Right click the offending text, select "Inspect Element". The dev tools should popup. On the right hand side there should be a "Fonts" tab. If you click this it'll show you the font being used.
- tyingq 6y agoCurious why redo wouldn't just use a hash of the file contents if it's going to maintain a database. That's a much easier way to know something's changed if you're going to the trouble of keeping state.
- jeffbee 6y agoThe article gives a dozen paragraphs to answering your question.
- tyingq 6y agoIt also contradicts itself. "Checksumming every output after building it is somewhat slow." "Mostly this is not too serious: the file is probably already in disk cache (since you just wrote it a moment ago!)"
- NateEag 6y agoYou quoted the wrong bit, if you're looking for his argument against hashing. The strong argument is in the section labeled: > Checksumming every input file before building is very slow
- tyingq 6y agoI'd be interested in comparing a fast CRC checksum versus their approach of stat() plus pulling a sequence number. I'm sure it's slower, but I'm curious how much slower for tens of thousands of files.
- jeffbee 6y agoFar and away the fastest CRC on my local box is Adler's checksum that uses the Intel hardware CRC32 instruction (introduced a long time ago, almost universally supported at this date). It only needs 40ms CPU time per GB, about 50x faster than md5sum. If you have a very new CPU you might have hardware SHA support.
- 6y ago
- iforgotpassword 6y agoIt's annoying, we all ran into it, but usually if you get weird results you just make clean and make again and move on. Deployments should always build from scratch. All those fancy checks plus database could be added to make too if you really wanted. It still wouldn't support all the fancy features of redo, but improve your experience with all those existing projects based on make.
- klodolph 6y agoYou don't have to build from scratch if you have high confidence that an incremental build is correct. You get this from modern build systems like Bazel.
- Joker_vD 6y agoWell, getting "weird results" is one particularly strong hint that the incremental build is not, in fact, correct. Weird linker errors, or better yet, weird resulting binaries. "I've just changed this routine and built the app, but when I run it the behaviour is the same?.. <attaches gdb> WTF, it has the old version of the routine! kill, q, make clean && make && gdb run, ah, now it works. Apparently, computers are haunted by the angry ghosts".
- iforgotpassword 6y agoI've done this for long enough to not have that confidence. It's not just whether the build system is correct, but also all the external things you do with it, random files laying around that aren't referenced by the build system but make it into the release bundle and mess up things at runtime, moon phase, whatnot. Even if Bazel is the ultimate, final, perfect build system, I don't trust everyone, including myself, to use it properly.
- klodolph 6y agoI’ve used Bazel for long enough that I’ve regained that confidence. I don’t think it’s some “ultimate, final, perfect build system”, but it is a build system that doesn't have insidious failure modes when used improperly. Execution of build rules is sandboxed by default so you generally don’t run into the problem of implicit dependencies screwing up your builds—not that you can’t, but you generally don’t.
- lmilcin 6y agoThis is an optimization. Production builds should be a full build. It is customary to have various optimizations in modify/save/rebuild/execute/verify loop. I, for one, don't even restart the app for most changes and use REPL to modify application at run time. I only rebuild it from scratch and restart from time to time to verify it all still works as intended.
- ed25519FUUU 6y agoWhy not just rebuild and redeploy? That seems easier to me than an in-place hot fix.
- lmilcin 6y agoBecause when you rebuild and redeploy you loose the state that you have in your application and then you need to put it in the correct state for you to test if your change worked correctly. Here is Shaun Mahood explaining this while building an application to do the presentation without even having to reload the page: https://www.youtube.com/watch?v=cDzjlx6otCU https://www.youtube.com/watch?v=cDzjlx6otCU
- klodolph 6y agoProduction builds don't have to be a full build if you have high confidence that an incremental build is correct. You can achieve this in practice with Bazel and other similar build systems. I would have agreed with you five years ago. Depending on how long it takes to do a full build versus an incremental build, doing incremental builds for releases can make a massive improvement in developer experience. Just speaking from my own experiences here. Large C++ codebases in particular suffer from long build times.
- saagarjha 6y agoProduction and development builds typically differ in optimization flags, though.
- 6y ago
- codetrotter 6y ago> Annoyingly, when you update the content of a file, the mtime of its containing directory is not changed. [...] And anyway, purists might argue that the "content" of a directory doesn't change when the files it points to change; the content is merely a list of filenames and inode numbers, after all, and those stay the same, no matter what happens inside those inodes. Purists make me sad. I don’t think it’s just about purism. For some it might be. But to me, ever since I learned some of the details about how for example the FreeBSD VFS is implemented, from reading the book The Design and Implementation of the FreeBSD Operating System by M. K. McKusick, it makes sense. Prior to that my own mental model of how file systems work on for example FreeBSD was very fuzzy. But reading about the technical details, like what kinds of structs are being used in the VFS and what fields those structs have, I realized why a lot of things are the way that they are. In the same paragraph that I quoted above, in the part I omitted for relevance, he also said: > All sorts of very convenient tree traversals would be possible if the directory mtime were updated (recursively to the root) when contained files changed, but no. This is probably because of hardlinks: since the kernel doesn't, in general, know all the filenames of an open file, it literally cannot update all the containing directories because it doesn't know what they are either. On this I have two thoughts. 1. Doing this means adding more overhead every time a file is written to. The overhead might be negligible, or it might be significant. We’d have to run a lot of benchmarks to really know. (For me the answer is that with the mental model of the file system that I have as mentioned above, I don’t desire this anyways, so I am not personally even going to do that type of benchmarking.) 2. Consider that one or more directories in the path between the root and your current working directory may be mounted from a file system different from the root file system. They might not even be local to the system! For example, perhaps your home directory is mounted over NFS or SMB. That’s certainly common in a lot of deployments at universities and big companies. From a philosophical point of view, does it even make sense to modify the mtime of every directory from the root down to your current working directory when you modified a file that is being written to a storage medium that isn’t even local to your system? And from a practical point of view, what good does it do that the desktop computer on desk A in floor 2 had all of its mtimes in its path modified when you head on up to floor 3 and sit down at the computer on desk K? Any tools you have that were relying on being able to start at the root file system and scan mtimes to find modified files would end up sometimes picking up changes (because someone else had recently modified another file in their respective home directory while logged onto computer 3K) and sometimes not (because at other times no one had been changing any files in their home directories between the time that you last did so on computer 3K and when you came back to it after having worked on your files from computer 2A). As a workaround you might say, okay so let’s stop at file system boundaries. And that might work out, but it’s a bit hairy. And still it could be that a longer component of the remote path is sometimes mounted and sometimes a shorter component. Then again, if the remote system is also doing this recursive mtime modification then that by itself is not a problem. Furthermore, as a consequence of this, in order to make things workable, you should also be changing mtimes of all parent directories when mounting any file system to the most recent mtime of the file system you are mounting, if it is more recent than the mtime of the directory you are mounting it on in order to make the change visible. And if that makes sense then it means that mounting and unmounting file systems are considered as a change, so actually when mounting always set the mtime of all parents to now, and when unmounting update all mtimes again. It seems like a lot of hassle, and potentially bad for performance. Though again, measurements would need to be done. And in the end I maintain that it boils down to what your mental model is for how the file systems that you use work.
- comex 6y ago> Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures. Note that the link is outdated and refers to HFS+. HFS+ originated in classic Mac OS and originally didn't support hard links at all, so they had to be added after the fact [1] using a hacky format based on a hidden directory. APFS has a cleaner design, but it still tracks the list of hard links to an inode. [2] [1] http://www.wsanchez.net/papers/USENIX_2000/ http://www.wsanchez.net/papers/USENIX_2000/ [2] https://developer.apple.com/support/downloads/Apple-File-System-Reference.pdf https://developer.apple.com/support/downloads/Apple-File-Sys..., search for "Siblings"
- deleted 6y ago[deleted]
- dspillett 6y ago> when you update the content of a file, the mtime of its containing directory is not changed ... if the directory mtime were updated (recursively to the root) when contained files changed ... This is probably because of hardlinks It might be partly due to hardlinks, but I suspect it is mostly an efficiency issue. Modifying a file could mean updating quite a few directories every time if you have a deep tree. For some uses this could be quite a performance hit, and for some media would potentially reduce the life of your hardware too. IIRC even when updating atime for every access was the usual default for many filesystems, only the current object had its time updated not all its chain of parents.
- Taniwha 6y agoYes imagine recording a video stream, you'd have to update every directory above a file every few mS when files are written, and because file metadata usually gets priority on a spinning disk the heads would go crazy and throughput would go way down - probably couldn't keep up with the incoming stream
- apenwarr 6y agoYou're assuming the inodes of all the parent directories would be flushed to disk every time they change. This is already not the case. It would be just a matter of touching the data structure in memory and eventually flushing it out.
- dspillett 6y agoYou are assuming a modern filesystem and/or cacheing architecture in a machine with enough memory to spare to run it effectively (though in fairness the post you replied to directly did mention a fairly modern use case). Back when the behaviours being discussed became the norm the landscape was pretty different.
- stuhood 6y agoGreat post. A point that neither this article nor the recent ninja retrospective (http://neugierig.org/software/blog/2020/05/ninja.html http://neugierig.org/software/blog/2020/05/ninja.html) touch on though is that hashing of files is an important step if you'd like to keep caches locally, but especially if you'd like to keep caches remotely. A key reason that Bazel-alikes (including http://www.pantsbuild.org http://www.pantsbuild.org, which I work on) hash things is for use in cache keys. And many of those systems use file watching facilities (notify, fsevents) to determine whether files need to be rehashed. If done properly (i.e.: no false negatives), this is significantly faster than checking file mtimes, and it allows for "early cutoff", where if after hashing the digest of a file has not actually changed, you can skip rebuilding its dependents.
- dang 6y agoDiscussed at the time: https://news.ycombinator.com/item?id=18473744 https://news.ycombinator.com/item?id=18473744
- ape4 6y agoMaybe this is unpopular. mtime and make work in non-edge cases. eg no NFS, no FUSE, no clock changes, etc. And you don't need a database of hashes.
- JdeBP 6y agoYou have a lot of background reading to catch up on. Start at http://jdebp.uk./FGA/introduction-to-redo.html http://jdebp.uk./FGA/introduction-to-redo.html .
- usr1106 6y agoI agree. I would estimate 90% of all Linux packages (over all distros) are based on make. Well, for the distro it does not matter, because packages are typically built from scratch and not incrementally. But everybody hacking on such a package will be affected by the mtime issues. I would claim it hardly ever happens. Of course if it happens it might cause great frustration and waste of half a day.
- adrianmonk 6y agoThe article describes a third way which is neither mtime nor hashes. It's basically a database of (mtime, size, inode number, file mode, owner, group) tuples. So it still requires keeping a database. But because all of that would come out of inodes, it avoids all the I/O of reading the files' contents. Compared to mtime, it's harder to implement, but more resilient and at a similar cost. Compared to hashes, it's almost as resilient but much faster.
- jojo14 6y agoTry scons then: https://scons.org/ https://scons.org/
- usr1106 6y agobitbake (openembedded and Yocto) does not use timestamps, but hashes all inputs. Well, it's a bit different scope. It builds packages and each package use make (or some replacement depending on package maintainer's choice) internally. So the number of files to checksum is small even so if their size can be big, typically mostly the source tarball of the package.
- quattrofan 6y agoFirst thing I do when I install a new app is turn off all notifications in settings.
- m463 6y agoSun used to have a version of make that WOULD keep track of commandlines, and would rebuild targets if they changed. Meaning with the command: foo.o: foo.c cc -o foo.o $(CFLAGS) foo.c make would save away "cc -o foo.o -g -DFOO foo.c" as part of the rebuild check. Also, I was under the (mistaken?) impression that some make actions came from a dependency tree, not a timestamp. Otherwise everything would be breaking during any invocation. One other point about ATIME -- turn it off if you value your SSD.
- aidenn0 6y agoThe make actions come from a dependency tree, and whether or not a target is "dirty" is if: 1. The target doesn't exist 2. Any children are dirty 3. The mtime of any children is more recent than the mtime of the target. All recursively propagated from leaves up.