9 ms·
The No-Order File System (2012)
- ro_bit 6y agoTitle could use a (2012)
- not2b 6y agoThe publications they point to are from 2012 and 2013. Did this work ever go anywhere after that?
- alextheparrot 6y agoRemzi and Andrea [0] have a pretty well regarded OS book used to teach the OS course at Wisconsin [1-2]. Hope they’re doing well, Andrea’s course teaching Scratch to 4th and 5th graders at a local elementary school was a highlight of my time there. [0] I would use honorifics, but they are married which makes it a bit confusing. [1] Which I never took but was well regarded when I was an undergrad [2] http://pages.cs.wisc.edu/~remzi/OSTEP/ http://pages.cs.wisc.edu/~remzi/OSTEP/
- pmiller2 6y agoIf you think that's confusing, I had two professors in grad school who were married and shared the same last name. She had a PhD when I started there, so I called her "Doctor," but he didn't, so I called him "Steve." Well, Steve later got his PhD, so I was incredibly confused at that point. We collectively resolved the dilemma by all agreeing to be on a first name basis. Now that I've spent significant time in industry and have worked with more people with PhDs than were in my department in grad school, I've come to realize I was being a little bit silly. In my experience, at least once you get to the graduate level, anybody who insists on being called "doctor" seems a little full of it to me. That said, I still think it's appropriate for undergrads to call their professors "doctor," when applicable, and I was always careful to do so whenever I was in the presence of any undergrads.
- leetcrew 6y agoI always just called my professors "professor {lastname}". that way I didn't have to keep track of who was a grad student or lecturer with a masters and who actually had a phd.
- skissane 6y agoThat works in the US, given in the US students call all academics "professor". However, in other countries, you would not call someone "professor X" unless they actually had the word "professor" in their job title. Here in Australia, a lot of academics don't – you start out as a "Lecturer", then get promoted to "Senior Lecturer", then "Associate Professor", then finally "Professor", and calling a lecturer "professor" is not done. And you certainly wouldn't use the word "professor" when addressing a PhD student.
- pmiller2 6y agoIt's not correct at all for graduate student instructors, though.
- skissane 6y agoBut yes for any instructor who has a PhD?
- pmiller2 6y agoYes, PhD = "doctor." That's what the "D" is actually for. PhD is shortened from philosophiae doctor, meaning "doctor of philosophy." Anyone who holds a legitimate PhD can use the title "doctor," and, if they so desire, ask people to call them "doctor" legitimately.
- skissane 6y agoYes, I agree it is okay to address people with doctorates (including PhD) by the term "doctor", at least in certain contexts. But my question was about "professor" though, not "doctor". You say you wouldn't call a grad student instructor a "professor", but what about an instructor with a PhD (but without any formal academic title of "professor")?
- tentacleuno 6y agoDidn't Plan9 do something like this?
- jxy 6y agoplan 9 likes those WORM filesystems, Venti, cwfs, or this early incantation https://9p.io/sys/doc/fs/fs.html https://9p.io/sys/doc/fs/fs.html I guess you can modify the WORM fs to do the same as this No-Order fs does.
- KMag 6y agoAs I remember, Plan9 also had a concept of a write-only filesystem, and a read-write filesystem was mounted as a union mount of a write-only fs and a WORM fs with the same partition.
- pwinnski 6y agoIt's a fascinating concept, but nothing seems to have happened in the eight years since. This seems like it might gain a bit of performance in exchange for slightly-higher filesystem overhead, though it's not clear there are any stats on either. It's also not clear how exactly performance increases would surface: are reads slower but writes faster?
- pwinnski 6y agoI clicked through to read the paper, and some of my questions are answered there, but not (from my initial skim) all. My intuition was wrong: reads are generally speedy, while writes are a bit slower than ext3.
- Animats 6y agoThat's progress. I've previously suggested that operating systems should have stronger file integrity guarantees. "Unit" files (rewriting replaces the whole file atomically, no reader ever sees a partially written file). That's the default. "Log" files (always end at a clean end point, don't tail off into junk). "Temp" files (disappear on reboot). And, for databases, "Managed" files. Managed files have more I/O functions. In particular, you get two completion events on writes - "copy complete" (the caller can reuse the buffer) and "safely stored" (the data has reached its final resting place, all links are complete, etc.). Programs like databases would use that. Those are the semantics databases want, and struggle to get by flushing, waiting, and various workarounds. When I mention this, what usually happens is that people get lost in complicated workarounds for simulating unit files. Different approaches are needed for Linux, Windows, NTFS, and various VM systems. This should Just Work. This isn't my invention; it's from Popek's kernel in 1985 at UCLA, later seen as UCLA-Locus and as an IBM product. They had explicit commit and revert functions for file systems. I'd suggest having the default be commit on normal close or normal program exit, but if the program aborts or crashes or is killed, unit files don't commit and remain unchanged.
- rodgerd 6y agoMeanwhile ext4 is plowing even further down the path of "we get great benchmarks if we make data integrity a userspace problem LOL"
- Dylan16807 6y agoHow so? They had that recent batch of safe optimizations, and trying to make fsync act closer to what it actually says is a pretty good thing.
- wmf 6y agoThe fact that write(), close(), rename() isn't safe unless you fsync is pretty annoying.
- 6y ago
- tytso 6y agoI can think of a number of reasons why this may not have gotten a lot of traction in the intervening eight years. One is that DIF/DIX disks are not easily available at reasonable prices. The other potential shortcoming is that it provides substantially weaker consistency guarantees than what people are used to. After all of the scan threads are finished with the recovery, yes, the file system metadata will be self-consistent; but that's all you can count upon. Suppose that a particular file is getting updated at the time of the crash. There might be several blocks that were newly allocated right before the crash, and the a larger number of data blocks that were getting overwritten right before the crash. There is no guarantee which set of data blocks will be persistent across the reboot, and which newly allocated blocks will actually be attached to the file. There might also be newly allocated blocks that were attached to the file, but the data might not be written to the block, such that stale data (the previous contents of the block, which mgiht be another user's medical data, or private e-mail, etc.) that would become visible to the file across the crash.
- deleted 6y ago[deleted]
- jzer0cool 6y ago> Modern file systems use ordering points to maintain consistency in the face of system crashes. Could someone shed more light? In modern or older O/S, what exactly happens during the crash?
- tyingq 6y agoI'd put it in 3 buckets of "not journaled", "metadata only journaled", and "data and metadata journaled". In the "not journaled" bucket, you have to hope something like fsck can make sense of the state of the filesystem. Often, the filesystem could be so "fscked" that you can't mount/boot. In the "metadata only journaled" bucket, if you use order...meaning writing the data before you write the metadata, you will have a consistent filesystem that can be mounted, albeit possibly missing some data. In the last bucket, you're also getting a consistent filesystem that can be mounted, but with less data loss. At the cost of some performance. I think they are saying both buckets 2 and 3 are "modern". I certainly encountered lots of fsck giving up in the 90's on "bucket 1" type filesystems.
- jzer0cool 6y agoThanks! Is bucket 1 also related to hard drive won't boot. Eventually be able to recover mbr and proper boot location on and to the OS?
- tyingq 6y agoYes, "won't boot" meaning the root filesystem isn't recoverable, can't fully boot without that. You could have, for example, a working /boot, but a dead /. In that case, the kernel will boot up and then panic when it can't mount root. A bad mbr is also possible, for reasons unrelated to any filesystem. And a journaling filesystem doesn't always help with drive errors, etc. I was trying to scope down to filesystem issues due to an unorderly shutdown, where the disk drive itself is fine.
- j16sdiz 6y agoThere are softupdate filesystem too. They are less common now, but still exist. https://en.wikipedia.org/wiki/Soft_updates https://en.wikipedia.org/wiki/Soft_updates
- gray_-_wolf 6y ago> Sorry! The URL you requested was not found on our server. The link seems to be dead :/
- bytematic 6y agoWorks for me
- chrismorgan 6y ago404 for me on accessing from both India and Australia.
- Ericson2314 6y agoAll I ask from storage: please give me a b-epsilon tree for content-addressed data. I will handle naming, mutation, etc. myself. The number one problem is all the current abstractions are stupefied, with everyone conway's law-ing around everyone else.
- peter_d_sherman 6y agoI got an idea when reading this for a file system for rotational (aka, "spinning rust") old-skool hard drives (which are probably pretty dated by this point!). Basically, you create a B-tree. Everything except the leaf nodes (that is, the root and all of the internal nodes) are stored outwardly from the center track of the hard drive, so if that's track 40, then it would be 40 then 39, then 41, then 38, then 42, etc. Now, each leaf node is stored on the start of each track, and represents all of the blocks (after the blocks it itself takes up) on that track. Now, when the OS needs to write or delete a block from a file, then the OS repositions the head to that track, writes the block, and then updates the leaf node (it's already on the same track -- so it's faster than moving the hard drive head to another track to do the update). Yes, the rest of the B-Tree (on other tracks) might need to be updated as well, but if you were writing a whole bunch of blocks to an empty track at the same time, I think it could save some time... If we were writing to a pre-existing file, and the B-Tree already had that track's leaf-node included in its pointers (that is, let's say that we're already writing to a file that already had one block on that track), then all we'd have to do is: 1) Move the hard drive head to that track 2) Write the new block 3) Update the leaf node of the B-Tree to say that the block is now part of the file. It gets even better. If there's a single block of the file on that track, and the track is otherwise empty, then if we have enough data to fill the track, then write all of the blocks, and cache writing to the B-Tree until all of that's done... in other words, a single write! B-Tree entries are a key comprising FileID + DiskBlockID. Files are ordered in DiskBlockID order. Should be extremely fast -- for old spinning hard drives... (Yes, I know... nobody uses spinning hard drives anymore! <g>)
- ashkankiani 6y agoKafka's entire design is based upon leveraging spinning disks and the OS's file cache. So the whole "nobody uses spinning hard drives anymore" is very incorrect. In fact, Kafka in some cases can be faster on spinning disks than SSDs.
- peter_d_sherman 6y agoI stand corrected! <g> On a more serious note however, Kafka sounds interesting! I will have to check it out...