3 ms·
The escalator never went down in the first place. Files are a weird, unclean semi-abstraction (growable virtual sparse block devices addressed as seekable byte-
by derefr 7y ago
The escalator never went down in the first place. Files are a weird, unclean semi-abstraction (growable virtual sparse block devices addressed as seekable byte-streams with heavy metadata and OS-level memory caching?) that we only feel a degree of primacy about because of how common they've been.
Consider: unikernels (like, say, any old cartridge game ROM) don't have any need for files. They have precisely three abstractions they deal with data in terms of:
• a .data section in ROM (maybe needing bank-switching to get in place);
• some kind of byte-addressable NVRAM (like battery-backed "save RAM", or CMOS memory) either bus-mapped, or through MMIO.
• tape or (floppy) disk, sometimes at an extremely low level (send commands to the drive motor, write guard nybbles, etc.), sometimes through a DOS where you can just request to seek to a given track then read or write a given sector on that track. Either way, more like a block device than a filesystem.
---
For today's kids, I'd suggest: don't start with files. Teach key-value storage first. Interact with a library like LMDB without explaining where it's putting the data. They'll understand this just fine.
Then, teach object storage in terms of key-value storage. Object storage—especially once you add object versioning—is much closer to the modern metaphor that user-facing apps expose. You compose a complete new version of an object in an in-memory scratch buffer, and then it atomically replaces the previous object. You can't corrupt an object by half-saving it. Etc. Again, don't bother explaining how this works yet; just give them a scripting runtime hooked up to a Minio instance.
After they get that, you can ask them what they'd do if they needed to create an object that wouldn't fit in memory. Then you can explain block devices, as a "place where large scratch buffers can live"—but don't force them to figure out how to allocate those buffers from the block device! That's gonna pull in a whole bunch of prerequisite teaching. Instead, pull out another API: Linux's LVM. Logical volume management takes block devices in, and spits block devices out. The logical volumes are the scratch buffers. Explain mmap(2), and how these buffers end up a lot slower that memory buffers. Explain how these buffers survive a disk crash.
After you get to that point, then you can explain that all the other ways computers durably store data are built on top of these logical-volume durable scratch buffers. You can explain how LMDB works in terms of disk pages; and then you can explain how a content-addressable storage works in terms of combining an LMDB-like KV store with hashing and splitting.
And, after that—if you like—you can explain that sometimes, when we need something that's like object storage but where everything in the storage bucket is actually a tiny scratch buffer, we use filesystems. You can explain what an "extent" is by talking about how something like LMDB, that has a "freelist" of pages from its underlying logical-volume, can reserve a contiguous set of those pages and then let something else access them. Then you can explain a filesystem as a key-value store that has buckets called "inodes" with page-range keys, and extent-address values. (And an associated versioned object store of directory-objects, where each object is a serialized list of records (dirents), each containing a reference to an inode and giving it a name and other stuff.)
Filesystems are hard!
- astrobe_ 7y ago> Filesystems are hard! Huh, no. Just show how the FAT file system works. It's still used on USB drives. Now that you have a simple model for the low level, you can gradually add complexity such as standard file operations, caching, wear-leveling, etc. You say Minio instance, I say hex editor.
- derefr 7y agoI didn't say the implementation of a filesystem was hard. You can certainly communicate, as a complete standalone fact, the brute engineering perspective on what makes a particular filesystem work. Probably even in a single lesson. But the point isn't to teach students about a filesystem; the point is to teach students what files are when they don't understand why you'd even want the abstraction that "files" represent; when "files" aren't an abstraction they're familiar with compared to other abstractions. You have to justify "files" in the context of things they do know—to explain what purpose files would serve if they were invented today; what reason you'd introduce a filesystem into a data architecture that had all these other things (KV storage, object storage, logical volumes) but not files. Students would ask: Why are files streams? Why are the streams arbitrarily writable, when this means files can become corrupted from a partial write? Why are files extendable, but only at the end, not in the middle or the beginning? Why do files have names? (And, once you explain inodes+dirents, they'll ask why there's a directory tree if hard links exist; and why it's the dirents that have the names, but the files themselves that have the other metadata—xattrs; alternate data streams, etc.) What is the difference between a file stored in a directory, a file stored in a filesystem stored in a mounted loopback image stored in another file, and a file stored in a subvolume or filesystem container? Why do directories need special system calls to read and write them? Why is a virtual filesystem used in many OSes to represent kernel objects like named pipes and device nodes? Etc. (This isn't even a hypothetical; engineers at AWS and GCP probably had to answer this very question when asked to justify building EFS and Cloud Filestore. Why do we need a filesystem as a service, when we already have these other services that provide these other abstractions?) This is not altogether unlike teaching a student what a programming language is and why you'd even want one of those, when they're immersed in an environment where software is created without them. Would you just sit down and show the kids C, because it's small and easy-ish to understand? A filesystem is a data-storage abstraction, like C is a machine abstraction. But neither are primitive abstractions. They build on numerous other, theoretically purer abstractions. It's much easier to explain the motivation for the creation of a language like C, if you already understand the motivation of the creation of a simpler language that rests upon fewer other abstractions—like ASM, or like Scheme. Likewise, it's much easier to understand the motivation behind the creation of the abstraction that is filesystems, if you already understand the motivation behind the creation of simpler data-storage abstractions, like logical volumes or memory-page-backed search trees.