4 ms·
> It just (mostly) decouples discarding state from power failures, making it intentional rather than accidental. Just wondering, but how is an OS supposed to a
by zozbot234 5y ago
> It just (mostly) decouples discarding state from power failures, making it intentional rather than accidental.
Just wondering, but how is an OS supposed to achieve this without fsync()ing all writes to memory and being slow as dog$h!t? Having a "single level" of storage means it becomes harder to tell what might be purely ephemeral, and can thus be disregarded altogether if RAM or disk caches are lost in a failure. You might think you can do this via new "volatile segments" API's but then you're just reintroducing the volatile storage that you were trying to get rid of.
- kragen 5y agoBoth KeyKOS and EUMEL used periodic globally consistent checkpoints supplemented by an explicit fsync() call for use on things that required durability like database transaction journals. So when the system crashed you would lose your last few minutes of work; EUMEL ran on floppies, so the interval could be as long as 15 minutes, but I think even KeyKOS only took a checkpoint every minute or so, perhaps because it would pause the whole system for close to a second on the IBM 370s it ran on. (Many years later, Norm Hardy showed me a SPARC workstation with I think 32 or 64 MiB of RAM; he said it was the biggest machine KeyKOS had ever run on, twice the size of the mainframe that was the previous record holder. Physically it was small enough to hold in one hand.) I assume L3 used the same approach as EUMEL. I don't know enough about OS/400 or Multics to comment intelligently, but at least OS/400 is evidently reasonably performant and reliable, so I'd be surprised if it isn't a similar strategy. There's a latency/bandwidth tradeoff: if you take checkpoints less often, more of the data you would have checkpointed gets overwritten, so your checkpoint bandwidth gets smaller. NetApp's WAFL (not a single-level store, but a filesystem using a similar globally-consistent-snapshot strategy) originally (01996) took a checkpoint ("snapshot") once a second and stored its journals in battery-backed RAM, but with modern SSDs a snapshot every 10 milliseconds or so seems feasible, depending on write bandwidth. Most of these systems did have "volatile segments" actually, but not for the reason you suggest, which is performance; rather, it was for device drivers. You can't swap out your device drivers (in most cases) and they don't have state that would be useful across power failures anyway. So they would be, as EUMEL called it, "resident": exempted from both the snapshot mechanism and from paging.
- zozbot234 5y ago> So when the system crashed you would lose your last few minutes of work So, pretty much how disk storage is managed today? But then if you're going to checkpoint application programs from time to time, it might better to design them so that they explicitly serialize their program state to a succinct representation that they can reload from, and only have OS-managed checkpoint and restore as a last resort for programs that aren't designed to do this...Which is again quite distinct from having everything on a "single level". > but with modern SSDs a snapshot every 10 milliseconds or so seems feasible, depending on write bandwidth. Not so. Even swapping to SSD is generally avoided because every single write creates wear-and-tear that impacts lifetime of the device.
- kragen 5y agoWell, I don't think it's "pretty much how disk storage is managed today," among other things because you don't have a multi-minute reboot process, and you usually lose some workspace context in a power failure. The bigger advantage of carefully designed formats for persistent data is usually not succinctness but schema upgrade. It's true that too much write bandwidth will destroy your SSDs prematurely, and that might indeed be a drawback of a single-level store, because you might end up spending that bandwidth on things that aren't worth it. This is especially true now that DWPD has collapsed from 10 or more down to, say, 0.2. But they're fast enough that you can do submillisecond durable commits, potentially hundreds of them per second. In the limit where each snapshot writes a single 4 KiB page, 100 snapshots per second is 0.4 MB/s. If you have a 3-year-life 1TiB SSD with 0.2 DWPD, you can write about 220 TiB to it before it fails. So if you have 4 GiB of RAM, in the worst case where every checkpoint dirties every page, you can only take 56000 checkpoints before killing the drive: about an hour and a half if you're doing it every 10 ms. But actually the drive probably isn't that fast. If you limit yourself to an Ultra DMA 2 speed of 33 MB/s on average, while still permitting bursts of the SSD's maximum speed, that's 2.8 months, but a maximum checkpoint latency of more than 2 minutes. You have to scale back to 7.7 MB/s to guarantee a disk life of a year, at which point your worst-case checkpoint time rises to over 9 minutes.
- Animats 5y agoJust wondering, but how is an OS supposed to achieve this without fsync()ing all writes to memory and being slow as dog$h!t? Separate from the persistent storage thing, I've wanted traditional file systems that work like this: * Unit files. The unit of file consistency is the file. The default case. Open, create, close, no other open can read it before close, on close it replaces the old file for all later opens. On crashes, you always get a consistent file, the old version if the close hadn't committed yet. * Log files. The unit of file consistency is one write. Append only. On crashes, you get a consistent file up to the end of a previous write. No trailing off into junk. * Temp files. There is no file consistency guaranteed. Random access. On crashes, deleted. * Managed files. A more complex API is available. Random access. On a write, you get two completion events. The first one is "buffer has been copied". The second is "data is safely in permanent storage". This is for databases, which really care about when the data is safely stored. The second full completion event might also be offered for other file types, primarily log files. That covers the standard use cases and isn't too hard to do.
- zozbot234 5y ago> The unit of file consistency is the file. The default case. Open, create, close, no other open can read it before close "$PROGRAM cannot open document $FILENAME because it's already open in $PROGRAM. Try closing any open applications and open $FILENAME again." Who thinks this is actually sane behavior? Unix does it right.
- kragen 5y agoInstead of giving you a mutual exclusion error, a newly created file written this way might simply be invisible until it's complete, at which point it atomically replaces the old version of the file (which might still be accessible through mechanisms like VMS version numbers, WAFL snapshots, Linux reflinks, or simply having opened it before the replacement). This is probably what a transactional filesystem would give you: the creation, filling, and naming of the file would normally be part of the same transaction. But, with optimistic synchronization, transactions that were reading an old version of the file might require redoing from the start when a new version was written.