4 ms·
I'd argue that UNIX-type file systems should offer several types of files: * Unit files. When you create a file and write it, it's not visible for other opens
by Animats 7y ago
I'd argue that UNIX-type file systems should offer several types of files:
* Unit files. When you create a file and write it, it's not visible for other opens until you close it. If you open a file with O_CREAT|O_WRONLY|O_TRUNC, you create a new file, which replaces the old one on close. In the event of a program or system crash, or exiting via "abort" without closing first, the old file remains. So there's always one completely written file. Creating a unit file is an atomic operation. Most files are unit files. (This isn't original with me; it comes from a distributed UNIX variant developed at UCLA in the 1980s.)
Replacing an existing file currently requires elaborate renaming gyrations which vary from OS to OS and file system to file system. At least for Linux, this should Just Work in the normal case.
* Log files. If you open a file with O_APPEND, you can only write at the end. The file system should guarantee that, even after a crash, you get the file as written out to some previous write. If you call "fsync", the recovery guarantee should include everything written up to the "fsync" point. No seeking backwards and overwriting on a log file.
* Temporary files. You can do all the file operations, and the file disappears on a reboot.
* Managed files. These are for databases and such. They have some extra API functions, for async reads and writes and commits. Async I/O should have two callbacks - "the data has been taken and you can now reuse the buffer", and "this write is definitely committed and will survive a system crash" That's what databases really need, and try to fake with "fsync".
Only a few programs will use this, but those are important programs.
- ianai 7y agoHow about sandboxed files? Ie can be assigned to a process - full acl per process/application binary. Maybe even checksum the binary before loading to ram and granting access.
- skissane 7y ago> * Temporary files. You can do all the file operations, and the file disappears on a reboot. It would be nice to be able to have a process tree own a temporary file, such that when the last process in the tree exits (not necessarily the process which created the file), the file is automatically deleted, rather than having to wait for the next reboot.
- Animats 7y agoThis is mostly about defining behavior for the bad cases - program abort and system restart. Those are classically undefined behavior, which causes problems.
- vbezhenar 7y agoCan't you create file, unlink it and then fork as much as necessary? I think that OS will maintain reference counter for that inode and will delete it when all processes will close that handle (explicitly or implicitly with exit).
- IshKebab 7y agoSadly not because then the file doesn't have a filename, and there are a lot of things that only work with file names, not descriptors. Even basic stuff like std::fstream needs a filename. Yes there are workarounds (library extensions, signals, /proc/fd, etc.) But you'd think a basic function like "delete this file when this process exits" wouldn't be too hard.
- TeMPOraL 7y agoI would prefer it to not be the default; it's sometimes useful to keep temporary files in the event of a process crash, especially on a server that would then immediately restart said process. I had such a case very recently, and was thankful for the existing behavior of tmp files.
- skissane 7y agoRather than delete the file straight away, put it in a “trash can” or “recycle bin”. A background process deletes files from “trash can” at a later date. It could normally give them a grace period (e.g. 7 days, configurable) but the files could be deleted early if storage space is running low. The same feature could be used to provide undelete for non-temporary files too.
- JetSpiegel 7y ago
- sargun 7y agoNagle, I believe I last read your comment on this (https://news.ycombinator.com/item?id=13964053 https://news.ycombinator.com/item?id=13964053), progress has been made . We have the temporary files you're asking for. https://lwn.net/Articles/619146/ https://lwn.net/Articles/619146/ With O_TMPFILE, you can also write new data, and then automatically replace a file on disk.
- Animats 7y agoWith O_TMPFILE, you can also write new data, and then automatically replace a file on disk. How? Using "linkat"? That's not automatic.
- sargun 7y agorenameat2 is meant to be atomic. From the man page: If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing. However, there will probably be a window in which both oldpath and newpath refer to the file being renamed.
- Animats 7y agoYes, that's one of the many workarounds for not having unit files. There are different workarounds in the Windows world, and the Linux workround does not work on some file system types.
- silvestrov 7y ago> Unit files. ... you create a new file, which replaces the old one on close. Classic MacOS-9 had the PBExchangeFiles call which did this perfectly. Before call: dirEntryA -> fileContentsA dirEntryB -> fileContentsB after call dirEntryA -> fileContentsB dirEntryB -> fileContentsA This meant that the user kept all meta info for files, e.g. tags, window position, custom icons, etc. So when saving a new document you wrote it to a new hidden temp file, and when everything was written, you called PBExchangeFiles to swap the contents of the old and new files. After this you deleted your temp file which now contained the old document content. http://mirror.informatimago.com/next/developer.apple.com/documentation/mac/Files/Files-254.html http://mirror.informatimago.com/next/developer.apple.com/doc...
- TeMPOraL 7y agoCommon Lisp standard also strongly suggests this behavior for the :if-exist :supersede argument to #'open: "The existing file is superseded; that is, a new file with the same name as the old one is created. If possible, the implementation should not destroy the old file until the new stream is closed." http://clhs.lisp.se/Body/f_open.htm http://clhs.lisp.se/Body/f_open.htm I imagine they lifted this idea from some prior Lisps, which suggests it's a pretty old concept.
- tobias3 7y agoLike renameat2 on Linux with RENAME_EXCHANGE ?
- silvestrov 7y agoNo renameat2 moves the meta info (e.g. xattrs)as it’s an atomic “mv a temp; mv b a; mv tmp b” So tags from a will end up on b. If b is a temp file, then the tags will be lost after the temp file is deleted.
- mehrdadn 7y agoFun fact for those who don't know: Windows already has a database API. I've never used it, but it's called the Extensible Storage Engine (JetCreateDatabase, etc.).
- fanf2 7y agoIs this the same storage engine that used to provide the back end for Visual Source Safe and Exchange? Which has a 2GB limit per storage unit and corrupts itself irreparably if you hit the limit?
- mehrdadn 7y agoI don't know!
- skissane 7y agoWindows also now bundles SQLite [1]. So Windows has more than one “database API”. [1] C:\windows\system32\winsqlite3.dll
- zzzcpan 7y agoIt's not that easy to come up with a decent filesystem API. Consider that filesystem data cannot be synced to underlying storage in a such way that it can be reliably read later and backups by definition are asynchronous and provide eventual consistency. This means that the best meaningful guarantee an application can get after machine crash is being able to read data written up to some point in the past, but not after every successful fsync before crash. Usually, though, even that guarantee is hard to achieve, as bad blocks happen and redundancy is not there. Given all that it's ok to relax filesystem APIs behavior to proper physical constraints. Like fsync is only meaningful as an ordering operation, no need to actually flush anything to disk immediately on fsync or any operation. Next is multi process and multi threaded scenarios. Should O_APPEND only work correctly from a single thread, should each write be atomic and to what size, we certainly can't have gigabytes in an atomic append or should there be some synchronization mechanism that blocks others? Same for temporary files and unit files. And what to do on bad blocks? Should there be redundancy within a single disk, should block device underneath be log structured storage with block remapping and scrubbing and provide reliable storage layer to the outside by sacrificing space? Maybe for desktop machines it should, but not for servers, at least not all of servers, they need a different API. I'm not even touching performance considerations here, that depend on performance-friendly APIs a lot.
- asveikau 7y ago> Log files. I wonder if this would ideally also provide a rotation API. Rotating log files in the presence of multiple writers is messy, and maybe shouldn't be reimplemented at the application layer every time. > Temporary files Windows is not always a model of filesystem elegance, but it has a "delete on close" flag in their equivalent to open, which makes it go away on the last close (handle can still be duplicated or inherited, so you get some reference counting through that). Actually I think you can do similar to this on Unix by unlinking after the open, but keeping an fd open for the lifetime of the temp file.
- Animats 7y agoI wonder if this would ideally also provide a rotation API. Rotating log files in the presence of multiple writers is messy, and maybe shouldn't be reimplemented at the application layer every time. That feature was in UNIVAC EXEC8, now OS-2200. It was called F-cycles.[1] Over half a century old, and still working. [1] https://public.support.unisys.com/2200/docs/CP18.0/PDF/78307949-029.pdf https://public.support.unisys.com/2200/docs/CP18.0/PDF/78307...