13 ms·
> On Windows, once the file is open, it is that filename that is open; You can't rename or delete it; This is wrong... there's no clear-cut thing like the "fil
by mehrdadn 9y ago
> On Windows, once the file is open, it is that filename that is open; You can't rename or delete it;
This is wrong... there's no clear-cut thing like the "file name" or "file stream" that you can specify as "in-use". It depends on the specifics of how the file is opened; often you can rename but not delete files that are open. Some (but AFAIK not all) in-use DLLs are like this. They can be renamed but not deleted. And then there's FILE_SHARE_DELETE which allows deletion, but then the handle starts returning errors when the file is deleted (as opposed to keeping the file "alive").
To make it even more confusing, you can pretty much always even create hardlinks to files that are "in use", but once you do that the new name cannot be deleted unless the old name can also be deleted (i.e. they follow the same rules). This should also make it clear that it's not the "name" that's in use, but the "actual file" (whatever that means... on NTFS I'd suppose it corresponds to the "file record" in the MFT).
The rule that Windows always abides by is that everything that takes up space on the disk must be reachable via some path. So you can't delete in-use files entirely because then they would be allocated disk space but unreachable via any path.
> What this means is, that even though you don't NEED to restart anything for most upgrades in Unix/Linux, you're still running the old version until you restart the program that uses it.
What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. Is this correct? Because it seems to me that the Windows method might be less flexible but is likely to be more stable, since there's a single coherent global view of the file system at any given time.
- tialaramex 9y agoYes, in principle what you've said about the Unix approach here is correct, if you upgrade one half of a system and not the other half and now they're talking different protocols, that might not work. But keep in mind that if your system can't cope with this what you've done there is engineer in unreliability, you've made a system that's deliberately not very robust, unless it's very, very tightly integrated (e.g. two sub-routines inside the same running program) the cost savings had better be _enormous_ or what you're doing is just amplifying a problem and giving it to somebody else, like "solving" a city's waste problem by just dumping all the raw sewage into a neighbouring city's rivers. Now, the "you can't delete things because then the disk space is unreachable" argument makes plenty of sense for, say, FAT, a filesystem from the 1980s. But (present year argument) this is 2018. Everybody's main file systems are journalled. Sure enough, both systems _can_ write a record to the journal which will cause the blocks to be freed on replay and then remove that journal entry if the blocks actually get freed up before then. The difference is that Windows doesn't bother doing this.
- mehrdadn 9y agoAaaaaand here comes the Linux defending! OK... > But keep in mind that if your system can't cope with this what you've done there is engineer in unreliability It's weird that you're blaming my operating system's problems on me. "My system" is something a ton of other people wrote, and this is the case for pretty much every user of every OS. I'm not engineering anything into (or out of) my system so I don't get the "you've made a system that [basically, sucks]" comments. > [other arguments] I wasn't trying to go down this rabbit hole of Linux-bashing (I was just trying to present it as as objective of a flexibility-vs.-reliability trade-off as I could), but given the barrage of comments I've been receiving: I don't know about you, but it happens more often than I would like that I update Linux (Ubuntu) and, lo and behold, I can't really use any programs until I reboot. Sometimes the window rendering gets messed up, sometimes I get random error pop-ups, sometimes stuff just doesn't run. I don't get why it happens in every instance, and there might be lots of different reasons in different instances. IPC mismatch is my best guess for a significant fraction of the incidents. All I know is it happens and it's less stable than what you (or I) would hope or expect. Yet from everyone's comments here I'm guessing I must be the only one who encounters this. Sad for me, but I'm happy for you guys I guess.
- rlpb 9y ago> ...but it happens more often than I would like that I update Linux (Ubuntu) and, lo and behold, I can't really use any programs until I reboot... Ubuntu developer here. This doesn't happen to me in practice. Most updates don't cause system instability. I rarely reboot. Firefox is the most noticeable thing. After updating Firefox (usually it's a security update), Firefox often starts misbehaving until restarted. But I am very rarely forced to restart the login session or the entire system. I should, to get updates to actually take effect, but as a developer I'm usually aware of the specifics, so I can afford to be more selective than the average user. Are you sure you aren't comparing apples to oranges here, and are actually complaining about the stability of updates while running the development release, which involves ABIs changing and so forth?
- mehrdadn 9y ago
- kelnos 9y ago> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. Is this correct? Only if the libraries that use IPC have changed their wire format between versions, which would be a pretty bad practice, so I wouldn't expect that to happen often (if ever). If something that's already running has its data files moved around or changed sufficiently, and it later tries to open (that is, the app was running but the data file wasn't open when the upgrade happened) what it thinks is an old data file, but is either new and different or just missing, that could cause problems. > Because it seems to me that the Windows method might be less flexible but is likely to be more stable, since there's a single coherent global view of the file system at any given time. In practice I've never had an issue with this (nearly 20 years using various Linux desktop and server distros). Upgrade-in-place is generally the norm, and most people will only reboot if there's a kernel update or an update to the init system.
- pjc50 9y ago> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. In theory, but Linux systems tend to do very little IPC other than X11, pipelines, and IP-based communication, where the protocols tend to support running with different versions. In practice you can achieve multi-year uptimes with systems until you get a mandatory kernel security update.
- mehrdadn 9y agoX11 (or other window-related tooling) was exactly what I was thinking of actually, because every time I do a major Linux (Ubuntu) update I can't really launch programs and use my computer normally until I reboot. It always gets finicky and IPC mismatch is the best explanation I can think of.
- rleigh 9y agoThat's D-BUS being a terribly specified and poorly implemented mess. Everything underneath should be solid.
- mehrdadn 9y agoI mean, OK, but if Windows's GUI (or PowerShell, or whatever) crashed or misbehaved upon update, would you be satisfied with "that's win32k/DWM/whatever being a poorly implemented mess; everything underneath should be solid"?
- rleigh 9y agoNo. D-BUS is a travesty and blight upon the Linux desktop, and with systemd, every Linux system. It's the most fragile and nasty IPC system I've encountered. There are several better alternatives implemented by competent people, so there's really no excuse for its many defects.
- andrewshadura 9y ago
- jacoblambda 9y ago*nixes have a system for handing off to the newer version such as kpatch and kgraft. kgraft for example swaps off each syscall for a process while it is not being used. This lets the OS slowly transfer to the new kernel as it is running. kpatch does it all in one go but locks up the system for a few milliseconds. The version that is currently merged into 4.0+ kernels is a hybrid of the two developed by the authors of both systems.
- cat199 9y agoRuntime kernel patching is a new thing. "*nixes" do not generally have these systems. Some proprietary technologies exist which enable specific platforms to use runtime kernel patching exist.
- vetinari 9y ago> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. While this is true, I've never seen this to be a problem. If two programs use IPC, they usually use either stable, or compatible protocol. To make things even more complicated, you can have two programs, either each in it's own container, or statically linked, or with their private bundles of libraries, doing IPC and then they are free to have different versions of the underlying libraries, while the users still expect them to work fine.
- postingatwork 9y ago> What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. Is this correct? At the first glance this is true, but you can guard against this in several ways. If your process only forks children then it already inherits the loaded libraries from the parent as part of the forked address space. Alternatively you can pass open file descriptors between processes. Another option is to use file-system snapshots, at least if the filesystem supports them. Yet another option is to not replace individual files but complete directories and swap them out via RENAME_EXCHANGE (an atomic swap, available since kernel 3.15). As long as the process keeps a handle on its original working directory it can keep working with the old version even if it has been replaced with a new one. Some of those approaches are tricky, but if you want to guard against such inconsistencies at least it is possible. And if your IPC interfaces provide a stable API it shouldn't be necessary. > And then there's FILE_SHARE_DELETE which allows deletion That has some issues when the file is mmaped. If I recall correctly you can't replace it as long as a mapping is open.
- jrs235 9y ago>So you can't delete in-use files entirely because then they would be allocated disk space but unreachable via any path. Ah. This must be why I can't permanently delete files in use by a program but can sometimes "delete" them and send them to the recycle bin.
- tzahola 9y ago>What I expect it also means is that you'll get inconsistencies when doing inter-process communication, since they'll be using different libraries with potential mismatches. That's why you should restart those programs that were using the library. You can find this out via `lsof`.
- mehrdadn 9y agoReally? Somehow procure a list of all libraries that were updated in a system update, go through each one, find out which program was using it, and kill that program? Every single time I update? You can't be serious.
- beagle3 9y agoApt does this automatically for you on uphrade
- jlgaddis 9y ago"checkrestart" also exists (on some distributions) for exactly this same purpose.
- avar 9y agoWhat you're describing is trivially done on any *nix system with a mature package manager, if it isn't doing this already: 1. Do the upgrade, this changes the files. 2. You have a log of what packages got upgraded. 3. Run the package system's introspection commands to see what files belonged to that package before & after 4. Run the package system's introspection commands to see what reverse depends on those packages, or use the lsof trick mentioned upthread. 5. For each of those things restart any running instance of the application / daemon. Even if something didn't implement this already (most mature systems to) this is at most a rather trivial 30 line shellscript.
- ptero 9y agoIn principle, yes, IPC can fail between different versions of the same software. However, the chances that communication will fail between new version and some other utility are IMO much higher. A surprise comm failure between two copies of the same software (even different revisions) usually makes developers look pretty bad. Some versions are known to be incompatible and most Linux distributions do a very good job of recommending and doing a restart of affected services in a way transparent to users. I have been running Linux and home and at work for years, almost never restart those workstations and, as far as I can tell, never had problems from piecemeal upgrades. My 2c.
- beagle3 9y ago> So you can't delete in-use files entirely because then they would be allocated disk space but unreachable via any path. Isn't there a $MFT\\{file entry number} virtual directory that gives an alternate access path to each file? Wouldn't that qualify as "a way to access the file?" Also, you might say that in practice Linux abides by the same rule - the old file can be referenced through some /proc/$pid/fd/$fd entry.
- code_duck 9y agoIn a case where one could get some sort of inconsistency because of different library versions, you restart the applications. That’s the point, that this can be handled by restarting the applications and not the entire operating system.
- ksk 9y agoIs there a way to get a nice helpful popup telling me which applications and services to restart?
- code_duck 9y agoYes, there are many ways to determine that on the command line. As far as a GUI, I couldn’t say as that isn’t how I do system administration. I’m not sure what scenario you are envisioning. Usually upgrades are handled via the distribution and its package manager, and the maintainers take care of library issues. It’s not like windows where you go all over downloading packages from websites and installing them over each other.
- ksk 9y agoI was responding to your own comment >In a case where one could get some sort of inconsistency because of different library versions, you restart the applications. I am envisioning the same scenario you replied to!
- code_duck 9y agoYes, but for some reason you want a GUI alert to tell you important things, which is a foreign concept to me. What I envision is knowing what is running on your server and updating programs purposefully, with knowledge of how they interact and what problems version inconsistency could cause.
- ksk 9y agoYou're the one proposing the enumeration of affected programs and services and restarting them as a solution!! How is using a GUI a foreign concept? Are you debating the merits of a GUI in 2018? In any case, my assumption here is we're trying to help the user and give them an easy way to know what to do, instead of leaving their software in an undefined state.
- asveikau 9y agoHere is the simplest way I can put it: When you delete a file in NT, any NtCreateFile() on its name will fail with STATUS_DELETE_PENDING until the last handle is closed.[1] Unix will remove the name for this case and the name is re-usable for any number of unrelated future files. [1] Note that is not the same as your "must be reachable via some path". It is literally inaccessible by name after delete. Try to access by name and you get STATUS_DELETE_PENDING. This is unrelated to the other misfeature of being able to block deletes by not including FILE_SHARE_DELETE.
- mehrdadn 9y ago"Reachable" doesn't mean "openable". Reachable just means there is a path that the system identifies the file with. There are files you cannot open but which are nevertheless reachable by path. Lots of reasons can exist for this and a pending delete is just one of them. Others can include having wrong permissions or being special files (e.g. hiberfil.sys or even $MFTMirr).
- asveikau 9y agoI would be kind of surprised if "the system" cares much about the name of a delete pending file. NT philosophy is to discard the name as soon as possible and work with handles. I was under the impression that ntfs.sys only has this behavior because older filesystems led everybody to expect it.
- mehrdadn 9y agoWell if you look at the scenario you described, I don't believe the parent folder can be deleted while the child is pending deletion. And if the system crashes, I'd expect the file to be there (but haven't tested). So the path components do have to be kept around somewhere...
- asveikau 9y agoIt's true that NT won't let you remove a directory if a child has a handle open. But I suspect you are getting the reasoning backwards. The directory is not empty as long as that delete pending file is there. Remove this ill-conceived implementation detail (and it is that) then this and other problems go away. There is also an API that retrieves a filename from a handle. I don't think it guarantees the name be usable though. It's easy to imagine a system that works the way I would have it, because it exists: Unix. You can unlink and keep descriptors open. NT is very close to being there too, except for these goofy quirks which are kind of artificial.