41 ms·

> My "favorite" is the inability to delete a file that is open by another process Because of this Linux "feature" I experienced data loss - I zipped a file and

by 323 4y ago

> My "favorite" is the inability to delete a file that is open by another process Because of this Linux "feature" I experienced data loss - I zipped a file and deleted it, but the process which created it was still writing to it. But all the data that it wrote now ended up in the void since the file didn't exist anymore. And there was no error/warning that the process was writing to a non-linked file, so I discovered the data loss much later.

codedokode 4y ago
Because the file was actually existing, but removed from directory list.
- iso1210 4y ago
  And you can still access it while it's open from the /proc directory
P5fRxh5kUvp2th 4y ago
you CAN lock files under linux, it's just not the default.
- shakow 4y ago
  But locks are suggestive, not mandatory. So if the other process does not take them into account, they just have no effect.
  - chasil 4y ago
    Mandatory locking has profound security impacts. I can update glibc or other core system libraries on Linux without rebooting. Processes using the old library will show "(deleted)" in their /proc/self/maps files, but they will continue to use the old code. New processes will use the new library, and downtime was not required. This is not possible on Windows because of mandatory locking on core system libraries. As a consequence, Windows must reboot for every Patch Tuesday. There are only a few places that Linux has mandatory locking, and this is a good thing. Here is a script that will show you all the deleted libraries that are still in use on Linux: # cat stale_libraries.sh #!/bin/sh awk '$NF=="(deleted)" && $(NF-1) ~ /[.]so/ { sub(/;.*$/,"",$(NF-1)) print $(NF-1), $NF}' /proc/*/maps | sort -u Here is a script that will show you all PIDs that are running with deleted libraries: # cat stale_programs.sh #!/bin/sh grep '[.]so.*deleted)' /proc/*/maps | sed 's/[:][^/]*//' | sort -u | while read -r line do pid=${line#/} pid=${pid#*/} pid=${pid%%/*} xargs -0 printf '%s ' < "/proc/$pid/cmdline" | sed 's/[ ]$//' printf :%s\\n "$line" done Some of the maps files can only be read by root.
    - 323 4y ago
      I don't know about Windows core system libraries, but for user libraries you can do the same thing - rename old library, and put new library in it's place. Running processes will continue using the old library, and new ones will pick up the new library. The only difference is that instead of deleting the old library you need to rename it (and then delete it later). Linux doesn't need reboots only in theory. I have an Ubuntu server box and not a week passes in which I don't see a "system reboot is required" prompt when I SSH into it.
      - chasil 4y ago
        Right - POSIX is generally "advisory locking" rather than "mandatory locking," so an update process is free to overwrite any library. Windows does not allow this, which is why we have Patch Tuesday outages. Also, Oracle KSplice was the first free tool to apply updates to a running Linux kernel without rebooting. KSplice is free on Ubuntu. I think other free services have become available since. https://ksplice.oracle.com/try/desktop https://ksplice.oracle.com/try/desktop
      - P5fRxh5kUvp2th 4y ago
        then stop updating the kernel once/week. There are mechanisms for updating the kernel in-place, and I believe canonical is one of the leaders in that domain, but if you're choosing not to use it and you don't want to reboot once/week, you can still keep your system level libraries up to date without rebooting. The above poster didn't say you could update absolutely everything in linux without a reboot, just that the locking mechanisms in windows means you have to reboot to update things that don't require a reboot in linux.
        323 4y ago
        I didn't choose anything, it's the default Ubuntu 22 image from a big cloud provider. They enabled auto-updates, including for the kernel. They didn't enable whatever update kernel in-place mechanism Ubuntu has. I assume they know what they're doing. My point was that in theory Linux doesn't need reboots and Windows does, but in practice my Ubuntu box needs a weekly reboot, while my Windows box just once a month.
Karellen 4y ago
But if the zip process had happened a bit quicker, the delete would have worked on either platform. Getting the error on Windows would only have been due to a race you could never guarantee would work out in your favour. So, you experienced data loss because you deleted a file you didn't mean to. ...and?
- 323 4y ago
  I don't understand what race on Windows, when you open a file for writing it's (typically) protected against deletion until you close it. In the example I gave, the other process was continuously writing to this file while I was zipping it.
  - Karellen 4y ago
    The race is between the process writing the zip file, and you trying to delete it. If the process creating the zip file runs faster, and finishes, and closes the file, your attempt to delete it would have worked anyway. Because the program that created it didn't have it open any more.
    - 323 4y ago
      The program which created the file was running all the time, before I created the zip, during the zip creation, after the zip creation ended, and after I deleted the file. The zip creation and me deleting the file were in parallel with another process writing to the file. And I didn't delete the zip file, I deleted the original file that I zipped.
      - trelane 4y ago
        > I didn't delete the zip file, I deleted the original file that I zipped. This is why you should always check the contents of your archive before deleting the originals. There are myriad reasons why the archive may be incomplete or incorrect. You were hit by one of them that happens to not happen on Windows.
      - 4y ago
Bilal_io 4y ago
This is hardly Linux's fault, nor its responsibility. As far as I know Linux gives you mechanism to lock files and directories, so the compression program you used should have locked the directory by default until it was done or canceled. And if that option was not the default in the program you've used, it could be present as an optional flag. Someone with more knowledge please correct me if I got the above wrong. That being said, the issue with Windows is that it prevents me from deleting a file if it's open in a text editor or any program, which should not be the default in my opinion. A directory gets locked simply because you have explorer open in that path, or a terminal cd'd into that path, which is annoying and counter productive most of the time.
- 323 4y ago
  > so the compression program you used The standard Linux compression programs, zip, gzip, tar, don't lock by default. I don't even think they have such an option.
- cesarb 4y ago
  > As far as I know Linux gives you mechanism to lock files and directories, so the compression program you used should have locked the directory by default until it was done or canceled. File locking on Linux is advisory, it excludes other processes which try to lock the same region of the same file, but not operations other than locking. Mandatory locks were optional (required a mount option, see https://lwn.net/Articles/667210/ https://lwn.net/Articles/667210/ for details), and were removed in 5.15 (https://lwn.net/Articles/874493/ https://lwn.net/Articles/874493/).
  - Bilal_io 4y ago
    Thank you for the links. I did a bit of reading to learn a bit and my understanding of the advisory file locking is that it can be ignored by programs that don't respect it. Is it fair to say that the the shell or gui the user used should have respected the advisory lock? Of course that's assuming the compression program used does attempt to apply file lock, which may not be the case according to another comment.
trelane 4y ago
How did you lose data? It looks like it was WAI. You were writing to a file and deleted it. How did you expect that to go down?
- 323 4y ago
  I was expecting that the process writing to the now deleted file would throw some kind of error/exception that the file doesn't exist anymore. I would have noticed that. But instead it silently continued writing into the void.
  - trelane 4y ago
    I still don't understand the loss. The original data that was going into the zip was still there. Or did you the write pipeline delete it at the end of something? More precisely, what data was lost, and how did it get lost?
    - scbrg 4y ago
      There are three processes involved. Process P, the data producer, writes to file foo. Process Z, the zip-process, reads file foo and produces foo.zip Process rm, removes file foo. I imagine it went a bit like this: P --output foo & Z < foo > foo.zip rm foo # time passes # process P terminates Any data written to foo since Z was run is now lost. User 323 assumed that either rm would fail, letting them know that foo cannot be deleted, or that P would throw an error when the file was deleted. (I don't think this is a reasonable expectation, and it's a failure on 323's hand of not learning the UNIX file model, but there's the situation).
      - trelane 4y ago
        In this case, it would have been deleted anyway. I think I can see the loss: 1. Start zip. 2. In parallel, copy the intermediate zip file. 3. Delete the zip 4. Delete originals The data was deleted in step 4 and is the loss. It notably would also had occurred had the zip terminated early or not zipped _all_ the files for some reason. The process was not resilient.
Tobu 4y ago
I don't see why you'd blame Linux for deleting a file you asked it to. However, if the process is sufficiently long-running, it is possible to relink the file by looking inside `/proc/$(pgrep yourprocess)`.
pdntspa 4y ago
Sounds like a pretty good feature to me. Annoying, for sure, but also a safeguard that prevents some weird instability.
elzbardico 4y ago
You can open files non-exclusively on windows. If your antivirus doesn't do it, it is not a problem with the operating system but with the application. Frankly, the windows behavior, while annoying in some situations because of badly behaved applications, is much more logical to me and avoids issues like what you described here. People are used to Unix. I know because I am too; the only non-linux or mac os machines I have at home are the windows laptops of my kids. But yet, on this regard, I think that the windows behavior, while annoying because of badly written apps, it the best approach.
- ethbr0 4y ago
  The problem with Windows is that most applications, including Microsoft's own app-level stuff, are badly behaving. And Microsoft's been trying to manage that insanity since 1995.

by 323 4y ago

codedokode 4y ago

Because the file was actually existing, but removed from directory list.

iso1210 4y ago
And you can still access it while it's open from the /proc directory

P5fRxh5kUvp2th 4y ago

you CAN lock files under linux, it's just not the default.

shakow 4y ago
But locks are suggestive, not mandatory. So if the other process does not take them into account, they just have no effect.
- chasil 4y ago
  Mandatory locking has profound security impacts. I can update glibc or other core system libraries on Linux without rebooting. Processes using the old library will show "(deleted)" in their /proc/self/maps files, but they will continue to use the old code. New processes will use the new library, and downtime was not required. This is not possible on Windows because of mandatory locking on core system libraries. As a consequence, Windows must reboot for every Patch Tuesday. There are only a few places that Linux has mandatory locking, and this is a good thing. Here is a script that will show you all the deleted libraries that are still in use on Linux: # cat stale_libraries.sh #!/bin/sh awk '$NF=="(deleted)" && $(NF-1) ~ /[.]so/ { sub(/;.*$/,"",$(NF-1)) print $(NF-1), $NF}' /proc/*/maps | sort -u Here is a script that will show you all PIDs that are running with deleted libraries: # cat stale_programs.sh #!/bin/sh grep '[.]so.*deleted)' /proc/*/maps | sed 's/[:][^/]*//' | sort -u | while read -r line do pid=${line#/} pid=${pid#*/} pid=${pid%%/*} xargs -0 printf '%s ' < "/proc/$pid/cmdline" | sed 's/[ ]$//' printf :%s\\n "$line" done Some of the maps files can only be read by root.
  - 323 4y ago
    I don't know about Windows core system libraries, but for user libraries you can do the same thing - rename old library, and put new library in it's place. Running processes will continue using the old library, and new ones will pick up the new library. The only difference is that instead of deleting the old library you need to rename it (and then delete it later). Linux doesn't need reboots only in theory. I have an Ubuntu server box and not a week passes in which I don't see a "system reboot is required" prompt when I SSH into it.
    - chasil 4y ago
      Right - POSIX is generally "advisory locking" rather than "mandatory locking," so an update process is free to overwrite any library. Windows does not allow this, which is why we have Patch Tuesday outages. Also, Oracle KSplice was the first free tool to apply updates to a running Linux kernel without rebooting. KSplice is free on Ubuntu. I think other free services have become available since. https://ksplice.oracle.com/try/desktop https://ksplice.oracle.com/try/desktop
    - P5fRxh5kUvp2th 4y ago
      then stop updating the kernel once/week. There are mechanisms for updating the kernel in-place, and I believe canonical is one of the leaders in that domain, but if you're choosing not to use it and you don't want to reboot once/week, you can still keep your system level libraries up to date without rebooting. The above poster didn't say you could update absolutely everything in linux without a reboot, just that the locking mechanisms in windows means you have to reboot to update things that don't require a reboot in linux.
      - 323 4y ago
        I didn't choose anything, it's the default Ubuntu 22 image from a big cloud provider. They enabled auto-updates, including for the kernel. They didn't enable whatever update kernel in-place mechanism Ubuntu has. I assume they know what they're doing. My point was that in theory Linux doesn't need reboots and Windows does, but in practice my Ubuntu box needs a weekly reboot, while my Windows box just once a month.

Karellen 4y ago

But if the zip process had happened a bit quicker, the delete would have worked on either platform. Getting the error on Windows would only have been due to a race you could never guarantee would work out in your favour. So, you experienced data loss because you deleted a file you didn't mean to. ...and?

323 4y ago
I don't understand what race on Windows, when you open a file for writing it's (typically) protected against deletion until you close it. In the example I gave, the other process was continuously writing to this file while I was zipping it.
- Karellen 4y ago
  The race is between the process writing the zip file, and you trying to delete it. If the process creating the zip file runs faster, and finishes, and closes the file, your attempt to delete it would have worked anyway. Because the program that created it didn't have it open any more.
  - 323 4y ago
    The program which created the file was running all the time, before I created the zip, during the zip creation, after the zip creation ended, and after I deleted the file. The zip creation and me deleting the file were in parallel with another process writing to the file. And I didn't delete the zip file, I deleted the original file that I zipped.
    - trelane 4y ago
      > I didn't delete the zip file, I deleted the original file that I zipped. This is why you should always check the contents of your archive before deleting the originals. There are myriad reasons why the archive may be incomplete or incorrect. You were hit by one of them that happens to not happen on Windows.
    - 4y ago

Bilal_io 4y ago

This is hardly Linux's fault, nor its responsibility. As far as I know Linux gives you mechanism to lock files and directories, so the compression program you used should have locked the directory by default until it was done or canceled. And if that option was not the default in the program you've used, it could be present as an optional flag. Someone with more knowledge please correct me if I got the above wrong. That being said, the issue with Windows is that it prevents me from deleting a file if it's open in a text editor or any program, which should not be the default in my opinion. A directory gets locked simply because you have explorer open in that path, or a terminal cd'd into that path, which is annoying and counter productive most of the time.

323 4y ago
> so the compression program you used The standard Linux compression programs, zip, gzip, tar, don't lock by default. I don't even think they have such an option.
cesarb 4y ago
> As far as I know Linux gives you mechanism to lock files and directories, so the compression program you used should have locked the directory by default until it was done or canceled. File locking on Linux is advisory, it excludes other processes which try to lock the same region of the same file, but not operations other than locking. Mandatory locks were optional (required a mount option, see https://lwn.net/Articles/667210/ https://lwn.net/Articles/667210/ for details), and were removed in 5.15 (https://lwn.net/Articles/874493/ https://lwn.net/Articles/874493/).
- Bilal_io 4y ago
  Thank you for the links. I did a bit of reading to learn a bit and my understanding of the advisory file locking is that it can be ignored by programs that don't respect it. Is it fair to say that the the shell or gui the user used should have respected the advisory lock? Of course that's assuming the compression program used does attempt to apply file lock, which may not be the case according to another comment.

trelane 4y ago

How did you lose data? It looks like it was WAI. You were writing to a file and deleted it. How did you expect that to go down?

323 4y ago
I was expecting that the process writing to the now deleted file would throw some kind of error/exception that the file doesn't exist anymore. I would have noticed that. But instead it silently continued writing into the void.
- trelane 4y ago
  I still don't understand the loss. The original data that was going into the zip was still there. Or did you the write pipeline delete it at the end of something? More precisely, what data was lost, and how did it get lost?
  - scbrg 4y ago
    There are three processes involved. Process P, the data producer, writes to file foo. Process Z, the zip-process, reads file foo and produces foo.zip Process rm, removes file foo. I imagine it went a bit like this: P --output foo & Z < foo > foo.zip rm foo # time passes # process P terminates Any data written to foo since Z was run is now lost. User 323 assumed that either rm would fail, letting them know that foo cannot be deleted, or that P would throw an error when the file was deleted. (I don't think this is a reasonable expectation, and it's a failure on 323's hand of not learning the UNIX file model, but there's the situation).
    - trelane 4y ago
      In this case, it would have been deleted anyway. I think I can see the loss: 1. Start zip. 2. In parallel, copy the intermediate zip file. 3. Delete the zip 4. Delete originals The data was deleted in step 4 and is the loss. It notably would also had occurred had the zip terminated early or not zipped _all_ the files for some reason. The process was not resilient.

Tobu 4y ago

I don't see why you'd blame Linux for deleting a file you asked it to. However, if the process is sufficiently long-running, it is possible to relink the file by looking inside `/proc/$(pgrep yourprocess)`.

pdntspa 4y ago

Sounds like a pretty good feature to me. Annoying, for sure, but also a safeguard that prevents some weird instability.

elzbardico 4y ago

You can open files non-exclusively on windows. If your antivirus doesn't do it, it is not a problem with the operating system but with the application. Frankly, the windows behavior, while annoying in some situations because of badly behaved applications, is much more logical to me and avoids issues like what you described here. People are used to Unix. I know because I am too; the only non-linux or mac os machines I have at home are the windows laptops of my kids. But yet, on this regard, I think that the windows behavior, while annoying because of badly written apps, it the best approach.

ethbr0 4y ago
The problem with Windows is that most applications, including Microsoft's own app-level stuff, are badly behaving. And Microsoft's been trying to manage that insanity since 1995.