5 ms·

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary pla

This means that like a regular app, you lose data if the app crashes or there is a power loss. It's much better to save after each operation in a temporary place (probably in ~/.local/share/application/yourapp, using XDG directories), and when the user clicks save, just copy the file into the desired location. That way, if there is a power loss and you reopen the app, it opens right back where it was doing (losing maybe he last few seconds of changes, but not all unsaved data)

avereveard 3y ago
Good ole .filename.swp
Someone 3y ago
> and when the user clicks save, just copy the file into the desired location. To be perfectly safe, you want to rename it, not copy it. If there’s a power loss during copying, you may endcup with corrupted data. Renaming is, to coin a phrase, “more atomic” than copying (on Linux, the OS says it is atomic. ISO C says it, too, but POSIX doesn’t (https://pubs.opengroup.org/onlinepubs/000095399/functions/rename.html https://pubs.opengroup.org/onlinepubs/000095399/functions/re...: “This rename() function is equivalent for regular files to that defined by the ISO C standard. Its inclusion here expands that definition to include actions on directories and specifies behavior when the new parameter names a file that already exists. That specification requires that the action of the function be atomic”) Also, filesystems may have bugs, hardware may lie about syncing to disk, and network shares can be finicky. Doing this properly isn’t as easy as one would think. You’ve to make sure to sync the file to be written and you’ll have to handle the case where the save location is on a different file system than your temporary file. If so, you’ll have to create a copy on that file system first. I think many tools do not check whether they need to work cross filesystem and just write their scratch files to the save directory with a different name and then rename them. Of course, that means you always need twice the disk space on the target disk to do a save. That used to be a problem almost everywhere, but nowadays mostly is restricted to embedded systems and USB sticks. In this case, however, SQLite will do a lot for you, and probably better than you would do it. It claims (https://www.sqlite.org/atomiccommit.html#_multi_file_commit https://www.sqlite.org/atomiccommit.html#_multi_file_commit): “SQLite allows a single database connection to talk to two or more database files simultaneously through the use of the ATTACH DATABASE command. When multiple database files are modified within a single transaction, all files are updated atomically. In other words, either all of the database files are updated or else none of them are. Achieving an atomic commit across multiple database files is more complex that doing so for a single file. This section describes how SQLite works that bit of magic.” However, about VACUUM INTO, it says (https://www.sqlite.org/lang_vacuum.html https://www.sqlite.org/lang_vacuum.html): “The VACUUM INTO command is transactional in the sense that the generated output database is a consistent snapshot of the original database. However, if the VACUUM INTO command is interrupted by an unplanned shutdown or power lose, then the generated output database might be incomplete and corrupt. Also, SQLite does not invoke fsync() or FlushFileBuffers() on the generated database to ensure that it has reached non-volatile storage before completing.” So, I don’t think doing “VACUUM INTO” is sufficient to guarantee that you get a good copy of your data on disk.
- cduzz 3y ago
  Well, copying is simply not atomic in linux; directory entry operations (rename, link, unlink) are atomic. There is a definition somewhere that says how many bytes may be written atomically; that's it -- past that writes are not atomic. The prior comment of "model user interactions in the database" seems spot on -- just keep track of what the user's doing as unsaved data in the database and commit it (in the appropriate way) to the DB as it happens; save is just another user action. Presumably sqlite has figured out how to write to the filesystem without corrupting itself even in a variety of adverse scenarios? If not, commit to a temporary copy of the DB that gets renamed to the "main" name periodically or when the app closes. There's an xzzzbit meme there somewhere, yo.
  - naniwaduni 3y ago
    > The prior comment of "model user interactions in the database" seems spot on -- just keep track of what the user's doing as unsaved data in the database and commit it (in the appropriate way) to the DB as it happens; save is just another user action. The trouble is that often in the wild, the content of the file on the filesystem is a user-facing interface. Users will copy it around and attach the whole document onto emails. When they do, they do not expect the file to contain data that they didn't want to save. (Yes, they sometimes also expect the file to contain data that they wanted to but didn't explicitly save. This is not a contradiction.)
    - cduzz 3y ago
      That's certainly true, and also the case in many modern file formats. For example there's the story of the academic studies with falsified data [1] where forensics on the included excel documents showed they'd gone through and replaced data with "randomized" data (if I remember correctly); there are tons of examples of "redacted" data in pdf docs being visible under the blacked-out rectangles. I'm not disagreeing with you, btw, such actions are certainly problematic, but hopefully kids will grow up knowing they need to run an export to sanitize their data if they don't want to show the whole world their transaction logs... [1] https://www.npr.org/transcripts/1190568472 https://www.npr.org/transcripts/1190568472
scherlock 3y ago
If you have a db, why not just model it as unsaved data? I.e. all changes get stored to the db, but have a flag of unsaved. If you open up a file and there are unsaved changes, you can prompt the user to either make them saved or discard them.
- mort96 3y ago
  That feels like it requires the data model to be very different? The file format would essentially need to be a list of changes, with a "committed" flag. Like, if someone changes some text in a paragraph, you can't just model that as "this paragraph now contains this new text". You have to model it as "this paragraph used to contain this text, but an uncommitted change changed it to this other text". User deletes an image? You have to still store the image and all the references to it, but with an uncommitted change to delete the image and remove the references to it. And maybe that's a good thing, maybe a git-like system where the history of every change is tracked is what you want. But it certainly doesn't feel like it'd be appropriate for every application and file format.
  - julesnp 3y ago
    You wouldn't necessarily need to track every change, you could just have 2 tables, one which contains the last "saved" version of the document, and one which contains the last modified version of the document. Upon opening after a crash, if there is a more recent modified version, the program will ask if you want to load that version.
    - mixmastamyk 3y ago
      Databases handle all this natively with transactions and WALs. i.e. Don't need to build a Flintstones version yourself. Also binary documents are a lousy fit with git, smashing square peg into round hole makes little sense.
SanderNL 3y ago
You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.
- josephg 3y ago
  Users shouldn’t ever need to adapt to computer crashes like this. Software should always auto save or have recovery files or something. As a principle, software should hold anything a user inputs with reverence.
  - SanderNL 3y ago
    I agree. Maybe as a dev I've become cynical and don't trust anything. Least of all some app holding my document. Makes me think of the "Voting software" xkcd: https://xkcd.com/2030 https://xkcd.com/2030 "I don't quite know how to put this, but our entire field is bad at what we do, and if you rely on us, everyone will die."
    - belenos46 3y ago
      You think that's cynical? I must be some avatar curmudgeonliness then. I'm pretty sure that what's actually going on is that everything we think of as a 'profession' is that way, and that people are in fact dying because of it. The difference is that devs are honest about it.
  - mixmastamyk 3y ago
    Yes, however you have to be sure every single program across decades is written with those rules in mind. Hard on any OS, but a lost cause on */Linux.
- chrisshroba 3y ago
  I used to have that instinct but lost it in the age of auto save. The applications (web or native) I use most often all do it for me: Google docs, Dropbox paper, notion, vscode. I don’t think I’m alone in this!
p4bl0 3y ago
Yes, I am aware of that, and you are right about this in general. In my particular case however, it is preferable to loose some work in the rare cases were a crash occurs than to have a copy of the file in some place that the users are not aware of. Of course if crashes were frequent the trade-off would be different.

avereveard 3y ago

Good ole .filename.swp

Someone 3y ago

> and when the user clicks save, just copy the file into the desired location. To be perfectly safe, you want to rename it, not copy it. If there’s a power loss during copying, you may endcup with corrupted data. Renaming is, to coin a phrase, “more atomic” than copying (on Linux, the OS says it is atomic. ISO C says it, too, but POSIX doesn’t (https://pubs.opengroup.org/onlinepubs/000095399/functions/rename.html https://pubs.opengroup.org/onlinepubs/000095399/functions/re...: “This rename() function is equivalent for regular files to that defined by the ISO C standard. Its inclusion here expands that definition to include actions on directories and specifies behavior when the new parameter names a file that already exists. That specification requires that the action of the function be atomic”) Also, filesystems may have bugs, hardware may lie about syncing to disk, and network shares can be finicky. Doing this properly isn’t as easy as one would think. You’ve to make sure to sync the file to be written and you’ll have to handle the case where the save location is on a different file system than your temporary file. If so, you’ll have to create a copy on that file system first. I think many tools do not check whether they need to work cross filesystem and just write their scratch files to the save directory with a different name and then rename them. Of course, that means you always need twice the disk space on the target disk to do a save. That used to be a problem almost everywhere, but nowadays mostly is restricted to embedded systems and USB sticks. In this case, however, SQLite will do a lot for you, and probably better than you would do it. It claims (https://www.sqlite.org/atomiccommit.html#_multi_file_commit https://www.sqlite.org/atomiccommit.html#_multi_file_commit): “SQLite allows a single database connection to talk to two or more database files simultaneously through the use of the ATTACH DATABASE command. When multiple database files are modified within a single transaction, all files are updated atomically. In other words, either all of the database files are updated or else none of them are. Achieving an atomic commit across multiple database files is more complex that doing so for a single file. This section describes how SQLite works that bit of magic.” However, about VACUUM INTO, it says (https://www.sqlite.org/lang_vacuum.html https://www.sqlite.org/lang_vacuum.html): “The VACUUM INTO command is transactional in the sense that the generated output database is a consistent snapshot of the original database. However, if the VACUUM INTO command is interrupted by an unplanned shutdown or power lose, then the generated output database might be incomplete and corrupt. Also, SQLite does not invoke fsync() or FlushFileBuffers() on the generated database to ensure that it has reached non-volatile storage before completing.” So, I don’t think doing “VACUUM INTO” is sufficient to guarantee that you get a good copy of your data on disk.

cduzz 3y ago
Well, copying is simply not atomic in linux; directory entry operations (rename, link, unlink) are atomic. There is a definition somewhere that says how many bytes may be written atomically; that's it -- past that writes are not atomic. The prior comment of "model user interactions in the database" seems spot on -- just keep track of what the user's doing as unsaved data in the database and commit it (in the appropriate way) to the DB as it happens; save is just another user action. Presumably sqlite has figured out how to write to the filesystem without corrupting itself even in a variety of adverse scenarios? If not, commit to a temporary copy of the DB that gets renamed to the "main" name periodically or when the app closes. There's an xzzzbit meme there somewhere, yo.
- naniwaduni 3y ago
  > The prior comment of "model user interactions in the database" seems spot on -- just keep track of what the user's doing as unsaved data in the database and commit it (in the appropriate way) to the DB as it happens; save is just another user action. The trouble is that often in the wild, the content of the file on the filesystem is a user-facing interface. Users will copy it around and attach the whole document onto emails. When they do, they do not expect the file to contain data that they didn't want to save. (Yes, they sometimes also expect the file to contain data that they wanted to but didn't explicitly save. This is not a contradiction.)
  - cduzz 3y ago
    That's certainly true, and also the case in many modern file formats. For example there's the story of the academic studies with falsified data [1] where forensics on the included excel documents showed they'd gone through and replaced data with "randomized" data (if I remember correctly); there are tons of examples of "redacted" data in pdf docs being visible under the blacked-out rectangles. I'm not disagreeing with you, btw, such actions are certainly problematic, but hopefully kids will grow up knowing they need to run an export to sanitize their data if they don't want to show the whole world their transaction logs... [1] https://www.npr.org/transcripts/1190568472 https://www.npr.org/transcripts/1190568472

scherlock 3y ago

If you have a db, why not just model it as unsaved data? I.e. all changes get stored to the db, but have a flag of unsaved. If you open up a file and there are unsaved changes, you can prompt the user to either make them saved or discard them.

mort96 3y ago
That feels like it requires the data model to be very different? The file format would essentially need to be a list of changes, with a "committed" flag. Like, if someone changes some text in a paragraph, you can't just model that as "this paragraph now contains this new text". You have to model it as "this paragraph used to contain this text, but an uncommitted change changed it to this other text". User deletes an image? You have to still store the image and all the references to it, but with an uncommitted change to delete the image and remove the references to it. And maybe that's a good thing, maybe a git-like system where the history of every change is tracked is what you want. But it certainly doesn't feel like it'd be appropriate for every application and file format.
- julesnp 3y ago
  You wouldn't necessarily need to track every change, you could just have 2 tables, one which contains the last "saved" version of the document, and one which contains the last modified version of the document. Upon opening after a crash, if there is a more recent modified version, the program will ask if you want to load that version.
  - mixmastamyk 3y ago
    Databases handle all this natively with transactions and WALs. i.e. Don't need to build a Flintstones version yourself. Also binary documents are a lousy fit with git, smashing square peg into round hole makes little sense.

SanderNL 3y ago

You are right and like you explained this is trivially easily fixed by autosaving regularly. What I have trouble imagining is people working with documents on computers for more than a few years yet somehow failing to develop the Always Save Instinct. I regularly catch myself saving unreasonably often.

josephg 3y ago
Users shouldn’t ever need to adapt to computer crashes like this. Software should always auto save or have recovery files or something. As a principle, software should hold anything a user inputs with reverence.
- SanderNL 3y ago
  I agree. Maybe as a dev I've become cynical and don't trust anything. Least of all some app holding my document. Makes me think of the "Voting software" xkcd: https://xkcd.com/2030 https://xkcd.com/2030 "I don't quite know how to put this, but our entire field is bad at what we do, and if you rely on us, everyone will die."
  - belenos46 3y ago
    You think that's cynical? I must be some avatar curmudgeonliness then. I'm pretty sure that what's actually going on is that everything we think of as a 'profession' is that way, and that people are in fact dying because of it. The difference is that devs are honest about it.
- mixmastamyk 3y ago
  Yes, however you have to be sure every single program across decades is written with those rules in mind. Hard on any OS, but a lost cause on */Linux.
chrisshroba 3y ago
I used to have that instinct but lost it in the age of auto save. The applications (web or native) I use most often all do it for me: Google docs, Dropbox paper, notion, vscode. I don’t think I’m alone in this!

p4bl0 3y ago

Yes, I am aware of that, and you are right about this in general. In my particular case however, it is preferable to loose some work in the rare cases were a crash occurs than to have a copy of the file in some place that the users are not aware of. Of course if crashes were frequent the trade-off would be different.