5 ms·

Why do you use a secondary, volatile database ? Performance-wise you won't gain a lot more (we're talking about a user editing a file, so not even 1 write per s

by rakoo 3y ago

Why do you use a secondary, volatile database ? Performance-wise you won't gain a lot more (we're talking about a user editing a file, so not even 1 write per second). A proposal: write directly, and automatically in the database. No more Save button. There are multiple advantages: - the system is crash-resistant. I like taking the approach of CouchDB where the only correct way to close the system is to crash it. That way a crash is an expected situation that you actually account for, not a special case that you might forget - there is only one database. Less code, fewer bugs. - it is safe. A write to SQLite works or doesn't work, there is no in-between. As said in the VACUUM doc you point to: "However, if the VACUUM INTO command is interrupted by an unplanned shutdown or power lose, then the generated output database might be incomplete and corrupt" - it is how SQLite was intended to work. And because of that, you won't have to think about it for the lifetime of SQLite

knome 3y ago
a save button is still good, as it allows you to keep specific checkpoints. but the save button could simply tag specific save points in a larger table. if the format can roll up changes to compress them, they also indicate where which variants need to be kept indefinitely.
- rakoo 3y ago
  In this view the save action is more like a commit, where the user manually checkpoints and also offers a simple description of why this is an important point. But in my view all intermediary points also need to be saved, because the user might have forgotten to explicitly checkpoint, and might still want some undo/redo capability that is more granular than just checkpoints.
- theamk 3y ago
  Auto-save is nice, but I think it works much better when it's a separate file. This way "main" document file (which might be checked in to git, or shared via dropbox or read by some document) only contains nice, clean, saved version. And yet if your computer crashes for whatever reason, the data is still not lost and can be trivially recovered.
fluidcruft 3y ago
There is nothing I hate more than an app that modifies files secretly when I open them. Then I have to get all defensive to copy files before I open them to keep them intact. You may not see the problem with changing the checksum or hash of a file, but silently tampering with files is a nightmare in many domains. If you open a file and accidentally change something trivial (some apps like to store things like presentation state i. e. window positions, last page viewed, zoom level, ...) For example in many regulated domains such as human subjects research files must be approved and only approved files may be used. "Is this version of the consent document the version that the IRB approved?" Well let's see... (1) file modification date is after the approval date and (2) checksums do not match. Not to mention that writing a single byte of content to a filesystems marks the entire blob as needing backup. The fact is the filesystem is the user's database, save is commit, and it should be under the users control because application developers do not have the faintest idea about user context.
- rakoo 3y ago
  You are right, this is a use case I absolutely did not take into account, but I want to separate user-defined actions and app-defined actions. A level of zoom is something a user does to read a document, but I wouldn't consider that as data to be persisted automatically, unlike characters typed or a font chosen. I value the idea of persisting it, but that would be a user-specific action ("Save view" or something like that) In the case of checksums in a database, that is why read-only modes should be used and I don't see what automatically saving would change. If anything, when the user zooms on a document in read-only mode, either it shouldn't be stored or storing it should trigger the same flow as modifying the document
  - fluidcruft 3y ago
    I see what your are saying, but in other use cases the presentation state needs to be considered as part of the document. This is one of the reasons zip/jar containers work somewhat well. You can audit different chunks of data separately and cryptographically sign them. sqlite actually has an archive format[1] that is interesting to think about and I have pondered using it for some applications (store the files and also store tables of metadata/analysis) [1] https://sqlite.org/sqlar.html https://sqlite.org/sqlar.html
p4bl0 3y ago
> Why do you use a secondary, volatile database ? For the exact reason I gave in the comment you are replying to: I want to keep a usual workflow for users. Principle of least surprise. Users are okay with change being autosaved when there is a single "thing" that can be edited to the point that you don't even have to open it, it's just there, it can be seen as a property (as in ownership) of the application more than of the user. For example, your music library in your jukebox application. On the contrary, when the user have to open the "thing" with your application and can choose between many of their files that can be edited with your application, users do not expect their files to be automatically modified at all. For example users may start doing some heavy editing and then at the moment of saving their work, they might make a backup of the previous state file before saving, or choose to "save as…" in order to keep the old version just in case. Crashes are not something that happen that often. It can become an actual problem when you have tens of thousands of users and rare-events do happen, but in the particular case of the application I working on, I do not actually have to worry about that (on the contrary, any solution would have downsides that are worst in the particular case of this application than having to do some work again because of a crash if it ever happens).
- rakoo 3y ago
  You can keep a distinction between old and new version inside the same database, by having a pointer to the "current" version, and updating the pointer when clicking on "save". You could store all changes in the database in a "staging area", such that when you reopen the app you can load the changes and you don't need a recovery phase, but with the "save" button active meaning that something changed since last save.
  - p4bl0 3y ago
    I could, but as you said in your previous comment (emphase is mine): > Less code, fewer bugs.
- infogulch 3y ago
  Another option is to explicitly start in read-only mode (modification buttons hidden / grayed out, some distinct mode indicator "Viewing Document" next to a button to "Start Editing", etc), and when the user chooses then switch into autosave mode. At this point many users are used to autosave and don't pay due attention to the document saved state. With Microsoft Windows' habit of rebooting your system overnight without your explicit permission, I'm concerned that this might lead to a lot of lost work.
asalahli 3y ago
> I like taking the approach of CouchDB where the only correct way to close the system is to crash it. The term you're looking for is (aptly named) crash-only software.[0] 0. https://en.m.wikipedia.org/wiki/Crash-only_software https://en.m.wikipedia.org/wiki/Crash-only_software

by rakoo 3y ago

knome 3y ago

a save button is still good, as it allows you to keep specific checkpoints. but the save button could simply tag specific save points in a larger table. if the format can roll up changes to compress them, they also indicate where which variants need to be kept indefinitely.

rakoo 3y ago
In this view the save action is more like a commit, where the user manually checkpoints and also offers a simple description of why this is an important point. But in my view all intermediary points also need to be saved, because the user might have forgotten to explicitly checkpoint, and might still want some undo/redo capability that is more granular than just checkpoints.
theamk 3y ago
Auto-save is nice, but I think it works much better when it's a separate file. This way "main" document file (which might be checked in to git, or shared via dropbox or read by some document) only contains nice, clean, saved version. And yet if your computer crashes for whatever reason, the data is still not lost and can be trivially recovered.

fluidcruft 3y ago

There is nothing I hate more than an app that modifies files secretly when I open them. Then I have to get all defensive to copy files before I open them to keep them intact. You may not see the problem with changing the checksum or hash of a file, but silently tampering with files is a nightmare in many domains. If you open a file and accidentally change something trivial (some apps like to store things like presentation state i. e. window positions, last page viewed, zoom level, ...) For example in many regulated domains such as human subjects research files must be approved and only approved files may be used. "Is this version of the consent document the version that the IRB approved?" Well let's see... (1) file modification date is after the approval date and (2) checksums do not match. Not to mention that writing a single byte of content to a filesystems marks the entire blob as needing backup. The fact is the filesystem is the user's database, save is commit, and it should be under the users control because application developers do not have the faintest idea about user context.

rakoo 3y ago
You are right, this is a use case I absolutely did not take into account, but I want to separate user-defined actions and app-defined actions. A level of zoom is something a user does to read a document, but I wouldn't consider that as data to be persisted automatically, unlike characters typed or a font chosen. I value the idea of persisting it, but that would be a user-specific action ("Save view" or something like that) In the case of checksums in a database, that is why read-only modes should be used and I don't see what automatically saving would change. If anything, when the user zooms on a document in read-only mode, either it shouldn't be stored or storing it should trigger the same flow as modifying the document
- fluidcruft 3y ago
  I see what your are saying, but in other use cases the presentation state needs to be considered as part of the document. This is one of the reasons zip/jar containers work somewhat well. You can audit different chunks of data separately and cryptographically sign them. sqlite actually has an archive format[1] that is interesting to think about and I have pondered using it for some applications (store the files and also store tables of metadata/analysis) [1] https://sqlite.org/sqlar.html https://sqlite.org/sqlar.html

p4bl0 3y ago

> Why do you use a secondary, volatile database ? For the exact reason I gave in the comment you are replying to: I want to keep a usual workflow for users. Principle of least surprise. Users are okay with change being autosaved when there is a single "thing" that can be edited to the point that you don't even have to open it, it's just there, it can be seen as a property (as in ownership) of the application more than of the user. For example, your music library in your jukebox application. On the contrary, when the user have to open the "thing" with your application and can choose between many of their files that can be edited with your application, users do not expect their files to be automatically modified at all. For example users may start doing some heavy editing and then at the moment of saving their work, they might make a backup of the previous state file before saving, or choose to "save as…" in order to keep the old version just in case. Crashes are not something that happen that often. It can become an actual problem when you have tens of thousands of users and rare-events do happen, but in the particular case of the application I working on, I do not actually have to worry about that (on the contrary, any solution would have downsides that are worst in the particular case of this application than having to do some work again because of a crash if it ever happens).

rakoo 3y ago
You can keep a distinction between old and new version inside the same database, by having a pointer to the "current" version, and updating the pointer when clicking on "save". You could store all changes in the database in a "staging area", such that when you reopen the app you can load the changes and you don't need a recovery phase, but with the "save" button active meaning that something changed since last save.
- p4bl0 3y ago
  I could, but as you said in your previous comment (emphase is mine): > Less code, fewer bugs.
infogulch 3y ago
Another option is to explicitly start in read-only mode (modification buttons hidden / grayed out, some distinct mode indicator "Viewing Document" next to a button to "Start Editing", etc), and when the user chooses then switch into autosave mode. At this point many users are used to autosave and don't pay due attention to the document saved state. With Microsoft Windows' habit of rebooting your system overnight without your explicit permission, I'm concerned that this might lead to a lot of lost work.

asalahli 3y ago

> I like taking the approach of CouchDB where the only correct way to close the system is to crash it. The term you're looking for is (aptly named) crash-only software.[0] 0. https://en.m.wikipedia.org/wiki/Crash-only_software https://en.m.wikipedia.org/wiki/Crash-only_software