13 ms·

Please don’t. When people are sending or sharing a document, they don’t expect to leak edit history of that document. And SQLite inherently leaks that edit his

Please don’t. When people are sending or sharing a document, they don’t expect to leak edit history of that document. And SQLite inherently leaks that edit history, in database freed pages and unallocated space. You can VACUUM the database after each save, but that’ll negate the performance benefits. Also VACUUM was designed to save space, not sure it reliably cleans up unwanted data. OTOH, these zipped XMLs are free from unwanted data, they (hopefully) only contain the data user wants to save, i.e. the content of the last version of the document.

chj 9y ago
People should publish PDF version of a document. But if they send the working copy around, then the history may be helpful for other collaborators. Think of git. They can even merge the changes easily.
- aaronchall 9y ago
  Agreed. History should be considered a feature.
  - joosters 9y ago
    And it is. That's fine, as long as the user actively turns that feature on, and by default no data is leaked.
- kbenson 9y ago
  Unfortunately, companies don't do that. They send the doc/docx. They also send the xls/xlsx instead of the csv. Protecting users from accidentally doing a very wrong thing is better than being slightly more helpful as a default. Security should be the default. IIRC there have been cases where confidential information has been found by looking through the history in government released documents.
  - Const-me 9y ago
    docx/xlsx are better, they are zipped XMLs too, just like odf. Older ones, doc & xls are closer to sqlite, for the same reason, performance. I remember finding confidential information in these files received from third parties they probably didn’t want to disclose, e.g. offer price that was before the last edit.
  - bigiain 9y ago
    You know you can pop calc with a csv file opened in Excel? from http://georgemauer.net/2017/10/07/csv-injection.html http://georgemauer.net/2017/10/07/csv-injection.html : UserId,BillToDate,ProjectName,Description,DurationMinutes 1,2017-07-25,Test Project,Flipped the jibbet,60 2,2017-07-25,Important Client,"Bop, dop, and giglip", 240 2,2017-07-25,Important Client,"=2+5+cmd|' /C calc'!A0", 240 Yeah - that "=2+5+cmd|' \c calc'!A0" gets Windows to execute code...
- gumby 9y ago
  > People should publish PDF version of a document. Oh please no! PDF is quite user-unfriendly (e.g. can't reflow, typically can't cut/paste properly) designed to emulate an ancient medium that has benefits and drawbacks different from screens. And PDF, while executable, is essentially compiler output, with all the destruction of semantics that implies.
  - abtinf 9y ago
    It depends on what your are trying to do. Short of exporting to an image format, PDF is the best format for preserving and sharing a final rendered output. All of the things you list as drawbacks are virtues if your purpose is static presentation.
  - pjmlp 9y ago
    I only have issues with ebooks, outside them it is just perfect. Cut/paste only fails on me, when whoever created the PDF had the brilliant idea of using images instead of actual text, e.g. scans. Something that using another format won't help. Maybe you should use another PDF reader?
  - Fnoord 9y ago
    For a resume, PDF is ace. I don't want the reader to see version history or copy/paste (even though its peanuts with a half decent PDF reader). I want them to print it.
SQLite 9y ago
The "PRAGMA secure_delete=ON;" command will fix that for you. Or you can make secure-delete the default at compile-time using the -DSQLITE_SECURE_DELETE option. There is a (small) performance hit, since extra writes have to occur to overwrite the obsolete data with zeros, but it does work. See https://www.sqlite.org/pragma.html#pragma_secure_delete https://www.sqlite.org/pragma.html#pragma_secure_delete for further information.
- Const-me 9y ago
  Not enough. See my next comment: https://news.ycombinator.com/item?id=15607650 https://news.ycombinator.com/item?id=15607650
Const-me 9y ago
P.S. Even fragmentation can leak data. E.g. search for the most fragmented portion of the document in the DB, and you'll likely find out the most heavily edited paragraph of the document.
- JohnBooty 9y ago
  I believe that you raise excellent concerns. I also believe the solutions are perhaps just a bit of thoughtful UI design away. The options could be something like: ----Performance & Privacy----- Document Save Mode [x] Balanced. Document history is not shared. (Default) [ ] Fastest. Not recommended for sensitive data [ ] Maximum privacy. Slowest, but most secure Click for more information on what these choices mean The first option would zero out previous edits. This would be the default. This would still be faster than re-zipping the entire archive -- you'd only be zero'ing out a single file. Metadata could still be inferred via fragmentation as you say, but I do not believe this is a concern for most use cases. The second option would optimize purely for speed. The third option would optimize purely for privacy. Perhaps the entire document database would be rewritten from scratch on every save. This probably wouldn't be too much slower than the current XML dance. I understand the value of this kind of "paranoid" mode but I think it's overkill for most situations. Personally, the situations where I'd want this kind of behavior are extremely rare. Clearly some people would need/want this and that's fine; it would be only a click away. And then all future documents could inherit this choice until the app is told otherwise.
  - panic 9y ago
    This UI solves a problem for the computer, not the user. Whenever you find yourself making a UI like this, you should go back and change your program so the computer doesn't have that problem any more. In this case, there's surely a way to write files quickly enough while still maintaining user privacy -- just do that instead of bothering me with a wall of text every time I save a file!
    - JohnBooty 9y ago
      just do that instead of bothering me with a wall of text every time I save a file! I certainly wouldn't want to be hit with this wall of text every time I save a file. I was thinking that this sort of option ought to be a global setting that could perhaps be altered on a per-document basis if (and only if) the user cares to muck about with it. It could, perhaps, be a choice that is presented to the user upon new document creation. This UI solves a problem for the computer, not the user. [...] In this case, there's surely a way to write files quickly enough while still maintaining user privacy Life is full of these choices. There's a gas pedal in my car. I can mash it (to go fast but burn a lot of fuel) or be gentle with it (and save gas, but I won't go as fast). Ideally the car should be crazy fast and crazy safe and crazy fuel efficient no matter how you drive the sucker, but that's a tough engineering challenge and ultimately we (today) leave that choice to the drive. That said, you are 100% correct. Maybe the privacy/paranoia steps are something that could be done on a background thread. The file could be saved quickly, and then the database could be privacy-optimized in the background immediately thereafter.
kbenson 9y ago
I agree. People are vastly underestimating the information already exposed by people not understanding what they are sharing when they hand a modern document format out. New features need to examined for unintended consequences, and the fact that this feature might entail data leakage through fragmented included data should be a red flag.

chj 9y ago

People should publish PDF version of a document. But if they send the working copy around, then the history may be helpful for other collaborators. Think of git. They can even merge the changes easily.

aaronchall 9y ago
Agreed. History should be considered a feature.
- joosters 9y ago
  And it is. That's fine, as long as the user actively turns that feature on, and by default no data is leaked.
kbenson 9y ago
Unfortunately, companies don't do that. They send the doc/docx. They also send the xls/xlsx instead of the csv. Protecting users from accidentally doing a very wrong thing is better than being slightly more helpful as a default. Security should be the default. IIRC there have been cases where confidential information has been found by looking through the history in government released documents.
- Const-me 9y ago
  docx/xlsx are better, they are zipped XMLs too, just like odf. Older ones, doc & xls are closer to sqlite, for the same reason, performance. I remember finding confidential information in these files received from third parties they probably didn’t want to disclose, e.g. offer price that was before the last edit.
- bigiain 9y ago
  You know you can pop calc with a csv file opened in Excel? from http://georgemauer.net/2017/10/07/csv-injection.html http://georgemauer.net/2017/10/07/csv-injection.html : UserId,BillToDate,ProjectName,Description,DurationMinutes 1,2017-07-25,Test Project,Flipped the jibbet,60 2,2017-07-25,Important Client,"Bop, dop, and giglip", 240 2,2017-07-25,Important Client,"=2+5+cmd|' /C calc'!A0", 240 Yeah - that "=2+5+cmd|' \c calc'!A0" gets Windows to execute code...
gumby 9y ago
> People should publish PDF version of a document. Oh please no! PDF is quite user-unfriendly (e.g. can't reflow, typically can't cut/paste properly) designed to emulate an ancient medium that has benefits and drawbacks different from screens. And PDF, while executable, is essentially compiler output, with all the destruction of semantics that implies.
- abtinf 9y ago
  It depends on what your are trying to do. Short of exporting to an image format, PDF is the best format for preserving and sharing a final rendered output. All of the things you list as drawbacks are virtues if your purpose is static presentation.
- pjmlp 9y ago
  I only have issues with ebooks, outside them it is just perfect. Cut/paste only fails on me, when whoever created the PDF had the brilliant idea of using images instead of actual text, e.g. scans. Something that using another format won't help. Maybe you should use another PDF reader?
- Fnoord 9y ago
  For a resume, PDF is ace. I don't want the reader to see version history or copy/paste (even though its peanuts with a half decent PDF reader). I want them to print it.

SQLite 9y ago

The "PRAGMA secure_delete=ON;" command will fix that for you. Or you can make secure-delete the default at compile-time using the -DSQLITE_SECURE_DELETE option. There is a (small) performance hit, since extra writes have to occur to overwrite the obsolete data with zeros, but it does work. See https://www.sqlite.org/pragma.html#pragma_secure_delete https://www.sqlite.org/pragma.html#pragma_secure_delete for further information.

Const-me 9y ago
Not enough. See my next comment: https://news.ycombinator.com/item?id=15607650 https://news.ycombinator.com/item?id=15607650

Const-me 9y ago

P.S. Even fragmentation can leak data. E.g. search for the most fragmented portion of the document in the DB, and you'll likely find out the most heavily edited paragraph of the document.

JohnBooty 9y ago
I believe that you raise excellent concerns. I also believe the solutions are perhaps just a bit of thoughtful UI design away. The options could be something like: ----Performance & Privacy----- Document Save Mode [x] Balanced. Document history is not shared. (Default) [ ] Fastest. Not recommended for sensitive data [ ] Maximum privacy. Slowest, but most secure Click for more information on what these choices mean The first option would zero out previous edits. This would be the default. This would still be faster than re-zipping the entire archive -- you'd only be zero'ing out a single file. Metadata could still be inferred via fragmentation as you say, but I do not believe this is a concern for most use cases. The second option would optimize purely for speed. The third option would optimize purely for privacy. Perhaps the entire document database would be rewritten from scratch on every save. This probably wouldn't be too much slower than the current XML dance. I understand the value of this kind of "paranoid" mode but I think it's overkill for most situations. Personally, the situations where I'd want this kind of behavior are extremely rare. Clearly some people would need/want this and that's fine; it would be only a click away. And then all future documents could inherit this choice until the app is told otherwise.
- panic 9y ago
  This UI solves a problem for the computer, not the user. Whenever you find yourself making a UI like this, you should go back and change your program so the computer doesn't have that problem any more. In this case, there's surely a way to write files quickly enough while still maintaining user privacy -- just do that instead of bothering me with a wall of text every time I save a file!
  - JohnBooty 9y ago
    just do that instead of bothering me with a wall of text every time I save a file! I certainly wouldn't want to be hit with this wall of text every time I save a file. I was thinking that this sort of option ought to be a global setting that could perhaps be altered on a per-document basis if (and only if) the user cares to muck about with it. It could, perhaps, be a choice that is presented to the user upon new document creation. This UI solves a problem for the computer, not the user. [...] In this case, there's surely a way to write files quickly enough while still maintaining user privacy Life is full of these choices. There's a gas pedal in my car. I can mash it (to go fast but burn a lot of fuel) or be gentle with it (and save gas, but I won't go as fast). Ideally the car should be crazy fast and crazy safe and crazy fuel efficient no matter how you drive the sucker, but that's a tough engineering challenge and ultimately we (today) leave that choice to the drive. That said, you are 100% correct. Maybe the privacy/paranoia steps are something that could be done on a background thread. The file could be saved quickly, and then the database could be privacy-optimized in the background immediately thereafter.

kbenson 9y ago

I agree. People are vastly underestimating the information already exposed by people not understanding what they are sharing when they hand a modern document format out. New features need to examined for unintended consequences, and the fact that this feature might entail data leakage through fragmented included data should be a red flag.