3 ms·
> In 2022, we shouldn't be reserving any area of a file as 'spare space' like this. In 2022, we should have file formats that are not naive of the underlying f
by dotancohen 5y ago
> In 2022, we shouldn't be reserving any area of a file as 'spare space' like this.
In 2022, we should have file formats that are not naive of the underlying filesystem. I store my text notes in Git, but I also have two LibreOffice documents in there because org-mode doesn't handle their contents well. It would be nice of only 5% of the file had to be rewritten when I change a single word, rather than the current 91% of the file. A 5% hit to storage for a 91% reduction each time I update it would be a great trade off.
- nyanpasu64 5y agoOne option is to store files as a database like SQLite, which I assume is designed around not rewriting the whole file on every change. For example Audacity 3.x does this. However SQLite stores multiple files on disk for crash persistence, complicating matters. But then again so does Audacity 2, and Microsoft Office to an extent (file locking rather than data integrity). Microsoft Word used to only append changes to the document on save rather than rewriting the whole file, but this was disabled since if you deleted text and saved the file, it remained in the document. https://www.cnet.com/culture/microsoft-disabling-word-2003s-fast-save-feature/ https://www.cnet.com/culture/microsoft-disabling-word-2003s-...
- EricE 5y agoModern file systems address all of that and more - without the added complexity of a whole freaking RDBMS system! If you are going to change an application it makes far more sense to just update it to leverage the features of a modern file system.
- coder543 5y agoUnder the hood, odt and docx files are zip files.[0] The “91%” change has nothing to do with rewriting the file; it mostly has to do with zip files and how poorly git handles binary diffing. Plaintext files are rewritten on disk every time you save them in virtually every editor and that doesn’t cause problems for git. I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again. Alternatively, you could just switch to a text-based format like rtf, as long as you don’t need any specific features from odt or docx. (EDIT: someone else mentions a promising sounding “flat xml” format, which I’ve never encountered before.) [0]: https://superuser.com/a/1356829 https://superuser.com/a/1356829
- 0xffff2 5y ago>I’ve never tried this, so take it with a heap of salt, but you might have better luck with the diffing if you developed a process that unzipped the file contents into your repo after you’re done editing it, and then zipped the contents back up with the right extension when you want to edit it again. I have a current project where one of the other developers has tried this. It might work for a solo project, but with multiple developers on Windows (Excel) and Linux (Open Office), I find that Open Office feels a need to continually fuck with unrelated fields. Editing a single field in an xslx file results in changes in a dozen files in the unpacked version. Making any more complicated changes results in a diff that's impossible to manage.
- coder543 5y agoDo you actually mean OpenOffice, or do you mean LibreOffice? I would hope everyone is using the much more improved LibreOffice by now instead of OpenOffice. Still, interesting to hear some feedback on how well (or not) that idea has worked for some. The Flat XML option seems to be the best approach if you need to edit documents more complex than what can be represented in either markdown or rtf (rich text format).
- sjhuang26 5y agoIn your case, LibreOffice allows you to save the file as Flat XML (e.g., .fodt), and the document will be written as a single uncompressed file. Unfortunately, few apps have a feature like this.
- ktpsns 5y agoWow, thanks for the Flat OpenDocument file type! Such kind of files are much better to treat with git (i.e. human readable diffs, for instance). However, unfortunately, these files tend to get very large (200kB ODS vs. 7MB FODS). What a pity the XML itself is not less verbose.
- lelanthran 5y agoI dunno if that will help. I know Dia lets you store the uncompressed XML, but I have seen it sometimes have a 100% diff due to the serialiser moving *everything* around when all that happened was that I rearranged half the blocks on the diagram.
- dotancohen 5y agoThis is perfect for my use case, thank you! The .fodt file is 150K, vs 117K for the .odt file. But after changing a word less than 20 of the 1983 lines in the file have changed. $ ls -hl todo.fodt -rw-rw-r-- 1 dotancohen dotancohen 150K Apr 7 16:41 todo.fodt $ ls -hl todo.odt -rwxrwxr-x 1 dotancohen dotancohen 117K Apr 6 12:01 todo.odt $ wc -l todo.fodt 1983 todo.fodt $ cp todo.fodt todo-edit-one-line.fodt $ libreoffice todo-edit-one-line.fodt $ diff todo.fodt todo-2.fodt | grep "+ " | wc -l 19