6 ms·
How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. (I'm guessing there mu
by fulafel 10y ago
How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form.
(I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code).
Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards.
- gcp 10y agoYou also need to handle all corner cases where one of the intermediary diffs gets corrupted (you can't generally assume in a program like Firefox that data you write out is going to be readable in the future because lol common hardware). Or where the diffs are larger than a fresh snapshot. And you absolutely can't get it wrong. It's something that sounds easy until you actually try to get it coded up and shipping. I explained below that this thing isn't a factor for Android because the program gets suspended.
- fulafel 10y agoFirefox does generally assume that the filesystem is reliable (as long as you use fsync properly etc). Witness eg its reliance on SQLite for data storage.
- gcp 10y agoSQLite is used exactly because it is so robust even on bad hardware. And you still need to check the data that SQLite is giving you.
- cbhl 10y agoUntil you have a NFS homedir...
- derefr 10y agoOn the other hand, SQLite with concurrent local and SSHFS-remote updates (where the SSHFS server is run with caching disabled) is entirely stable.
- alphapapa 10y agoI'm curious, could you expand on this comment? Do you mean that you run Firefox from a profile loaded over SSHFS? I can only imagine that being unbearably slow because of I/O latency. And shouldn't disabling caching in SSHFS make it much, much worse?
- Dylan16807 10y agoDoes the lack of proper locking on NFS matter when you're only running a single instance of firefox?
- Dylan16807 10y agoIn the general case that's a huge pain. In this case, where you're writing a single blob to disk, you can stick a checksum at the end and you're good to go.
- gcp 10y agoIf you read the bug discussion, you'll see this is one of the most problematic parts.
- Dylan16807 10y agoThis bug? https://bugzilla.mozilla.org/show_bug.cgi?id=1304389 https://bugzilla.mozilla.org/show_bug.cgi?id=1304389 It doesn't talk about the difficulties in getting data safely to disk. There's just a worry that taking a hash of the entire session state is expensive. I'm skeptical that a fast hash would take long compared to the time spent serializing to JSON in the first place, let alone time spent diffing.
- chris_wot 10y agoAs Microsoft have discovered in their binary patches through Windows update. If you've ever run sfc /scannow and then dism /online /cleanup-image /scanhealth then you are almost certainly seeing corrupted binary patches.
- Yoric 10y ago> How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. > > (I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code). That is actually a good idea we haven't considered yet. A bit too brute force for my tastes, but relatively easy to implement. We would need to determine how much CPU is needed for a diff between two 300Mb JSON files, though (yes, some users have these). Of course, we're back to the issue of manpower, but definitely worth trying out. > Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards. The implementation of Session Restore for Android is largely independent, so I'm not sure how it works these days.
- reitzensteinm 10y agoSurely you'd diff the data structures in memory, not the serialized JSON; and would it really be faster to blindly write the 300mb each time than perform this diff?
- Yoric 10y agoTo be benchmarked.
- bsdetector 10y agoYou could at least just compress the data with LZF or any of the really fast text compressors. It'll compress to about 30% of the size, it's easy to do, and it shouldn't add more than a tiny CPU overhead over formatting the JSON itself. It solves half the problem with like 15 minutes of work.
- Yoric 10y agoActually, we had a prototype doing this. In the end, we didn't because it broke some add-ons, but it might be time to try again.
- mlvljr 10y agoLike Word did? :)