4 ms·

> How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. > > (I'm guessing the

by Yoric 10y ago

> How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. > > (I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code). That is actually a good idea we haven't considered yet. A bit too brute force for my tastes, but relatively easy to implement. We would need to determine how much CPU is needed for a diff between two 300Mb JSON files, though (yes, some users have these). Of course, we're back to the issue of manpower, but definitely worth trying out. > Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards. The implementation of Session Restore for Android is largely independent, so I'm not sure how it works these days.

reitzensteinm 10y ago
Surely you'd diff the data structures in memory, not the serialized JSON; and would it really be faster to blindly write the 300mb each time than perform this diff?
- Yoric 10y ago
  To be benchmarked.
bsdetector 10y ago
You could at least just compress the data with LZF or any of the really fast text compressors. It'll compress to about 30% of the size, it's easy to do, and it shouldn't add more than a tiny CPU overhead over formatting the JSON itself. It solves half the problem with like 15 minutes of work.
- Yoric 10y ago
  Actually, we had a prototype doing this. In the end, we didn't because it broke some add-ons, but it might be time to try again.
  - Someone 10y ago
    Broke some add-ons that read or, worse, write data behind the browser's back, you mean? Also, what do these add-ons do? The only use case I can think of is figuring out whether the user has a tab open to a given site, and that's going around the browser's security model, so breaking that would be a good thing.
- holygoat 10y ago
  15 minutes of coding, perhaps. Then an hour of writing good tests. Then lots of manual and automated testing on four or five platforms, and fixing the weird issues you get on Windows XP SP2, or only on 64-bit Linux running in a VM, or whatever. Then making sure you don't regress startup performance (which you probably will unless you have a really, really slow disk). Then implementing rock solid migration so you can land this in Nightly through to Beta. Then a rollback/backout plan, because you might find a bug, and you don't want users to lose their session between Beta 2 and Beta 3. Large-scale software engineering is mostly not about writing code.
  - bsdetector 10y ago
    They already have Google's Brotli imported, they'd only need a small tweak to also include the encoder. Or use Snappy which is also in the codebase already. Add the code that's able to load compressed session backups and leave it in for a couple versions. Once enough versions have passed enable the code that writes compressed session backups. It's really not that hard to do unless you want to enable it now.
  - acqq 10y ago
    > regress startup performance No, for example, LZ4 is unbelievably fast: https://github.com/Cyan4973/lz4 https://github.com/Cyan4973/lz4 almost 2 GB per second in decompression! I've just tried compressing some backupXX.session file (the biggest I've managed to find, just around 2 MB) and it compressed to 70% of the original, probably not enough to implement the compression -- and I suspect the reason is that the file contains too much base64 encode image files which can't be much compressed? So the answer to having sane session files can be first to stop storing the favicons (and other images(?)) there? I still believe somebody should analyze the big files carefully to get the most relevant facts. For the start, somebody should make a tool that extracts all the pictures from the .session file (should be easy with python or Perl, for example), just that we know what's inside.
    - acqq 10y ago
      So "somebody" was me at the end, I've rolled up my sleeves and extracted the damn pictures: in my 1.7 MB session file approximately 1 MB were the base64 encodings of exactly 57 jpeg pictures that were all the result of the Google image search (probably of the single search). There were additionally a few pngs, representing "facebook, twitter, gplus" images and one "question mark sign" gif too.
acqq 10y ago
Why is there so much data in the session restore anyway? If the goal is to have the URLs of the currently opened tabs, I'd expect that just the given URLs should be enough? I think I've seen some unexpected stuff there like the images base64 encoded? Maybe there's enough users that would be satisfied just with the URLs? At least for them the "rewrite" would be seldom needed. Or, maybe to reformulate, which wild scenarios does Firefox want to support now? I can imagine that the user's experience wouldn't match the wishes. Some people that use session restore claimed they "lost everything" from time to time, and I had to fish "just the urls" from their session store files which looked strange ("full of everything"), but automatically restored to nothing.
- cpncrunch 10y ago
  Agreed. If my computer crashes (once in a blue moon) I would be happy for it to just open the urls again. I don't care whatsoever about it keeping the current state of every tab. If any users do care about that, it should be optional (perhaps a tickbox in the preferences window).
  - chime 10y ago
    My FF has crashed many times on OS X and while it does restore the URLs, it has never restored the state. I know it because sometimes I am in the middle of writing a comment and it crashes. On recovery, the comment is gone.
    - Senji 10y ago
      Get Lazarus https://addons.mozilla.org/en-US/firefox/addon/lazarus-form-recovery/ https://addons.mozilla.org/en-US/firefox/addon/lazarus-form-...
- holygoat 10y ago
  Session store contains open tabs, windows, history for each tab, form fields, referrers (so we can re-request the page correctly), titles (so we can restore tabs without re-fetching every page), favicons, charsets, some timestamps, extension data, some kinds of site storage, scroll positions, and a few other things. The goal of session restore is to restore your session -- your open tabs should come back, the same pages should load, scrolled to the same place, and with the right content.

by Yoric 10y ago

reitzensteinm 10y ago

Surely you'd diff the data structures in memory, not the serialized JSON; and would it really be faster to blindly write the 300mb each time than perform this diff?

Yoric 10y ago
To be benchmarked.

bsdetector 10y ago

You could at least just compress the data with LZF or any of the really fast text compressors. It'll compress to about 30% of the size, it's easy to do, and it shouldn't add more than a tiny CPU overhead over formatting the JSON itself. It solves half the problem with like 15 minutes of work.

Yoric 10y ago
Actually, we had a prototype doing this. In the end, we didn't because it broke some add-ons, but it might be time to try again.
- Someone 10y ago
  Broke some add-ons that read or, worse, write data behind the browser's back, you mean? Also, what do these add-ons do? The only use case I can think of is figuring out whether the user has a tab open to a given site, and that's going around the browser's security model, so breaking that would be a good thing.
holygoat 10y ago
15 minutes of coding, perhaps. Then an hour of writing good tests. Then lots of manual and automated testing on four or five platforms, and fixing the weird issues you get on Windows XP SP2, or only on 64-bit Linux running in a VM, or whatever. Then making sure you don't regress startup performance (which you probably will unless you have a really, really slow disk). Then implementing rock solid migration so you can land this in Nightly through to Beta. Then a rollback/backout plan, because you might find a bug, and you don't want users to lose their session between Beta 2 and Beta 3. Large-scale software engineering is mostly not about writing code.
- bsdetector 10y ago
  They already have Google's Brotli imported, they'd only need a small tweak to also include the encoder. Or use Snappy which is also in the codebase already. Add the code that's able to load compressed session backups and leave it in for a couple versions. Once enough versions have passed enable the code that writes compressed session backups. It's really not that hard to do unless you want to enable it now.
- acqq 10y ago
  > regress startup performance No, for example, LZ4 is unbelievably fast: https://github.com/Cyan4973/lz4 https://github.com/Cyan4973/lz4 almost 2 GB per second in decompression! I've just tried compressing some backupXX.session file (the biggest I've managed to find, just around 2 MB) and it compressed to 70% of the original, probably not enough to implement the compression -- and I suspect the reason is that the file contains too much base64 encode image files which can't be much compressed? So the answer to having sane session files can be first to stop storing the favicons (and other images(?)) there? I still believe somebody should analyze the big files carefully to get the most relevant facts. For the start, somebody should make a tool that extracts all the pictures from the .session file (should be easy with python or Perl, for example), just that we know what's inside.
  - acqq 10y ago
    So "somebody" was me at the end, I've rolled up my sleeves and extracted the damn pictures: in my 1.7 MB session file approximately 1 MB were the base64 encodings of exactly 57 jpeg pictures that were all the result of the Google image search (probably of the single search). There were additionally a few pngs, representing "facebook, twitter, gplus" images and one "question mark sign" gif too.

acqq 10y ago

Why is there so much data in the session restore anyway? If the goal is to have the URLs of the currently opened tabs, I'd expect that just the given URLs should be enough? I think I've seen some unexpected stuff there like the images base64 encoded? Maybe there's enough users that would be satisfied just with the URLs? At least for them the "rewrite" would be seldom needed. Or, maybe to reformulate, which wild scenarios does Firefox want to support now? I can imagine that the user's experience wouldn't match the wishes. Some people that use session restore claimed they "lost everything" from time to time, and I had to fish "just the urls" from their session store files which looked strange ("full of everything"), but automatically restored to nothing.

cpncrunch 10y ago
Agreed. If my computer crashes (once in a blue moon) I would be happy for it to just open the urls again. I don't care whatsoever about it keeping the current state of every tab. If any users do care about that, it should be optional (perhaps a tickbox in the preferences window).
- chime 10y ago
  My FF has crashed many times on OS X and while it does restore the URLs, it has never restored the state. I know it because sometimes I am in the middle of writing a comment and it crashes. On recovery, the comment is gone.
  - Senji 10y ago
    Get Lazarus https://addons.mozilla.org/en-US/firefox/addon/lazarus-form-recovery/ https://addons.mozilla.org/en-US/firefox/addon/lazarus-form-...
holygoat 10y ago
Session store contains open tabs, windows, history for each tab, form fields, referrers (so we can re-request the page correctly), titles (so we can restore tabs without re-fetching every page), favicons, charsets, some timestamps, extension data, some kinds of site storage, scroll positions, and a few other things. The goal of session restore is to restore your session -- your open tabs should come back, the same pages should load, scrolled to the same place, and with the right content.