8 ms·

Hi, I’m one of the Firefox developers who was in charge of Session Restore, so I’m one of the culprits of this heavy SSD I/O. To make a long story short: we are

by Yoric 10y ago

Hi, I’m one of the Firefox developers who was in charge of Session Restore, so I’m one of the culprits of this heavy SSD I/O. To make a long story short: we are aware of the problem, but fixing it for real requires completely re-architecturing Session Restore. That’s something we haven’t done yet, as Session Restore is rather safety-critical for many users, so this would need to be done very carefully, and with plenty of manpower. I hope we can get around to doing it someday. Of course, as usual in an open-source project, contributors welcome :)

nograpes 10y ago
What would an updated version of Session Restore look like? I mean, what would be your vision for a new version?
- Yoric 10y ago
  I haven't worked on Session Restore in some time. But from the top of my head, it would be as follows. On the back-end, I believe that Session Restore should be backed by a database, with each tab updated independently, rather than a big bunch of JSON data. The rationale being that: - it's performance-critical; - it's safety-critical; - we have users with 300+ Mb of data in their Session Restore and JSON isn't meant for this scale of data; - we wouldn't need to rewrite x Mb of data every 15 seconds, just a per-tab update; - if we're using a relational database, it would be easier to trust the code to not screw up with the data; - we wouldn't need to load the entire Session Restore upon startup. (On the minus side, this might make backups a bit more complicated.) On the front-end, we would need a high-level API for Session Restore, which would let us do things such as accessing per-tab data, (de)hibernating tabs, etc. Oh, and it would need to be accessible by WebExtensions. In the middle, we would need to re-engineer Session Restore to make sure that we don't need to maintain this huge object representing the entire state of the session. We would also need improvements e.g. to cookie management, to avoid having to re-collect cookies so often.
  - Dylan16807 10y ago
    How does that much data get into Session Restore? Mine are only about 3KB per tab, and Firefox starts melting at more than a thousand tabs.
    - Yoric 10y ago
      I seem to remember that DOM Session Storage and Session Cookies are pretty big offenders. Other websites (e.g. Google Docs) use lots and lots and lots of hidden forms to remember lots and lots and lots of data. Also, people who keep Session Restore tabs open (you know, the tab that lets you restore your session, when you have crashed) and continue browsing – Firefox needs to store several nested Session Restore JSON files.
      - db48x 10y ago
        I've always been impressed by that, because it's essentially bullet-proof.
        alphapapa 10y ago
        I wish it were, but every now and then, I restart Firefox and get empty, title-only tabs that go to about:blank when I reload them, completely losing the tab. I have to use a Python script to dump the session data (which still appears in the JSON file, even though Firefox won't use it), eyeball the list to figure out which tab was broken, and reopen that URL manually.
  - alphapapa 10y ago
    Using a SQLite database for Session Restore seems to make sense. However, it would make it even more difficult to get data out of it when necessary. As ugly as a mess of JSON is, at least it's text, and I don't have to write a SQL query to grep for URLs. Why not just use the filesystem? Imagine a layout like this: + profilename.default |--+ sessionrestore |--+ <timestamp> |--+ window1 | |--+ tab1 | | |--- cookies | | |--- formdata | | |--- title | | |--- url | ---+ tab2 | |--- cookies | |--- formdata | |--- title | |--- url ---+ window2 |--+ tab1 | |--- cookies | |--- formdata | |--- title | |--- url ---+ tab2 |--- cookies |--- formdata |--- title |--- url One directory per timestamp, containing one directory per window, containing one directory per tab, containing one file per data type. All plain-text. Change a tab, just write to that tab's files, and update the timestamp directory name. Every x minutes, make a new timestamp tree, keeping y old ones as backups. No serializing and writing entire sessions to disk at once. Only small writes to small files. All plain-text, easy to read outside of the browser. Easy to copy, backup, modify, troubleshoot. No complex JSON, serializing, or database code. Use the write-to-temp-file-then-rename-over-existing-files paradigm to get atomic updates. Need to know if part of a session on disk is stale? Check the mtime for that file, see if it's older than the last time that data was changed in memory. Simple stuff. Straightforward. No overengineering. No bloat. Why not do this?
brador 10y ago
> Session Restore is rather safety-critical for many users Maybe you could give users the option? (until it can be fixed fully)
- gcp 10y ago
  That's what the interval config setting is for?
  - adam12 10y ago
    I think brador is referring to the options page. about:config shows a big warning message "This may void your warranty".
    - gcp 10y ago
      Which warranty?
      - Yoric 10y ago
        Have you looked at about:config recently? :)
        gcp 10y ago
        Woosh :-)
fulafel 10y ago
How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. (I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code). Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards.
- gcp 10y ago
  You also need to handle all corner cases where one of the intermediary diffs gets corrupted (you can't generally assume in a program like Firefox that data you write out is going to be readable in the future because lol common hardware). Or where the diffs are larger than a fresh snapshot. And you absolutely can't get it wrong. It's something that sounds easy until you actually try to get it coded up and shipping. I explained below that this thing isn't a factor for Android because the program gets suspended.
  - fulafel 10y ago
    Firefox does generally assume that the filesystem is reliable (as long as you use fsync properly etc). Witness eg its reliance on SQLite for data storage.
    - gcp 10y ago
      SQLite is used exactly because it is so robust even on bad hardware. And you still need to check the data that SQLite is giving you.
      - cbhl 10y ago
        Until you have a NFS homedir...
        derefr 10y ago
        On the other hand, SQLite with concurrent local and SSHFS-remote updates (where the SSHFS server is run with caching disabled) is entirely stable.
chmike 10y ago
Why writing to disk if firefox is idle and its state doesn't change ? Is it repeatedly rewriting the same data over and over in a storage working like a log database or journal file ?
- Yoric 10y ago
  Unless there is a bug, Session Restore is not written if Firefox is idle. Of course, many applications are not idle even if you're not using them, and with the current architecture, just one tab using, say, DOM Storage, or updating a session cookie or a hidden form field, is enough to force a rewrite.
  - dewiz 10y ago
    As an heavy user of session restore and tons of open tabs, I would be OK ignoring updates to tabs that have not been active for a while. For example, let's say I have 10 tabs open but I have been using only the latest tab during the last 5 minutes. In this scenario I don't care about the cookies (eg due to ongoing ajax) nor the state of the other 9 tabs. If the browser crashes I'm OK with those 9 tabs being restored with a 5 minutes old snapshot. So for instance FF should be smart to save the state of new/recent tabs, and should slow down progressively on old/inactive tabs.
    - posterboy 10y ago
      don't generalize your personal use case
deleted 10y ago
[deleted]
markcerqueira 10y ago
Why not leave the existing code (SR1) as is and rebuild Session Restore 2 (SR2); keep using SR1 until it has nothing to restore and then use SR2 moving forward.
- Yoric 10y ago
  That sounds pretty hard to pull off. What's the expected benefit?
  - Drdrdrq 10y ago
    That's a pretty standard M.O. for server-side software development. But in the case of desktop software such as Firefox I see no gain.
asimuvPR 10y ago
Long time Netscape/ firefox user here: 1. Thanks for your work. 2. How do I really contribute? Could you link to what I need to do to start working on this now? I'm affected by this problem ame want to figure out if I can fix it.
- holygoat 10y ago
  http://whatcanidoformozilla.org http://whatcanidoformozilla.org is a good starting point.
  - labster 10y ago
    "Please enable JavaScript in your browser!" That advice was a lot simpler than I thought it would be.
darkhorn 10y ago
How can I disable it completely?
- bhrgunatha 10y ago
  http://kb.mozillazine.org/Browser.sessionstore.enabled http://kb.mozillazine.org/Browser.sessionstore.enabled
makomk 10y ago
Session Restore is already a little buggy. Better than it used to be certainly, but it occasionally and apparently irreversably turns the odd window into a window full of empty New Tabs rather than their original contents.
xorblurb 10y ago
OTOH SSD lifetime is rather safety-critical for many users...
- cwillu 10y ago
  _Safety critical_? I doubt that...
gcb0 10y ago
link to bugzilla?
vsespb 10y ago
That's not SSD-only problem. HDD affected too. FF UI is just freezing when using HDD (linux; tried WD RE4, 7200rpm or WD blue 7200 rpm), several years ago I wrote script for myself to keep profile data on RAM disk and sync back to HDD, and problem gone.
pllbnk 10y ago
I think you are safe to rewrite it. It is not very reliable anyway as I have lost my tabs after crashes more than once. It might have been possible to save them but after a crash first thing I would do is restart Firefox and it would sometimes just completely overwrite the recovery.js. To mitigate this I have created a daemon to copy the file to a separate directory every few hours. I then delete those old versions manually every year or so.
- alphapapa 10y ago
  I have had the same problem. I finally made a git repo in the sessionrestore directory, setup a cronjob to commit it every 5 minutes, and then use this script to dump the session contents when necessary: https://gist.github.com/alphapapa/bc7f3b25025fb99dad56 https://gist.github.com/alphapapa/bc7f3b25025fb99dad56 The git repo grows in size rapidly, but can be compressed way down. (Today it was 4.3 GB, but "git gc" compressed it to 230 MB). Every now and then you can blow away the repo or filter-branch to get rid of outdated sessions.

by Yoric 10y ago

nograpes 10y ago

What would an updated version of Session Restore look like? I mean, what would be your vision for a new version?

Yoric 10y ago
I haven't worked on Session Restore in some time. But from the top of my head, it would be as follows. On the back-end, I believe that Session Restore should be backed by a database, with each tab updated independently, rather than a big bunch of JSON data. The rationale being that: - it's performance-critical; - it's safety-critical; - we have users with 300+ Mb of data in their Session Restore and JSON isn't meant for this scale of data; - we wouldn't need to rewrite x Mb of data every 15 seconds, just a per-tab update; - if we're using a relational database, it would be easier to trust the code to not screw up with the data; - we wouldn't need to load the entire Session Restore upon startup. (On the minus side, this might make backups a bit more complicated.) On the front-end, we would need a high-level API for Session Restore, which would let us do things such as accessing per-tab data, (de)hibernating tabs, etc. Oh, and it would need to be accessible by WebExtensions. In the middle, we would need to re-engineer Session Restore to make sure that we don't need to maintain this huge object representing the entire state of the session. We would also need improvements e.g. to cookie management, to avoid having to re-collect cookies so often.
- Dylan16807 10y ago
  How does that much data get into Session Restore? Mine are only about 3KB per tab, and Firefox starts melting at more than a thousand tabs.
  - Yoric 10y ago
    I seem to remember that DOM Session Storage and Session Cookies are pretty big offenders. Other websites (e.g. Google Docs) use lots and lots and lots of hidden forms to remember lots and lots and lots of data. Also, people who keep Session Restore tabs open (you know, the tab that lets you restore your session, when you have crashed) and continue browsing – Firefox needs to store several nested Session Restore JSON files.
    - db48x 10y ago
      I've always been impressed by that, because it's essentially bullet-proof.
      - alphapapa 10y ago
        I wish it were, but every now and then, I restart Firefox and get empty, title-only tabs that go to about:blank when I reload them, completely losing the tab. I have to use a Python script to dump the session data (which still appears in the JSON file, even though Firefox won't use it), eyeball the list to figure out which tab was broken, and reopen that URL manually.
- alphapapa 10y ago
  Using a SQLite database for Session Restore seems to make sense. However, it would make it even more difficult to get data out of it when necessary. As ugly as a mess of JSON is, at least it's text, and I don't have to write a SQL query to grep for URLs. Why not just use the filesystem? Imagine a layout like this: + profilename.default |--+ sessionrestore |--+ <timestamp> |--+ window1 | |--+ tab1 | | |--- cookies | | |--- formdata | | |--- title | | |--- url | ---+ tab2 | |--- cookies | |--- formdata | |--- title | |--- url ---+ window2 |--+ tab1 | |--- cookies | |--- formdata | |--- title | |--- url ---+ tab2 |--- cookies |--- formdata |--- title |--- url One directory per timestamp, containing one directory per window, containing one directory per tab, containing one file per data type. All plain-text. Change a tab, just write to that tab's files, and update the timestamp directory name. Every x minutes, make a new timestamp tree, keeping y old ones as backups. No serializing and writing entire sessions to disk at once. Only small writes to small files. All plain-text, easy to read outside of the browser. Easy to copy, backup, modify, troubleshoot. No complex JSON, serializing, or database code. Use the write-to-temp-file-then-rename-over-existing-files paradigm to get atomic updates. Need to know if part of a session on disk is stale? Check the mtime for that file, see if it's older than the last time that data was changed in memory. Simple stuff. Straightforward. No overengineering. No bloat. Why not do this?

brador 10y ago

> Session Restore is rather safety-critical for many users Maybe you could give users the option? (until it can be fixed fully)

gcp 10y ago
That's what the interval config setting is for?
- adam12 10y ago
  I think brador is referring to the options page. about:config shows a big warning message "This may void your warranty".
  - gcp 10y ago
    Which warranty?
    - Yoric 10y ago
      Have you looked at about:config recently? :)
      - gcp 10y ago
        Woosh :-)

fulafel 10y ago

How about this: 29 out of 30 times, save only a diff to the previous data. 1 out of 30 times, save the complete data in compressed form. (I'm guessing there must already be functionality to diff a bunch of JSON somewhere in the millions of lines of code). Though I'm sure this doesn't make usually make a dent in a SSD's lifetime. But there are still people running Firefox on low end Android phones with meager flash, and Raspberry Pis with SD cards.

gcp 10y ago
You also need to handle all corner cases where one of the intermediary diffs gets corrupted (you can't generally assume in a program like Firefox that data you write out is going to be readable in the future because lol common hardware). Or where the diffs are larger than a fresh snapshot. And you absolutely can't get it wrong. It's something that sounds easy until you actually try to get it coded up and shipping. I explained below that this thing isn't a factor for Android because the program gets suspended.
- fulafel 10y ago
  Firefox does generally assume that the filesystem is reliable (as long as you use fsync properly etc). Witness eg its reliance on SQLite for data storage.
  - gcp 10y ago
    SQLite is used exactly because it is so robust even on bad hardware. And you still need to check the data that SQLite is giving you.
    - cbhl 10y ago
      Until you have a NFS homedir...
      - derefr 10y ago
        On the other hand, SQLite with concurrent local and SSHFS-remote updates (where the SSHFS server is run with caching disabled) is entirely stable.

chmike 10y ago

Why writing to disk if firefox is idle and its state doesn't change ? Is it repeatedly rewriting the same data over and over in a storage working like a log database or journal file ?

Yoric 10y ago
Unless there is a bug, Session Restore is not written if Firefox is idle. Of course, many applications are not idle even if you're not using them, and with the current architecture, just one tab using, say, DOM Storage, or updating a session cookie or a hidden form field, is enough to force a rewrite.
- dewiz 10y ago
  As an heavy user of session restore and tons of open tabs, I would be OK ignoring updates to tabs that have not been active for a while. For example, let's say I have 10 tabs open but I have been using only the latest tab during the last 5 minutes. In this scenario I don't care about the cookies (eg due to ongoing ajax) nor the state of the other 9 tabs. If the browser crashes I'm OK with those 9 tabs being restored with a 5 minutes old snapshot. So for instance FF should be smart to save the state of new/recent tabs, and should slow down progressively on old/inactive tabs.
  - posterboy 10y ago
    don't generalize your personal use case

deleted 10y ago

[deleted]

markcerqueira 10y ago

Why not leave the existing code (SR1) as is and rebuild Session Restore 2 (SR2); keep using SR1 until it has nothing to restore and then use SR2 moving forward.

Yoric 10y ago
That sounds pretty hard to pull off. What's the expected benefit?
- Drdrdrq 10y ago
  That's a pretty standard M.O. for server-side software development. But in the case of desktop software such as Firefox I see no gain.

asimuvPR 10y ago

Long time Netscape/ firefox user here: 1. Thanks for your work. 2. How do I really contribute? Could you link to what I need to do to start working on this now? I'm affected by this problem ame want to figure out if I can fix it.

holygoat 10y ago
http://whatcanidoformozilla.org http://whatcanidoformozilla.org is a good starting point.
- labster 10y ago
  "Please enable JavaScript in your browser!" That advice was a lot simpler than I thought it would be.

darkhorn 10y ago

How can I disable it completely?

bhrgunatha 10y ago
http://kb.mozillazine.org/Browser.sessionstore.enabled http://kb.mozillazine.org/Browser.sessionstore.enabled

makomk 10y ago

Session Restore is already a little buggy. Better than it used to be certainly, but it occasionally and apparently irreversably turns the odd window into a window full of empty New Tabs rather than their original contents.

xorblurb 10y ago

OTOH SSD lifetime is rather safety-critical for many users...

cwillu 10y ago
_Safety critical_? I doubt that...

gcb0 10y ago

link to bugzilla?

vsespb 10y ago

That's not SSD-only problem. HDD affected too. FF UI is just freezing when using HDD (linux; tried WD RE4, 7200rpm or WD blue 7200 rpm), several years ago I wrote script for myself to keep profile data on RAM disk and sync back to HDD, and problem gone.

pllbnk 10y ago

I think you are safe to rewrite it. It is not very reliable anyway as I have lost my tabs after crashes more than once. It might have been possible to save them but after a crash first thing I would do is restart Firefox and it would sometimes just completely overwrite the recovery.js. To mitigate this I have created a daemon to copy the file to a separate directory every few hours. I then delete those old versions manually every year or so.

alphapapa 10y ago
I have had the same problem. I finally made a git repo in the sessionrestore directory, setup a cronjob to commit it every 5 minutes, and then use this script to dump the session contents when necessary: https://gist.github.com/alphapapa/bc7f3b25025fb99dad56 https://gist.github.com/alphapapa/bc7f3b25025fb99dad56 The git repo grows in size rapidly, but can be compressed way down. (Today it was 4.3 GB, but "git gc" compressed it to 230 MB). Every now and then you can blow away the repo or filter-branch to get rid of outdated sessions.