10 ms·

LocalStorage or WebSQL unexpectedly cleared

147 points by ssddanbrown 7y ago

McDev 7y ago
Ouch! A recent Reddit post immediately came to mind: https://www.reddit.com/r/androiddev/comments/ea86jt/users_reporting_wiped_data/ https://www.reddit.com/r/androiddev/comments/ea86jt/users_re...
Lex-2008 7y ago
looks like a big f_ckup... From comment #26: > why it was missed: test apps that don't happen to use local storage will not have a file created :(
- C14L 7y ago
  Shouldn't test apps test _all_ possible files? Sounds like a case of "who's testing the tests".
  - Thorrez 7y ago
    I think in this case the test coverage metric would have been handy to find things that aren't being tested.
low_key 7y ago
I was bit by this in a cordova app on a past update. After that I backed up api keys, etc to native storage using a cordova plugin and added a wrapper that will fall back to it if localstorage lookups fail. I would advise others to do the something similar since unless you are packaging the browser too (like crosswalk did), then that piece is really out of your control and you shouldn't expect it to be retained. A similar issue can be seen on iOS under low storage conditions where the operating system will clear the browser cache to reclaim space.
- webappguy 7y ago
  Which plugin?
  - martin_bech 7y ago
    If would guess the cordova sqlite plugin
  - low_key 7y ago
    This plugin has worked reliably on both Android and iOS and has a simple api: https://github.com/TheCocoaProject/cordova-plugin-nativestorage https://github.com/TheCocoaProject/cordova-plugin-nativestor...
- traspler 7y ago
  I can only recommend you switch fully to SQLite or native storage. localStorage has always been unreliable and could be cleared any time on iOS.
- the_gipsy 7y ago
  > a wrapper that will fall back to it if localstorage lookups fail. If I understand correctly, the scenario is that saved data in localStorage is lost, so that won't help.
  - makeramen 7y ago
    “It” in this case is native storage, not localStorage
    - the_gipsy 7y ago
      Yes, that was clear. You say it uses localstorage, and only fallback to native when localstorage is not available. This is not the scenario of the bug. Here, localstorage is available, it just has been wiped out.
      - 7y ago
dmitriid 7y ago
This also servers as a warning for PWAs: despite all the hype and virtues (real or perceived), your are at the hands of very fickle gods
- webappguy 7y ago
  We all are at the fikle hands of Apple and Google sadly.
- cromwellian 7y ago
  Don't you have the same problems with native? It's like like no OS update ever broke apps, or that Apple doesn't capriciously remove APIs, or reject you from the app store sometimes for unexplained reasons. Hell, people get broken by OSS ecosystem libraries a lot.
  - webappguy 7y ago
    Wish Apple and Google would put more browser collaboration time in, for the benefit of all..constant one upmanship hurts both and millions of man hours in bugs, pollyfills, API repairs, and on and on. Webview, WKWebview teams need more rigorous standards and safety checks clearly for consistency and reliability. Progressive development benefits both. Web-apps are a critical, large and important part of the app developer ecosystem.
  - dmitriid 7y ago
    Yup. In the end you are at the mercy of whoever controls the platform and its development.
  - tsimionescu 7y ago
    True to some extent, though I don't think there have been many OS updates which deleted all storage for all apps (well, moved to an inaccessible place)... This really is a massive failure of QA and dev.
    - wbl 7y ago
      A friend of mine made a backup disk for her mac case sensitive. It backed up fine, but refuses to restore.
WA 7y ago
Chrome had a similar fuckup in April 2018. They changed the minimum granted quota for IndexedDb. Suddenly, all Android devices with low free storage space (< 1GB roughly) had a granted quota of 0. This of course led to all Android WebView apps relying on IndexedDb to have no access to existing stored items and no way to save new data. It took them 6 weeks to push a fix. I used the opportunity to switch to Sqlite and can only recommend others the same.
- webappguy 7y ago
  Ftw SQLite primary with Indexed DB and LokiJS wrapper for in-mem.
- EGreg 7y ago
  How did sqlite help in this case? We are talking about the Web.
  - WA 7y ago
    We are talking about WebView apps, i. e. Cordova-based with SQLite plugin.
    - EGreg 7y ago
      But on the Web...? Your data is just lost if you saved it in ANY local container
      - Thorrez 7y ago
        This bug only affects webview, not regular websites.
- apatheticonion 7y ago
  This actually hit me really hard. I made a PWA I used to track certain stats but the data wasn't so important that it needed to be stored in a database. Instead I had an import/export for manual backups and assumed that the data inside indexedDb wouldn't be dropped. Bad assumption.
jsnell 7y ago
Just how fast are Chrome rollouts on Android? Given how destructive this appears to have been, it seems amazing that they got to a 50% rollout before the first bug report.
webappguy 7y ago
Blaming people for not using beta version for six weeks to test and inform them prior of the issue. Rich. Millions of apps affected.
- swsieber 7y ago
  That seems like an uncharitbale reading. A manager let people know that they'll be able to tell if a new chrome release will affect them by testing with beta. The overwhelming sentiment was "we're sorry".
  - ken 7y ago
    When Google introduced Chrome in 2008 (via comic!), they claimed that within 20-30 minutes of a beta build, it was automatically tested on tens of thousands of the most popular webpages. Is that no longer true? Does that not include any which use local storage?
    - pg_is_a_butt 7y ago
      They were just testing to make sure AdSense worked.
    - gsnedders 7y ago
      AFAIK it is still true, but this bug is specifically with upgrading. Basically all the testing occurs with an empty profile; migrations are specifically undertested. Also note that this bug only affected WebViews and not Chrome. Not sure how testing of migrations compares between the two.
- deleted 7y ago
  [deleted]
orf 7y ago
Not to downplay the issue, but what a ridiculous bug report: > It causes such a big problem for our products!!!! Who is in charge? I'd have expected more from a tech focused company like pocketpos.
- beart 7y ago
  This could be an exhausted Customer Service manager. They tend to be a lot less technical.
- oauea 7y ago
  In general, app developers are of surprisingly low quality To any app developers reading this, caring enough to read HN or similar puts you leagues above the kind of people I'm talking about.
kbutler 7y ago
Apple taught this lesson a few years ago, when iOS 5.1 changed the local storage location: https://phonegap.com/blog/2012/04/18/ios-5-1-and-the-embedded-uiwebview-with-cordova/ https://phonegap.com/blog/2012/04/18/ios-5-1-and-the-embedde... But this behavior is within the range of spec compliance: "User agents may, possibly in a manner configured by the user, automatically delete stored data after a period of time." https://www.w3.org/TR/webdatabase/#privacy https://www.w3.org/TR/webdatabase/#privacy https://www.w3.org/TR/webstorage/#user-tracking https://www.w3.org/TR/webstorage/#user-tracking So these browser storage mechanisms must be considered cache, not primary storage of important data.
- finaliteration 7y ago
  > So these browser storage mechanisms must be considered cache, not primary storage of important data. Maybe I’m an exception, but I’ve always viewed these storage mechanisms as unreliable. I primarily use them for storing things like user filter settings on a list view. It never even occurred to me to use them for anything critical.
r721 7y ago
Real life consequences (from comment #63): >I heard from a company that uses localstorage for offline no-connection available that had local record of animals getting vaccionation. The update "erased" all the data. They don't know which animals got vaccine and can't repeat on all of them. Serious stuff. https://bugs.chromium.org/p/chromium/issues/detail?id=1033655#c63 https://bugs.chromium.org/p/chromium/issues/detail?id=103365...
- tcd 7y ago
  You should apply a rule similar to 3-2-1: 3 copies of the data 2 copies on different storage mediums 1 backup in a separate location. Ideally, a backup solution on the OS would target the browser profile itself at regular intervals, and you could allow the user to export the data into a file that can be downloaded by the browser (or copy/pasted). It should be obvious that any form of browser storage is unreliable, and MUST be treated as such, and it should be EXPECTED to be deleted at a moments notice - setting such expectations encourages good practice at backing up your data incase of a failure. Don't rely on others to ensure business critical data is stored and accessible - that's partially on the users/application to offer what they can.
- the_gipsy 7y ago
  Sounds like it's their fault for not backing up such important information.
  - jessaustin 7y ago
    "offline no-connection available" Probably the veterinary work is not taking place at Starbucks...
megous 7y ago
Who uses localStorage for more than a local cache, or a short lived log of records to be synced somewhere into some serious database asap? That people are storing/losing: - years of financial data - animal vaccination records - etc. because of this is so weird.
- tracker1 7y ago
  Even then, it can still be used for offline data... you collect/perform tasks without an internet connection and it syncs later. Chrome updates before data sync, boom, all your data is gone. Now, as for years of data, I mostly agree, this should absolutely be backed up and/or synced more regularly.
  - curryst 7y ago
    That still seems like a poor design. This would also happen if you cleared localStorage before the data is synced. If, for example, when you came back online and were trying to do something on an unrelated site, hit a bug and tech support asked you to clear your browser. For something as critical as financial records or vaccine records, the sensible thing might be to not work in offline mode, or to persist to native storage. I think it's fine as a cache for offline data for a lot of use cases (i.e. task trackers, note apps, shopping lists, etc). But it creates an unacceptable risk for business critical data, imo.
- Exuma 7y ago
  Storing financial data in localstorage. Lol how stupid can people be
- swiley 7y ago
  Developer writes POC/Demo/quick hack without the necessary warnings. Much later a user (or less aware developer) finds the thing and starts using it. Maybe software needs GHS labels like chemicals have.
kmlx 7y ago
i still don't understand the core issue: why is anyone saving essential data to localstorage? isn't that supposed to be a state-related, somewhat temporary (due to its nature) storage in-between the UI and the API/database? don't you lose all your data when you use a different pc or smartphone? i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)? i've worked on pwa's in the past but server sync was always a priority in my algorithms. does anyone have any realistic use cases where localstorage is so essential that clearing it would somehow lose tons of data?
- capableweb 7y ago
  Depends on the context. I've built Cordova/Phonegap apps in the past where you want things to work offline and sync once the user goes online. If the time from user going offline to online again is too long / device storage too little, that data that would eventually need to be synced, would be lost. Always ways of working around it, like saving to files or whatever, but localStorage is a nice interface that works both for Cordova and client-side web apps.
  - kmlx 7y ago
    agree. for the use case that you mentioned there would be a problem. but as you said, it could be overcome by using native apis that store state. it's even imperative in case of secrets, like access tokens and the like. for client side web apps there's a bigger problem with localstorage. from my experience end users switch browsers and platforms. and they expect consistency from their web sites/apps. this expectation is much more intense than for native apps. so, while practical for developers, i'm not sure it's that welcomed in web land except for always having a local backup or as a temporary storage to alleviate network issues.
- zmzrr 7y ago
  >i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)? Yes. Why not? You can write a Cordova app that doesn't even have a backend.
erikrothoff 7y ago
This has been happening in Chrome extensions for years now. I’ve tried reporting it so many times. In our case it happens randomly so it’s impossible to reproduce, which inevitably leads to ”issue closed - couldn’t reproduce”.

147 points by ssddanbrown 7y ago

Ouch! A recent Reddit post immediately came to mind: https://www.reddit.com/r/androiddev/comments/ea86jt/users_reporting_wiped_data/ https://www.reddit.com/r/androiddev/comments/ea86jt/users_re...

Lex-2008 7y ago

looks like a big f_ckup... From comment #26: > why it was missed: test apps that don't happen to use local storage will not have a file created :(

C14L 7y ago
Shouldn't test apps test _all_ possible files? Sounds like a case of "who's testing the tests".
- Thorrez 7y ago
  I think in this case the test coverage metric would have been handy to find things that aren't being tested.

low_key 7y ago

I was bit by this in a cordova app on a past update. After that I backed up api keys, etc to native storage using a cordova plugin and added a wrapper that will fall back to it if localstorage lookups fail. I would advise others to do the something similar since unless you are packaging the browser too (like crosswalk did), then that piece is really out of your control and you shouldn't expect it to be retained. A similar issue can be seen on iOS under low storage conditions where the operating system will clear the browser cache to reclaim space.

webappguy 7y ago
Which plugin?
- martin_bech 7y ago
  If would guess the cordova sqlite plugin
- low_key 7y ago
  This plugin has worked reliably on both Android and iOS and has a simple api: https://github.com/TheCocoaProject/cordova-plugin-nativestorage https://github.com/TheCocoaProject/cordova-plugin-nativestor...
traspler 7y ago
I can only recommend you switch fully to SQLite or native storage. localStorage has always been unreliable and could be cleared any time on iOS.
the_gipsy 7y ago
> a wrapper that will fall back to it if localstorage lookups fail. If I understand correctly, the scenario is that saved data in localStorage is lost, so that won't help.
- makeramen 7y ago
  “It” in this case is native storage, not localStorage
  - the_gipsy 7y ago
    Yes, that was clear. You say it uses localstorage, and only fallback to native when localstorage is not available. This is not the scenario of the bug. Here, localstorage is available, it just has been wiped out.
    - 7y ago

dmitriid 7y ago

This also servers as a warning for PWAs: despite all the hype and virtues (real or perceived), your are at the hands of very fickle gods

webappguy 7y ago
We all are at the fikle hands of Apple and Google sadly.
cromwellian 7y ago
Don't you have the same problems with native? It's like like no OS update ever broke apps, or that Apple doesn't capriciously remove APIs, or reject you from the app store sometimes for unexplained reasons. Hell, people get broken by OSS ecosystem libraries a lot.
- webappguy 7y ago
  Wish Apple and Google would put more browser collaboration time in, for the benefit of all..constant one upmanship hurts both and millions of man hours in bugs, pollyfills, API repairs, and on and on. Webview, WKWebview teams need more rigorous standards and safety checks clearly for consistency and reliability. Progressive development benefits both. Web-apps are a critical, large and important part of the app developer ecosystem.
- dmitriid 7y ago
  Yup. In the end you are at the mercy of whoever controls the platform and its development.
- tsimionescu 7y ago
  True to some extent, though I don't think there have been many OS updates which deleted all storage for all apps (well, moved to an inaccessible place)... This really is a massive failure of QA and dev.
  - wbl 7y ago
    A friend of mine made a backup disk for her mac case sensitive. It backed up fine, but refuses to restore.

WA 7y ago

Chrome had a similar fuckup in April 2018. They changed the minimum granted quota for IndexedDb. Suddenly, all Android devices with low free storage space (< 1GB roughly) had a granted quota of 0. This of course led to all Android WebView apps relying on IndexedDb to have no access to existing stored items and no way to save new data. It took them 6 weeks to push a fix. I used the opportunity to switch to Sqlite and can only recommend others the same.

webappguy 7y ago
Ftw SQLite primary with Indexed DB and LokiJS wrapper for in-mem.
EGreg 7y ago
How did sqlite help in this case? We are talking about the Web.
- WA 7y ago
  We are talking about WebView apps, i. e. Cordova-based with SQLite plugin.
  - EGreg 7y ago
    But on the Web...? Your data is just lost if you saved it in ANY local container
    - Thorrez 7y ago
      This bug only affects webview, not regular websites.
apatheticonion 7y ago
This actually hit me really hard. I made a PWA I used to track certain stats but the data wasn't so important that it needed to be stored in a database. Instead I had an import/export for manual backups and assumed that the data inside indexedDb wouldn't be dropped. Bad assumption.

jsnell 7y ago

Just how fast are Chrome rollouts on Android? Given how destructive this appears to have been, it seems amazing that they got to a 50% rollout before the first bug report.

webappguy 7y ago

Blaming people for not using beta version for six weeks to test and inform them prior of the issue. Rich. Millions of apps affected.

swsieber 7y ago
That seems like an uncharitbale reading. A manager let people know that they'll be able to tell if a new chrome release will affect them by testing with beta. The overwhelming sentiment was "we're sorry".
- ken 7y ago
  When Google introduced Chrome in 2008 (via comic!), they claimed that within 20-30 minutes of a beta build, it was automatically tested on tens of thousands of the most popular webpages. Is that no longer true? Does that not include any which use local storage?
  - pg_is_a_butt 7y ago
    They were just testing to make sure AdSense worked.
  - gsnedders 7y ago
    AFAIK it is still true, but this bug is specifically with upgrading. Basically all the testing occurs with an empty profile; migrations are specifically undertested. Also note that this bug only affected WebViews and not Chrome. Not sure how testing of migrations compares between the two.
deleted 7y ago
[deleted]

orf 7y ago

Not to downplay the issue, but what a ridiculous bug report: > It causes such a big problem for our products!!!! Who is in charge? I'd have expected more from a tech focused company like pocketpos.

beart 7y ago
This could be an exhausted Customer Service manager. They tend to be a lot less technical.
oauea 7y ago
In general, app developers are of surprisingly low quality To any app developers reading this, caring enough to read HN or similar puts you leagues above the kind of people I'm talking about.

kbutler 7y ago

Apple taught this lesson a few years ago, when iOS 5.1 changed the local storage location: https://phonegap.com/blog/2012/04/18/ios-5-1-and-the-embedded-uiwebview-with-cordova/ https://phonegap.com/blog/2012/04/18/ios-5-1-and-the-embedde... But this behavior is within the range of spec compliance: "User agents may, possibly in a manner configured by the user, automatically delete stored data after a period of time." https://www.w3.org/TR/webdatabase/#privacy https://www.w3.org/TR/webdatabase/#privacy https://www.w3.org/TR/webstorage/#user-tracking https://www.w3.org/TR/webstorage/#user-tracking So these browser storage mechanisms must be considered cache, not primary storage of important data.

finaliteration 7y ago
> So these browser storage mechanisms must be considered cache, not primary storage of important data. Maybe I’m an exception, but I’ve always viewed these storage mechanisms as unreliable. I primarily use them for storing things like user filter settings on a list view. It never even occurred to me to use them for anything critical.

r721 7y ago

Real life consequences (from comment #63): >I heard from a company that uses localstorage for offline no-connection available that had local record of animals getting vaccionation. The update "erased" all the data. They don't know which animals got vaccine and can't repeat on all of them. Serious stuff. https://bugs.chromium.org/p/chromium/issues/detail?id=1033655#c63 https://bugs.chromium.org/p/chromium/issues/detail?id=103365...

tcd 7y ago
You should apply a rule similar to 3-2-1: 3 copies of the data 2 copies on different storage mediums 1 backup in a separate location. Ideally, a backup solution on the OS would target the browser profile itself at regular intervals, and you could allow the user to export the data into a file that can be downloaded by the browser (or copy/pasted). It should be obvious that any form of browser storage is unreliable, and MUST be treated as such, and it should be EXPECTED to be deleted at a moments notice - setting such expectations encourages good practice at backing up your data incase of a failure. Don't rely on others to ensure business critical data is stored and accessible - that's partially on the users/application to offer what they can.
the_gipsy 7y ago
Sounds like it's their fault for not backing up such important information.
- jessaustin 7y ago
  "offline no-connection available" Probably the veterinary work is not taking place at Starbucks...

megous 7y ago

Who uses localStorage for more than a local cache, or a short lived log of records to be synced somewhere into some serious database asap? That people are storing/losing: - years of financial data - animal vaccination records - etc. because of this is so weird.

tracker1 7y ago
Even then, it can still be used for offline data... you collect/perform tasks without an internet connection and it syncs later. Chrome updates before data sync, boom, all your data is gone. Now, as for years of data, I mostly agree, this should absolutely be backed up and/or synced more regularly.
- curryst 7y ago
  That still seems like a poor design. This would also happen if you cleared localStorage before the data is synced. If, for example, when you came back online and were trying to do something on an unrelated site, hit a bug and tech support asked you to clear your browser. For something as critical as financial records or vaccine records, the sensible thing might be to not work in offline mode, or to persist to native storage. I think it's fine as a cache for offline data for a lot of use cases (i.e. task trackers, note apps, shopping lists, etc). But it creates an unacceptable risk for business critical data, imo.
Exuma 7y ago
Storing financial data in localstorage. Lol how stupid can people be
swiley 7y ago
Developer writes POC/Demo/quick hack without the necessary warnings. Much later a user (or less aware developer) finds the thing and starts using it. Maybe software needs GHS labels like chemicals have.

kmlx 7y ago

i still don't understand the core issue: why is anyone saving essential data to localstorage? isn't that supposed to be a state-related, somewhat temporary (due to its nature) storage in-between the UI and the API/database? don't you lose all your data when you use a different pc or smartphone? i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)? i've worked on pwa's in the past but server sync was always a priority in my algorithms. does anyone have any realistic use cases where localstorage is so essential that clearing it would somehow lose tons of data?

capableweb 7y ago
Depends on the context. I've built Cordova/Phonegap apps in the past where you want things to work offline and sync once the user goes online. If the time from user going offline to online again is too long / device storage too little, that data that would eventually need to be synced, would be lost. Always ways of working around it, like saving to files or whatever, but localStorage is a nice interface that works both for Cordova and client-side web apps.
- kmlx 7y ago
  agree. for the use case that you mentioned there would be a problem. but as you said, it could be overcome by using native apis that store state. it's even imperative in case of secrets, like access tokens and the like. for client side web apps there's a bigger problem with localstorage. from my experience end users switch browsers and platforms. and they expect consistency from their web sites/apps. this expectation is much more intense than for native apps. so, while practical for developers, i'm not sure it's that welcomed in web land except for always having a local backup or as a temporary storage to alleviate network issues.
zmzrr 7y ago
>i see a lot of mentions of cordova apps. are people publishing apps without a backend (and save everything in localstorage)? Yes. Why not? You can write a Cordova app that doesn't even have a backend.

erikrothoff 7y ago

This has been happening in Chrome extensions for years now. I’ve tried reporting it so many times. In our case it happens randomly so it’s impossible to reproduce, which inevitably leads to ”issue closed - couldn’t reproduce”.