4 ms·

I do find it disappointing that there is no reliable local storage system for the web. I get the resistance to trackers but there should be a way to request per

I do find it disappointing that there is no reliable local storage system for the web. I get the resistance to trackers but there should be a way to request permission to store data that isn't deleted except for explicit action by users. It means that you effectively have to store the data in "the Cloud" which means 1. I have to pay for it 2. If I shut down the service you are screwed and 3. I have at least some access to it (encryption aside). I would also like to see a synced version since most browsers these days support syncing settings and passwords. But creating a generic syncing solution that is actually useful is hard.

richardwhiuk 5y ago
Can you allow the user to load and save files?
- kevincox 5y ago
  I mean sure, you can get the user to download/upload files, but this is very awkward and not suited for storing every change. (But is good for migrating and backing up the data). Having to get the user to manually break the data out of the browser is not a good UX for day-to-day work. I know that Chrome is pushing for a filesystem API but I don't know if that will be exempt from the usual ephemerality. IIRC it is just a private storage space with a filesystem-like API.
  - easrng 5y ago
    Chrome has an API that allows you to save to files without a new download every time. They made a pretty nice library that wraps that API on Chrome and it gracefully degrades on other browsers. https://web.dev/browser-fs-access/ https://web.dev/browser-fs-access/
    - jcims 5y ago
      i noticed this behavior in a drawing app called excalidraw. first save opens a file dialog, subsequent saves just update the file, basically like a standard local text editor. i keep doing 'save as' to create new files because i don't trust it lol
jefftk 5y ago
> there should be a way to request permission to store data that isn't deleted except for explicit action by users I think that's "request permission to use files on disk". This is in progress as https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API https://developer.mozilla.org/en-US/docs/Web/API/File_System..., though there's still more work before there's a version all the browsers like (Mozilla likes the ability to work with files, but thinks cross-site access should not be included: https://mozilla.github.io/standards-positions/#native-file-system https://mozilla.github.io/standards-positions/#native-file-s...)
- danShumway 5y ago
  > but it is wrapped together with aspects for which we do not think meaningful end user consent is possible to obtain (in particular cross-site access to the end user's local file system) This is a really difficult problem to solve, and I get Mozilla's hesitation. I'm also frankly very hesitant about Google leading the charge on this, not because I'm paranoid about them sneaking in tracking, just because I think Google tends to create less thoughtful web specifications sometimes. But... cross-site file access is really important for data portability and open standards, and Google's current proposal isn't bad, it might be rough but it's definitely workable. Mozilla really should try to figure out a way to move forward on this. We've seen the difference in data portability between mobile and desktop apps, and a big part of the difference between those two platforms is being able to very easily have multiple sources working on your data at the same time. Siloing data has downsides. It's tough to embrace a Unix-style philosophy without allowing programs to operate on the same data. And having Unix-style smaller webapps that work with each other is a good way of fighting against data silos and in some cases a good way of fighting against anti-user and anti-privacy services in general. I'd love to see more progress made on this, but who knows how that will work out. Caution is probably warranted for the moment, I'm just disappointed that the language suggests Mozilla would never consider a proposal that included this. It's also very important that this expose user-accessible file system access and not just a virtual filesystem in the browser; otherwise it just becomes another data-silo in the web browser. This is something that Google's proposal really gets right, and it's disappointing to see what appears to be pushback on the idea that users should be able to open up the directories that a web browser is writing to, inspect the files, and open them or move them around the filesystem, or even write to them from native apps. That to me is an essential part of the proposal.
_fat_santa 5y ago
> I do find it disappointing that there is no reliable local storage system for the web. I get the resistance to trackers but there should be a way to request permission to store data that isn't deleted except for explicit action by users. I've built a number of apps in the last year or two that use browser local storage. It also annoys me that the idea around "storing data in your browser" is automatically attributed to ad tracking. I have to go out of my way in my apps to inform users that yes this app uses localStorage/cookies, but no this is not used for any ad tracking, rather for actually storing your app's data.
- kevincox 5y ago
  How do you deal with the face that the browser may just decide to wipe the storage at any time?
  - jitl 5y ago
    The most important thing to do is get your users data somewhere safe as quickly as possible. For the vast majority of users that means your cloud database. As the user generates new data, spray it to your servers as well as writing it to the syncable IndexedDB local storage, and to an in-memory buffer. Make your backend handle writes idempotently, and retry all failures a few times. (Eg, IndexedDB disk might be full or flaking out, so retry writing the memory buffer to disk.) As long as the write path is quick, users can tolerate the browser nuking offline storage cache because they can re-download all the data that made it up to your server. Hopefully soon the browser vendors will allow more durable file system access with appropriate user controls. Chromium built out the file system access API (https://web.dev/file-system-access/ https://web.dev/file-system-access/) but it’s not supported in Firefox or Safari.
    - chrisfinazzo 5y ago
      Is this (generally) reliable? I know terrible, awful bugs eventually doomed WebSQL from getting any traction and IndexedDB seems to be a more competent replacement, but the fact that Google is leaning on FSA seems like a non-starter. It just feels like there's no way in hell Webkit will ever implement this stuff - not because of the divide between the App Store and PWA's - but due to the implications for privacy. Hard pass.
cdbattags 5y ago
Not much more to say other than Noms was my favorite project (https://github.com/attic-labs/noms https://github.com/attic-labs/noms) for a while until acquisition and the engineers are now the ones behind Replicache (https://replicache.dev/ https://replicache.dev/). I think this is going to be the next "Realm" that works everywhere.
- aboodman 5y ago
  aww, thanks.
- nacs 5y ago
  Looks like replicache is pretty expensive though. If you have more than 500 users, the price is $500/mo (and it goes up from there).
EamonnMR 5y ago
What's wrong with offering the user a file to "download" (actually creating a file on the fly) or "upload" (actually loads it into the local web app?) Edit: Here's the stack overflow copypasta I used to achieve it: https://github.com/EamonnMR/Flythrough.Space/blob/master/src/file_utils.js https://github.com/EamonnMR/Flythrough.Space/blob/master/src...
nathcd 5y ago
`navigator.storage.persist()` will prompt the user to allow persistent storage for your site, so data won't be evicted under storage pressure (for IndexedDB, service worker registrations, localStorage, sessionStorage, etc.) https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist https://developer.mozilla.org/en-US/docs/Web/API/StorageMana...
- simiones 5y ago
  That doesn't work on iOS, nor on Safari on Mac for any released version as far as I understand: https://caniuse.com/mdn-api_storagemanager_persist https://caniuse.com/mdn-api_storagemanager_persist
- marcosdumay 5y ago
  The problem runs deeper than what the GP complains about. If you rely on that, and write important things on the browser storage, a week later your user access some moronic site that doesn't work, and their support tells him to clear his browser cache, your data will almost certainly be cleaned with it. The problem is that browsers do not even consider that they may be storing important data. There is no clear way to recover or backup that data, and it is tangled with what comes from every other site.
oblib 5y ago
A few years ago a tried using the browser's IndexedDB for my invoicing app and loaded a 1000 documents into it at a time. It did ok up to about 2-3k documents but choked the browser to the point of being unusable at 5k documents. That was on a Late `09 Mac Mini so maybe newer or more powerful PCs would do better, but that's not an issue at all if you're using CouchDB to store the data on the client side. I use CouchDB installed on the client side to implement Offline First data storage. This works for Desktop PCs and if you run it on local device that's accessible via your local network, like a Raspberry Pi in-house for example, you can also use it with mobile devices in-house too. The local CouchDB will sync with the Cloud based CouchDB as soon as they're both online. CouchDB will decide which version to keep and deliver it from that point. It's certainly not perfect and doesn't provide "real time collaboration" but so far that's been out of reach anyway and may not be a very good approach at all. The notion of several people editing the same document at the same time seems to me to be chaotic no matter how you approach it. The biggest downside to this approach is that the user has to install and configure CouchDB. I made a simple web app to help with this, but it's a bit too much to expect users to install and configure it. What we need is a client side DB pre-installed that any web app can access and the data for that app is sandboxed and can only access the DB assigned to it. But it's not reasonable to add that to a web browser. CouchDB can do that now.
hinkley 5y ago
sqlite bindings in browsers came and went in the time between when I learned about them and finally found a nail that needed that hammer. I started on a design and literally within a few weeks Firefox announced it was deprecated.

richardwhiuk 5y ago

Can you allow the user to load and save files?

kevincox 5y ago
I mean sure, you can get the user to download/upload files, but this is very awkward and not suited for storing every change. (But is good for migrating and backing up the data). Having to get the user to manually break the data out of the browser is not a good UX for day-to-day work. I know that Chrome is pushing for a filesystem API but I don't know if that will be exempt from the usual ephemerality. IIRC it is just a private storage space with a filesystem-like API.
- easrng 5y ago
  Chrome has an API that allows you to save to files without a new download every time. They made a pretty nice library that wraps that API on Chrome and it gracefully degrades on other browsers. https://web.dev/browser-fs-access/ https://web.dev/browser-fs-access/
  - jcims 5y ago
    i noticed this behavior in a drawing app called excalidraw. first save opens a file dialog, subsequent saves just update the file, basically like a standard local text editor. i keep doing 'save as' to create new files because i don't trust it lol

jefftk 5y ago

> there should be a way to request permission to store data that isn't deleted except for explicit action by users I think that's "request permission to use files on disk". This is in progress as https://developer.mozilla.org/en-US/docs/Web/API/File_System_Access_API https://developer.mozilla.org/en-US/docs/Web/API/File_System..., though there's still more work before there's a version all the browsers like (Mozilla likes the ability to work with files, but thinks cross-site access should not be included: https://mozilla.github.io/standards-positions/#native-file-system https://mozilla.github.io/standards-positions/#native-file-s...)

danShumway 5y ago
> but it is wrapped together with aspects for which we do not think meaningful end user consent is possible to obtain (in particular cross-site access to the end user's local file system) This is a really difficult problem to solve, and I get Mozilla's hesitation. I'm also frankly very hesitant about Google leading the charge on this, not because I'm paranoid about them sneaking in tracking, just because I think Google tends to create less thoughtful web specifications sometimes. But... cross-site file access is really important for data portability and open standards, and Google's current proposal isn't bad, it might be rough but it's definitely workable. Mozilla really should try to figure out a way to move forward on this. We've seen the difference in data portability between mobile and desktop apps, and a big part of the difference between those two platforms is being able to very easily have multiple sources working on your data at the same time. Siloing data has downsides. It's tough to embrace a Unix-style philosophy without allowing programs to operate on the same data. And having Unix-style smaller webapps that work with each other is a good way of fighting against data silos and in some cases a good way of fighting against anti-user and anti-privacy services in general. I'd love to see more progress made on this, but who knows how that will work out. Caution is probably warranted for the moment, I'm just disappointed that the language suggests Mozilla would never consider a proposal that included this. It's also very important that this expose user-accessible file system access and not just a virtual filesystem in the browser; otherwise it just becomes another data-silo in the web browser. This is something that Google's proposal really gets right, and it's disappointing to see what appears to be pushback on the idea that users should be able to open up the directories that a web browser is writing to, inspect the files, and open them or move them around the filesystem, or even write to them from native apps. That to me is an essential part of the proposal.

_fat_santa 5y ago

> I do find it disappointing that there is no reliable local storage system for the web. I get the resistance to trackers but there should be a way to request permission to store data that isn't deleted except for explicit action by users. I've built a number of apps in the last year or two that use browser local storage. It also annoys me that the idea around "storing data in your browser" is automatically attributed to ad tracking. I have to go out of my way in my apps to inform users that yes this app uses localStorage/cookies, but no this is not used for any ad tracking, rather for actually storing your app's data.

kevincox 5y ago
How do you deal with the face that the browser may just decide to wipe the storage at any time?
- jitl 5y ago
  The most important thing to do is get your users data somewhere safe as quickly as possible. For the vast majority of users that means your cloud database. As the user generates new data, spray it to your servers as well as writing it to the syncable IndexedDB local storage, and to an in-memory buffer. Make your backend handle writes idempotently, and retry all failures a few times. (Eg, IndexedDB disk might be full or flaking out, so retry writing the memory buffer to disk.) As long as the write path is quick, users can tolerate the browser nuking offline storage cache because they can re-download all the data that made it up to your server. Hopefully soon the browser vendors will allow more durable file system access with appropriate user controls. Chromium built out the file system access API (https://web.dev/file-system-access/ https://web.dev/file-system-access/) but it’s not supported in Firefox or Safari.
  - chrisfinazzo 5y ago
    Is this (generally) reliable? I know terrible, awful bugs eventually doomed WebSQL from getting any traction and IndexedDB seems to be a more competent replacement, but the fact that Google is leaning on FSA seems like a non-starter. It just feels like there's no way in hell Webkit will ever implement this stuff - not because of the divide between the App Store and PWA's - but due to the implications for privacy. Hard pass.

cdbattags 5y ago

Not much more to say other than Noms was my favorite project (https://github.com/attic-labs/noms https://github.com/attic-labs/noms) for a while until acquisition and the engineers are now the ones behind Replicache (https://replicache.dev/ https://replicache.dev/). I think this is going to be the next "Realm" that works everywhere.

aboodman 5y ago
aww, thanks.
nacs 5y ago
Looks like replicache is pretty expensive though. If you have more than 500 users, the price is $500/mo (and it goes up from there).

EamonnMR 5y ago

What's wrong with offering the user a file to "download" (actually creating a file on the fly) or "upload" (actually loads it into the local web app?) Edit: Here's the stack overflow copypasta I used to achieve it: https://github.com/EamonnMR/Flythrough.Space/blob/master/src/file_utils.js https://github.com/EamonnMR/Flythrough.Space/blob/master/src...

nathcd 5y ago

`navigator.storage.persist()` will prompt the user to allow persistent storage for your site, so data won't be evicted under storage pressure (for IndexedDB, service worker registrations, localStorage, sessionStorage, etc.) https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist https://developer.mozilla.org/en-US/docs/Web/API/StorageMana...

simiones 5y ago
That doesn't work on iOS, nor on Safari on Mac for any released version as far as I understand: https://caniuse.com/mdn-api_storagemanager_persist https://caniuse.com/mdn-api_storagemanager_persist
marcosdumay 5y ago
The problem runs deeper than what the GP complains about. If you rely on that, and write important things on the browser storage, a week later your user access some moronic site that doesn't work, and their support tells him to clear his browser cache, your data will almost certainly be cleaned with it. The problem is that browsers do not even consider that they may be storing important data. There is no clear way to recover or backup that data, and it is tangled with what comes from every other site.

oblib 5y ago

A few years ago a tried using the browser's IndexedDB for my invoicing app and loaded a 1000 documents into it at a time. It did ok up to about 2-3k documents but choked the browser to the point of being unusable at 5k documents. That was on a Late `09 Mac Mini so maybe newer or more powerful PCs would do better, but that's not an issue at all if you're using CouchDB to store the data on the client side. I use CouchDB installed on the client side to implement Offline First data storage. This works for Desktop PCs and if you run it on local device that's accessible via your local network, like a Raspberry Pi in-house for example, you can also use it with mobile devices in-house too. The local CouchDB will sync with the Cloud based CouchDB as soon as they're both online. CouchDB will decide which version to keep and deliver it from that point. It's certainly not perfect and doesn't provide "real time collaboration" but so far that's been out of reach anyway and may not be a very good approach at all. The notion of several people editing the same document at the same time seems to me to be chaotic no matter how you approach it. The biggest downside to this approach is that the user has to install and configure CouchDB. I made a simple web app to help with this, but it's a bit too much to expect users to install and configure it. What we need is a client side DB pre-installed that any web app can access and the data for that app is sandboxed and can only access the DB assigned to it. But it's not reasonable to add that to a web browser. CouchDB can do that now.

hinkley 5y ago

sqlite bindings in browsers came and went in the time between when I learned about them and finally found a nail that needed that hammer. I started on a design and literally within a few weeks Firefox announced it was deprecated.