5 ms·

Show HN: IronDB – a resilient key-value store for the browser

73 points by grun 8y ago

tptacek 8y ago
It's not clear to me how this is more privacy-respecting than Evercookies, except that it doesn't use browser extensions.
- grun 8y ago
  Evercookie's goal is to store data indelibly, regardless of user intention. IronDB's goal is to store data reliably, e.g. in the face of storage eviction, but not against user intention. If the user clears all browsing data, that willful action is respected. Thank you for your feedback. I'll clarify this in the docs.
  - laurent123456 8y ago
    Why do we need a library to store data reliably? Aren't browsers API already reliable? Does the LocalStorage API for example sometimes fails to store its data properly?
    - grun 8y ago
      Clearing cookies is a common user action. Even for relatively non-technical users. And under storage pressure, browsers evict data stored in IndexedDB, LocalStorage, and/or SessionStorage. See https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa#how_does_cache_eviction_work https://developers.google.com/web/fundamentals/instant-and-o... IronDB is resilient in the face of such events. Thank you for your feedback. I'll add an explanation about such in the documentation.
      - tptacek 8y ago
        That's the answer I was looking for, thanks. The intuition I have is that if someone deletes something from localStorage (or clears localStorage altogether), they want that data gone.
      - chx 8y ago
        What about https://developers.google.com/web/updates/2016/06/persistent-storage https://developers.google.com/web/updates/2016/06/persistent...
        grun 8y ago
        Persistent storage requires explicit user permission. IronDB doesn't. See https://developers.google.com/web/updates/2016/06/persistent-storage#how_do_i_make_my_storage_persistent https://developers.google.com/web/updates/2016/06/persistent.... Great point, though. I'll highlight this in the documentation. Thank you.
      - staticvar 8y ago
        Those docs are dated unfortunately. For example, in my tests on multiple android devices and Mac OS the storage limit on Chrome is closer to 20% of total disk size and you don't need to prompt the user for persistent storage like you prompt for access to gps.
      - staticvar 8y ago
        If anyone could point to an up to date docs around storage eviction in browsers it would be greatly appreciated. It's been a big hit to our team's confidence in the "use the platform" mentality that this is not accurately documented.
        edoceo 8y ago
        I remember reading a 5MB limit some place (ages ago) and I'm not able to find the reference at the time. My team has been assuming that is true (for local,session)Storage and so far haven't been bitten. Only occasionally have we even gotten close to that size. You can also shove loads of data in the DOM. Edit: one of the old refs - 10MB https://www.html5rocks.com/en/tutorials/offline/quota-research/ https://www.html5rocks.com/en/tutorials/offline/quota-resear...
  - deleted 8y ago
    [deleted]
bazizbaziz 8y ago
> When a value is retrieved via its key, IronDB... > Looks up that key in every store. > Counts each unique, returned value. > Determines the most commonly returned value as the 'correct' value. > Returns this most common correct value > Then IronDB self heals: if any store(s) returned a value different than the determined correct value, or no value at all, the correct value is rewritten to that store. In this way, consensus, reliability, and redundancy is maintained. I'm not sure any of these properties are ensured the way anyone would want. Why do this? What are the failure modes this protects against? If only one of the backing stores is active when a result is written, and then the others later become active with no data, is blank data returned for the prior result? It seems like recording timestamps would fix this problem nicely on a single system and make this thing overall quite reliable.
- grun 8y ago
  Cookies are frequently cleared by users, and the other datastores -- IndexedDB, LocalStorage, and SessionStorage -- can be unceremoniously purged by the browser under storage pressure. See my other comment, here: https://news.ycombinator.com/item?id=18297177 https://news.ycombinator.com/item?id=18297177 I'll explicate this in the documentation. Thank you for your feedback, bazizbaziz.
  - bazizbaziz 8y ago
    Thanks for the response! FWIW I'm overall really interested in this, as I maintain an application that uses localStorage to keep very important data while the app is offline until it can be uploaded. I recently tested localForage [0] to see if it could be more reliable and get around storage limitations of localStorage. Unfortunately, LocalForage has a very annoying problem where it picks a single storage backend to use, but the one it chooses can switch on page reloads (yes, even on the same device/browser.) and this switching causes data loss as keys stored in the other backends are unaccessible. I'm very interested to see if IronDB can help here! Thanks for working on this. [0] https://github.com/localForage/localForage https://github.com/localForage/localForage
    - 8y ago
ggregoire 8y ago
What are the differences with localForage (or similar solutions)? The fallback to cookies if no localStorage? https://github.com/localForage/localForage https://github.com/localForage/localForage
- grun 8y ago
  localForage only uses a single datastore. E.g. LocalStorage or SessionStorage or IndexedDB. Thus, stored data is lost forever if the browser unceremoniously purges that datastore due to storage pressure. https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa#how_does_cache_eviction_work https://developers.google.com/web/fundamentals/instant-and-o... IronDB redundantly stores data in multiple datastores -- cookies, IndexedDB, LocalStorage, and SessionStorage -- and self-heals if the data in any thereof is deleted or corrupted. For example, stored data is safeguarded in the event the user clears their cookies or IndexedDB is purged due to storage pressure. tl;dr: IronDB is resilient in the face of data deletion. localForage isn't.
  - ggregoire 8y ago
    Thank you for the great answer. :)
  - lgessler 8y ago
    It's kind of ridiculous that a library needs to exist to guard against this browser behavior, although I empathize with the browser vendors' argument, which is that browser storage should essentially be used for caching and that browsers should somewhat aggressively defend against disk space consumption that most users don't even know about. It would be nice to have an in-between where if a user authorized an app to permanently consume up to X amount of disk space the browser would allow it. But pulling off that user interaction well (i.e., getting the user to make an informed decision rather than blindly clicking yes or no) seems quite hard.
gaimrox 8y ago
Confused. Why would you name your K/V database irondb when there is already a time-series database called exactly that? Think this is their site: https://www.irondb.io https://www.irondb.io
- thanatos_dem 8y ago
  Yup, I was wondering how the hell a timeseries database would run in the browser. Nope, this is just syntactic sugar on top of calling (up to) 4 JavaScript functions.
proneb1rd 8y ago
What’s the point of storing the same data in three distinct storages? What kind of disaster are you trying to avert? ps: this will back fire real bad when safari and friends add permission prompts for each specific storage that this library fills up in a for loop.
- latchkey 8y ago
  The usecase I've seen more recently is that paywall news sites now store the number of page reads in multiple places. You end up having to clear more than just your cookies to 'reset' the read count.
kreetx 8y ago
Maybe this is useful tool, but several things strike as weird to me: - save all values everywhere: why not select the one that would last the longest (that people delete least often)? Or maybe two, if you want a cookie but also want to survive over "delete all cookies". But then again, if a user wants to delete cookies then you really shouldn't try to keep it. - "Data is stored resiliently but can also be voluntarily purged if the user designedly clears cookies and application storage." So you delete data when the user wants to delete data. Which database doesn't have this feature? - doesn't use Flash, Silverlight, or Java -- which do? - "healing" is usually used in the context of distributed computing, but here it means that it gets around user wanting to delete stuff from it's browser, but perhaps isn't in-the-know enough to delete everything but just the cookies. No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained?
- grun 8y ago
  > save all values everywhere: why not select the one that would last the longest (that people delete least often)? For improved resiliency. You can also configure IronDB to only use any two datastores of your choice, if you so desire: IronStorage's constructor takes an Array of storage implementations of your choice. See https://github.com/gruns/irondb#api https://github.com/gruns/irondb#api. > So you delete data when the user wants to delete data. Which database doesn't have this feature? A database where the data therein can be deleted without warning. Browsers unceremoniously delete IndexedDB, LocalStorage, and SessionStorage under storage pressure. See https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa#how_does_cache_eviction_work https://developers.google.com/web/fundamentals/instant-and-o... > doesn't use Flash, Silverlight, or Java -- which do? Evercookie, a similar library, uses Flash, Silverlight, and/or Java. See https://github.com/samyk/evercookie https://github.com/samyk/evercookie. > No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained? No offense taken. Thank you for your feedback, kreetx. Don't hesitate to let me know if I can answer any other questions, or if there's anything else I can do for you.

73 points by grun 8y ago

It's not clear to me how this is more privacy-respecting than Evercookies, except that it doesn't use browser extensions.

grun 8y ago
Evercookie's goal is to store data indelibly, regardless of user intention. IronDB's goal is to store data reliably, e.g. in the face of storage eviction, but not against user intention. If the user clears all browsing data, that willful action is respected. Thank you for your feedback. I'll clarify this in the docs.
- laurent123456 8y ago
  Why do we need a library to store data reliably? Aren't browsers API already reliable? Does the LocalStorage API for example sometimes fails to store its data properly?
  - grun 8y ago
    Clearing cookies is a common user action. Even for relatively non-technical users. And under storage pressure, browsers evict data stored in IndexedDB, LocalStorage, and/or SessionStorage. See https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa#how_does_cache_eviction_work https://developers.google.com/web/fundamentals/instant-and-o... IronDB is resilient in the face of such events. Thank you for your feedback. I'll add an explanation about such in the documentation.
    - tptacek 8y ago
      That's the answer I was looking for, thanks. The intuition I have is that if someone deletes something from localStorage (or clears localStorage altogether), they want that data gone.
    - chx 8y ago
      What about https://developers.google.com/web/updates/2016/06/persistent-storage https://developers.google.com/web/updates/2016/06/persistent...
      - grun 8y ago
        Persistent storage requires explicit user permission. IronDB doesn't. See https://developers.google.com/web/updates/2016/06/persistent-storage#how_do_i_make_my_storage_persistent https://developers.google.com/web/updates/2016/06/persistent.... Great point, though. I'll highlight this in the documentation. Thank you.
    - staticvar 8y ago
      Those docs are dated unfortunately. For example, in my tests on multiple android devices and Mac OS the storage limit on Chrome is closer to 20% of total disk size and you don't need to prompt the user for persistent storage like you prompt for access to gps.
    - staticvar 8y ago
      If anyone could point to an up to date docs around storage eviction in browsers it would be greatly appreciated. It's been a big hit to our team's confidence in the "use the platform" mentality that this is not accurately documented.
      - edoceo 8y ago
        I remember reading a 5MB limit some place (ages ago) and I'm not able to find the reference at the time. My team has been assuming that is true (for local,session)Storage and so far haven't been bitten. Only occasionally have we even gotten close to that size. You can also shove loads of data in the DOM. Edit: one of the old refs - 10MB https://www.html5rocks.com/en/tutorials/offline/quota-research/ https://www.html5rocks.com/en/tutorials/offline/quota-resear...
- deleted 8y ago
  [deleted]

bazizbaziz 8y ago

> When a value is retrieved via its key, IronDB... > Looks up that key in every store. > Counts each unique, returned value. > Determines the most commonly returned value as the 'correct' value. > Returns this most common correct value > Then IronDB self heals: if any store(s) returned a value different than the determined correct value, or no value at all, the correct value is rewritten to that store. In this way, consensus, reliability, and redundancy is maintained. I'm not sure any of these properties are ensured the way anyone would want. Why do this? What are the failure modes this protects against? If only one of the backing stores is active when a result is written, and then the others later become active with no data, is blank data returned for the prior result? It seems like recording timestamps would fix this problem nicely on a single system and make this thing overall quite reliable.

grun 8y ago
Cookies are frequently cleared by users, and the other datastores -- IndexedDB, LocalStorage, and SessionStorage -- can be unceremoniously purged by the browser under storage pressure. See my other comment, here: https://news.ycombinator.com/item?id=18297177 https://news.ycombinator.com/item?id=18297177 I'll explicate this in the documentation. Thank you for your feedback, bazizbaziz.
- bazizbaziz 8y ago
  Thanks for the response! FWIW I'm overall really interested in this, as I maintain an application that uses localStorage to keep very important data while the app is offline until it can be uploaded. I recently tested localForage [0] to see if it could be more reliable and get around storage limitations of localStorage. Unfortunately, LocalForage has a very annoying problem where it picks a single storage backend to use, but the one it chooses can switch on page reloads (yes, even on the same device/browser.) and this switching causes data loss as keys stored in the other backends are unaccessible. I'm very interested to see if IronDB can help here! Thanks for working on this. [0] https://github.com/localForage/localForage https://github.com/localForage/localForage
  - 8y ago

ggregoire 8y ago

What are the differences with localForage (or similar solutions)? The fallback to cookies if no localStorage? https://github.com/localForage/localForage https://github.com/localForage/localForage

grun 8y ago
localForage only uses a single datastore. E.g. LocalStorage or SessionStorage or IndexedDB. Thus, stored data is lost forever if the browser unceremoniously purges that datastore due to storage pressure. https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa#how_does_cache_eviction_work https://developers.google.com/web/fundamentals/instant-and-o... IronDB redundantly stores data in multiple datastores -- cookies, IndexedDB, LocalStorage, and SessionStorage -- and self-heals if the data in any thereof is deleted or corrupted. For example, stored data is safeguarded in the event the user clears their cookies or IndexedDB is purged due to storage pressure. tl;dr: IronDB is resilient in the face of data deletion. localForage isn't.
- ggregoire 8y ago
  Thank you for the great answer. :)
- lgessler 8y ago
  It's kind of ridiculous that a library needs to exist to guard against this browser behavior, although I empathize with the browser vendors' argument, which is that browser storage should essentially be used for caching and that browsers should somewhat aggressively defend against disk space consumption that most users don't even know about. It would be nice to have an in-between where if a user authorized an app to permanently consume up to X amount of disk space the browser would allow it. But pulling off that user interaction well (i.e., getting the user to make an informed decision rather than blindly clicking yes or no) seems quite hard.

gaimrox 8y ago

Confused. Why would you name your K/V database irondb when there is already a time-series database called exactly that? Think this is their site: https://www.irondb.io https://www.irondb.io

thanatos_dem 8y ago
Yup, I was wondering how the hell a timeseries database would run in the browser. Nope, this is just syntactic sugar on top of calling (up to) 4 JavaScript functions.

proneb1rd 8y ago

What’s the point of storing the same data in three distinct storages? What kind of disaster are you trying to avert? ps: this will back fire real bad when safari and friends add permission prompts for each specific storage that this library fills up in a for loop.

latchkey 8y ago
The usecase I've seen more recently is that paywall news sites now store the number of page reads in multiple places. You end up having to clear more than just your cookies to 'reset' the read count.

kreetx 8y ago

Maybe this is useful tool, but several things strike as weird to me: - save all values everywhere: why not select the one that would last the longest (that people delete least often)? Or maybe two, if you want a cookie but also want to survive over "delete all cookies". But then again, if a user wants to delete cookies then you really shouldn't try to keep it. - "Data is stored resiliently but can also be voluntarily purged if the user designedly clears cookies and application storage." So you delete data when the user wants to delete data. Which database doesn't have this feature? - doesn't use Flash, Silverlight, or Java -- which do? - "healing" is usually used in the context of distributed computing, but here it means that it gets around user wanting to delete stuff from it's browser, but perhaps isn't in-the-know enough to delete everything but just the cookies. No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained?

grun 8y ago
> save all values everywhere: why not select the one that would last the longest (that people delete least often)? For improved resiliency. You can also configure IronDB to only use any two datastores of your choice, if you so desire: IronStorage's constructor takes an Array of storage implementations of your choice. See https://github.com/gruns/irondb#api https://github.com/gruns/irondb#api. > So you delete data when the user wants to delete data. Which database doesn't have this feature? A database where the data therein can be deleted without warning. Browsers unceremoniously delete IndexedDB, LocalStorage, and SessionStorage under storage pressure. See https://developers.google.com/web/fundamentals/instant-and-offline/web-storage/offline-for-pwa#how_does_cache_eviction_work https://developers.google.com/web/fundamentals/instant-and-o... > doesn't use Flash, Silverlight, or Java -- which do? Evercookie, a similar library, uses Flash, Silverlight, and/or Java. See https://github.com/samyk/evercookie https://github.com/samyk/evercookie. > No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained? No offense taken. Thank you for your feedback, kreetx. Don't hesitate to let me know if I can answer any other questions, or if there's anything else I can do for you.