15 ms·

Offline-First Database Comparison

415 points by typingmonkey 5y ago

porcc 5y ago
gunjs?
- tommiegannert 5y ago
  I've tried to read the code 3-4 times now. Every time I have to give up. The "chain" thing is really bad code, even admitted by the author. If you're considering GunJS, take a look at the code and verify it's something you'd want to debug, before going all-in.
  - porcc 5y ago
    I did go all in and it worked fine, great even. The devs are responsive on their Gitter for sorting things out and it was generally simple enough that I wouldn't need to mess with the internals
  - jamil7 5y ago
    I saw the author present once, I couldn't follow anything they were talking about or the code samples presented.
dSebastien 5y ago
I've used CouchDB and PouchDB on a previous project, and it was a blast. The built-in features like replication and HTTP API are great. My only regret is the limited support (at the time) for full text search and complex queries. I suppose I like joins a bit too much.. :) I deployed CouchDB in a Kubernetes cluster (not with HA as I didn't have high availability requirements), and it was working great.
- psychometry 5y ago
  Yeah although the docs claim to support things like regex-based searches, it's so horrendously slow it shouldn't even be listed as a feature.
redwood 5y ago
Surprised not to see Realm included (www.realm.io)
carabiner 5y ago
Some charts for that table would be so awesome.
- smcnally 5y ago
  I'm not the OP or author. This sheet[0] has the Metrics and Feature Map data for charting, sorting, etc. directly from their GH page. [0] https://docs.google.com/spreadsheets/d/12ReO-4_bZ2BaLj9P6oJT7m0CdXpz_Z9ie_F1ShzUHHI/edit?usp=sharing https://docs.google.com/spreadsheets/d/12ReO-4_bZ2BaLj9P6oJT...
karmelapple 5y ago
So glad to see PouchDB included. We use it and have generally had a great experience! We use it with CouchDB on the backend, and Couch seems like a fantastic way to go for use cases involving syncing data between devices with an offline mode and syncing between clients. It was built from the ground up with replication in mind. Biggest bummer of CouchDB? If you’re not hosting it yourself, there’s only one major player in the market that I know of: IBM Cloudant. They contribute much to Apache CouchDB though, and hosting it yourself doesn’t seem too difficult, especially for small, simple use cases. Anyone else using CouchDB?
- knubie 5y ago
  I use the PouchDB/CouchDB combo for exactly the use case your describing. The query language for CouchDB leaves a little to be desired, but ultimately it has worked well for me. I'm self hosting on a digital ocean droplet.
  - tata71 5y ago
    Is DO offering anything close to enterprise grade?
    - kbenson 5y ago
      "enterprise grade" covers a a much larger spectrum of uses and needs with regard to features and stability than non-enterprise grade, making it kinda hard to answer that generally. They offer VMs, containers, managed DB instance offerings, block storage, multiple regions and datacenters, load balancing, etc. They have an API to control all those things, and modules available in many popular languages. But that's basically what many people consider table stakes for a service like that, and indeed there are competitors like Vultr and Upcloud that offer the same. Will any of them have quite the same level of offerings that AWS or GCE or Azure offer? Probably not. But for a great many people what they have is all the enterpise level stuff they'll need or use, and it is decidedly easier to just start up a cheap linux VM and take care of it on one of these services compared to AWS or GCE or Azure, so if what you want are somewhat manually managed cloud VMs, I highly recommend one of these services over one of the big names. I've used all the services named, and I still prefer DO for just throwing up a cheap $5-$10 VM for personal stuff, or to spin up a temporary VM for testing something out. On DO that's a couple second process when you do it manually by clicking around.
FractalHQ 5y ago
I imagine that Supabase would be a perfect candidate for this project! I would love to get the authors opinion on it too.
- typingmonkey 5y ago
  I considered to add Supabase, RethinkDB and meteor. But they are not really client side databases. They realtime stream query results from the server to the client.
  - knes 5y ago
    Did you look into https://ditto.live/ https://ditto.live/? Its and off-line first db too. I never used it but looks very interesting and would be interested how it compare to the technologies you picked
  - irae 5y ago
    I love RethinkDB. I really believe it is a database that deserves more attention. We've been running on it for six years and it is flawless, easy to use. Like any other database you need to understand how it performs and how to query efficiently. But it does has everything required for streaming and syncing data in a modern way. You can open streaming queries and even with a backlog of data, so syncing should be pretty easy. Maybe would be feasible to fork PouchDB to use RethinkDB as a backend?
  - PKop 5y ago
    Firestore isn't really a client side database then either. It is certainly also querying results from a server.
- jonplackett 5y ago
  I second that! Would like to see supabase since that’s what I’ve been using a lot recently and really liking!
  - mastazi 5y ago
    I have a question, how do you manage the fact that Supabase is based on a relational db[1]? Do you just put all fields (except the primary key) in a JSON column[2]? Does it work well if you use it that way? We were investigating it as a way to escape Firebase's vendor lock-in but the fact that we would have to manage schema migrations was a bit of a deal breaker based on our use case. I'm also interested in hearing about any alternatives to Supabase that use a document-based nosql db. [1] https://supabase.io/database https://supabase.io/database [2] https://www.postgresql.org/docs/13/datatype-json.html https://www.postgresql.org/docs/13/datatype-json.html
dvdhnt 5y ago
I really enjoy examples like this, thanks, I’ll be exploring it. As an aside… I would truly love to explore a collection of interesting ways to use SQLite. It’s such an impressive piece of technology that I’d like to use more often. Please share if you have something similar!
- heyzk 5y ago
  I had a really great experience building an EAV store with Datalog as the query interface on top of SQLite for embedding in native mobile apps. Pros: querying complex data hierarchies was easy, and was able to skip the pain typically associated with managing a SQL schema.
  - phyrex 5y ago
    I would love to hear more! Did you write the datalog layer or is there one somewhere? Is there any code available I could see?
    - dunham 5y ago
      You might be interested in the now defunct Mentat project from Mozilla. They made an EAV store with syncing on top of sqlite. It ran datalog queries by translating them into sql. https://github.com/mozilla/mentat https://github.com/mozilla/mentat
      - phyrex 5y ago
        I’m aware of mentat and eternally disappointed that it got shelved :(
  - Mertax 5y ago
    What’s the application for this? EAV is often an anti-pattern when a schema could be defined, but I’m actually using it as well. Our application is an end-user-defined database for mobile data collection. The EAV model in SQLite is a bit of a cognitive burden but makes offline sync and conflict resolution pretty straight forward. It’s almost a crude CRDT implementation.
throwaway743 5y ago
Am I wrong to think that pouchdb uses indexeddb? Over the last 8 months, I've been working on a react/capacitor based android app (potentially ios later on) and was originally using idb-keyval, which uses indexeddb for key val storage, and things were great since it's a local storage solution compatible with react/capacitor, and one of my goals is to not rely on a remote data storage solution. As said, things were going great, but then a couple weeks ago things went to shit when all the data that was stored in the prototype on my android device was wiped. Apparently, both android and ios tend to wipe browser/web-view local storage at random/when space is needed(?). Dug around since looking for an alternative solution (would love a capacitor compatible mongodb solution), came across pouchdb via rxdb, but could've sworn there was mention that it relies on indexeddb. So just to be safe switched to sqlite and been rewriting components since. Lesson of the story, even if it isn't dependent on indexeddb, if you're looking for a local storage option for a mobile app and happen to be using a js framework with capacitor or anything that utilizes a web-view, stay away from anything that uses indexeddb. If a wipe like this were to happen post release, the chance that your app succeeds afterwards would be near 0% Edit: so yeah, just double checked/was reading through the readme of this project, and pouchdb via rxdb is reliant on indexeddb
- typingmonkey 5y ago
  Author here. I am using RxDB with Capacitor (iOS and Android app). You can use the SQLite based pouchdb adapter with capacitor. It keeps your data and is (sometimes) faster. Here [1] I have documented a whole section about how to use RxDB+SQLite in Capacitor. [1] https://rxdb.info/adapters.html https://rxdb.info/adapters.html
  - throwaway743 5y ago
    Ah okay cool. Thank you for pointing this out. Now would I use the adapter for react native or for cordova? If react native, I've been writing with react js and have been under the impression that react native specific plugins aren't compatible with react js. Is that wrong? If cordova, it's totally compatible with capacitor? Sorry just want to make sure before jumping in
sushsjsuauahab 5y ago
How far the world has fallen from Linux, MySQL, SpringBoot, and HTML/CSS/JS
- spdebbarma 5y ago
  You can only live in the past for so long. To bystanders, you're falling behind by being cynical and ignorant of newer solutions. Don't get me wrong. The world still uses the technology you mentioned at large, but the industry has in-fact built upon and grown alongside older technology.
  - mhd 5y ago
    I don't want to seem like I'm defending SpringBoot, ferchrissakes, but still fail to see a big impact of PWAs. Especially a positive one for the user, not mere ad clicks. In general, the whole rich web app space reminds me of the "You're not making Christianity better, you're making Rock'n Roll worse" meme.
    - sushsjsuauahab 5y ago
      I am genuinely shocked that people don't like spring boot? It is very performant, open, and ergonomic in my experience.
      - mhd 5y ago
        2021 Spring and Maven could buy me chocolates and flowers all day long, I wouldn't forget what they did to me in the past... But seriously, I'm okay with working with SB when I have to, but quite often in those situations Java wouldn't be my first choice, and it's still all the putrescence of proper Spring underneath. A framework for a framework, with a bit too much magic for me. Explicit is better than implicit.
  - sushsjsuauahab 5y ago
    While I certainly can fathom that other newer technologies are capable of solving interesting problems, I think my assertion is that the additional value they provide is not worth the risks involved in trying them.
shoo 5y ago
From the perspective of someone less familiar with this kind of thing, this comparison would be easier to understand with a bit of an introductory explanation about what job we're trying to do or problem we're trying to solve, and the assumed context or constraints.
- amw-zero 5y ago
  Offline-first is a well-known term. You can search for it elsewhere. When writing, it's important to choose who you're speaking to exactly so that you can avoid sharing context, since that takes up time and bandwidth. The truth is, communication is much more efficient if you don't re-explain every single concept that you are talking about.
  - wolfram74 5y ago
    A compromise solution would be linking to a glossary or introductory material if someone is generally versed but not in a particular subject, we get a lot of, say, pure math people on here who find software engineering interesting.
    - y4mi 5y ago
      you'd still have to draw a line at some point which terms you'd explain as otherwise your article only consist of glossary information. i'm pretty sure the term offline-first wouldn't have met the cutoff point, as its _really_ well known from my experience.
      - typingmonkey 5y ago
        I wrote so much things and posted it on HN over the years. There is just no way to make everyone happy, someone always complains that some infos are missing or too much. I now leave things out that can be googled and are already known by "most" readers.
        amw-zero 5y ago
wanderingmind 5y ago
Any information on how scalable are these databases compared to traditional SQL databases? Or specifically, when should you use this (in prototyping or production)
- rendall 5y ago
  IMO the principal consideration here is that these are offline/local databases for browsers and (probably?) Electron, so they are not intended to be scalable at all. If you have a progressive web app (PWA) and you need a queryable database for some reason, then you would use these. Otherwise, stick your queries behind API endpoints.
- janl 5y ago
  CouchDB is a lot more scalable than SQL databases because it has a distributed scaling model built in (just add nodes), no need to mess with read-replicate and finicky hot-failover, it all just works out of the box (Dynamo style).
  - karmelapple 5y ago
    It’s more scalable in theory, and I talked its praises in a different comment, but our team has hit scaling issues with our one-user-per-database approach. It was a mess to sort out, but Cloudant support was very helpful. Our major issue: we write many small documents, and we write them over every user’s database fairly frequently. And Cloudant’s default settings don’t like that with a one-user-per-database approach. In fact, they discourage anyone from the one-db-per-user approach these days: https://www.ibm.com/cloud/blog/cloudant-best-and-worst-practices-part-1 https://www.ibm.com/cloud/blog/cloudant-best-and-worst-pract... That blog post calls it an anti-pattern, but I would respectfully disagree. It is an absolutely great pattern to keep a native app and a web app in sync across multiple devices with intelligent conflict resolution. A solution was to reduce the number of shards that a database was split out over, since our database’s data is pretty small overall and we didn’t need each database split out so much across our cluster.
    - janl 5y ago
nyanpasu64 5y ago
Not directly related to the post (which is focused on which database to host for your app), but I'm writing desktop apps (think DAWs) in C++ and Rust (not JS), and want to synchronize settings through a Dropbox or Google Drive (so I don't have to host my own cloud sync servers). What's a good library or schema to achieve this? Personally I usually don't have multiple instances of the same app open on multiple machines, but other people might open the same or different files on their desktop and laptop. - Not all settings should be synchronized (don't include machine-specific "recent files" paths). - How should settings be stored locally (if I may have multiple instances of my app open on a single machine)? Registry (Windows-only)? INI with atomic saving (requires care and locking to prevent multiple instances from trampling or racing with each other)? SQLite? IMO Stylus is a pretty good implementation of offline-first cloud settings sync over Dropbox/etc. It's currently based around one JSON file per CSS file (Dropbox/Apps/Stylus - Userstyles Manager/docs/uuid.json), and what appears to be a transaction log (Dropbox/Apps/Stylus - Userstyles Manager/changes/number.json). Cloud sync has been 100% reliable in my experience, though I do notice temporary file lock errors when switching between different machines in my dual-boot setup (but sync seems to be eventually consistent nonetheless). uBlock Origin is worse. Instead of merging settings, it expects the user to upload and download the entire settings blob at once (and pulling an old blob can erase changes you've made locally). And in the past it's entirely failed to sync because the blob was too big to upload to Mozilla's servers. (Right now it "works" but takes several minutes for one computer to see a config uploaded from another computer.)
- lbhdc 5y ago
  I would imagine using a CRDT would be appropriate for the data you want to sync would let you sync state between multiple open clients.
- franga2000 5y ago
  I've done something pretty ridiculous to solve this problem and I'm not sure I'd recommend it, but here it is: - A directory is synced with the server with no conflict resolution - The application creates its config file in that directory, named by a random UUID, which is stored outside the synced folder - The config file stores the setting overrides (defaults were compiled-in) in any format (I used YAML) - Each setting override includes a "locked" and "lastModifed" - On startup (sync was external) all files in the directory are read and merged starting with the local one, then skipping any settings that are locked (locally or remotely), last modified wins Some deployments used a daily rsync cronjob, some had a mounted network share (with hilarious broken file locking) and of course it worked with direct bind mounts as well. I also briefly experimented turning the "locked" field into a "group" field to enable multiple "sync groups" with some keys shared globally and some only with other group members (even different groups for different settings), but it ended up not being useful for my use case, although it did work.
jdc 5y ago
See also Google's Lovefield SQL browser database: https://google.github.io/lovefield https://google.github.io/lovefield
- rutierut 5y ago
  Second this. Oh and it’s not reaaaaly abandoned, they just called it done. I looked into using this as well but eventually decided against it due to the lack of active development.
- psychometry 5y ago
  No commits in 2 years. Seems Google lost interest like usual.
LAC-Tech 5y ago
Last write wins is not a strategy for conflict resolution, it's a surrender. So I'm glad to hear something else apart from Pouch actually handles it. Anyone familiar with rxdb and can chime in on how they do it?
- netghost 5y ago
  It sounds like rxdb is built on top of pouch, so probably the same set of options with the possibility of some opinionated design or sensible defaults though I can't find anything. obvious.
  - typingmonkey 5y ago
    Yes RxDB conflict resolution is equal to PouchDBs. At least for now, there are plans to improve from there where you have a global resoluting function instead of listening for conflicts in the changestream.
y4mi 5y ago
My biggest gripe with offline-first capability of PWAs is speed. a network roundtrip is generally faster then fetching the same information from indexeddb, and you still have to sync first because indexeddb tends to get wiped at inopportune times. its great if you're writing a traditional app though, but really unfortunate wrt to the PWAs
- getcrunk 5y ago
  Really? I've never used index db but to confirm your saying to fetch data from local memory is slower than network round trip?
  - tehbeard 5y ago
    indexedDB is quite a low level in what the interface gives you. This includes what options you have available for querying (just accessing a key range, upper/lower bound, forward or backwards (backwards can be much slower), rather than a full query language like SQL etc.. So you need to design your app / data format and pre-plan any queries so you'll have the indexes you need, or do some glue logic to combine indexes as needed.
  - dmw_ng 5y ago
    The complaint was about IndexedDB access, not local memory. Folk mentioning this are usually referring to the disastrous implementation in Chrome: see https://dev.to/skhmt/why-are-indexeddb-operations-significantly-slower-in-chrome-vs-firefox-1bnd https://dev.to/skhmt/why-are-indexeddb-operations-significan... and https://jlongster.com/future-sql-web https://jlongster.com/future-sql-web
    - getcrunk 5y ago
      Wow. Orders of magnitude slower than ff.
- tehbeard 5y ago
  Why is that?. Is this because you're on a fibre link? indexedDB is being used with several layers to add a comfy SQL and ORM layer for you? I've used indexedDB on a couple of projects at work, while there are definitely downsides with its indexing design, limited querying options and the menagerie of fuckups by Team Fruit(TM), it works well as a local cache when our clients are out on site with their customers and all they've got is a crappy intermittent 3/4g signal.
peterthehacker 5y ago
What kind of consistency models [0] do Offline-first databases like RxDB and PouchDB have? I was thinking read uncommitted, but they might allow dirty writes. Maybe there’s some CRDTs under the hood… I can’t find any documentation on consistency though, anyone here know? [0] https://jepsen.io/consistency https://jepsen.io/consistency
- genewitch 5y ago
  Is the reason for all of these sorts of comments some sort of "never lose data" quality assurance? There's consistency, write first, latency, replication, and then acronyms. I just implement hardware and OS stuff, but I like the DB people to be happy. What am I missing?
- janl 5y ago
  CouchDB docs have you covered: https://docs.couchdb.org/en/stable/replication/intro.html https://docs.couchdb.org/en/stable/replication/intro.html
  - peterthehacker 5y ago
    Thanks! Very cool. I haven’t used couchdb in prod. I’ll read into this more.
spankalee 5y ago
Wait, this is benchmarking the Firestore emulator. How is that relevant?
- typingmonkey 5y ago
  It is benchmarking the firestore JavaScript library that runs at the client.
  - spankalee 5y ago
    Maybe, but I don't see where the browser is brought offline to enforce that. It looks like the emulator is still running and there's still a connection to it. Can you show where you force the browser offline?
globular-toast 5y ago
I've used watermelondb and found it quite nice to use. I mainly chose it because it has observable queries. I can live without conflict resolution and transactions on the client side because it's an electron app. Having a sqlite backend (via the electron process) would overcome some of the shortcoming wrt to the in memory database (and still allow in memory via sqlite if speed is desired). I wrote a backend sync for it which was quite easy although I haven't fully implemented support for everything because I don't need it (notably deletes).
joshxyz 5y ago
I wonder why multitab isnt supported on watermelon, crosstab updates thru localstorage api works great
- franga2000 5y ago
  > WatermelonDB uses the LokiJS adapter which is an in memory database that regularly persists the data to IndexedDB either on interval, or when the browser tab is closed
- radex 5y ago
  You can use multiple tabs but to have more than one write to the DB and not be overwritten relies on (online) sync. The problem is that Watermelon assumes a consistent view of the entire database, so you can't have multiple writers - at least not without synchronous notifications from IndexedDB (not a thing), leader election (cannot be made reliable), or some design sacrifices. To be reconsidered in the future...
RobertRoberts 5y ago
What is the advantages of using one of these client side databases vs using indexeddb directly?
- radex 5y ago
  IndexedDB is a joke of a database. Yes, it can store data, and you can create a very simple index, so it's _technically_ a database… But its ability to express queries is borderline useless for all but simplest use cases, it's slow, and it's very inconvenient to use. So solutions exist that range from giving IDB a simpler, more modern API, all the way to using IDB as a dumb storage medium to a fast in-memory database.
- typingmonkey 5y ago
  Replication, Encryption, Conflict Handling, Multi-Tab-Support, Compression, Observability and many more..
chadcmulligan 5y ago
I often feel I'm on another planet when I see timings for browser things people are recommending - Insert one message - best 9ms, insert 20 messages - best 33ms, worse - >8 Seconds! Why bother? just write a native client.
- radex 5y ago
  Browser-based apps have use cases even when you reject PWAs generally as a replacement for native apps. Trying out a new tool quickly, or short-term when using a tool with a customer, in enterprise where you can't install native tools, etc...
  - chadcmulligan 5y ago
    Well make a binary that doesn't require installation, this is a very narrow use case for a technology thats being advocated for more than that use case
    - tatersolid 5y ago
      Most enterprise users can’t download and run an arbitrary binary due to numerous security controls and regulations. This is true even if that binary is signed - if it isn’t on an allow-list maintained by high priests it can’t be used. The real world demands web apps in the enterprise even in 2021.
      - deleted 5y ago
        [deleted]
      - chadcmulligan 5y ago
        This is a poor reason in 2021 - app stores exist. Software developers have an obligation, as professionals, to present the costs of web apps vs binaries when developing new projects, the browser is not an operating system and has real limitations - as this project demonstrates.
- typingmonkey 5y ago
oblib 5y ago
For offline-first use PouchDB can connect directly to a CouchDB installed on your desktop PC as opposed to storing user data in your browser's built-in IndexedDB and syncing that with a Cloud based CouchDB. Syncing that local CouchDB to your web based CouchDB is very fast and it's done in the background so it doesn't affect the performance of actually using the app. You can click "Save" and move on with no waiting at all. So in this scenerio the speed of the DB is not necessarily a reflection of the speed of the app for the user.
- typingmonkey 5y ago
  > Syncing that local CouchDB to your web based CouchDB is very fast This is not true. When you compare the CouchDB replication with other replication protocols, it is slow. The reason is that CouchDB supports replication with many instances at the same time. This creates big overhead in handling revision trees. Many requests have to be made all the time. You can observe that by starting the PouchDB subproject in the comparison repo. Watch the network tab in dev tools. Another problem is that CouchDB does not support Websocket replication, everything is long polling and plain http requests. Other replications that only support many-clients-to-one-server are way faster. Both, on the initial load and on ongoing changes. This was the main reason why I build GraphQL replication for RxDB.
  - oblib 5y ago
    >>This is not true. When you compare the CouchDB replication with other replication protocols, it is slow. That may be true but when a single user is working with an offline-first app connected to a CouchDB installed on their desktop pc that happens entirely in the background so they don't experience any lag. And while I've not done benchmark studies with CouchDB I have monitored the logs to watch those syncs and we're not talking painfully "slow" in real world use. It is reliable though. I've been using it for about 5 years now and it's been solid. And so has the work they've done to improve it. And I was not aware of RxDB, which is certainly interesting, so thank you for sharing that!
ofrzeta 5y ago
I never see Mozilla's Kinto mentioned.

415 points by typingmonkey 5y ago

porcc 5y ago

gunjs?

tommiegannert 5y ago
I've tried to read the code 3-4 times now. Every time I have to give up. The "chain" thing is really bad code, even admitted by the author. If you're considering GunJS, take a look at the code and verify it's something you'd want to debug, before going all-in.
- porcc 5y ago
  I did go all in and it worked fine, great even. The devs are responsive on their Gitter for sorting things out and it was generally simple enough that I wouldn't need to mess with the internals
- jamil7 5y ago
  I saw the author present once, I couldn't follow anything they were talking about or the code samples presented.

dSebastien 5y ago

I've used CouchDB and PouchDB on a previous project, and it was a blast. The built-in features like replication and HTTP API are great. My only regret is the limited support (at the time) for full text search and complex queries. I suppose I like joins a bit too much.. :) I deployed CouchDB in a Kubernetes cluster (not with HA as I didn't have high availability requirements), and it was working great.

psychometry 5y ago
Yeah although the docs claim to support things like regex-based searches, it's so horrendously slow it shouldn't even be listed as a feature.

redwood 5y ago

Surprised not to see Realm included (www.realm.io)

carabiner 5y ago

Some charts for that table would be so awesome.

smcnally 5y ago
I'm not the OP or author. This sheet[0] has the Metrics and Feature Map data for charting, sorting, etc. directly from their GH page. [0] https://docs.google.com/spreadsheets/d/12ReO-4_bZ2BaLj9P6oJT7m0CdXpz_Z9ie_F1ShzUHHI/edit?usp=sharing https://docs.google.com/spreadsheets/d/12ReO-4_bZ2BaLj9P6oJT...

karmelapple 5y ago

So glad to see PouchDB included. We use it and have generally had a great experience! We use it with CouchDB on the backend, and Couch seems like a fantastic way to go for use cases involving syncing data between devices with an offline mode and syncing between clients. It was built from the ground up with replication in mind. Biggest bummer of CouchDB? If you’re not hosting it yourself, there’s only one major player in the market that I know of: IBM Cloudant. They contribute much to Apache CouchDB though, and hosting it yourself doesn’t seem too difficult, especially for small, simple use cases. Anyone else using CouchDB?

knubie 5y ago
I use the PouchDB/CouchDB combo for exactly the use case your describing. The query language for CouchDB leaves a little to be desired, but ultimately it has worked well for me. I'm self hosting on a digital ocean droplet.
- tata71 5y ago
  Is DO offering anything close to enterprise grade?
  - kbenson 5y ago
    "enterprise grade" covers a a much larger spectrum of uses and needs with regard to features and stability than non-enterprise grade, making it kinda hard to answer that generally. They offer VMs, containers, managed DB instance offerings, block storage, multiple regions and datacenters, load balancing, etc. They have an API to control all those things, and modules available in many popular languages. But that's basically what many people consider table stakes for a service like that, and indeed there are competitors like Vultr and Upcloud that offer the same. Will any of them have quite the same level of offerings that AWS or GCE or Azure offer? Probably not. But for a great many people what they have is all the enterpise level stuff they'll need or use, and it is decidedly easier to just start up a cheap linux VM and take care of it on one of these services compared to AWS or GCE or Azure, so if what you want are somewhat manually managed cloud VMs, I highly recommend one of these services over one of the big names. I've used all the services named, and I still prefer DO for just throwing up a cheap $5-$10 VM for personal stuff, or to spin up a temporary VM for testing something out. On DO that's a couple second process when you do it manually by clicking around.

FractalHQ 5y ago

I imagine that Supabase would be a perfect candidate for this project! I would love to get the authors opinion on it too.

typingmonkey 5y ago
I considered to add Supabase, RethinkDB and meteor. But they are not really client side databases. They realtime stream query results from the server to the client.
- knes 5y ago
  Did you look into https://ditto.live/ https://ditto.live/? Its and off-line first db too. I never used it but looks very interesting and would be interested how it compare to the technologies you picked
- irae 5y ago
  I love RethinkDB. I really believe it is a database that deserves more attention. We've been running on it for six years and it is flawless, easy to use. Like any other database you need to understand how it performs and how to query efficiently. But it does has everything required for streaming and syncing data in a modern way. You can open streaming queries and even with a backlog of data, so syncing should be pretty easy. Maybe would be feasible to fork PouchDB to use RethinkDB as a backend?
- PKop 5y ago
  Firestore isn't really a client side database then either. It is certainly also querying results from a server.
jonplackett 5y ago
I second that! Would like to see supabase since that’s what I’ve been using a lot recently and really liking!
- mastazi 5y ago
  I have a question, how do you manage the fact that Supabase is based on a relational db[1]? Do you just put all fields (except the primary key) in a JSON column[2]? Does it work well if you use it that way? We were investigating it as a way to escape Firebase's vendor lock-in but the fact that we would have to manage schema migrations was a bit of a deal breaker based on our use case. I'm also interested in hearing about any alternatives to Supabase that use a document-based nosql db. [1] https://supabase.io/database https://supabase.io/database [2] https://www.postgresql.org/docs/13/datatype-json.html https://www.postgresql.org/docs/13/datatype-json.html

dvdhnt 5y ago

I really enjoy examples like this, thanks, I’ll be exploring it. As an aside… I would truly love to explore a collection of interesting ways to use SQLite. It’s such an impressive piece of technology that I’d like to use more often. Please share if you have something similar!

heyzk 5y ago
I had a really great experience building an EAV store with Datalog as the query interface on top of SQLite for embedding in native mobile apps. Pros: querying complex data hierarchies was easy, and was able to skip the pain typically associated with managing a SQL schema.
- phyrex 5y ago
  I would love to hear more! Did you write the datalog layer or is there one somewhere? Is there any code available I could see?
  - dunham 5y ago
    You might be interested in the now defunct Mentat project from Mozilla. They made an EAV store with syncing on top of sqlite. It ran datalog queries by translating them into sql. https://github.com/mozilla/mentat https://github.com/mozilla/mentat
    - phyrex 5y ago
      I’m aware of mentat and eternally disappointed that it got shelved :(
- Mertax 5y ago
  What’s the application for this? EAV is often an anti-pattern when a schema could be defined, but I’m actually using it as well. Our application is an end-user-defined database for mobile data collection. The EAV model in SQLite is a bit of a cognitive burden but makes offline sync and conflict resolution pretty straight forward. It’s almost a crude CRDT implementation.

throwaway743 5y ago

Am I wrong to think that pouchdb uses indexeddb? Over the last 8 months, I've been working on a react/capacitor based android app (potentially ios later on) and was originally using idb-keyval, which uses indexeddb for key val storage, and things were great since it's a local storage solution compatible with react/capacitor, and one of my goals is to not rely on a remote data storage solution. As said, things were going great, but then a couple weeks ago things went to shit when all the data that was stored in the prototype on my android device was wiped. Apparently, both android and ios tend to wipe browser/web-view local storage at random/when space is needed(?). Dug around since looking for an alternative solution (would love a capacitor compatible mongodb solution), came across pouchdb via rxdb, but could've sworn there was mention that it relies on indexeddb. So just to be safe switched to sqlite and been rewriting components since. Lesson of the story, even if it isn't dependent on indexeddb, if you're looking for a local storage option for a mobile app and happen to be using a js framework with capacitor or anything that utilizes a web-view, stay away from anything that uses indexeddb. If a wipe like this were to happen post release, the chance that your app succeeds afterwards would be near 0% Edit: so yeah, just double checked/was reading through the readme of this project, and pouchdb via rxdb is reliant on indexeddb

typingmonkey 5y ago
Author here. I am using RxDB with Capacitor (iOS and Android app). You can use the SQLite based pouchdb adapter with capacitor. It keeps your data and is (sometimes) faster. Here [1] I have documented a whole section about how to use RxDB+SQLite in Capacitor. [1] https://rxdb.info/adapters.html https://rxdb.info/adapters.html
- throwaway743 5y ago
  Ah okay cool. Thank you for pointing this out. Now would I use the adapter for react native or for cordova? If react native, I've been writing with react js and have been under the impression that react native specific plugins aren't compatible with react js. Is that wrong? If cordova, it's totally compatible with capacitor? Sorry just want to make sure before jumping in

sushsjsuauahab 5y ago

How far the world has fallen from Linux, MySQL, SpringBoot, and HTML/CSS/JS

spdebbarma 5y ago
You can only live in the past for so long. To bystanders, you're falling behind by being cynical and ignorant of newer solutions. Don't get me wrong. The world still uses the technology you mentioned at large, but the industry has in-fact built upon and grown alongside older technology.
- mhd 5y ago
  I don't want to seem like I'm defending SpringBoot, ferchrissakes, but still fail to see a big impact of PWAs. Especially a positive one for the user, not mere ad clicks. In general, the whole rich web app space reminds me of the "You're not making Christianity better, you're making Rock'n Roll worse" meme.
  - sushsjsuauahab 5y ago
    I am genuinely shocked that people don't like spring boot? It is very performant, open, and ergonomic in my experience.
    - mhd 5y ago
      2021 Spring and Maven could buy me chocolates and flowers all day long, I wouldn't forget what they did to me in the past... But seriously, I'm okay with working with SB when I have to, but quite often in those situations Java wouldn't be my first choice, and it's still all the putrescence of proper Spring underneath. A framework for a framework, with a bit too much magic for me. Explicit is better than implicit.
- sushsjsuauahab 5y ago
  While I certainly can fathom that other newer technologies are capable of solving interesting problems, I think my assertion is that the additional value they provide is not worth the risks involved in trying them.

shoo 5y ago

From the perspective of someone less familiar with this kind of thing, this comparison would be easier to understand with a bit of an introductory explanation about what job we're trying to do or problem we're trying to solve, and the assumed context or constraints.

amw-zero 5y ago
Offline-first is a well-known term. You can search for it elsewhere. When writing, it's important to choose who you're speaking to exactly so that you can avoid sharing context, since that takes up time and bandwidth. The truth is, communication is much more efficient if you don't re-explain every single concept that you are talking about.
- wolfram74 5y ago
  A compromise solution would be linking to a glossary or introductory material if someone is generally versed but not in a particular subject, we get a lot of, say, pure math people on here who find software engineering interesting.
  - y4mi 5y ago
    you'd still have to draw a line at some point which terms you'd explain as otherwise your article only consist of glossary information. i'm pretty sure the term offline-first wouldn't have met the cutoff point, as its _really_ well known from my experience.
    - typingmonkey 5y ago
      I wrote so much things and posted it on HN over the years. There is just no way to make everyone happy, someone always complains that some infos are missing or too much. I now leave things out that can be googled and are already known by "most" readers.
      - amw-zero 5y ago

wanderingmind 5y ago

Any information on how scalable are these databases compared to traditional SQL databases? Or specifically, when should you use this (in prototyping or production)

rendall 5y ago
IMO the principal consideration here is that these are offline/local databases for browsers and (probably?) Electron, so they are not intended to be scalable at all. If you have a progressive web app (PWA) and you need a queryable database for some reason, then you would use these. Otherwise, stick your queries behind API endpoints.
janl 5y ago
CouchDB is a lot more scalable than SQL databases because it has a distributed scaling model built in (just add nodes), no need to mess with read-replicate and finicky hot-failover, it all just works out of the box (Dynamo style).
- karmelapple 5y ago
  It’s more scalable in theory, and I talked its praises in a different comment, but our team has hit scaling issues with our one-user-per-database approach. It was a mess to sort out, but Cloudant support was very helpful. Our major issue: we write many small documents, and we write them over every user’s database fairly frequently. And Cloudant’s default settings don’t like that with a one-user-per-database approach. In fact, they discourage anyone from the one-db-per-user approach these days: https://www.ibm.com/cloud/blog/cloudant-best-and-worst-practices-part-1 https://www.ibm.com/cloud/blog/cloudant-best-and-worst-pract... That blog post calls it an anti-pattern, but I would respectfully disagree. It is an absolutely great pattern to keep a native app and a web app in sync across multiple devices with intelligent conflict resolution. A solution was to reduce the number of shards that a database was split out over, since our database’s data is pretty small overall and we didn’t need each database split out so much across our cluster.
  - janl 5y ago

nyanpasu64 5y ago

Not directly related to the post (which is focused on which database to host for your app), but I'm writing desktop apps (think DAWs) in C++ and Rust (not JS), and want to synchronize settings through a Dropbox or Google Drive (so I don't have to host my own cloud sync servers). What's a good library or schema to achieve this? Personally I usually don't have multiple instances of the same app open on multiple machines, but other people might open the same or different files on their desktop and laptop. - Not all settings should be synchronized (don't include machine-specific "recent files" paths). - How should settings be stored locally (if I may have multiple instances of my app open on a single machine)? Registry (Windows-only)? INI with atomic saving (requires care and locking to prevent multiple instances from trampling or racing with each other)? SQLite? IMO Stylus is a pretty good implementation of offline-first cloud settings sync over Dropbox/etc. It's currently based around one JSON file per CSS file (Dropbox/Apps/Stylus - Userstyles Manager/docs/uuid.json), and what appears to be a transaction log (Dropbox/Apps/Stylus - Userstyles Manager/changes/number.json). Cloud sync has been 100% reliable in my experience, though I do notice temporary file lock errors when switching between different machines in my dual-boot setup (but sync seems to be eventually consistent nonetheless). uBlock Origin is worse. Instead of merging settings, it expects the user to upload and download the entire settings blob at once (and pulling an old blob can erase changes you've made locally). And in the past it's entirely failed to sync because the blob was too big to upload to Mozilla's servers. (Right now it "works" but takes several minutes for one computer to see a config uploaded from another computer.)

lbhdc 5y ago
I would imagine using a CRDT would be appropriate for the data you want to sync would let you sync state between multiple open clients.
franga2000 5y ago
I've done something pretty ridiculous to solve this problem and I'm not sure I'd recommend it, but here it is: - A directory is synced with the server with no conflict resolution - The application creates its config file in that directory, named by a random UUID, which is stored outside the synced folder - The config file stores the setting overrides (defaults were compiled-in) in any format (I used YAML) - Each setting override includes a "locked" and "lastModifed" - On startup (sync was external) all files in the directory are read and merged starting with the local one, then skipping any settings that are locked (locally or remotely), last modified wins Some deployments used a daily rsync cronjob, some had a mounted network share (with hilarious broken file locking) and of course it worked with direct bind mounts as well. I also briefly experimented turning the "locked" field into a "group" field to enable multiple "sync groups" with some keys shared globally and some only with other group members (even different groups for different settings), but it ended up not being useful for my use case, although it did work.

jdc 5y ago

See also Google's Lovefield SQL browser database: https://google.github.io/lovefield https://google.github.io/lovefield

rutierut 5y ago
Second this. Oh and it’s not reaaaaly abandoned, they just called it done. I looked into using this as well but eventually decided against it due to the lack of active development.
psychometry 5y ago
No commits in 2 years. Seems Google lost interest like usual.

LAC-Tech 5y ago

Last write wins is not a strategy for conflict resolution, it's a surrender. So I'm glad to hear something else apart from Pouch actually handles it. Anyone familiar with rxdb and can chime in on how they do it?

netghost 5y ago
It sounds like rxdb is built on top of pouch, so probably the same set of options with the possibility of some opinionated design or sensible defaults though I can't find anything. obvious.
- typingmonkey 5y ago
  Yes RxDB conflict resolution is equal to PouchDBs. At least for now, there are plans to improve from there where you have a global resoluting function instead of listening for conflicts in the changestream.

y4mi 5y ago

My biggest gripe with offline-first capability of PWAs is speed. a network roundtrip is generally faster then fetching the same information from indexeddb, and you still have to sync first because indexeddb tends to get wiped at inopportune times. its great if you're writing a traditional app though, but really unfortunate wrt to the PWAs

getcrunk 5y ago
Really? I've never used index db but to confirm your saying to fetch data from local memory is slower than network round trip?
- tehbeard 5y ago
  indexedDB is quite a low level in what the interface gives you. This includes what options you have available for querying (just accessing a key range, upper/lower bound, forward or backwards (backwards can be much slower), rather than a full query language like SQL etc.. So you need to design your app / data format and pre-plan any queries so you'll have the indexes you need, or do some glue logic to combine indexes as needed.
- dmw_ng 5y ago
  The complaint was about IndexedDB access, not local memory. Folk mentioning this are usually referring to the disastrous implementation in Chrome: see https://dev.to/skhmt/why-are-indexeddb-operations-significantly-slower-in-chrome-vs-firefox-1bnd https://dev.to/skhmt/why-are-indexeddb-operations-significan... and https://jlongster.com/future-sql-web https://jlongster.com/future-sql-web
  - getcrunk 5y ago
    Wow. Orders of magnitude slower than ff.
tehbeard 5y ago
Why is that?. Is this because you're on a fibre link? indexedDB is being used with several layers to add a comfy SQL and ORM layer for you? I've used indexedDB on a couple of projects at work, while there are definitely downsides with its indexing design, limited querying options and the menagerie of fuckups by Team Fruit(TM), it works well as a local cache when our clients are out on site with their customers and all they've got is a crappy intermittent 3/4g signal.

peterthehacker 5y ago

What kind of consistency models [0] do Offline-first databases like RxDB and PouchDB have? I was thinking read uncommitted, but they might allow dirty writes. Maybe there’s some CRDTs under the hood… I can’t find any documentation on consistency though, anyone here know? [0] https://jepsen.io/consistency https://jepsen.io/consistency

genewitch 5y ago
Is the reason for all of these sorts of comments some sort of "never lose data" quality assurance? There's consistency, write first, latency, replication, and then acronyms. I just implement hardware and OS stuff, but I like the DB people to be happy. What am I missing?
janl 5y ago
CouchDB docs have you covered: https://docs.couchdb.org/en/stable/replication/intro.html https://docs.couchdb.org/en/stable/replication/intro.html
- peterthehacker 5y ago
  Thanks! Very cool. I haven’t used couchdb in prod. I’ll read into this more.

spankalee 5y ago

Wait, this is benchmarking the Firestore emulator. How is that relevant?

typingmonkey 5y ago
It is benchmarking the firestore JavaScript library that runs at the client.
- spankalee 5y ago
  Maybe, but I don't see where the browser is brought offline to enforce that. It looks like the emulator is still running and there's still a connection to it. Can you show where you force the browser offline?

globular-toast 5y ago

I've used watermelondb and found it quite nice to use. I mainly chose it because it has observable queries. I can live without conflict resolution and transactions on the client side because it's an electron app. Having a sqlite backend (via the electron process) would overcome some of the shortcoming wrt to the in memory database (and still allow in memory via sqlite if speed is desired). I wrote a backend sync for it which was quite easy although I haven't fully implemented support for everything because I don't need it (notably deletes).

joshxyz 5y ago

I wonder why multitab isnt supported on watermelon, crosstab updates thru localstorage api works great

franga2000 5y ago
> WatermelonDB uses the LokiJS adapter which is an in memory database that regularly persists the data to IndexedDB either on interval, or when the browser tab is closed
radex 5y ago
You can use multiple tabs but to have more than one write to the DB and not be overwritten relies on (online) sync. The problem is that Watermelon assumes a consistent view of the entire database, so you can't have multiple writers - at least not without synchronous notifications from IndexedDB (not a thing), leader election (cannot be made reliable), or some design sacrifices. To be reconsidered in the future...

RobertRoberts 5y ago

What is the advantages of using one of these client side databases vs using indexeddb directly?

radex 5y ago
IndexedDB is a joke of a database. Yes, it can store data, and you can create a very simple index, so it's _technically_ a database… But its ability to express queries is borderline useless for all but simplest use cases, it's slow, and it's very inconvenient to use. So solutions exist that range from giving IDB a simpler, more modern API, all the way to using IDB as a dumb storage medium to a fast in-memory database.
typingmonkey 5y ago
Replication, Encryption, Conflict Handling, Multi-Tab-Support, Compression, Observability and many more..

chadcmulligan 5y ago

I often feel I'm on another planet when I see timings for browser things people are recommending - Insert one message - best 9ms, insert 20 messages - best 33ms, worse - >8 Seconds! Why bother? just write a native client.

radex 5y ago
Browser-based apps have use cases even when you reject PWAs generally as a replacement for native apps. Trying out a new tool quickly, or short-term when using a tool with a customer, in enterprise where you can't install native tools, etc...
- chadcmulligan 5y ago
  Well make a binary that doesn't require installation, this is a very narrow use case for a technology thats being advocated for more than that use case
  - tatersolid 5y ago
    Most enterprise users can’t download and run an arbitrary binary due to numerous security controls and regulations. This is true even if that binary is signed - if it isn’t on an allow-list maintained by high priests it can’t be used. The real world demands web apps in the enterprise even in 2021.
    - deleted 5y ago
      [deleted]
    - chadcmulligan 5y ago
      This is a poor reason in 2021 - app stores exist. Software developers have an obligation, as professionals, to present the costs of web apps vs binaries when developing new projects, the browser is not an operating system and has real limitations - as this project demonstrates.
typingmonkey 5y ago

oblib 5y ago

For offline-first use PouchDB can connect directly to a CouchDB installed on your desktop PC as opposed to storing user data in your browser's built-in IndexedDB and syncing that with a Cloud based CouchDB. Syncing that local CouchDB to your web based CouchDB is very fast and it's done in the background so it doesn't affect the performance of actually using the app. You can click "Save" and move on with no waiting at all. So in this scenerio the speed of the DB is not necessarily a reflection of the speed of the app for the user.

typingmonkey 5y ago
> Syncing that local CouchDB to your web based CouchDB is very fast This is not true. When you compare the CouchDB replication with other replication protocols, it is slow. The reason is that CouchDB supports replication with many instances at the same time. This creates big overhead in handling revision trees. Many requests have to be made all the time. You can observe that by starting the PouchDB subproject in the comparison repo. Watch the network tab in dev tools. Another problem is that CouchDB does not support Websocket replication, everything is long polling and plain http requests. Other replications that only support many-clients-to-one-server are way faster. Both, on the initial load and on ongoing changes. This was the main reason why I build GraphQL replication for RxDB.
- oblib 5y ago
  >>This is not true. When you compare the CouchDB replication with other replication protocols, it is slow. That may be true but when a single user is working with an offline-first app connected to a CouchDB installed on their desktop pc that happens entirely in the background so they don't experience any lag. And while I've not done benchmark studies with CouchDB I have monitored the logs to watch those syncs and we're not talking painfully "slow" in real world use. It is reliable though. I've been using it for about 5 years now and it's been solid. And so has the work they've done to improve it. And I was not aware of RxDB, which is certainly interesting, so thank you for sharing that!

ofrzeta 5y ago

I never see Mozilla's Kinto mentioned.