14 ms·

CouchDB 3.0

266 points by ifcologne 7y ago

newfeatureok 7y ago
One interesting thing you can do with CouchDB is that you can have a webapp where a user can specify their own database and credentials and it works over HTTP(s). That's pretty unique. I'd love to see a SaaS using CouchDB and their "on-premise" offering just means the user provides their own database. I'm not sure how payment would work though - perhaps some verification proxy? Firebase is the gold-standard for offline apps (as a service). CouchDB replaces Cloud Firestore, and Keycloak replaces Authentication. I haven't seen OSS equivalents of Cloud Functions, ML Kit, and the other things (e.g. In-App messaging, and Cloud Messaging). It'd be nice to have the entire stack of Firebase bundled as a group of OSS projects, including CouchDB. Sad to see that per doc access control didn't make it in 3.0. Hopefully it'll be in 3.1.
- pachico 7y ago
  I didn't understand. You mean it's unique to work over http(s)?
  - newfeatureok 7y ago
    I mean your CouchDB instance itself is represented by a host and port and your application's data could be stored there and a native HTTP-based API to access said data. This is contrasted to most where you would need a driver and it's accessible only in the "back-end".
    - tomnipotent 7y ago
      > itself is represented by a host and port and your application's data could be stored there All databases are represented by a host and a port. I think you mean CouchDB offers a HTTP-based API that allows queries to be run without requiring a database-specific library and that because it's HTTP, it can be accessed via a browser.
      - newfeatureok 7y ago
        Haha, yes, exactly. I omitted the most important part - the [HTTP-based API].
      - pachico 7y ago
        That was my point exactly. Http API us very cool but very far from being unique to couchdb (see InfluxDB, ClickHouse, Prometheus, etc.)
    - Lio 7y ago
      To take that further, I really like the idea of running something locally like PouchDB and then letting it sync with a remote CouchDB using the replication protocol.
      - WorldMaker 7y ago
        That's where I fell into love with the CouchDB world. Building offline-first databases in PouchDB, and letting that sync to any HTTP address that speaks the replication protocol is sometimes a dream. In practice there are so many hurdles, sadly. (CORS, CSPs, firewalls, not enough things speaking the replication protocol that should, ...)
        didericis 7y ago
        I’m just getting into pouchdb and am liking it. I love the idea, for sure. I ran into a replication issue running through a proxy that had something to do with sessions being cached, but that was more the fault of the proxy. My biggest current concern is client side search and size. I’ve been developing a private journalling/notes app with some fairly particular bells and whistles for my own personal use. Although it’s mostly just text, I want to store a lot of text. I would much prefer the search not happen on the server, as I’d like to encrypt all data that hits the server. Have you used pouchdb quick search? If so, in your experience, can it handle full text search on about 1,000-5,000 documents with about 10kb worth of text each? Ideally I’d also like to store data uri for png sketches, and maybe photos. But I know photos especially would balloon the database size quite a bit/am worried I’d hit client side storage limits extremely quickly (I think I read some mobile devices have a 50mb limit, but I haven’t researched it that thoroughly yet)
        WorldMaker 7y ago
        I've not tried quick search. For the most part in the applications I've worked on I've just relied on the main primary key index (the _id field) for most lookups. Generally I'm using a `folder/structure/ULID` approach to keys and its really easy with start_key and end_key on allDocs to grab an entire "folder" at a time. I've had some pretty large "folders" and not seen too much trouble. At this point the biggest application I worked on pulls a lot of folders into Redux on startup and so far (knock on wood) performance seems strong. (ULIDs [1] are similar to GUIDs but are timestamp ordered lexicographically so synchronizations leave a stable sort within the folder when just pulling by _id order.) At least as far as my queries have been and what my applications needs have been, PouchDB is as fast or faster than the equivalent server-side queries (accounting for HTTPS time of flight), especially now that all modern browsers have good IndexedDB support. (There were some performance concerns I had previously when things fell back to WebSQL or worse, such as various iOS IndexedDB polyfills built on top of bad WebSQL polyfill implementations, and also a brief attempt that did not go well to use Couchbase Mobile on iOS only.) Photos have been the bane of my applications' existence, but not for client-side reasons. I had PouchDB on top of IndexedDB handling hundreds of photos without breaking a sweat and those size limits all have nice opt-ins permission dialogs for IndexedDB if you exceed them. Where I found all of the pain in working with photos was server side. CouchDB supports binary attachments, but the Replication Protocol is really dumb at handling them. Trying to replicate/synchronize photos was always filled with HTTP timeouts due to hideously bloated JSON requests (because things often get serialized as Base64), to the point where I was restricting PouchDB to only synchronize a single document at a time (and that was painfully slow). Binary attachments would balloon CouchDB's own B-Tree files badly and its homegrown database engine is not great with that (sharding in 3.0 would help, presumably). Other replication protocol servers had their own interesting limits on binary attachments; Couchbase in my tests didn't handle them well either and Cloudant turned out to have attachment size limits that weren't obvious and would result in errors, though at least their documentation also kindly pointed out that Cloudant was not intended to be a good Blob store and recommended against using binary attachments (despite CouchDB "supporting" them). (It sounds like the proposed move to FoundationDB in CouchDB 4.0 would also hugely shake up the binary attachment game. The 8 MB document limit already eliminates some of the photos I was seeing from iOS/Android cameras.) I'd imagine you'd have all the same replication problems with large data URIs (as it was the Base64 encoding during transfers that seemed the biggest trouble), without the benefits of how well PouchDB handles binary attachments (because of how well the browsers today have optimized IndexedDB handle binary Blobs). The approach I've been slowly moving towards is using `_local` documents (which don't replicate) with attached photos in PouchDB, metadata documents that do replicate with name, date, captions, ULID, resource paths/bucket IDs (and comments or whatever else makes sense) and a Blurhash [2] so there's at least a placeholder to show when photos haven't replicated, and side-banding photo replication to some other Blob storage option (S3 or Azure Storage). It's somewhat disappointing to need two entirely different replication paths (and have to secure both) and multiple storage systems in play, but I haven't found a better approach. [1] https://github.com/ulid/spec https://github.com/ulid/spec [2] https://blurha.sh/ https://blurha.sh/
- Graphguy 7y ago
  Cloudant on IBM Cloud is CouchDB API/replication compatible and offers support for Apache CouchDB (1). Also, OpenWhisk integrates nicely with CouchDB/Cloudant and can even be a backing persistence for it (2) (1) https://www.ibm.com/cloud/blog/announcements/announcing-support-and-a-kubernetes-operator-for-apache-couchdb https://www.ibm.com/cloud/blog/announcements/announcing-supp... (2)https://github.com/apache/openwhisk/blob/master/tools/db/README.md https://github.com/apache/openwhisk/blob/master/tools/db/REA...
  - newfeatureok 7y ago
    Cloudant is awesome, but it's way too expensive IMHO.
    - Graphguy 7y ago
      Send me an email (in profile.) Would love to chat and see what we can do for you.
      - mauflows 7y ago
        If you indeed work for Cloudant, please consider trying to convince someone to invest in PouchDB. It looks mostly unmaintained and it would be in IBM's and the community's interest to keep it running!
    - splatcollision 7y ago
      Partitioned dbs are supposed to allow you to query more cheaply, haven't implemented those yet.
    - karmelapple 7y ago
      They’ve recently shifted their pricing scheme to be more on-demand; before that, you needed to do multi-tenant at very small scale, or buy dedicated clusters. We have dedicated clusters on Cloudant and they’ve run quite smoothly for many years. Someday we might switch to the on-demand IBM Cloud pricing, but haven’t done it yet.
- deleted 7y ago
  [deleted]
- chasers 7y ago
  I'm doing this with BigQuery for Logflare (logflare.app).
- WorldMaker 7y ago
  Yeah, I'm still disappointed that the MongoDB API outpaced the CouchDB Replication Protocol in general adoption. As nice as Cloudant can be some of the time, I know that my IT group would be a lot happier if we could use Cosmos DB (and/or if Cloudant would just directly support Azure data centers again). Every now and again I wonder if I could implement the CouchDB Replication Protocol on top of Cosmos DB with a presumably hairy ball of Azure Functions and hoping someone beats me to needing that to exist and scratches that itch for me. (Cosmos DB's changes feed is so almost right for the job it hurts because it sounds like it should be easy, and yet I assume it won't be.)
- moenzuel 7y ago
  For a Cloud Functions like project, OpenFaas seems like a promising project that I’ve been watching but have not yet had the chance to use.
- kache_ 7y ago
  Cloudant off IBM cloud. Full disclosure; I utilized it to support the application layer on IBM cloud.
- matlin 7y ago
  I'm really can't wait for the per-doc permissions because I'm building something very similar to what you're describing and with CouchDB!focusing on the database and auth side first and then adding functions. So shameless plug if you're interested in signing up for the alpha: https://www.aspen.cloud https://www.aspen.cloud
canada_dry 7y ago
Maybe I'm being petty, but it doesn't fill me with confidence when the ssl certificate on their website isn't even configured properly (valid for uberspace.de domain). To clarify: seems their main site is on apache.org. But, their www.couchdb.org site (hosted on uberspace.de) doesn't have a correct cert.
- lars_francke 7y ago
  For me I get a valid Let's Encrypt certificate that has blog.couchdb.org in its SAN list.
- Hedja 7y ago
  Their blog is hosted on Wordpress.com which seems to be using Let's Encrypt to generate one certificate for multiple different, unrelated custom domain names. Maybe you encountered a bug where it served the wrong cert for a different batch of custom domains.
- deleted 7y ago
  [deleted]
- canada_dry 7y ago
  Update: someone has now fixed it.
knubie 7y ago
CouchDB is awesome and feels way ahead of its time. Its design docs are extremely powerful, to the point that you can build entire web apps with CouchDB alone (not that that's recommended anymore). Plus with PouchDB you can create offline-first apps that sync with a remote CouchDB instance.
- code-is-code 7y ago
  If you like PouchDB, you should also check out RxDB. It is build on top of PouchDB and is optimised for realtime-applications where you can subscribe to queries and stuff. https://github.com/pubkey/rxdb https://github.com/pubkey/rxdb
- jfkebwjsbx 7y ago
  Ahead of its time? PL/SQL also allowed (and allows) you to create entire apps within a database.
e12e 7y ago
Oh wow, this is great news. I though the project was effectively long dead. Is there a new/up-to-date "couchapp" too? > Default installations are now secure and locked down. More good news! Anyone have recent experience with couchdb? I see the (quickstart) docs use plain http - should one terminate ssl in front, eg with a recent version of haproxy?
- Graphguy 7y ago
  For anyone else looking to quickstart but on Kube, https://operatorhub.io/operator/couchdb-operator https://operatorhub.io/operator/couchdb-operator. Should add 3.0 soon.
- mauflows 7y ago
  I wish couch was used whenever users ask for an app to "sync to Dropbox". I don't know if this changes with 3.0 but couch is naturally database per user, took me five minutes to install on my rpi with docker, very good admin interface, the database is the frontend (no driver or separate process), and let's the application layer handle conflicts.
- janl 7y ago
  We use https://github.com/jo/couchdb-bootstrap https://github.com/jo/couchdb-bootstrap successfully. CouchDB does SSL natively, but we do recommend HAProxy.
tbrock 7y ago
At this point why would you use CouchDB over something like MongoDB? Seriously asking... Over the past 5 years MongoDB has gotten a great storage engine, transactions, distributed transactions, multi master replication, first class change streams and is very very solid as a foundational piece of infrastructure you can rely on while CouchDB has languished. I can’t imagine reaching for it in my tool belt when I need a document store over MongoDB but I’m obviously biased so I’m wondering if there is a lot I’m missing. Obviously it’s cool from a more open source databases standpoint — I love learning about how things are built and evolve over time.
- newfeatureok 7y ago
  The main reason most people use CouchDB is because of the HTTP API and offline support with Couchbase Mobile and PouchDB. Doesn't CouchDB have most of those things already from 2.3?
  - mauflows 7y ago
    I don't think couchbase has couchdb in mind for the mobile client anymore
    - Volundr 7y ago
      Correct. The newest version of CouchBase mobile no longer supports CouchDB as a replication target. It can still be accomplish with the CouchBase Sync Gateway, but get complicated quickly.
      - jamil7 7y ago
        I evaluated Couchbase mobile about a year ago and found although it worked well once setup there was a lot of overhead and the docs seemed a little all over the place and the fact that you can't also use the same DB on the web anymore with PouchDB meant I ultimately dropped it. It's a shame because there isn't really anything open source / self hosted like it for mobile.
  - 7y ago
hajile 7y ago
Reducing max document size from 4GB down to 8MB seems hyper-restrictive. For those interested, looks like the guts of CouchDB are going to be swapped out for FoundationDB. https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-prepare-for-4-0/ https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-...
- splatcollision 7y ago
  If you're trying to store single GB documents in couch, you're doing it wrong... Unless those are binaries you can usually fragment data logically across many documents, then write custom views to aggregate however you need to. Updates on huge docs would be painful!
  - hajile 7y ago
    I agree that 4GB is more than a sane person should probably be using. I don't think that going above 8MB is very hard or uncommon though. If I'm going to spread everything across many different documents and document types and then join them all together, I then have to make a case for why I'm still choosing Couch over a RDBMS.
  - meddlepal 7y ago
    I agree in large with your point that multi-GB documents is perhaps excessive but this does create a heck of a migration problem for a lot of users that aren't even close to 4GB.
    - Volundr 7y ago
      It does, but keep in mind for 3.0 this is a change to the default settings, not a hard cap. The idea is to give people lots of warning and time to do any migrations necessary prior to 4.0.
- newfeatureok 7y ago
  8MB is just the default, you can switch it back to 4GB if you want, but you won't have an easy time switching to 4.0 due to the 8MB limit imposed by FoundationDB.
splatcollision 7y ago
CouchDB is awesome, full stop. While it's missing some popularity from MongoDB and having wide adoption of things like mongoose in lots of open source CMS-type projects, it wins for the (i believe) unique take on map / reduce and writing custom javascript view functions that run on every document, letting you really customize the way you can query slice and access parts of your data... Example: I'm building a document analysis app that does topic + keyword frequency vectorization of a corpus of documents, only a few thousand for now. I end up with a bunch of documents that have "text": "here is my document text..." and "vector": [ array of floating point values ...]. What I can do with couchdb is store that 20d vector and emit integers of it as a query key: var intVectors = doc.vector.map(function(val){ return Math.floor(val) }) emit(intVectors, 1); Then I can match an input document's vector (calculated the same as corpus documents), calculate a 'range' of those vectors, pass it as start and end keys, and super quickly get a result from the database of 'here are documents that have vectors similar to your input'... Super fun, quick and flexible to work with!
- xwowsersx 7y ago
  it's = "it is" its = possessive form of "it" So should be "it wins for its unique..".
  - splatcollision 7y ago
    Yep, thanks.
  - devhead 7y ago
    there's always one...as in there is always one who has to poke people for grammer. sighs.
    - xwowsersx 7y ago
      In his original message right after "it's" he put "(I believe?)", implying he was unsure about the usage and inviting feedback so I responded to that. If I misread, my bad, but I thought he was specifically asking.
Phillips126 7y ago
I haven't heard of CouchDB in quite some time, great to see it still improving. I used it years ago when I was experimenting with Ionic[0]. What appealed to me was that I could use CouchDB (cloud) and PouchDB[1] (device) to and have a replicated copy of the data locally. The application was used in areas where network connection was very limited. Using this strategy I was able to ensure the mobile devices data was as recent as the last time it had a network connection. [0] - https://ionicframework.com/ https://ionicframework.com/ [1] - https://pouchdb.com/ https://pouchdb.com/
- lytefm 7y ago
  I can confirm that the stack still works well :) We've been developing a cross-platform app for the German market - therefore the need of offline capability - since 2017 and never had any real issues with Pouch/Couch, that part just worked. The upgrade from Ionic 3 to 4 was was quite painful though. For user authentication I've forked the nowadays unmaintained superlogin package [1], which still does a great job when keeping the dependencies up to date. [1] https://github.com/LyteFM/superlogin https://github.com/LyteFM/superlogin
crudbug 7y ago
Congrats to the whole team. Looking forward to CouchDB 4.0/FoundationDB goodies. Do we have any roadmap details on this.
agumonkey 7y ago
anybody acquainted with pouchdb devs ? just to know if there are plans to migrate already or not
- gtirloni 7y ago
  https://github.com/pouchdb/pouchdb/issues/7987 https://github.com/pouchdb/pouchdb/issues/7987
- Volundr 7y ago
  For those who don't want to follow the link, there are no changes to the replication protocol in Couch 3.0, so PouchDB already works.
smoyer 7y ago
I built two products on CouchDB 1.x starting in 2010 ... version three is another amazing step forward! For my more recent projects, I've replaced CouchDB with clustered PostgreSQL using JSON columns as I really enjoy the ability to write SQL queries for against the JSON and to use the built-in full-text search capabilities. I think both CouchDB and clustered PostgreSQL are amazing tools and it's nice to be able to choose between them as needed. The best advice I've heard is to choose CouchDB when you know your queries ahead of time and the data "schema"[1] is variable and choose PostgreSQL when you know your data ahead of time and your queries are variable. [1] In this case, a JSON document but either with a JSON-schema or marshaled/unmarshaled into a strict type.
- jimstr 7y ago
  I've gotten the impression that clustered Postgres still isn't very straightforward to run. Do you mind elaborating on your ideal setup and point to some resources? Thanks!
  - smoyer 7y ago
    It's not straightforward at all but it's better than it was five years ago ... you can use something more "meta" like SymmetricDS (https://www.symmetricds.org/ https://www.symmetricds.org/). I haven't used it personally but a dirt simple way to get an HA, scalable PostgreSQL instance would be to use Amazon's Aurora DB.
- karmelapple 7y ago
  JSON Schema has been a big benefit for our use case. Our iOS, Android, and web app all pull in a schema from one repo, which serves up that schema via Cocoapods, Gradle, or npm. We built it years ago and it’s worked smoothly ever since.
LoSboccacc 7y ago
is the lucene search indexer synchronous with couchdb days updates? I'm wondering how people solve the common search after create pattern when using external indexes
- janl 7y ago
  Yup, works with clustering and everything: https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-easy-fulltext-search/ https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-...
anonyfox 7y ago
Has anyone here tried to use couchdb directly within an elixir/erlang OTP application? As like, „mix install“? Would kill for couchdb as a library!
yatsyk 7y ago
CouchDB/PouchDB looks very promising for offline first apps, but I can’t understand how to restrict bad clients. Client potentially could insert document of huge size or execute expensive query and degrade experience of other clients on the same server. Is it any way to prevent this?
- newfeatureok 7y ago
  You resolve that issue the same way you would resolve the same issue if you were using Postgres - you introduce some back-end. For your example specifically I'd use a proxy.
  - yatsyk 7y ago
    Custom backend means no synchronisation and no advantages over postgres. Do you propose to create proxy that parses query and estimates complexity? I think this task at least as hard as implementing couchdb myself (actually harder) Is there any secure open source code with pouchdb/couchdb integrations?
    - daleharvey 7y ago
      Your backend can be a reverse proxy that authenticates requests then passes them off to CouchDB (or PouchDB, since that also runs on the server). I have an example up @ https://github.com/daleharvey/noted https://github.com/daleharvey/noted. The server is 200 lines and does signup / email authentication etc.
      - yatsyk 7y ago
        This server can't prevent authenticated user from uploading huge document of running expensive query.
        daleharvey 7y ago
        I wasnt worried about that since it is a basic proof of concept, adding that would make it ~210 lines of code.
johnchristopher 7y ago
> – Updated to modern JavaScript engine SpiderMonkey 60 Yes ^^ ! Congrats to the team. These people are some of the nicest and most supportive devs I know of in the OSS community (or whatev'). They show a great deal of patience in their slack channel and are always welcoming and answering stupid questions from idiots like me.
- janl 7y ago
  <3
gigatexal 7y ago
Can't wait to see what CouchDB 4.0 with FoundationDB at it's core does for the db. This is a great release too!
pawelk 7y ago
Other than being a great solution for some problems I wanted to highlight the fact that CouchDB has commited to SpiderMonkey (the Mozilla JS engine) since the very beginning and is one of the few projects helping to fend away the V8 monoculture.
wildchild 7y ago
CouchDB, couchbase, etc are all useless junk.
couchdb_ouchdb 7y ago
I'm surprised to see so much love for CouchDB in this thread. I don't think it's been widely adopted in corporate america and has lost the war to MongoDB closed source or not. I joined a company where it's being used backing a mobile app with couch/pouch in production. We can't wait to get off of it. Writes are slow. Reads are worse. Having a DB per user is a scaling and backup nightmare. If you run into any issues, it's a ghost town. I'm glad the CouchDB Team is forging ahead, but who is really using this database?
- staticautomatic 7y ago
  Would you be willing to say more? Inquiring minds want to know.
- janl 7y ago
  I sadly can’t name names, but rest assured the fortune 500 is heavily involved. OTOH, publicly known big companies using CouchDB include Apple and IBM. And I worked on a team that used CouchDB’s offline capability in the 2015 Ebola crisis in West Africa. That work also lead to the first Ebola vaccine ever. That’s why we do CouchDB :)
mark_l_watson 7y ago
I haven't used CouchDB in years. I just downloaded and installed it. Interesting that there are no apparent links to client libraries in different languages. Perhaps most people just use the HTTP API Reference and roll their own.
- mikekchar 7y ago
  In Ruby there is a CouchRest gem, which I've used, but to be honest a REST interface that talks JSON is so easy to use that I've often thought we'd be better off without anything specific.
  - mark_l_watson 7y ago
    Thanks, that makes sense.
- james_s_tayler 7y ago
  There are a couple of client libraries in .NET Some are no longer maintained. Some still work.
haolez 7y ago
I once read that the right way to use CouchDB is for every user to have its own database. However, how does this work with BI? Or with public data that should be known by all users? Do I create a single centralized DB just for that kind of data? Maybe aggregate data from all users' DBs? Genuinely curious.
- CameronNemo 7y ago
  For public data, you can try to partition it in such a way that writes can be merged without any potential conflicts. E.g. a user's posts are in a separate partition. I have never done this with CouchDB, but the technique is described in Martin Kleppman's __Designing Data Intensive Applications__.
- janl 7y ago
  You can replicate all per-user DBs into a central database today. We are working on per-document-access-control at the moment, to support this use-case out of the box
seigel 7y ago
CouchDB is good. Yes. I still dream of the day when the cluster will balance shards automatically and recover better from losing and replacing nodes. :D
fiatjaf 7y ago
Many commenters here still think CouchDB is the same thing it was many years ago. CouchDB was a simple but very powerful idea (that still needed improvements), but it was coopted into something not very nice nor good nor useful. See my old rant about it and why it failed: http://web.archive.org/web/20170530122143/http://entulho.fiatjaf.alhur.es/notes/about-couchdb/ http://web.archive.org/web/20170530122143/http://entulho.fia...

266 points by ifcologne 7y ago

One interesting thing you can do with CouchDB is that you can have a webapp where a user can specify their own database and credentials and it works over HTTP(s). That's pretty unique. I'd love to see a SaaS using CouchDB and their "on-premise" offering just means the user provides their own database. I'm not sure how payment would work though - perhaps some verification proxy? Firebase is the gold-standard for offline apps (as a service). CouchDB replaces Cloud Firestore, and Keycloak replaces Authentication. I haven't seen OSS equivalents of Cloud Functions, ML Kit, and the other things (e.g. In-App messaging, and Cloud Messaging). It'd be nice to have the entire stack of Firebase bundled as a group of OSS projects, including CouchDB. Sad to see that per doc access control didn't make it in 3.0. Hopefully it'll be in 3.1.

pachico 7y ago
I didn't understand. You mean it's unique to work over http(s)?
- newfeatureok 7y ago
  I mean your CouchDB instance itself is represented by a host and port and your application's data could be stored there and a native HTTP-based API to access said data. This is contrasted to most where you would need a driver and it's accessible only in the "back-end".
  - tomnipotent 7y ago
    > itself is represented by a host and port and your application's data could be stored there All databases are represented by a host and a port. I think you mean CouchDB offers a HTTP-based API that allows queries to be run without requiring a database-specific library and that because it's HTTP, it can be accessed via a browser.
    - newfeatureok 7y ago
      Haha, yes, exactly. I omitted the most important part - the [HTTP-based API].
    - pachico 7y ago
      That was my point exactly. Http API us very cool but very far from being unique to couchdb (see InfluxDB, ClickHouse, Prometheus, etc.)
  - Lio 7y ago
    To take that further, I really like the idea of running something locally like PouchDB and then letting it sync with a remote CouchDB using the replication protocol.
    - WorldMaker 7y ago
      That's where I fell into love with the CouchDB world. Building offline-first databases in PouchDB, and letting that sync to any HTTP address that speaks the replication protocol is sometimes a dream. In practice there are so many hurdles, sadly. (CORS, CSPs, firewalls, not enough things speaking the replication protocol that should, ...)
      - didericis 7y ago
        I’m just getting into pouchdb and am liking it. I love the idea, for sure. I ran into a replication issue running through a proxy that had something to do with sessions being cached, but that was more the fault of the proxy. My biggest current concern is client side search and size. I’ve been developing a private journalling/notes app with some fairly particular bells and whistles for my own personal use. Although it’s mostly just text, I want to store a lot of text. I would much prefer the search not happen on the server, as I’d like to encrypt all data that hits the server. Have you used pouchdb quick search? If so, in your experience, can it handle full text search on about 1,000-5,000 documents with about 10kb worth of text each? Ideally I’d also like to store data uri for png sketches, and maybe photos. But I know photos especially would balloon the database size quite a bit/am worried I’d hit client side storage limits extremely quickly (I think I read some mobile devices have a 50mb limit, but I haven’t researched it that thoroughly yet)
        WorldMaker 7y ago
        I've not tried quick search. For the most part in the applications I've worked on I've just relied on the main primary key index (the _id field) for most lookups. Generally I'm using a `folder/structure/ULID` approach to keys and its really easy with start_key and end_key on allDocs to grab an entire "folder" at a time. I've had some pretty large "folders" and not seen too much trouble. At this point the biggest application I worked on pulls a lot of folders into Redux on startup and so far (knock on wood) performance seems strong. (ULIDs [1] are similar to GUIDs but are timestamp ordered lexicographically so synchronizations leave a stable sort within the folder when just pulling by _id order.) At least as far as my queries have been and what my applications needs have been, PouchDB is as fast or faster than the equivalent server-side queries (accounting for HTTPS time of flight), especially now that all modern browsers have good IndexedDB support. (There were some performance concerns I had previously when things fell back to WebSQL or worse, such as various iOS IndexedDB polyfills built on top of bad WebSQL polyfill implementations, and also a brief attempt that did not go well to use Couchbase Mobile on iOS only.) Photos have been the bane of my applications' existence, but not for client-side reasons. I had PouchDB on top of IndexedDB handling hundreds of photos without breaking a sweat and those size limits all have nice opt-ins permission dialogs for IndexedDB if you exceed them. Where I found all of the pain in working with photos was server side. CouchDB supports binary attachments, but the Replication Protocol is really dumb at handling them. Trying to replicate/synchronize photos was always filled with HTTP timeouts due to hideously bloated JSON requests (because things often get serialized as Base64), to the point where I was restricting PouchDB to only synchronize a single document at a time (and that was painfully slow). Binary attachments would balloon CouchDB's own B-Tree files badly and its homegrown database engine is not great with that (sharding in 3.0 would help, presumably). Other replication protocol servers had their own interesting limits on binary attachments; Couchbase in my tests didn't handle them well either and Cloudant turned out to have attachment size limits that weren't obvious and would result in errors, though at least their documentation also kindly pointed out that Cloudant was not intended to be a good Blob store and recommended against using binary attachments (despite CouchDB "supporting" them). (It sounds like the proposed move to FoundationDB in CouchDB 4.0 would also hugely shake up the binary attachment game. The 8 MB document limit already eliminates some of the photos I was seeing from iOS/Android cameras.) I'd imagine you'd have all the same replication problems with large data URIs (as it was the Base64 encoding during transfers that seemed the biggest trouble), without the benefits of how well PouchDB handles binary attachments (because of how well the browsers today have optimized IndexedDB handle binary Blobs). The approach I've been slowly moving towards is using `_local` documents (which don't replicate) with attached photos in PouchDB, metadata documents that do replicate with name, date, captions, ULID, resource paths/bucket IDs (and comments or whatever else makes sense) and a Blurhash [2] so there's at least a placeholder to show when photos haven't replicated, and side-banding photo replication to some other Blob storage option (S3 or Azure Storage). It's somewhat disappointing to need two entirely different replication paths (and have to secure both) and multiple storage systems in play, but I haven't found a better approach. [1] https://github.com/ulid/spec https://github.com/ulid/spec [2] https://blurha.sh/ https://blurha.sh/
Graphguy 7y ago
Cloudant on IBM Cloud is CouchDB API/replication compatible and offers support for Apache CouchDB (1). Also, OpenWhisk integrates nicely with CouchDB/Cloudant and can even be a backing persistence for it (2) (1) https://www.ibm.com/cloud/blog/announcements/announcing-support-and-a-kubernetes-operator-for-apache-couchdb https://www.ibm.com/cloud/blog/announcements/announcing-supp... (2)https://github.com/apache/openwhisk/blob/master/tools/db/README.md https://github.com/apache/openwhisk/blob/master/tools/db/REA...
- newfeatureok 7y ago
  Cloudant is awesome, but it's way too expensive IMHO.
  - Graphguy 7y ago
    Send me an email (in profile.) Would love to chat and see what we can do for you.
    - mauflows 7y ago
      If you indeed work for Cloudant, please consider trying to convince someone to invest in PouchDB. It looks mostly unmaintained and it would be in IBM's and the community's interest to keep it running!
  - splatcollision 7y ago
    Partitioned dbs are supposed to allow you to query more cheaply, haven't implemented those yet.
  - karmelapple 7y ago
    They’ve recently shifted their pricing scheme to be more on-demand; before that, you needed to do multi-tenant at very small scale, or buy dedicated clusters. We have dedicated clusters on Cloudant and they’ve run quite smoothly for many years. Someday we might switch to the on-demand IBM Cloud pricing, but haven’t done it yet.
deleted 7y ago
[deleted]
chasers 7y ago
I'm doing this with BigQuery for Logflare (logflare.app).
WorldMaker 7y ago
Yeah, I'm still disappointed that the MongoDB API outpaced the CouchDB Replication Protocol in general adoption. As nice as Cloudant can be some of the time, I know that my IT group would be a lot happier if we could use Cosmos DB (and/or if Cloudant would just directly support Azure data centers again). Every now and again I wonder if I could implement the CouchDB Replication Protocol on top of Cosmos DB with a presumably hairy ball of Azure Functions and hoping someone beats me to needing that to exist and scratches that itch for me. (Cosmos DB's changes feed is so almost right for the job it hurts because it sounds like it should be easy, and yet I assume it won't be.)
moenzuel 7y ago
For a Cloud Functions like project, OpenFaas seems like a promising project that I’ve been watching but have not yet had the chance to use.
kache_ 7y ago
Cloudant off IBM cloud. Full disclosure; I utilized it to support the application layer on IBM cloud.
matlin 7y ago
I'm really can't wait for the per-doc permissions because I'm building something very similar to what you're describing and with CouchDB!focusing on the database and auth side first and then adding functions. So shameless plug if you're interested in signing up for the alpha: https://www.aspen.cloud https://www.aspen.cloud

canada_dry 7y ago

Maybe I'm being petty, but it doesn't fill me with confidence when the ssl certificate on their website isn't even configured properly (valid for uberspace.de domain). To clarify: seems their main site is on apache.org. But, their www.couchdb.org site (hosted on uberspace.de) doesn't have a correct cert.

lars_francke 7y ago
For me I get a valid Let's Encrypt certificate that has blog.couchdb.org in its SAN list.
Hedja 7y ago
Their blog is hosted on Wordpress.com which seems to be using Let's Encrypt to generate one certificate for multiple different, unrelated custom domain names. Maybe you encountered a bug where it served the wrong cert for a different batch of custom domains.
deleted 7y ago
[deleted]
canada_dry 7y ago
Update: someone has now fixed it.

knubie 7y ago

CouchDB is awesome and feels way ahead of its time. Its design docs are extremely powerful, to the point that you can build entire web apps with CouchDB alone (not that that's recommended anymore). Plus with PouchDB you can create offline-first apps that sync with a remote CouchDB instance.

code-is-code 7y ago
If you like PouchDB, you should also check out RxDB. It is build on top of PouchDB and is optimised for realtime-applications where you can subscribe to queries and stuff. https://github.com/pubkey/rxdb https://github.com/pubkey/rxdb
jfkebwjsbx 7y ago
Ahead of its time? PL/SQL also allowed (and allows) you to create entire apps within a database.

e12e 7y ago

Oh wow, this is great news. I though the project was effectively long dead. Is there a new/up-to-date "couchapp" too? > Default installations are now secure and locked down. More good news! Anyone have recent experience with couchdb? I see the (quickstart) docs use plain http - should one terminate ssl in front, eg with a recent version of haproxy?

Graphguy 7y ago
For anyone else looking to quickstart but on Kube, https://operatorhub.io/operator/couchdb-operator https://operatorhub.io/operator/couchdb-operator. Should add 3.0 soon.
mauflows 7y ago
I wish couch was used whenever users ask for an app to "sync to Dropbox". I don't know if this changes with 3.0 but couch is naturally database per user, took me five minutes to install on my rpi with docker, very good admin interface, the database is the frontend (no driver or separate process), and let's the application layer handle conflicts.
janl 7y ago
We use https://github.com/jo/couchdb-bootstrap https://github.com/jo/couchdb-bootstrap successfully. CouchDB does SSL natively, but we do recommend HAProxy.

tbrock 7y ago

At this point why would you use CouchDB over something like MongoDB? Seriously asking... Over the past 5 years MongoDB has gotten a great storage engine, transactions, distributed transactions, multi master replication, first class change streams and is very very solid as a foundational piece of infrastructure you can rely on while CouchDB has languished. I can’t imagine reaching for it in my tool belt when I need a document store over MongoDB but I’m obviously biased so I’m wondering if there is a lot I’m missing. Obviously it’s cool from a more open source databases standpoint — I love learning about how things are built and evolve over time.

newfeatureok 7y ago
The main reason most people use CouchDB is because of the HTTP API and offline support with Couchbase Mobile and PouchDB. Doesn't CouchDB have most of those things already from 2.3?
- mauflows 7y ago
  I don't think couchbase has couchdb in mind for the mobile client anymore
  - Volundr 7y ago
    Correct. The newest version of CouchBase mobile no longer supports CouchDB as a replication target. It can still be accomplish with the CouchBase Sync Gateway, but get complicated quickly.
    - jamil7 7y ago
      I evaluated Couchbase mobile about a year ago and found although it worked well once setup there was a lot of overhead and the docs seemed a little all over the place and the fact that you can't also use the same DB on the web anymore with PouchDB meant I ultimately dropped it. It's a shame because there isn't really anything open source / self hosted like it for mobile.
- 7y ago

hajile 7y ago

Reducing max document size from 4GB down to 8MB seems hyper-restrictive. For those interested, looks like the guts of CouchDB are going to be swapped out for FoundationDB. https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-prepare-for-4-0/ https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-...

splatcollision 7y ago
If you're trying to store single GB documents in couch, you're doing it wrong... Unless those are binaries you can usually fragment data logically across many documents, then write custom views to aggregate however you need to. Updates on huge docs would be painful!
- hajile 7y ago
  I agree that 4GB is more than a sane person should probably be using. I don't think that going above 8MB is very hard or uncommon though. If I'm going to spread everything across many different documents and document types and then join them all together, I then have to make a case for why I'm still choosing Couch over a RDBMS.
- meddlepal 7y ago
  I agree in large with your point that multi-GB documents is perhaps excessive but this does create a heck of a migration problem for a lot of users that aren't even close to 4GB.
  - Volundr 7y ago
    It does, but keep in mind for 3.0 this is a change to the default settings, not a hard cap. The idea is to give people lots of warning and time to do any migrations necessary prior to 4.0.
newfeatureok 7y ago
8MB is just the default, you can switch it back to 4GB if you want, but you won't have an easy time switching to 4.0 due to the 8MB limit imposed by FoundationDB.

splatcollision 7y ago

CouchDB is awesome, full stop. While it's missing some popularity from MongoDB and having wide adoption of things like mongoose in lots of open source CMS-type projects, it wins for the (i believe) unique take on map / reduce and writing custom javascript view functions that run on every document, letting you really customize the way you can query slice and access parts of your data... Example: I'm building a document analysis app that does topic + keyword frequency vectorization of a corpus of documents, only a few thousand for now. I end up with a bunch of documents that have "text": "here is my document text..." and "vector": [ array of floating point values ...]. What I can do with couchdb is store that 20d vector and emit integers of it as a query key: var intVectors = doc.vector.map(function(val){ return Math.floor(val) }) emit(intVectors, 1); Then I can match an input document's vector (calculated the same as corpus documents), calculate a 'range' of those vectors, pass it as start and end keys, and super quickly get a result from the database of 'here are documents that have vectors similar to your input'... Super fun, quick and flexible to work with!

xwowsersx 7y ago
it's = "it is" its = possessive form of "it" So should be "it wins for its unique..".
- splatcollision 7y ago
  Yep, thanks.
- devhead 7y ago
  there's always one...as in there is always one who has to poke people for grammer. sighs.
  - xwowsersx 7y ago
    In his original message right after "it's" he put "(I believe?)", implying he was unsure about the usage and inviting feedback so I responded to that. If I misread, my bad, but I thought he was specifically asking.

Phillips126 7y ago

I haven't heard of CouchDB in quite some time, great to see it still improving. I used it years ago when I was experimenting with Ionic[0]. What appealed to me was that I could use CouchDB (cloud) and PouchDB[1] (device) to and have a replicated copy of the data locally. The application was used in areas where network connection was very limited. Using this strategy I was able to ensure the mobile devices data was as recent as the last time it had a network connection. [0] - https://ionicframework.com/ https://ionicframework.com/ [1] - https://pouchdb.com/ https://pouchdb.com/

lytefm 7y ago
I can confirm that the stack still works well :) We've been developing a cross-platform app for the German market - therefore the need of offline capability - since 2017 and never had any real issues with Pouch/Couch, that part just worked. The upgrade from Ionic 3 to 4 was was quite painful though. For user authentication I've forked the nowadays unmaintained superlogin package [1], which still does a great job when keeping the dependencies up to date. [1] https://github.com/LyteFM/superlogin https://github.com/LyteFM/superlogin

crudbug 7y ago

Congrats to the whole team. Looking forward to CouchDB 4.0/FoundationDB goodies. Do we have any roadmap details on this.

agumonkey 7y ago

anybody acquainted with pouchdb devs ? just to know if there are plans to migrate already or not

gtirloni 7y ago
https://github.com/pouchdb/pouchdb/issues/7987 https://github.com/pouchdb/pouchdb/issues/7987
Volundr 7y ago
For those who don't want to follow the link, there are no changes to the replication protocol in Couch 3.0, so PouchDB already works.

smoyer 7y ago

I built two products on CouchDB 1.x starting in 2010 ... version three is another amazing step forward! For my more recent projects, I've replaced CouchDB with clustered PostgreSQL using JSON columns as I really enjoy the ability to write SQL queries for against the JSON and to use the built-in full-text search capabilities. I think both CouchDB and clustered PostgreSQL are amazing tools and it's nice to be able to choose between them as needed. The best advice I've heard is to choose CouchDB when you know your queries ahead of time and the data "schema"[1] is variable and choose PostgreSQL when you know your data ahead of time and your queries are variable. [1] In this case, a JSON document but either with a JSON-schema or marshaled/unmarshaled into a strict type.

jimstr 7y ago
I've gotten the impression that clustered Postgres still isn't very straightforward to run. Do you mind elaborating on your ideal setup and point to some resources? Thanks!
- smoyer 7y ago
  It's not straightforward at all but it's better than it was five years ago ... you can use something more "meta" like SymmetricDS (https://www.symmetricds.org/ https://www.symmetricds.org/). I haven't used it personally but a dirt simple way to get an HA, scalable PostgreSQL instance would be to use Amazon's Aurora DB.
karmelapple 7y ago
JSON Schema has been a big benefit for our use case. Our iOS, Android, and web app all pull in a schema from one repo, which serves up that schema via Cocoapods, Gradle, or npm. We built it years ago and it’s worked smoothly ever since.

LoSboccacc 7y ago

is the lucene search indexer synchronous with couchdb days updates? I'm wondering how people solve the common search after create pattern when using external indexes

janl 7y ago
Yup, works with clustering and everything: https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-easy-fulltext-search/ https://blog.couchdb.org/2020/02/26/the-road-to-couchdb-3-0-...

anonyfox 7y ago

Has anyone here tried to use couchdb directly within an elixir/erlang OTP application? As like, „mix install“? Would kill for couchdb as a library!

yatsyk 7y ago

CouchDB/PouchDB looks very promising for offline first apps, but I can’t understand how to restrict bad clients. Client potentially could insert document of huge size or execute expensive query and degrade experience of other clients on the same server. Is it any way to prevent this?

newfeatureok 7y ago
You resolve that issue the same way you would resolve the same issue if you were using Postgres - you introduce some back-end. For your example specifically I'd use a proxy.
- yatsyk 7y ago
  Custom backend means no synchronisation and no advantages over postgres. Do you propose to create proxy that parses query and estimates complexity? I think this task at least as hard as implementing couchdb myself (actually harder) Is there any secure open source code with pouchdb/couchdb integrations?
  - daleharvey 7y ago
    Your backend can be a reverse proxy that authenticates requests then passes them off to CouchDB (or PouchDB, since that also runs on the server). I have an example up @ https://github.com/daleharvey/noted https://github.com/daleharvey/noted. The server is 200 lines and does signup / email authentication etc.
    - yatsyk 7y ago
      This server can't prevent authenticated user from uploading huge document of running expensive query.
      - daleharvey 7y ago
        I wasnt worried about that since it is a basic proof of concept, adding that would make it ~210 lines of code.

johnchristopher 7y ago

> – Updated to modern JavaScript engine SpiderMonkey 60 Yes ^^ ! Congrats to the team. These people are some of the nicest and most supportive devs I know of in the OSS community (or whatev'). They show a great deal of patience in their slack channel and are always welcoming and answering stupid questions from idiots like me.

janl 7y ago
<3

gigatexal 7y ago

Can't wait to see what CouchDB 4.0 with FoundationDB at it's core does for the db. This is a great release too!

pawelk 7y ago

Other than being a great solution for some problems I wanted to highlight the fact that CouchDB has commited to SpiderMonkey (the Mozilla JS engine) since the very beginning and is one of the few projects helping to fend away the V8 monoculture.

wildchild 7y ago

CouchDB, couchbase, etc are all useless junk.

couchdb_ouchdb 7y ago

I'm surprised to see so much love for CouchDB in this thread. I don't think it's been widely adopted in corporate america and has lost the war to MongoDB closed source or not. I joined a company where it's being used backing a mobile app with couch/pouch in production. We can't wait to get off of it. Writes are slow. Reads are worse. Having a DB per user is a scaling and backup nightmare. If you run into any issues, it's a ghost town. I'm glad the CouchDB Team is forging ahead, but who is really using this database?

staticautomatic 7y ago
Would you be willing to say more? Inquiring minds want to know.
janl 7y ago
I sadly can’t name names, but rest assured the fortune 500 is heavily involved. OTOH, publicly known big companies using CouchDB include Apple and IBM. And I worked on a team that used CouchDB’s offline capability in the 2015 Ebola crisis in West Africa. That work also lead to the first Ebola vaccine ever. That’s why we do CouchDB :)

mark_l_watson 7y ago

I haven't used CouchDB in years. I just downloaded and installed it. Interesting that there are no apparent links to client libraries in different languages. Perhaps most people just use the HTTP API Reference and roll their own.

mikekchar 7y ago
In Ruby there is a CouchRest gem, which I've used, but to be honest a REST interface that talks JSON is so easy to use that I've often thought we'd be better off without anything specific.
- mark_l_watson 7y ago
  Thanks, that makes sense.
james_s_tayler 7y ago
There are a couple of client libraries in .NET Some are no longer maintained. Some still work.

haolez 7y ago

I once read that the right way to use CouchDB is for every user to have its own database. However, how does this work with BI? Or with public data that should be known by all users? Do I create a single centralized DB just for that kind of data? Maybe aggregate data from all users' DBs? Genuinely curious.

CameronNemo 7y ago
For public data, you can try to partition it in such a way that writes can be merged without any potential conflicts. E.g. a user's posts are in a separate partition. I have never done this with CouchDB, but the technique is described in Martin Kleppman's __Designing Data Intensive Applications__.
janl 7y ago
You can replicate all per-user DBs into a central database today. We are working on per-document-access-control at the moment, to support this use-case out of the box

seigel 7y ago

CouchDB is good. Yes. I still dream of the day when the cluster will balance shards automatically and recover better from losing and replacing nodes. :D

fiatjaf 7y ago

Many commenters here still think CouchDB is the same thing it was many years ago. CouchDB was a simple but very powerful idea (that still needed improvements), but it was coopted into something not very nice nor good nor useful. See my old rant about it and why it failed: http://web.archive.org/web/20170530122143/http://entulho.fiatjaf.alhur.es/notes/about-couchdb/ http://web.archive.org/web/20170530122143/http://entulho.fia...