5 ms·

Three out of five answers so far just questions the legitimacy of using noSql. There are definitely use cases for noSQL and I clicked on this thread hoping for

by croo 6y ago

Three out of five answers so far just questions the legitimacy of using noSql. There are definitely use cases for noSQL and I clicked on this thread hoping for information and war stories about cockroach, mongo, redis, couchdb, the current state of noSql in postgres, and a few name I maybe never heard of. Let me derail the conversation of berating this technology. I've happen to had a requirement which needs a dinamically changing dB structure (saving lots of json data from an dinamically changeable form). Which noSql db would you recommend for me? Any pitfalls? I'm primarily looking for self hosted solutions.

pier25 6y ago
Maybe Mongo if you don't need ACID transactions [1]. https://twitter.com/jepsen_io/status/1261276984681754625 https://twitter.com/jepsen_io/status/1261276984681754625
frompdx 6y ago
Honestly Postgres is probably my first choice for storing some dynamic JSON and I have used it for exactly that in the past. In particular, I used to to build a simple time series db for reporting on data from JSON payloads that the potential to hold arbitrary data. The OP has harped quite a bit on connection limit issues and this is a valid concern but also something that you can mitigate by using connection pooling. Geographic replication is an issue and it's one of those things where I'm not really convinced any sql or nosql db offers a really good solution. For example, if you run mongodb in replica you must take extra steps to ensure your replicas do not suffer from split brain during a network partition. And as another commenter has pointed out, mongodb's transactions are flawed. I would not trust it for anything transactional beyond single document transactions in a non-replicated configuration. DynamoDB is pretty decent for a document DB but I personally dislike working with it. It requires you to make many sacrifices that RDBMS's like postgres offer out of the box. For example, if you want to fetch all records you will have to do it in a loop because there are limits to the number of records that can be fetched at the same time. But of course you can't self host it. Another hosted option is Datatomic. I've heard great things but have never used it so I can't really comment.
- pier25 6y ago
  > you can mitigate by using connection pooling Can you do connection pooling to Postgres from cloud functions? I know there is a Node driver that does it for MySQL [1] but I've never seen one for Postgres. [1] https://github.com/jeremydaly/serverless-mysql https://github.com/jeremydaly/serverless-mysql
  - frompdx 6y ago
    Run something like PgBouncer in front of your postgres instance. https://www.pgbouncer.org/ https://www.pgbouncer.org/ For what it is worth, the popular postgres and mysql libraries for nodejs support connection pooling out of the box. Cloud functions break this functionality because you are spinning up an instance per connection rather than handling multiple connections with a single instance of nodejs. In this case I think that longer lived containers or vms that can autoscale my be a better solution. That is not to say you should not use cloud functions. I understand they offer many benefits. Managing a fleet of vms or containers introduces its own challenges. In general I think this is a good example of how simplifying one thing can complicate other things. There are no silver bullets. It's all about determining the compromises you can and cannot accept.
    - pier25 6y ago
      Thanks for the link to Pgbouncer, I'll check that. I'm using Cloudflare Workers which are distributed on the CF network and run at the edge to reduce latency. Achieving the same thing with containers would be probably complicated and expensive. Right now I'm using Fauna DB which is also distributed and doesn't have connection limits, but I want to explore all my options. Edit: If anyone wants to know more about PGBouncer and pooling with postgres I found this article very informative: https://www.compose.com/articles/how-to-pool-postgresql-connections-with-pgbouncer/ https://www.compose.com/articles/how-to-pool-postgresql-conn...

by croo 6y ago

pier25 6y ago

Maybe Mongo if you don't need ACID transactions [1]. https://twitter.com/jepsen_io/status/1261276984681754625 https://twitter.com/jepsen_io/status/1261276984681754625

frompdx 6y ago

Honestly Postgres is probably my first choice for storing some dynamic JSON and I have used it for exactly that in the past. In particular, I used to to build a simple time series db for reporting on data from JSON payloads that the potential to hold arbitrary data. The OP has harped quite a bit on connection limit issues and this is a valid concern but also something that you can mitigate by using connection pooling. Geographic replication is an issue and it's one of those things where I'm not really convinced any sql or nosql db offers a really good solution. For example, if you run mongodb in replica you must take extra steps to ensure your replicas do not suffer from split brain during a network partition. And as another commenter has pointed out, mongodb's transactions are flawed. I would not trust it for anything transactional beyond single document transactions in a non-replicated configuration. DynamoDB is pretty decent for a document DB but I personally dislike working with it. It requires you to make many sacrifices that RDBMS's like postgres offer out of the box. For example, if you want to fetch all records you will have to do it in a loop because there are limits to the number of records that can be fetched at the same time. But of course you can't self host it. Another hosted option is Datatomic. I've heard great things but have never used it so I can't really comment.

pier25 6y ago
> you can mitigate by using connection pooling Can you do connection pooling to Postgres from cloud functions? I know there is a Node driver that does it for MySQL [1] but I've never seen one for Postgres. [1] https://github.com/jeremydaly/serverless-mysql https://github.com/jeremydaly/serverless-mysql
- frompdx 6y ago
  Run something like PgBouncer in front of your postgres instance. https://www.pgbouncer.org/ https://www.pgbouncer.org/ For what it is worth, the popular postgres and mysql libraries for nodejs support connection pooling out of the box. Cloud functions break this functionality because you are spinning up an instance per connection rather than handling multiple connections with a single instance of nodejs. In this case I think that longer lived containers or vms that can autoscale my be a better solution. That is not to say you should not use cloud functions. I understand they offer many benefits. Managing a fleet of vms or containers introduces its own challenges. In general I think this is a good example of how simplifying one thing can complicate other things. There are no silver bullets. It's all about determining the compromises you can and cannot accept.
  - pier25 6y ago
    Thanks for the link to Pgbouncer, I'll check that. I'm using Cloudflare Workers which are distributed on the CF network and run at the edge to reduce latency. Achieving the same thing with containers would be probably complicated and expensive. Right now I'm using Fauna DB which is also distributed and doesn't have connection limits, but I want to explore all my options. Edit: If anyone wants to know more about PGBouncer and pooling with postgres I found this article very informative: https://www.compose.com/articles/how-to-pool-postgresql-connections-with-pgbouncer/ https://www.compose.com/articles/how-to-pool-postgresql-conn...