6 ms·

Ok, old man yelling at clouds moment finally coming for me. Now that we've been through the document-database heyday and are out the other end, what have we lea

by kbd 3y ago

Ok, old man yelling at clouds moment finally coming for me. Now that we've been through the document-database heyday and are out the other end, what have we learned about where document databases are a good fit? At the time I looked at them like a fad. "These script kiddies want to write javascript and ignore schemas. Let's see how well that works out for them." As expected, most of what I ever hear is regret. Today, MongoDB has grown out of the reliability issues it had in the past, and Postgres has json features for the occasional times it's useful to store some loosely structured data along with otherwise relational data. Question is, what applications is a document-first database good for, outside of prototyping? Edit: and to make sure I understand, FerretDB is a layer reimplementing MongoDB on top of Postgres and its json features?

peterfarkas 3y ago
Yes, FerretDB is a layer which implements the MongoDB wire protocol on top of Postgres. Right now we are using JSONB, but this affects performance and we need to depart from this strategy in the long run. We have an article which explains the concept [1]. I wouldn't go into the document vs. relational argument, all arguments for and against would have merit. There are valid use cases for document databases (take e-commerce, for example), and we should not discount the fact that using a relational database is just more complicated. Using vanilla Postgres for a MongoDB use case will not be feasible for someone who's focus is, let's say, mobile application development. There is a reason behind MongoDB's popularity - it just provides a great developer experience. This is what we are aiming to recreate on top of Postgres. [1]: https://blog.ferretdb.io/pjson-how-to-store-bson-in-jsonb/ https://blog.ferretdb.io/pjson-how-to-store-bson-in-jsonb/
- Zababa 3y ago
  I'm curious about what makes document databases a good fit for e-commerce. I've never worked in that industry.
  - tylerchurch 3y ago
    Two things off the top of my head: 1. Denormalized data is often a boon. One of your key use cases is "give me all the information about this Product so I can show it to the user." Forget joining a ton of tables, just do 1 read of 1 product and go on your way. 2. A certain amount of more freeform data is expected. Different product categories will have different sets of information. T-shirts and drinkware both have sizes, but these sizes have nothing to do with each other. You can model all this in a traditional SQL database, but you have to stop and think really hard about it, and potentially end up with a plethora of tables. In a document database, it's much easier to just add the data and go on your way.
    - Zababa 3y ago
      That makes a lot of sense, thanks!
- yawnxyz 3y ago
  if we use Ferret for prototyping and eventually land on a schema, would it be possible to convert the FerretDB rows into more structured postgres? Would be so cool if it could just analyze and create a schema for you, you double check it, and it just works. I think being able to convert back and forth would make it so worthwhile!
  - aleksi 3y ago
    Yeah, we want to do that: https://github.com/FerretDB/FerretDB/issues/226 https://github.com/FerretDB/FerretDB/issues/226
- CodesInChaos 3y ago
  > we need to depart from this strategy in the long run What's the alternative you want to move to?
  - aleksi 3y ago
    Probably a custom PostgreSQL extension for BSON support. The biggest issue right now is that MongoDB compares and sorts values differently than PostgreSQL/jsonb. For that reason, we have to do a lot of filtering on the FerretDB side, and that can't be great for performance. Pushing more work on the database size should make FerretDB perform much better.
mitjam 3y ago
A use case I have for FerretDB is migrating existing apps off MongoDB without needing to change the code. I find it funny that you could call these now „legacy“ apps.
- peterfarkas 3y ago
  Exactly. Let's say a major company have a few applications which require MongoDB, and the rest of their applications are running on Postgres. With FerretDB, they can migrate the app off MongoDB as you said, and keep Postgres only. Therefore they don't need to maintain internal knowledge on how to run MongoDB, or pay for MongoDB to run it for them (in which case they are not in control of their data, because it is all under MongoDB's account...). This is one real world user example.
- kiney 3y ago
  Insert <look at me, I am the legacy app now> meme here
- ocdtrekkie 3y ago
  Yeah, anxiously waiting for FerretDB to support Meteor over here.
  - peterfarkas 3y ago
    FerretDB maintainer here: we are working on it. We are already getting reports of successful migrations with Meteor apps. [1] Would you mind sharing the specific Meteor app you are looking to get supported? MeteorJS is among the most requested applications/frameworks our users are looking to use with FerretDB. [1]: https://twitter.com/CowboyCaramel/status/1646089964126347264 https://twitter.com/CowboyCaramel/status/1646089964126347264
    - ocdtrekkie 3y ago
      I'm a contributor to the Sandstorm.io project, and Mongo continues to present a bit of a pickle for us. We can arguably write our way out of one upgrade, but upgrading to a later Mongo just punts the issue again. We'd much rather just leave Mongo behind.
tracker1 3y ago
Looks that way to me... I really liked a LOT about MongoDB, I don't think they had a good story for administration of scale + redundancy. RethinkDB, Cassandra and others have a ring + redundancy model which I think is easier to deal with. Where Mongo at least was limited to replication or sharding, and was a serious pain to deal with from the admin side imo. Understanding the advantages and shortfalls is sometimes harder too. Mongo having multiple secondary indexes is pretty nice as well. Haven't dug into FerretDB, my first question is if it will run over the top of CockroachDB, as that's how I would probably want it configured. Then, I'm not sure if I wouldn't just use the JSONB surface in PostgreSQL/CockroachDB directly. I think it just really depends on how/what you need to accomplish.
aranchelk 3y ago
> Question is, what applications is a document-first database good for, outside of prototyping? Some criteria I’d use: * Data is already naturally segregated and won’t be shared/joined much during normal usage * App already has transactions modeled in the application layer (or I guess if consistency really doesn’t matter) * App would benefit from being geographically distributed. * App is written in a language with a strong type system, has high code quality, test coverage, etc. In my case, my company makes a distributed project management application. All changes to projects are canonically ordered and applied in the application layer. Data is stored in Cloudflare Durable Objects and R2. Hard to classify DOs but they’re a lot closer to a document store than an SQL Db. There are some nice benefits that would be hard to replicate with a traditional SQL Db setup.

by kbd 3y ago

peterfarkas 3y ago

Yes, FerretDB is a layer which implements the MongoDB wire protocol on top of Postgres. Right now we are using JSONB, but this affects performance and we need to depart from this strategy in the long run. We have an article which explains the concept [1]. I wouldn't go into the document vs. relational argument, all arguments for and against would have merit. There are valid use cases for document databases (take e-commerce, for example), and we should not discount the fact that using a relational database is just more complicated. Using vanilla Postgres for a MongoDB use case will not be feasible for someone who's focus is, let's say, mobile application development. There is a reason behind MongoDB's popularity - it just provides a great developer experience. This is what we are aiming to recreate on top of Postgres. [1]: https://blog.ferretdb.io/pjson-how-to-store-bson-in-jsonb/ https://blog.ferretdb.io/pjson-how-to-store-bson-in-jsonb/

Zababa 3y ago
I'm curious about what makes document databases a good fit for e-commerce. I've never worked in that industry.
- tylerchurch 3y ago
  Two things off the top of my head: 1. Denormalized data is often a boon. One of your key use cases is "give me all the information about this Product so I can show it to the user." Forget joining a ton of tables, just do 1 read of 1 product and go on your way. 2. A certain amount of more freeform data is expected. Different product categories will have different sets of information. T-shirts and drinkware both have sizes, but these sizes have nothing to do with each other. You can model all this in a traditional SQL database, but you have to stop and think really hard about it, and potentially end up with a plethora of tables. In a document database, it's much easier to just add the data and go on your way.
  - Zababa 3y ago
    That makes a lot of sense, thanks!
yawnxyz 3y ago
if we use Ferret for prototyping and eventually land on a schema, would it be possible to convert the FerretDB rows into more structured postgres? Would be so cool if it could just analyze and create a schema for you, you double check it, and it just works. I think being able to convert back and forth would make it so worthwhile!
- aleksi 3y ago
  Yeah, we want to do that: https://github.com/FerretDB/FerretDB/issues/226 https://github.com/FerretDB/FerretDB/issues/226
CodesInChaos 3y ago
> we need to depart from this strategy in the long run What's the alternative you want to move to?
- aleksi 3y ago
  Probably a custom PostgreSQL extension for BSON support. The biggest issue right now is that MongoDB compares and sorts values differently than PostgreSQL/jsonb. For that reason, we have to do a lot of filtering on the FerretDB side, and that can't be great for performance. Pushing more work on the database size should make FerretDB perform much better.

mitjam 3y ago

A use case I have for FerretDB is migrating existing apps off MongoDB without needing to change the code. I find it funny that you could call these now „legacy“ apps.

peterfarkas 3y ago
Exactly. Let's say a major company have a few applications which require MongoDB, and the rest of their applications are running on Postgres. With FerretDB, they can migrate the app off MongoDB as you said, and keep Postgres only. Therefore they don't need to maintain internal knowledge on how to run MongoDB, or pay for MongoDB to run it for them (in which case they are not in control of their data, because it is all under MongoDB's account...). This is one real world user example.
kiney 3y ago
Insert <look at me, I am the legacy app now> meme here
ocdtrekkie 3y ago
Yeah, anxiously waiting for FerretDB to support Meteor over here.
- peterfarkas 3y ago
  FerretDB maintainer here: we are working on it. We are already getting reports of successful migrations with Meteor apps. [1] Would you mind sharing the specific Meteor app you are looking to get supported? MeteorJS is among the most requested applications/frameworks our users are looking to use with FerretDB. [1]: https://twitter.com/CowboyCaramel/status/1646089964126347264 https://twitter.com/CowboyCaramel/status/1646089964126347264
  - ocdtrekkie 3y ago
    I'm a contributor to the Sandstorm.io project, and Mongo continues to present a bit of a pickle for us. We can arguably write our way out of one upgrade, but upgrading to a later Mongo just punts the issue again. We'd much rather just leave Mongo behind.

tracker1 3y ago

Looks that way to me... I really liked a LOT about MongoDB, I don't think they had a good story for administration of scale + redundancy. RethinkDB, Cassandra and others have a ring + redundancy model which I think is easier to deal with. Where Mongo at least was limited to replication or sharding, and was a serious pain to deal with from the admin side imo. Understanding the advantages and shortfalls is sometimes harder too. Mongo having multiple secondary indexes is pretty nice as well. Haven't dug into FerretDB, my first question is if it will run over the top of CockroachDB, as that's how I would probably want it configured. Then, I'm not sure if I wouldn't just use the JSONB surface in PostgreSQL/CockroachDB directly. I think it just really depends on how/what you need to accomplish.

aranchelk 3y ago

> Question is, what applications is a document-first database good for, outside of prototyping? Some criteria I’d use: * Data is already naturally segregated and won’t be shared/joined much during normal usage * App already has transactions modeled in the application layer (or I guess if consistency really doesn’t matter) * App would benefit from being geographically distributed. * App is written in a language with a strong type system, has high code quality, test coverage, etc. In my case, my company makes a distributed project management application. All changes to projects are canonically ordered and applied in the application layer. Data is stored in Cloudflare Durable Objects and R2. Hard to classify DOs but they’re a lot closer to a document store than an SQL Db. There are some nice benefits that would be hard to replicate with a traditional SQL Db setup.