3 ms·

The tech stack is: Kafka + PostgreSQL + MongoDB + Redis + .NET Core

vb-8448 3y ago
just for curiosity: - what kind of data are you storing in mongodb? - why mongodb when you already have postgresql + redis? - it's a one-man business?
- victormacedo 3y ago
  - MongoDB is used for scheduled messages and retry (retry leverages the scheduled messages logic too). - Using MongoDB I can easily set messages to expire, then I listen to the change stream and I build the scheduled/delayed message functionality on top of that. Redis is used for general application caching, and I intend to build another module to provide caching functionality built on top of Redis, that's the main reason why I've setup a Redis cluster too. - It's not a business, but I wish it were =) and yes, it's only me, that's currently my hobby. P.S.: Since you asked more details about the DBs I'll also include details about the other parts in case you want to know =) Everything is in small scale at this moment, but PostgreSQL and MongoDB are running with replication, Redis also has a few nodes. All the setup I've made with Ansible and it's running in just one VM, but since the setup is made with Ansible, it's easy to add more machines if necessary. I've setup HAProxy to load balance the API instances, Postgres and Redis (I'm not using K8S yet). API and jobs run on docker, DBs and Kafka run on host, and in front of the applications there is NGINX. The applications were built using C# + .NET Core. The way the application handles data on Postgres is using eventual consistency, so it reads from replicas and writes on the master (I'm not using multi master). There are more details, and I haven't described the workflow here, but that's pretty much the solution overview.
  - vb-8448 3y ago
    Please don't get me wrong, it's just my point of view, but it seems pretty complicated to run and manage all these things together. At least one between mongo and postgres seems unnecessary.
    - victormacedo 3y ago
      Each database is used because of its strengths, and the complexity is not that high as it seems to be :)

just for curiosity: - what kind of data are you storing in mongodb? - why mongodb when you already have postgresql + redis? - it's a one-man business?

victormacedo 3y ago
- MongoDB is used for scheduled messages and retry (retry leverages the scheduled messages logic too). - Using MongoDB I can easily set messages to expire, then I listen to the change stream and I build the scheduled/delayed message functionality on top of that. Redis is used for general application caching, and I intend to build another module to provide caching functionality built on top of Redis, that's the main reason why I've setup a Redis cluster too. - It's not a business, but I wish it were =) and yes, it's only me, that's currently my hobby. P.S.: Since you asked more details about the DBs I'll also include details about the other parts in case you want to know =) Everything is in small scale at this moment, but PostgreSQL and MongoDB are running with replication, Redis also has a few nodes. All the setup I've made with Ansible and it's running in just one VM, but since the setup is made with Ansible, it's easy to add more machines if necessary. I've setup HAProxy to load balance the API instances, Postgres and Redis (I'm not using K8S yet). API and jobs run on docker, DBs and Kafka run on host, and in front of the applications there is NGINX. The applications were built using C# + .NET Core. The way the application handles data on Postgres is using eventual consistency, so it reads from replicas and writes on the master (I'm not using multi master). There are more details, and I haven't described the workflow here, but that's pretty much the solution overview.
- vb-8448 3y ago
  Please don't get me wrong, it's just my point of view, but it seems pretty complicated to run and manage all these things together. At least one between mongo and postgres seems unnecessary.
  - victormacedo 3y ago
    Each database is used because of its strengths, and the complexity is not that high as it seems to be :)