4 ms·

As much as I detest MongoDB immaturity in many respects, I found a lot of features that are actually making life easier when you design pretty large scale appli

As much as I detest MongoDB immaturity in many respects, I found a lot of features that are actually making life easier when you design pretty large scale applications (mine was typically doing 2GB/s of data out of the database, I like to think it is pretty large). One feature I like is change event stream which you can subscribe to. It is pretty fast and reliable and for good reason -- the same mechanism is used to replicate MongoDB nodes. I found you can use it as a handy notification / queueing mechanism (more like Kafka topics than RabbitMQ). I would not recommend it as any kind of interface between components but within an application, for its internal workings, I think it is pretty viable option.

necovek 4y ago
That's quite interesting: I wonder if someone has done something similar with Postgres WAL log streaming?
- twawaaay 4y ago
  MongoDB's change stream is accidentally very simple to use. You just call the database and get continuous stream of documents that you are interested in from the database. If you need to restart, you can restart processing from the chosen point. It is not a global WAL or anything like that, it is just a stream of documents with some metadata.
  - spmurrayzzz 4y ago
    > If you need to restart, you can restart processing from the chosen point One caveat to this is that you can only start from wherever the beginning of your oplog window is. So for large deployments and/or situations where your oplog ondisk size simply isn't tuned properly, you're SOL unless you build a separate mechanism for catching up.
    - twawaaay 4y ago
      Which is fine, queueing systems can't store infinity of messages either. In the end messages are stored somewhere so there is always some limit.
      - spmurrayzzz 4y ago
        Yep, absolutely. But the side effect I am referring to (and probably wasn't clear enough about) is that the oplog is globally shared across the replica set. So even if your queue collection tops out at like 10k documents max, if you have another collection in the same deployment thats getting 10mm docs/min, your queue window is also gonna be artificially limited. Putting the queue in its own deployment is a good insulation against this (assuming you don't need to use aggregate() with the queue across collections obviously).
        twawaaay 4y ago
        I do agree, but listen... this is supposed to be handy solution. You know, my app already uses MongoDB, why do I need another component if I can run my notifications with a collection? Also, I am firm believer that you should not put actual data through notifications. Notifications are meant to wake other systems up, not carry gigabytes of data. You can pack your data into another storage and notify "Hey, here is data of 10k new clients that needs to be processed. Cheers!" The message is meant to ensure correct processing flow (message has been received, processed, if it fails somebody else will process it, etc.), but it does not have to carry all the data. I have fixed at least one platform that "reached limits of Kafka" (their words not mine) and "was looking for expert help" to manage the problem. My solution? I got the component that publishes upload the data to compressed JSON to S3 and post the notification with some metadata and link to the JSON. And the client to parse the JSON. Bam, suddenly everything works fine, no bottlenecks anymore. For the cost of maybe three pages of code. There is few situation where you absolutely need to track so many individual objects that you have to start caring if they make hard drives large enough. And I managed some pretty large systems.
        spmurrayzzz 4y ago
- twic 4y ago
  If i understand correctly, then yes, they have. PostgreSQL supports "logical decoding", which is where you parse the WAL into logical events: https://www.postgresql.org/docs/15/logicaldecoding.html https://www.postgresql.org/docs/15/logicaldecoding.html Quite a number of bits of software use that to do things with these events. For example: https://debezium.io/documentation/reference/stable/connectors/postgresql https://debezium.io/documentation/reference/stable/connector...
Joel_Mckay 4y ago
Funny enough, we designed one subsystem to use RabbitMQ to enforce linear committed records into mongodb to avoid indices. I.e. the routes in rabbitMQ would ensure a GUID tagged record was spatially localized with other user data on the same host (the inter-host shovel traffic is minimized). Depends on the use-case, but the original article smells like FUD. This is because the connection C so lib allows you to select how the envelopes are bound/ack'ed on the queue/dead-letter-route in the AMQP client-consumer (you don't usually camp on the connection). Also, the expected runtime constraint should always be included when designing a job-queue regardless of the underlying method (again, expiry default routing is built into rabbitMQ)... Cheers =)
aPoCoMiLogin 4y ago
i've used that as a cache flush mechanism when some cached records were updated/deleted, the simplicity was the key.

necovek 4y ago

That's quite interesting: I wonder if someone has done something similar with Postgres WAL log streaming?

twawaaay 4y ago
MongoDB's change stream is accidentally very simple to use. You just call the database and get continuous stream of documents that you are interested in from the database. If you need to restart, you can restart processing from the chosen point. It is not a global WAL or anything like that, it is just a stream of documents with some metadata.
- spmurrayzzz 4y ago
  > If you need to restart, you can restart processing from the chosen point One caveat to this is that you can only start from wherever the beginning of your oplog window is. So for large deployments and/or situations where your oplog ondisk size simply isn't tuned properly, you're SOL unless you build a separate mechanism for catching up.
  - twawaaay 4y ago
    Which is fine, queueing systems can't store infinity of messages either. In the end messages are stored somewhere so there is always some limit.
    - spmurrayzzz 4y ago
      Yep, absolutely. But the side effect I am referring to (and probably wasn't clear enough about) is that the oplog is globally shared across the replica set. So even if your queue collection tops out at like 10k documents max, if you have another collection in the same deployment thats getting 10mm docs/min, your queue window is also gonna be artificially limited. Putting the queue in its own deployment is a good insulation against this (assuming you don't need to use aggregate() with the queue across collections obviously).
      - twawaaay 4y ago
        I do agree, but listen... this is supposed to be handy solution. You know, my app already uses MongoDB, why do I need another component if I can run my notifications with a collection? Also, I am firm believer that you should not put actual data through notifications. Notifications are meant to wake other systems up, not carry gigabytes of data. You can pack your data into another storage and notify "Hey, here is data of 10k new clients that needs to be processed. Cheers!" The message is meant to ensure correct processing flow (message has been received, processed, if it fails somebody else will process it, etc.), but it does not have to carry all the data. I have fixed at least one platform that "reached limits of Kafka" (their words not mine) and "was looking for expert help" to manage the problem. My solution? I got the component that publishes upload the data to compressed JSON to S3 and post the notification with some metadata and link to the JSON. And the client to parse the JSON. Bam, suddenly everything works fine, no bottlenecks anymore. For the cost of maybe three pages of code. There is few situation where you absolutely need to track so many individual objects that you have to start caring if they make hard drives large enough. And I managed some pretty large systems.
        spmurrayzzz 4y ago
twic 4y ago
If i understand correctly, then yes, they have. PostgreSQL supports "logical decoding", which is where you parse the WAL into logical events: https://www.postgresql.org/docs/15/logicaldecoding.html https://www.postgresql.org/docs/15/logicaldecoding.html Quite a number of bits of software use that to do things with these events. For example: https://debezium.io/documentation/reference/stable/connectors/postgresql https://debezium.io/documentation/reference/stable/connector...

Joel_Mckay 4y ago

Funny enough, we designed one subsystem to use RabbitMQ to enforce linear committed records into mongodb to avoid indices. I.e. the routes in rabbitMQ would ensure a GUID tagged record was spatially localized with other user data on the same host (the inter-host shovel traffic is minimized). Depends on the use-case, but the original article smells like FUD. This is because the connection C so lib allows you to select how the envelopes are bound/ack'ed on the queue/dead-letter-route in the AMQP client-consumer (you don't usually camp on the connection). Also, the expected runtime constraint should always be included when designing a job-queue regardless of the underlying method (again, expiry default routing is built into rabbitMQ)... Cheers =)

aPoCoMiLogin 4y ago

i've used that as a cache flush mechanism when some cached records were updated/deleted, the simplicity was the key.