3 ms·

I've used events successfully for an important fraud detection system. There are a set of known events. A fraud model could hook into the event chain at any poi

I've used events successfully for an important fraud detection system. There are a set of known events. A fraud model could hook into the event chain at any point. It would then produce an alert event with its findings. Downstream another handler listened for it. Coupled with RabbitMQ, the system is a pleasure to expand. All the fraud models are micro services. Just a start up script and boom, hooked in. We planned for duplication of messages. Operations are idempotent. If the message already ran, we could rerun the calculation safely. With Rabbit the only duplicate messages were the ones humans sent to rerun a model that broke.

Floegipoky 10y ago
Sounds like a system I built, but what I didn't account for is how often the RabbitMQ cluster would get tanked by my coworkers. The system fails when messages start getting dropped on the floor due to server reboots. Still haven't come up with a great solution, but what we have is "good enough" for that project's requirements. It was a great learning experience though, an uncomfortable reminder of the difference between reliability and durability, and it definitely helped inform subsequent architectural decisions.
- jononor 10y ago
  Separate out another RabbitMQ cluster for those other applications/people? And why are they rebooting all the time anyway??
deleted 10y ago
[deleted]
pjc50 10y ago
> We planned for duplication of messages This is important. There is no "exactly once" system; you can have at-most-once where things get occasionally lost, or at-least-once where you resend things that you can't be sure have been recieved. The latter loses less data but needs methods for detecting or otherwise ignoring duplicates. NFS goes for "idempotent operations", SMTP goes for message-IDs that can be used for deduplication.
edem 10y ago
I've written a system like that and it was also composed of microservices. I had one which was running on my computer for two months without a restart and it did not fail once. You can build robust systems on an AMQP backed MQ if you take care.

Floegipoky 10y ago

Sounds like a system I built, but what I didn't account for is how often the RabbitMQ cluster would get tanked by my coworkers. The system fails when messages start getting dropped on the floor due to server reboots. Still haven't come up with a great solution, but what we have is "good enough" for that project's requirements. It was a great learning experience though, an uncomfortable reminder of the difference between reliability and durability, and it definitely helped inform subsequent architectural decisions.

jononor 10y ago
Separate out another RabbitMQ cluster for those other applications/people? And why are they rebooting all the time anyway??

deleted 10y ago

[deleted]

pjc50 10y ago

> We planned for duplication of messages This is important. There is no "exactly once" system; you can have at-most-once where things get occasionally lost, or at-least-once where you resend things that you can't be sure have been recieved. The latter loses less data but needs methods for detecting or otherwise ignoring duplicates. NFS goes for "idempotent operations", SMTP goes for message-IDs that can be used for deduplication.

edem 10y ago

I've written a system like that and it was also composed of microservices. I had one which was running on my computer for two months without a restart and it did not fail once. You can build robust systems on an AMQP backed MQ if you take care.