3 ms·

I have been working on an engine to perform heavy computations that is distributed and my messaging choice has been gRPC + Flatbuffers + proxys and I was wonder

I have been working on an engine to perform heavy computations that is distributed and my messaging choice has been gRPC + Flatbuffers + proxys and I was wondering if someone could name some reasons or use cases to chose message brokers vs a lower level approach like gRPC. Is it mostly because of synchronous/asynchronous needs or there is something else such as latency or even ease of implementation?

ozarker 3y ago
Depends on your use case. Queueing messages enables the competing consumer flow which implicitly allows load-leveling which is great if your traffic is spiky and you need a little time to auto-scale or other scenarios like that. If you’re just looking to blast out data to all consumers or just load-balancing is fine, approaches like yours are probably better.
imachine1980_ 3y ago
grpc client a -> grpc server a for client a grpc client b -> grpc server a for client b grpc client a -> message broker(implemen a) --> grpc client b -> message broker(implemen b) --> message broker-> your app for the broker this make sense if dont make sense manage the quque in your app if ypu want to abstract the implementations is worth if you have n to 1 or n to n and all works different
pravus 3y ago
If your workers can hop onto the next task without having to reach out for any additional state or report back to anything synchronously a distributed log/spool/queue/broker works well. If you need light-weight reporting like a status code you can use RPC/reply mechanisms if the broker supports it (e.g. rabbitmq), build your own (custom headers), or use an API/gRPC call. Everything else starts needing some sort of more sophisticated coordination mechanism and gRPC will look attractive for that although there are usually other options as well.
stolsvik 3y ago
It is the asynchronous and decoupled nature of messaging that pulls architectures towards message brokers. In a "MOM Architecture", the producing of messages, and consuming of messages, is completely decoupled - both in "space" (HA, location transparency, replication etc) and in time (the async nature). This leads to a whole heap of benefits. I've outlined a few of them here, in the "Rationale for Mats3", my Message-Oriented Async RPC library: https://mats3.io/background/rationale-for-mats/ https://mats3.io/background/rationale-for-mats/ You will find that the Reactive Manifesto also lays out the same type of arguments (there's some links to it from the list in the first link): https://www.reactivemanifesto.org/ https://www.reactivemanifesto.org/ and its glossary: https://www.reactivemanifesto.org/glossary https://www.reactivemanifesto.org/glossary

ozarker 3y ago

Depends on your use case. Queueing messages enables the competing consumer flow which implicitly allows load-leveling which is great if your traffic is spiky and you need a little time to auto-scale or other scenarios like that. If you’re just looking to blast out data to all consumers or just load-balancing is fine, approaches like yours are probably better.

imachine1980_ 3y ago

grpc client a -> grpc server a for client a grpc client b -> grpc server a for client b grpc client a -> message broker(implemen a) --> grpc client b -> message broker(implemen b) --> message broker-> your app for the broker this make sense if dont make sense manage the quque in your app if ypu want to abstract the implementations is worth if you have n to 1 or n to n and all works different

pravus 3y ago

If your workers can hop onto the next task without having to reach out for any additional state or report back to anything synchronously a distributed log/spool/queue/broker works well. If you need light-weight reporting like a status code you can use RPC/reply mechanisms if the broker supports it (e.g. rabbitmq), build your own (custom headers), or use an API/gRPC call. Everything else starts needing some sort of more sophisticated coordination mechanism and gRPC will look attractive for that although there are usually other options as well.

stolsvik 3y ago

It is the asynchronous and decoupled nature of messaging that pulls architectures towards message brokers. In a "MOM Architecture", the producing of messages, and consuming of messages, is completely decoupled - both in "space" (HA, location transparency, replication etc) and in time (the async nature). This leads to a whole heap of benefits. I've outlined a few of them here, in the "Rationale for Mats3", my Message-Oriented Async RPC library: https://mats3.io/background/rationale-for-mats/ https://mats3.io/background/rationale-for-mats/ You will find that the Reactive Manifesto also lays out the same type of arguments (there's some links to it from the list in the first link): https://www.reactivemanifesto.org/ https://www.reactivemanifesto.org/ and its glossary: https://www.reactivemanifesto.org/glossary https://www.reactivemanifesto.org/glossary