4 ms·

I don't think you can have a serious conversation about microservices without talking about poison messages and ordering guarantees (both with respect to each o

by latch 5y ago

I don't think you can have a serious conversation about microservices without talking about poison messages and ordering guarantees (both with respect to each other and not). What does Pub/Sub provider to deal with those? (If you're interviewing somewhere and they say they have microservices, ask how they deal with poison messages. The answer can be telling).

aliswe 5y ago
Whats a good answer? I want more information ...
- latch 5y ago
  Tongue-in-cheek answer (but really what I believe): Don't use microservices. Short answer: It's extremely difficult and, there isn't always a good solution. Different approaches are used on a _per consumer_ basis. That's the gist of what you want to hear. Long answer: In some cases ordering and lost messages don't matter so much, so any failed messages can just be thrown out. For some queues lost messages matters, but not ordering. So you can put the message at the back of the queue (possibly delayed if that's support), so you don't block other messages. Partitioning helps isolated poison messages to a vertical (e.g., customer_id). Always adding id and timestamp to each message can let consumers behave more smartly (e.g. throwing away out of order or already processed messages). But when none of that helps and ordering and delivery matter (which I think will make up most cases for any non-trivial system)...I'd love to know too. Good alerts to wake up developers? Then there's how to scale this and making it reliable What's the ordering guarantee across N consumers? What if a consumer nack's a message. What if there's prefetching?
thatsamonad 5y ago
I’m not sure if this fully addresses your comment but, similar to AWS SQS, Google Cloud Pub/Sub supports both message ordering[0] and dead letter topics[1]. [0](https://cloud.google.com/pubsub/docs/ordering#receiving_messages_in_order https://cloud.google.com/pubsub/docs/ordering#receiving_mess...) [1](https://cloud.google.com/pubsub/docs/handling-failures#setting-a-dead-letter-topic https://cloud.google.com/pubsub/docs/handling-failures#setti...)

by latch 5y ago

aliswe 5y ago

Whats a good answer? I want more information ...

latch 5y ago
Tongue-in-cheek answer (but really what I believe): Don't use microservices. Short answer: It's extremely difficult and, there isn't always a good solution. Different approaches are used on a _per consumer_ basis. That's the gist of what you want to hear. Long answer: In some cases ordering and lost messages don't matter so much, so any failed messages can just be thrown out. For some queues lost messages matters, but not ordering. So you can put the message at the back of the queue (possibly delayed if that's support), so you don't block other messages. Partitioning helps isolated poison messages to a vertical (e.g., customer_id). Always adding id and timestamp to each message can let consumers behave more smartly (e.g. throwing away out of order or already processed messages). But when none of that helps and ordering and delivery matter (which I think will make up most cases for any non-trivial system)...I'd love to know too. Good alerts to wake up developers? Then there's how to scale this and making it reliable What's the ordering guarantee across N consumers? What if a consumer nack's a message. What if there's prefetching?

thatsamonad 5y ago

I’m not sure if this fully addresses your comment but, similar to AWS SQS, Google Cloud Pub/Sub supports both message ordering[0] and dead letter topics[1]. [0](https://cloud.google.com/pubsub/docs/ordering#receiving_messages_in_order https://cloud.google.com/pubsub/docs/ordering#receiving_mess...) [1](https://cloud.google.com/pubsub/docs/handling-failures#setting-a-dead-letter-topic https://cloud.google.com/pubsub/docs/handling-failures#setti...)