3 ms·

I'm putting this one on my new hire required reading list. So often people end up re-inventing message queues and not even noticing they are doing it.

sudhirj 6y ago
Interesting. This is actually sort of a distillation of all the new hire training / discussions I’ve done. It’s as much a “things I wish someone had told me 15 years ago” thing too.
brightball 6y ago
This is along the same lines, but something people don’t often think about when using and naming queues. https://www.brightball.com/articles/organizing-background-worker-queues https://www.brightball.com/articles/organizing-background-wo...
athoscouto 6y ago
Would you be willing to share the list with us?
- indymike 6y ago
  Here it is: https://mikeseidle.com/2021/01/03/my-developer-new-hire-reading-list/ https://mikeseidle.com/2021/01/03/my-developer-new-hire-read...
- indymike 6y ago
  Hmm. I really should find time to get my personal site back online. I took it down a month ago to re-face it and life has intervened.
  - loevborg 6y ago
    You could just dump it in a gist
yuppiepuppie 6y ago
> people end up re-inventing message queues and not even noticing they are doing it Amazing how people sucked down this rabbit hole. My work is doing this right now, and its driving me mad as there are well supported alternatives out there. But I suppose there is something intrinsically fun about building a complex communications system.
tyu2 6y ago
> I'm putting this one on my new hire required reading list. So, "I can hire people and this is how they must dance for me, regardless of how wrong I am."?
- sudhirj 6y ago
  If the poster is a point-haired-middle-manager archetype, maybe. More often though, it’s very useful for a manager or team leader to ensure that everyone on the team has a common base of knowledge, concepts and terminology. That way the team members can unambiguously understand jargon (at-least-once), communicate with high bandwidth (“let’s switch RMQ queue X to at-most-once mode, the persistent audit FIFO will handle missed messages”). Even if the team members disagree, at least everyone knows what the disagreement is about. The shared vocabulary is very useful.
- KaiserPro 6y ago
  thats one way to look at it. However if you want your employees to grow, you need to teach, encourage, guide and in some cases tell them to stop. doing this in a transparent and accessible way seems like a good idea to me
KaiserPro 6y ago
I've bumped into this recently. Someone waved their hand and said that x queue system didn't work for their specific use case. so they spent an innovation token on making a slow unmonitored queuing system. unsurprisingly the project is late, and it only now has started to work on the bits its supposed to handle. (three months after the public announcement) When I asked for the rationale, they said that x queue system didn't do ordered delivery. The queue system literally has the word "ordered" in it. Its not really their fault, they project manager should have stopped it. However its still not an excuse to re-invent the wheel.
- sudhirj 6y ago
  Yeah, a similar nightmare prompted a lot of this reasearch. A colleague would argue for what felt like hours against SQS because it was “random” ordering, and kept demanding we use something that had strict FIFO. Despite the clients who placed the jobs not being affected by or caring about execution order. The worst part was that we had paid a shit ton of money for many globally distributed executors to increase throughout - I simply didn’t have the vocabulary to explain how asking for strict FIFO would mean only one executor would be running at a time, wasting hundreds of thousands of dollars.
jstrong 6y ago
I've experienced excellent results building distributed systems where the nodes talk to each over via raw udp. often, in the context of message queues, there is much talk about delivery guarantees, but in many contexts the same goals can be accomplished by designing the system so it can handle lost messages. for example, rather than a node sending a single message notifying that it has started work on something, it sends a message once a second, saying, 'I am currently working on x'. is that reinventing the wheel? to me it just seems much simpler.

sudhirj 6y ago

Interesting. This is actually sort of a distillation of all the new hire training / discussions I’ve done. It’s as much a “things I wish someone had told me 15 years ago” thing too.

brightball 6y ago

This is along the same lines, but something people don’t often think about when using and naming queues. https://www.brightball.com/articles/organizing-background-worker-queues https://www.brightball.com/articles/organizing-background-wo...

athoscouto 6y ago

Would you be willing to share the list with us?

indymike 6y ago
Here it is: https://mikeseidle.com/2021/01/03/my-developer-new-hire-reading-list/ https://mikeseidle.com/2021/01/03/my-developer-new-hire-read...
indymike 6y ago
Hmm. I really should find time to get my personal site back online. I took it down a month ago to re-face it and life has intervened.
- loevborg 6y ago
  You could just dump it in a gist

yuppiepuppie 6y ago

> people end up re-inventing message queues and not even noticing they are doing it Amazing how people sucked down this rabbit hole. My work is doing this right now, and its driving me mad as there are well supported alternatives out there. But I suppose there is something intrinsically fun about building a complex communications system.

tyu2 6y ago

> I'm putting this one on my new hire required reading list. So, "I can hire people and this is how they must dance for me, regardless of how wrong I am."?

sudhirj 6y ago
If the poster is a point-haired-middle-manager archetype, maybe. More often though, it’s very useful for a manager or team leader to ensure that everyone on the team has a common base of knowledge, concepts and terminology. That way the team members can unambiguously understand jargon (at-least-once), communicate with high bandwidth (“let’s switch RMQ queue X to at-most-once mode, the persistent audit FIFO will handle missed messages”). Even if the team members disagree, at least everyone knows what the disagreement is about. The shared vocabulary is very useful.
KaiserPro 6y ago
thats one way to look at it. However if you want your employees to grow, you need to teach, encourage, guide and in some cases tell them to stop. doing this in a transparent and accessible way seems like a good idea to me

KaiserPro 6y ago

I've bumped into this recently. Someone waved their hand and said that x queue system didn't work for their specific use case. so they spent an innovation token on making a slow unmonitored queuing system. unsurprisingly the project is late, and it only now has started to work on the bits its supposed to handle. (three months after the public announcement) When I asked for the rationale, they said that x queue system didn't do ordered delivery. The queue system literally has the word "ordered" in it. Its not really their fault, they project manager should have stopped it. However its still not an excuse to re-invent the wheel.

sudhirj 6y ago
Yeah, a similar nightmare prompted a lot of this reasearch. A colleague would argue for what felt like hours against SQS because it was “random” ordering, and kept demanding we use something that had strict FIFO. Despite the clients who placed the jobs not being affected by or caring about execution order. The worst part was that we had paid a shit ton of money for many globally distributed executors to increase throughout - I simply didn’t have the vocabulary to explain how asking for strict FIFO would mean only one executor would be running at a time, wasting hundreds of thousands of dollars.

jstrong 6y ago

I've experienced excellent results building distributed systems where the nodes talk to each over via raw udp. often, in the context of message queues, there is much talk about delivery guarantees, but in many contexts the same goals can be accomplished by designing the system so it can handle lost messages. for example, rather than a node sending a single message notifying that it has started work on something, it sends a message once a second, saying, 'I am currently working on x'. is that reinventing the wheel? to me it just seems much simpler.