5 ms·
Building a message queue with only two UNIX signals
- bradleybuda 1y agoThis is awesome. Does POSIX guarantee the order of signal delivery? And I'm dying to see what the bandwidth / throughput of this channel is...
- cannonpalms 1y agoFor standard signals--no, but for real-time signals, yes. The latter are still a portability issue, though.
- bradleybuda 1y agoAnswering both of my questions, from the post: sleep 0.001 # Delay to allow the receiver to process the signal
- toast0 1y agoI don't know if POSIX has a position on signal order. But I'm pretty sure it allows signals to be coallesced... if a process is sent the same signal several times before the handler is invoked, it's in spec to only invoke it once.
- o11c 1y agoReal-time signals have guaranteed order: first by number (lowest first, i.e. `SIGRTMIN`), then by the order in which they are sent. Signals are generally the slowest IPC method, unless you're doing something stupid with a different method.
- president_zippy 1y agoGlad you had fun doing this! If you choose to take this experiment further and go deeper, you will discover something even more fun: reentrancy. ... That, and how software interrupts work at the kernel level. Happy hunting! EDIT: In anticipation of an eventual response, I just realized how condescending this sounds at first glance. I meant it in good faith.
- gleenn 1y agoFun article but the title is definitely overstating the huge amount of functionality lost if you replace Kafka. Immediately on my mind would be durability and broadcasting.
- dbacar 1y agoSuch titles should be flagged and banned to protect the innocent.
- briandw 1y agoYou have problem with too much traffic that your server can't handle. You add a queue. Now you have 2 problems :)
- foofoo12 1y agoThis has nothing to do with Kafka and it's not not really a functioning message queue except for theoretically speaking. The article is fine, but call it what it is: abusing the Unix signal system for shit and giggles. Nothing wrong with that.
- nijave 1y agoObligatory: Kafka is a log, not a message queue (although it turns out there's a lot of overlap)
- ape4 1y agoI hope this is a joke / hack. A named pipe (like Postfix sendmail uses) seems slightly more sane.
- liqilin1567 1y agoYeah, there are a lot of more elegant, simple and reliable ways to do that.
- kryword 1y agoI liked a lot your article, hacking with Unix signals is impressive, and was faster than I expected. I would have expected signals to be slower.
- cortesoft 1y agoWow, I didn't know! I will work on replacing my Kafka cluster handling 10 million msg/sec with this right away!
- ok123456 1y agoWhy use signals when POSIX defines a complete message queue interface: https://www.opensourceforu.com/2023/06/a-deep-dive-into-posix-message-queues/ https://www.opensourceforu.com/2023/06/a-deep-dive-into-posi...
- gldrk 1y agoUNIX signals *do not* queue. If two or more signals with the same number are sent faster than the receiving thread handles them (due to the signal being blocked and/or the thread not being scheduled), all but the last will be lost irrevocably. There is no mechanism to prevent this. https://ldpreload.com/blog/signalfd-is-useless https://ldpreload.com/blog/signalfd-is-useless
- jmalicki 1y agoRT signals do get queued... that is one of the major differences (and yes, the essay is not using them, so your point stands as it is written, but using RT signals is a mechanism to prevent it). https://davmac.org/davpage/linux/rtsignals.html https://davmac.org/davpage/linux/rtsignals.html
- nakamoto_damacy 1y agoNaively asking: what prevents RT Unix/Linux from being used in place of non-RT mainstream versions? Seems like a superset.
- jmalicki 1y agoRT signals are an extended API of POSIX you don't need actual RT Linux to use them.
- kscarlet 1y agoAre they guaranteed to be delivered in order?
- jmalicki 1y agoYes as long as you use sigqueue with SA_SIGINFO
- RedShift1 1y agoThat's easily solved, just create a queue for your signals.
- donatj 1y agoWe run a very simple filesystem based queue that processes around 1 billion events a day. Makes use of XFS for it's better handling of large numbers of files. Corporate tried to push us to replace it with SQS and it could not keep up / costs with through the roof
- RedShift1 1y agoHow does it work? Can you have multiple workers on the same payload? Do you archive the events afterwards?
- otterley 1y agoBefore y'all go nuts with the criticisms... "Yes, we built a message broker using nothing but UNIX signals and a bit of Ruby magic. Sure, it’s not production-ready, and you definitely shouldn’t use this in your next startup (please don’t), but that was never the point. "The real takeaway here isn’t the broker itself: it’s understanding how the fundamentals work. We explored binary operations, UNIX signals, and IPC in a hands-on way that most people never bother with. "We took something “useless” and made it work, just for fun. So next time someone asks you about message brokers, you can casually mention that you once built (or saw) one using just two signals. And if they look at you weird, well, that’s their problem. Now go build something equally useless and amazing. The world needs more hackers who experiment just for the fun of it."
- SchwKatze 1y agoUnfortunately I bet that 90% won't even reach at that part and just ragebait based on the title. The golden rule of modern age is always do the disclaimer as soon as possible.
- cortesoft 1y agoOr you could skip the rage bait title entirely?
- SchwKatze 1y agoMaybe, but I know Leandro, it was more a joke than anything else. People just don't chill, the post is cool
- bawolff 1y ago"Its just a joke, bro" is always a terrible defense for rude behaviour.
- SchwKatze 1y agoI'm truly curious to know why would this be rude, seriously. Maybe it's a cultural mismatch. For me ragebait and rudeness are things like: "X sucks, use Y", "If you aren't doing W you're losing money", etc. He never said that Kafka sucks, nor anything related, obviously you can't replace kafka with only two signals. I'm asking with all politeness as possible, I just wanna understand what other people consider improper behavior
- drob518 1y agoClick bait
- nialv7 1y agoyou can literally just attach data[1] to signals... you don't need to do this. [1]: https://man7.org/linux/man-pages/man3/sigqueue.3.html https://man7.org/linux/man-pages/man3/sigqueue.3.html
- lifthrasiir 1y agoInitially I assumed that the author was about to use something like shared memory for bulk transfer. Boy I was totally wrong...
- bawolff 1y ago> And if you came here just because of the clickbait title, I apologize and invite you to keep reading. It’ll be fun, I promise. I kind of hate this trend of making clickbait and then apologizing for it. I think its more annoying then just making clickbait. Is it really that hard to just accurately title your blog posts?
- johnisgood 1y agoI read the article. I think it is accurate enough.
- m-hodges 1y agoAttention is the new oil.
- bcrl 1y agoThis is the kind of article that deserves to be posted on April 1st, preferably with an accompanying RFC published at the IETF.
- hshdhdhehd 1y agoCan those two Unix signals run Doom?
- sheepscreek 1y agoI challenged ChatGPT the other day to design a bidirectional process interop in *nix and this was one of the suggestions. Until then I had only ever thought of pipes as unidirectional. I still thought it was bonkers. This looks like a neat prototype though.
- m-hodges 1y agoThis was a really fun article and many in the comments seem to have forgotten you're allowed to have fun with the computer.
- aaronrobert 1y ago[dead]
- hajimuz 1y agoGreat article for learning UNIX Signal
- florians 1y agoI enjoyed reading the article and found it interesting. Had no prior interest in the topic but it was an entertaining read. Last time I used binary and calculated with it was in high school. Didn’t know about UNIX signals and now I understand how processes are terminating.
- up2isomorphism 1y agoThe author doesn’t know why people use things like Kafka in the first place
- esseph 1y agoOne of my favorite things that happens on the internet is when it is clear as glass when someone didn't read the article. You nailed it!