6 ms·
I'm curious about that Rust-based message queue system
by Dowwie 4y ago
I'm curious about that Rust-based message queue system
- likeabbas 4y agofluvio.io ?
- andrewstuart 4y agoWhat do you want to know? It was a prototype. I was trying to learn Rust (didn't succeed), but I did manage to hack together a message queue that used HTTP for client interaction. I'd previously written a SQL database message queue in Python which worked with Postgres/MySQL and SQL server. This worked well but it was not fast enough for my liking. My goal was to build the fastest and simplest message queue server that exists, with zero configuration (I hate configuration). I used Rust with Actix and I tried two strategies - one strategy was to use the plain old file system as a storage back end, with each message in a single file. This was so fast that it easily maxed out the capability of the disk way before the CPU capabilities were maxed out. The advantage of using plain old file system as a storage back end is it requires no configuration at all. So I moved on to a RAM only strategy in which the message queue was entirely ephemeral, leaving the responsibility for message persistence/storage/reliability to the client. This was the configuration that got about 8 million messages a second. As far as I could tell my prototype left almost all message queue servers in the dust. This is because message queue servers seem to almost all integrate "reliable" message storage - that makes the entire solution much, much more complex and slow. My thinking was to separate the concerns of storage/reliability/delivery and focus my message queue only on message delivery, and push status information back to the client, which could then decide what to do about storage and retries. I gave up because I didn't see the point in the end because it wasn't going to make me any money, and I was finding Rust frustratingly hard to learn and I had other things to do.
- moloch-hai 4y agoIt seems very diplomatic of you to say you found Rust hard to learn, rather than that it was hard to make Rust do what you wanted. You seem very clear on what you wanted to do.
- andrewstuart 4y agoI managed to build it without really grasping Rust by hacking around and looking at examples of how other stuff worked that I wanted to do, and by avoiding doing things in Rust that I didn't understand - stuff as basic as function calls. The resulting code worked but was garbage and at the end of the day Rust had not clicked for me and being fluent in it still felt a distant goal. I love the idea of Rust but I don't like the implementation. I'm hoping in time there will be a new language created that has the memory and thread safety of Rust but is 50 times more simple.
- estebank 4y agoIf you don't mind, what where the things that were hard to deal with? You mention "function calls", but I'd like to understand what that actually means in practice. This kind of negative feedback is useful to improve Rust for any other newcomers, even if we have already soured you for good.
- andrewstuart 4y agoI could not even succeed in doing the simplest thing like putting code into a function and executing the function and getting a result back. I can't recall why.
- likeabbas 4y agoInterestingly, I started a side project to rebuild OneTimeSecret in Rust yesterday and got the basics working already https://github.com/likeabbas/RustOneTimeSecret https://github.com/likeabbas/RustOneTimeSecret The only other exposure I had to the language was attempting some Leetcode problems. I really like Rust so far. The type system is extremely powerful, the compiler gives robust error messages, and it has so many awesome features like pattern matching and destructuring.
- andrewstuart 4y agoYou are smarter than me.