6 ms·
Iggy.rs – building message streaming in Rust
- ramon156 3y agoLooks neat! Excited to play around with it when I get the time
- spetz 3y agoThank you, feel free to ping us anytime and/or join our open Discord!
- iddan 3y agoLove the idea, and the blog post writing. The author really comes off humble, honest and a constructive project leader. Good luck!
- spetz 3y agoThank you, I'd like to mention that the team is really awesome, as they've decided to join me in my efforts, just to have some fun as well :)
- cwales95 3y agoWould love to give it a go at some point (but I'll first need to learn Rust). On a side note, I love the aesthetic of the site.
- cosmic_quanta 3y agoThe blog post mentions SDKs for other programming languages, so you may be able to use this without learning Rust
- spetz 3y agoExactly, we have multiple SDKs, and the blog uses Rust Zola engine :)
- pdimitar 3y agoI'm still trying to learn Zola well. Do you plan to open-source the website?
- musjleman 3y agoIf you're talking about the blog, it's an open source Zola theme https://github.com/not-matthias/apollo https://github.com/not-matthias/apollo (the original site https://secret.club https://secret.club which it's based on/inspired by is closed source though)
- pdimitar 3y agoNo, I really need to see how did he structure his website exactly -- to learn how I would do it as well. I'll take a look at the theme as well, thank you.
- bsaul 3y agoHow fast can one realistically expect to be able to build decent quality system-level rust code, provided the person is already familiar with C-style programming languages and has 0 rust experience ?
- ddoolin 3y agoIt has gotten a little easier over time. IMO, a month or two depending on your aptitude.
- spoiler 3y agoIf you have a little bit of experience with any higher level language (eg Python, Haskell), then I'd say fairly quicky (say, a week to a week and a half for the books and rustlings course), if you want an opinionated (and good enough, I recommend the book!) intro to production code theres the zero2prod Rust book. You can read the zero2prod book while working on one or two pet project (anything more complicated than a calculator would probably do) to help inform decisions. That should be another week or two, depending on how quickly you pace through the book. That should get you started to writing decent code, maybe not aways idiomatic or neatly structured. That might take a few more weeks or months (depending on your interest domains and their inherent complexities). But like with all skills, it's an endless journey; you'll keep getting better as you're increasingly more familiar with the tools.
- deleted 3y ago[deleted]
- steveklabnik 3y agoThe closest thing I have to a non-anecdotal answer to this question is from Google: https://opensource.googleblog.com/2023/06/rust-fact-vs-fiction-5-insights-from-googles-rust-journey-2022.html https://opensource.googleblog.com/2023/06/rust-fact-vs-ficti... > All survey participants are professional software developers (or a related field), employed at Google. While some of them had prior Rust experience (about 13%), most of them are coming from C/C++, Python, Java, Go, or Dart. > Based on our studies, more than 2/3 of respondents are confident in contributing to a Rust codebase within two months or less when learning Rust. Further, a third of respondents become as productive using Rust as other languages in two months or less. Within four months, that number increased to over 50%. Anecdotally, these ramp-up numbers are in line with the time we’ve seen for developers to adopt other languages, both inside and outside of Google. > Overall, we’ve seen no data to indicate that there is any productivity penalty for Rust relative to any other language these developers previously used at Google.
- emmanueloga_ 3y agoLooks like a direct competitor to JetStream [1]? Impressive progress in less than a year of work! -- 1: https://docs.nats.io/nats-concepts/jetstream https://docs.nats.io/nats-concepts/jetstream
- spetz 3y agoJetStream, Kafka, Redpanda, RabbitMQ Streams, Fluvio - quite a few message streaming solutions out there. Thanks!
- the_duke 3y agoI'm not quite sure how this compares to Kafka and fluvio [1], a Kafka competitor also written in Rust? Is it more of a message queue like rabbitmq? [1] https://www.fluvio.io/ https://www.fluvio.io/
- spetz 3y agoIt's a message stream, so more like Kafka, Redpanda or RabbitMQ Streams (plugin). Fluvio is more mature (it's an actual product & company behind it), but we have our own ideas how to make Iggy a competitive message streaming solution :)
- esafak 3y agoDoesn't Fluvio aim to replace both Flink and Kafka? I just heard about it so I am trying to understand.
- debadyutirc 3y agoI lead product at Fluvio. If I remember correctly, I had seen iggy.rs in one of the recent applicant resumes to one of our Senior Rust Developer roles. Iggy seems like a cool project. All the best for the future of Iggy. Fluvio has been in open source development for nearly 5 years now. The team has written hundreds of thousands of lines of code. Fluvio core is a complete distributed streaming system for unbounded stream processing that is reliable, scalable, self healing. It’s the Kafka bit. The Flink bit is our Stateful Service Development Kit which is currently in Dev Preview. Stateful Service Development Kit offers a scaffolding to compose complex chained event driven data pipelines. Once we launch the stateful materialization bit, we will have a complete system for building end to end streaming data flows.
- vlovich123 3y agoI'm surprised about the selection of monoio. AFAIK it requires the usage of the nightly compiler which I've found to not be a great choice when maintaining a project.
- steveklabnik 3y agoIt does require nightly, but it only uses five features, one of which could be removed by adding an external crate, and the others are like, not anything super wild, for the most part. I haven't looked too deeply into the code but they all make sense to me, for example one of them is a standard library API for creating uninitialized containers, which allows you to eliminate a copy. I have not compared monoio with glommio, which does run on stable. Would be interesting!
- vlovich123 3y agoIt’s not that the features are unreasonable but more that anything that uses it as a dependency also inherits the nightly requirement which makes things a bit more unstable because nightly is a floating version. I did compare with glommio a bit and disliked the file API in monoio but that’s probably more a matter of taste.
- steveklabnik 3y agoYeah I hear you. I much prefer staying on stable for that reason. Sometimes (though much more rarely than non-Rust users seem to think) it is unavoidable though, and once you do go into nightly, you can kinda go two ways: * only use what is absolutely necessary * go wild and add tons of stuff As well as there being two kinds of nightly features: * stuff that's not stable but has a clear path to becoming stable on some time frame * stuff that's not stable but who knows when that will change, it may never And to me, this dependency feels kinda closer to choosing the first option from both lists, which is the most conservative way of using nightly.
- spetz 3y ago
- mattmmatthews 3y agoi'm a bit confused as to what exactly this is; a low-level, messaging system? what are the implications of it and the comparison of competitors? I'm obviously missing something. Could someone elaborate?
- spetz 3y agoThis is a message streaming infrastructure, similar to Kafka, Redpadna etc. You can think of it as a kind of a simple database (so-called Write Ahead Log), being responsible for appending the messages in a highly performant manner.
- jauntywundrkind 3y agoIt seems like such a sharp, smart move to start with QUIC. It provides really useful SCTP-like multi-streaming, which is a great base to start from. There's already so many good libraries the project or others can work from, and it'll just keep getting better (and better optimized, likely). https://github.com/xileteam/awesome-quic?tab=readme-ov-file#library--frameworks https://github.com/xileteam/awesome-quic?tab=readme-ov-file#... This seems like such a natural place where of course having a somewhat better transport protocol than we've had is a huge win. Looking forward to the next decade of QUIC.
- spetz 3y agoI've started with QUIC, as I wanted to try out something new. Still, TCP protocol we have in place is a bit faster than QUIC, but it might be due to the lack of some additional tuning or so. On a side note, QUIC on MacOS is slow, when compared to Linux.
- dochtman 3y agoIs TCP + TLS still faster than QUIC, or do you mean TCP without encryption? (I'm assuming you're using Quinn with rustls, so encryption is always enabled.) I'm also curious about your macOS vs Linux numbers. As a Quinn maintainer, we're always happy to hear feedback including about our performance!
- spetz 3y agoRight, I meant TCP without TLS, and when it comes to QUIC I only did benchmark using the "dangerous" mode (disabled local cert validation). I don't have the exact numbers, but AFAIR on MacOS raw TCP was a few times faster than QUIC, while on Linux (e.g. PopOS distro) TCP was maybe around 20-30% faster? Still, there are plenty of options to be configured in Quinn (buffer size, send/receive window etc.), so it might be that with some tweaking the performance would be similar or QUIC would be even faster than TCP - this is also one of the things that I'd like to spend more time with in the future. We expose most of these options in the server configuration file, so it's easy to adjust. Thank you for contributing to such a great library, really easy to work with :)
- amath 3y agoThese types of stories are what drove me to software in the first place. The ideal of a group of people working towards the same goal, for sometimes different reasons, but not for the sole purpose of monetary reward. Good luck with the project! I also echo some of the others, it would be great to see a sort of comparison with the alternatives so that we can better understand how the project fits in.
- spetz 3y agoThank you, that's exactly how it all started! :) At some point, we will try to incorporate some benchmarks and comparisons with the other tools.
- scxsky 3y agoI keep my fingers crossed, the project looks great and interesting!
- spetz 3y agoThank you :)
- internet101010 3y agoIs this project kind of an amalgamation of Kafka and something like Flight/nats?
- spetz 3y agoCan't say for sure, we share some of the same concepts that can be found in Kafka or RabbitMQ Streams, but at the same time, try to build something from the ground up and focus on low-level I/O improvements :)
- worik 3y agoDefinitely wins a prize for cute dog picture Love it
- spetz 3y agoThanks, I do agree, they're both lovely! :D
- thibauts 3y agoBuilt something similar in Go with a friend a few years ago. https://github.com/thibauts/styx https://github.com/thibauts/styx
- spetz 3y agoNice one, looks quite similar indeed! May I ask why did you discontinue the further work?
- xwolfi 3y agoProbably because there's a very low value in building message streaming framework once you've done the basics: how do you come to, say, a trading broker and say "your order management system could be just as fast with my other messaging framework". He did it to learn and have fun, but finishing it to be prod-ready would require at least one user.
- thibauts 3y agoThat was indeed a big parameter. We missed someone on the team able to connect to us to one or two large early adopters. We learned that building a good product and building a company selling it are two very different things !
- thibauts 3y agoLack of traction basically. We ran out of time and money. Also we, maybe wrongly, did not feel like we could compete with the big ones without clustering. Yet we had a lot of fun building it. It was a thrill to see our product deliver performance on par or above in-memory messaging systems while being fully persisted and fsync’ed. I’m sure you know what I mean ! Good luck with Iggy anyway. Looks like you are on the right track !
- lysecret 3y agoQuite an aspirational read especially about how the team came to be. I love open source.
- spetz 3y agoI did some dotnet OSS in the past, but there's something special about Rust community and I guess that if I'd picked another language instead, I wouldn't gather such an amazing team :)
- tizz98 3y agoReally cool idea and project! There are 2 things I would need to understand before trying it out though: 1. How can I run more than one instance of the server? 2. When running more than one instance of the server, how would the filesystem interaction work between servers?
- tortoise_in 3y agoWill there be any public servers like xmpp? So that people can use it like normally who don't have expertise in hosting servers.
- spetz 3y agoIggy is a message streaming infrastructure (think of Kafka for example), not a messaging platform like Discord or so :)
- pseudopersonal 3y agoHow would this stack up to the likes of KeyDB’s streaming? More durable persistence is my take.
- debadyutirc 3y agoThis post prompted me to resurface the Genesis of Fluvio. We are a small team going at it for a little while since we have long relationship with the data centric applications across multiple domains for the past decades and we are excited about data streaming on rust and web assembly instead of java and jvm. Here is a blog that is from June 2021 from our CTO laying out the vision for Fluvio. https://news.ycombinator.com/item?id=38880743 https://news.ycombinator.com/item?id=38880743 Since the comparison question keeps popping up, I would share whatever we have as well. It's a long drawn documentation exercise and I am happy to share what we have at Fluvio. Again - Great job with Iggy!