5 ms·

Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking

Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course. This logic > written in rust will have better performance, lower latency, ..., lower memory footprint is flawed and is cargo-cult programming unless you say what are you objectively comparing it against and how you intend to achieve those goals. Picking the right™ language just for the sake of these goals won't get you too far.

chenquan 2y ago
Why do you have to beat a native performance-oriented streaming engine written in C or C++? Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings. Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.
- menaerus 2y ago
  So if you don't want to beat native engines in performance what is it that you're trying to solve but Java-based engines don't have? I think it's pretty important to set a vision upfront otherwise you're going to set yourself a trap for a quick failure.
  - chenquan 2y ago
    Hi! Brother, I think I will seriously consider what you said and I am honored to communicate with you.
Jweb_Guru 2y ago
> Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course. There is absolutely nothing in Rust's semantics preventing you from writing high-performance data processing workloads in it, and in fact it's one of the best languages for that purpose. Beyond that, the usual barrier to entry for working on a product like this written in C++ is incredibly high in part because stability and safety are so critical for these products--which is one of the reasons that in practice they are often written in memory safe languages, where C++ is not even an option. Have you worked on any nontrivial Rust data processing product where "copy/clone imposed semantics" somehow prevented you from getting big performance wins? I'd be very curious to hear about this if so.
- menaerus 2y ago
  Stability and safety are the least of the concerns in data processing and database workloads. That's totally not the reason why we saw an increase of these systems during the 90s and early 00s written in Java or similar alternative languages. It was ease of use, low-entry bar into the ecosystem and generally developer pool accessibility. Otherwise, the cost is the main driver in infrastructure type of software and the reason why we see many of these rewritten exactly in C++. Rust is just another contender here, and it's usually because of the performance and a lot of hype recently, which is fair.
  - chenquan 2y ago
    > Stability and safety are the least of the concerns in data processing and database workloads. I'm curious how you came to this conclusion?
    - menaerus 2y ago
      I have professional working experience in this domain.
      - chenquan 2y ago
        Hi, then I'm curious what do you think people care most about in data processing and database workloads?
        menaerus 2y ago
        You're a servant to the business needs so whatever the business needs are at that moment. It's a vague answer probably not appealing to many engineers but that's what it really is. You're solving problems for your business stakeholder and for your business stakeholder clients. In another words, programming language is usually not at the very focus of daily development, given that there's always much bigger fish to fry in this domain, but if Rust provides such an undisputed benefit to your business model, while keeping the cost and risk of it viable for the business, then it's going to be a no-brainer. Chances are that this is going to be the case is very very low. So, my advice would rather be use the language whichever you prefer but don't dwell over it - rather put your focus on innovating workload-specific optimizations that are solving real-world issues that are palpable and easily proven/demonstrated. Study the challenges of storage or data processing engines or vectorized query execution algorithms. Depending on the domain problem you're trying to solve, make sure that your language of choice does not step in your way.

chenquan 2y ago

Why do you have to beat a native performance-oriented streaming engine written in C or C++? Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings. Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.

menaerus 2y ago
So if you don't want to beat native engines in performance what is it that you're trying to solve but Java-based engines don't have? I think it's pretty important to set a vision upfront otherwise you're going to set yourself a trap for a quick failure.
- chenquan 2y ago
  Hi! Brother, I think I will seriously consider what you said and I am honored to communicate with you.

Jweb_Guru 2y ago

> Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course. There is absolutely nothing in Rust's semantics preventing you from writing high-performance data processing workloads in it, and in fact it's one of the best languages for that purpose. Beyond that, the usual barrier to entry for working on a product like this written in C++ is incredibly high in part because stability and safety are so critical for these products--which is one of the reasons that in practice they are often written in memory safe languages, where C++ is not even an option. Have you worked on any nontrivial Rust data processing product where "copy/clone imposed semantics" somehow prevented you from getting big performance wins? I'd be very curious to hear about this if so.

menaerus 2y ago
Stability and safety are the least of the concerns in data processing and database workloads. That's totally not the reason why we saw an increase of these systems during the 90s and early 00s written in Java or similar alternative languages. It was ease of use, low-entry bar into the ecosystem and generally developer pool accessibility. Otherwise, the cost is the main driver in infrastructure type of software and the reason why we see many of these rewritten exactly in C++. Rust is just another contender here, and it's usually because of the performance and a lot of hype recently, which is fair.
- chenquan 2y ago
  > Stability and safety are the least of the concerns in data processing and database workloads. I'm curious how you came to this conclusion?
  - menaerus 2y ago
    I have professional working experience in this domain.
    - chenquan 2y ago
      Hi, then I'm curious what do you think people care most about in data processing and database workloads?
      - menaerus 2y ago
        You're a servant to the business needs so whatever the business needs are at that moment. It's a vague answer probably not appealing to many engineers but that's what it really is. You're solving problems for your business stakeholder and for your business stakeholder clients. In another words, programming language is usually not at the very focus of daily development, given that there's always much bigger fish to fry in this domain, but if Rust provides such an undisputed benefit to your business model, while keeping the cost and risk of it viable for the business, then it's going to be a no-brainer. Chances are that this is going to be the case is very very low. So, my advice would rather be use the language whichever you prefer but don't dwell over it - rather put your focus on innovating workload-specific optimizations that are solving real-world issues that are palpable and easily proven/demonstrated. Study the challenges of storage or data processing engines or vectorized query execution algorithms. Depending on the domain problem you're trying to solve, make sure that your language of choice does not step in your way.