7 ms·

Applications of Java and Rust do not overlap. As a big data engineer, Rust does not provide any advantage to me. Good luck rewritting all the big data tools fro

Applications of Java and Rust do not overlap. As a big data engineer, Rust does not provide any advantage to me. Good luck rewritting all the big data tools from Java (Elasticsearch, Spark, Kafka, Hadoop, Neo4j, Deeplearning4j, Cassandra, Solr, Arrow, OrientDB) and all the scientific stuff from C++ (Eigen, Tensorflow, PyTorch, Jax, Halide, OpenCV and the multitude of resources for CUDA and OpenCL) and databases (MySQL, MongoDB). If I want to contribute to LLVM or JVM, they are C++, the browsers are all C++. Unreal Engine and Unity (IL2CPP and engine) are all C++. Are you doing microVMs or drivers? Then probably Rust is a good investment (discutable), otherwise, I would wait for better alternatives. I will stick to Java, Scala, Python and C++ for the foreseeable future. EDIT: may the downvotes come from the idealists that hammer down pragmatists, like our opinion doesn't even count

coddle-hark 6y ago
With this logic there should be no new programming languages ever until there’s a paradigm shift in computing.
- johnnycerberus 6y ago
  > I always hoped Java would make a comeback, but Rust is looking like the real deal. I was mainly responding to this statement which came across quite aggressive. Java and C++ do not need comebacks and my comment was intended to be a wake-up call to reality. Yes, it is great to learn Rust on the side, but advocating something to be used everywhere is another story.
  - jodrellblank 6y ago
    > "but advocating something to be used everywhere is another story." Where is that in the comment you replied to?
    - johnnycerberus 6y ago
      > Where is that in the comment you replied to? I guess you don't know what the real deal is.
      - basicneo 6y ago
        I think you're being downvoted because this is a bit passive aggressive. That said, I sense a bit of Rust bias on HN.
geofft 6y ago
Why do you need to rewrite them? Rust can call C++ code, just like Java and Python can. (I note you list Python libraries like Tensorflow and PyTorch as C++, which is accurate because they're written in C++, but they're certainly heavily used from a non-C++ language.)
- johnnycerberus 6y ago
  It was more of a proof of what you can build with Java and C++, the infrastructure that provides Python users with capabilities.
- jpgvm 6y ago
  Rust definitely makes sense anywhere you are currently using C/C++ or Python/Ruby. Java is somewhat of a special case though, especially the Big Data ecosystem. Much of the fastest and most important Java tools are written in pure Java (or other JVM language) and as such can't be called from Rust in a reasonable way. Writing Big Data code in Rust is certainly possible from a performance perspective but is an uphill battle of lack of libraries, integration, etc. Some of this might change in time but I don't see Rust ever displacing Java from Big Data. One of the big reasons for this is the main downside of Java is it's GC, which is mainly problematic for latency sensitive applications. In the Big Data field this is substantially reduced as a problem as the code is usually throughput bound and executed outside of interactive contexts, hiding the cost of the GC where it might be more apparent in other applications. I love Rust and I think it has a lot of potential to replace some of the databases and networked services used in data infrastructure, just don't see it displacing Java as the top dog.
  - proverbialbunny 6y ago
    You've got it backwards. Rust is lower level than Java, so Java would be calling Rust, not the other way around. Rust isn't aiming to displace Java. They're two different levels of abstraction. It's the same with Python. Rust isn't aiming to displace Python just as Java isn't aiming to displace Python either. Rust one day could displace C++, but the odds are low, because in response to Rust C++ has stepped up its game and has changed as a language drastically. Most modern C++ features are concepts from Rust.
jjice 6y ago
I don't think rewrites should be the primary talking point, but rather using it for new applications, especially in memory safety critical environments like kernel modules.
g4nt1 6y ago
In my opinion data processing / ML / database development is one of the best use case for rust. You want stable memory management and performance . This is still something than often times is done in C/C++. There are several promising technologies being developer in rust in that space; tikv (key-value store), materialize (steaming DB), arrow/parquet has a rust implementation, Toshi (Elasticsearch), Tantivy (Lucene), etc
stagger87 6y ago
There will be people who downvote you because... - You are hijacking a top comment to insert your off-topic opinion. - You appear to be more territorial than presenting some sort of argument. - You are overly negative and somewhat dismissive/disrespectful. - You are commenting on downvotes, which is generally frowned upon
- johnnycerberus 6y ago
  My comment in on point, not off-topic. I had no intention to "hijack" the top comment, didn't even think about that until I've seen your comment. I don't see Rust as a replacement for Java and I also pointed out that Rust has to win the mindshare of C++ programmers first, before even reaching other ecosystems. Rust is mainly a replacement for C++.
  - danudey 6y ago
    Your comment that Java and Rust don't overlap is valid, but going on to discuss how you're going to keep using C++ and Python, and how Unreal Engine uses C++, is wildly irrelevant. Rust isn't going to replace Java, but it may supplant Java in a lot of fields where one wants both memory safety and an OO language with high performance. No one is (rationally) suggesting that Rust will replace every language. The tone of your post, however, makes it sound as though you have something against Rust, and that you're more interested in pointing out all the things it's not good at, or shouldn't be used for, rather than contributing to a constructive discussion. > I also pointed out that Rust has to win the mindshare of C++ programmers first, before even reaching other ecosystems Not entirely true; Rust is an alternative to C++, but it's vastly more accessible to people who've used other languages like Java, Python, Ruby, and so on, where you get types, OO, and memory safety. Instead of simply resulting in a 1:1 replacement of C++ with Rust, it may result in a minimal reduction in C++ programming and a notable increase in Rust programming which would have been done in another safe OO language instead, but at dramatically worse performance.
    - chc 6y ago
      Small clarification — Rust very deliberately does not do OO. It has features that look sort of like OO from a syntactic standpoint, but being object oriented is not a design goal for Rust.
saagarjha 6y ago
> the browsers are all C++ You mean one is already shipping Rust and another seems to be on track to shipping it soon?
- est31 6y ago
  Chrome is "on track"? Last I remember they had a document explaining why it's not a good idea for them. I disagree with it, and would very much wish for Chrome to ship Rust, but they don't seem to be interested for now.
  - TheDong 6y ago
    They're referring to this[0] I believe, which is not glowing, but does seem mildly optimistic. Specifically: > For now, Chrome investment in Rust will remain a background investigation (mostly directed towards prototyping these tools and techniques). If we become convinced this sort of interoperability is possible, we’ll revisit widespread use of Rust in Chrome [0]: https://www.chromium.org/Home/chromium-security/memory-safety/rust-and-c-interoperability https://www.chromium.org/Home/chromium-security/memory-safet...
    - MaxBarraclough 6y ago
      Right, and their issues there all pertain to interoperability, not to issues with the Rust language and its implementation. They seem to view Rust as production-ready, and a language that would bring relevant benefits to the Chromium project, were it not for the C++ interoperability difficulties.
    - fabrice_d 6y ago
      Looks like this will not happen soon: https://groups.google.com/a/chromium.org/g/blink-dev/c/gDA2SMPePv8 https://groups.google.com/a/chromium.org/g/blink-dev/c/gDA2S...
    - est31 6y ago
      Yes that's the same document I refer to as well. So nothing has changed. A "mildly optimistic" rejection is a rejection of Rust nonetheless. Rust is definitely not "on track" of being adopted in Chromium. For that you'd need more. Right now it's rather a "we are interested, maybe we'll add it". Also note that Chromium engineers also built some languages themselves that solve some of the same problems that Rust solves: https://github.com/google/wuffs https://github.com/google/wuffs
augusto-moura 6y ago
You are saying different things as one argument. Yes, Java applications and Rust mostly don't overlap, but there are exceptions in both ecosystems. Rust might offer some advantages for Data Engineers (the same way as CUDA applications in C++), not all of them as with every language else, and yes you can write Data centric libraries in Rust for Python (easier than C/C++ IIRC). And no, there's no need to rewrite existing services to Rust, it can interface with all the examples you cited. To me Rust is just a simpler and safer C++, if you said you would program C++, you probably will want to program Rust, writing C++ bindings for Rust is basic stuff nowadays
- bbatha 6y ago
  There are a lot of database-like services written Java (Kafka, Cassandra, ElasticSearch, Neo4J, HBase) that would likely be better in Rust. Yes the common denominator is Apache but they seem to have bought into Rust as well https://arrow.apache.org/blog/2019/02/04/datafusion-donation/ https://arrow.apache.org/blog/2019/02/04/datafusion-donation...
  - StreamBright 6y ago
    DataFusion is a tiny project maintained by few people. I am really hopeful that it takes off though.
deleted 6y ago
[deleted]
throwaway894345 6y ago
Your comment seems to assume that your use case “big data engineer” is the only use case or the only Java use case and that the only context in which Rust can grow is to rewrite the existing tools that you use today (as opposed to greenfield projects like firecracker or bottle rocket). Your edit also writes Rust off as “idealistic” and not “pragmatic” without support, which is very far from the impression I have (based on my usage and seeing it used in real applications). In general, your comment seems like generic programming language flamebait.
- johnnycerberus 6y ago
  Firecracker and Bottlerocket are fine pieces of engineering. If Rust wants to get a share of the pie when it comes to data engineering, then it needs to have at least a killer app with an ecosystem around it. We can talk all day and all night for why some languages are more successful than others but it seems like ecosystems are giving them a boost. Hopefully, Rust will be the language for microVMs and kernel-level modules. I was strictly referring to people that think Rust would displace Java as idealists.
  - throwaway894345 6y ago
    I generally agree--languages do better when they become the language of a particular ecosystem and that ecosystem becomes relevant. Go had this with containers and network software a bit more broadly. That said, I don't think anyone was suggesting Rust would displace Java. The OP was pretty clear IMO that his firm had experienced exponential growth in interest in Rust skills while he had been hoping that interest in Java would grow instead. This doesn't mean that Rust will replace Java for data engineering or any other domain in particular.
  - mr_toad 6y ago
    > it needs to have at least a killer app with an ecosystem around it Give it 20 years and I can see it doing to Python what Python did to R.
    - abenga 6y ago
chc 6y ago
20 years ago, almost none of that stuff existed. It's entirely possible that whatever exists 20 years from now will all be Rust. I don't think that's likely, but I think it's more likely than the landscape you describe still existing in any recognizable form.
hundchenkatze 6y ago
The part you seem to be replying to doesn't imply that Rust is or should replace Java, at least not to me. > I always hoped Java would make a comeback, but Rust is looking like the real deal. To me it seems like SoSoRoCoCo is hoping their preferred language would make a comeback in popularity while also acknowledging that Rust is gaining popularity and real-world demand.
twic 6y ago
I write Java and Rust at work, and i more or less agree. More or less because there is some overlap: for example, people write low-latency trading systems in both. But it's very hard to get Java to perform well at the kind of work where Rust shines naturally. There is also very little overlap between Rust and Go. There's far more between Java and Go than Rust and either!
woah 6y ago
Being as experienced as you are, couldn't you just learn Rust like any other language if you need it in your work (or just to try it out). Your post has a weird negative vibe, like it would cost you greatly to try it.
- basicneo 6y ago
  The cost to learning Rust might be low but the opportunity cost might be high, since there's many other ways someone can improve their career.
sjtindell 6y ago
A good point and a long list of tools. But imo this is not the industry to say “good luck rewriting X” in. It is inevitable someone will come along to rewrite or write a new and improved tool for everything on that list. And it might be in Rust.
patrec 6y ago
Debatable. Java's dominance in the data engineering space rests a bit on clay feet: it's unsuitable for scientific computing in a number of ways: for example you can neither write fast nor clean numerical code in java and it does not interface naturally with the languages that people do write numerical code in. This is not an absolute deal breaker, but it adds friction. Also, serveral big data tools like cassandra (scylladb), kafka (redpanda) etc. have basically already been rewritten in C++ not least because they suffer from java-typical performance problems (even if those problems are not necessarily inherent in the choice of java). But C++ is not a very attractive language to write code in if there are viable alternatives, because it is objectively unpleasant and expensive to use (long compile times, terrible build and packaging eco-system, high cognitive load, difficulty of avoiding undefined behavior, etc.). Rust shares some of these defects and has some of its own, but it also offers substantial benefits for some types of problems that seem relevant to data engineering and crucially it interfaces more naturally with Fortran, C++ and Python than Java does. So I think it has a chance to gradually make inroads.
- twsted 6y ago
  > C++ is not a very attractive language to write code in if there are viable alternatives, because it is objectively unpleasant and expensive to use From a current perspective I may agree, but in the ‘90 a number of programmers like me was really excited to be able to develop at a much higher abstraction level than C using classes and, later, templates, preserving efficiency and performances.
tutfbhuf 6y ago
> databases (MySQL, MongoDB) We are heading towards distributed high available SQL and NoSQL databases (with Redis, Postgres and MySQL compatibility for easier migration) designed to work exceptionally well in cloud environments like CockroachDB (Go), TiKV (Rust), Vitess (Go) to name a few. That is because SRE teams want to be able to scale horizontally and make a node, rack or even region failure to be a normal thing, their company don't need to worry about.
StreamBright 6y ago
> As a big data engineer, Rust does not provide any advantage to me. Good luck rewritting all the big data tools from Java (Elasticsearch, Spark, Kafka, Hadoop, Neo4j, Deeplearning4j, Cassandra, Solr, Arrow, OrientDB) Yep, this is true. Java has an upper hand in data engineering. There are better languages for targeting the Java libraries and JVM than Java itself. I always use Clojure instead of Java when doing data crunching because it is much easier to create parallel programs with it. Kotlin is also coming up in this space. Having said that. I like Rust for some reasons: - targeting multiple libc (musl, no libc at all) - targeting multiple platforms (ppc, mips, etc., even though some of those are tierN) - having ML features Why I do not like Rust: - accidental complexity and convoluted syntax (maybe it is only me, but it is very hard to follow what is happening) For me the ideal language would be something like F# with the good parts of Rust.
caffeine 6y ago
I am a major fan of Rust. I upvoted your comment because it is mostly factual, and correct! I agree that Rust is still early for most adopters. There are a subset of people who need to solve problems that would normally be solved in C or C++, but they don't know or don't want to use those languages. Those people generally love Rust (I am in this camp). Then there is everyone else, who either like Rust because it's a cool language, or don't for any number of reasons all of which are fine!
mynameisash 6y ago
> As a big data engineer, Rust does not provide any advantage to me I assume that this is meant to imply that Rust doesn't provide advantages in the data engineering space. So I'll add in here -- as a data engineer myself -- that Rust is definitely providing an advantage for me. Combining it with Python (using PyO3 bindings) lets me write performant, safe code that interops with Python in a project for which Scala+Spark isn't feasible. So while Rust may not provide an advantage for you, that doesn't mean it doesn't have a potential niche for data engineering.
dmit 6y ago
> Applications of Java and Rust do not overlap. Both are general purpose languages, so of course they overlap. > Good luck rewritting all the big data tools from Java (Elasticsearch, Spark, Kafka, Hadoop, Neo4j, Deeplearning4j, Cassandra, Solr, Arrow, OrientDB) Cassandra (https://scylladb.com https://scylladb.com) and Kafka (https://vectorized.io https://vectorized.io) have already been rewritten once in C++, with massive latency and throughput improvements. No reason why they couldn't get their superior Rust clones in the observable future. Materialize (https://materialize.com https://materialize.com), Noria (https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria), and Sled (https://github.com/spacejam/sled https://github.com/spacejam/sled) are just some of the Rust database projects that are aiming at unseating the de facto standard implementations in the space. InfluxDB (https://www.influxdata.com https://www.influxdata.com) is now doing major Rust development as well. The future is almost here. It's just not evenly distributed yet.
- StreamBright 6y ago
  > Cassandra (https://scylladb.com https://scylladb.com) and Kafka (https://vectorized.io https://vectorized.io) have already been rewritten once in C++, with massive latency and throughput improvements. It is not a single dimension problem as they try to frame it. Performance is a single dimension of systems. Java with GC is a memory safe solution compare to C++ (hello segmentation faults). The reason why the C++ implementation is faster is not that it is written in C++ but it is written using a different approach. There are plenty of Java HFT systems that run circles around C++ systems yet we cannot draw a conclusion that Java is better suited than C++ for HFT. Implementation details matter, code quality matters and performance does not come purely because you pick a particular programming language. Btw. most of the systems which I was working on using Cassandra instead of ScyllaDB because the companies were not ok switch up for some performance gains (because Cassandra was performant enough for their business use case). I think what you receive as superiority is not as clear cut as you try to make it.
oblio 6y ago
About microVMs, I'm not sure what this is referring to, but this literal post says that AWS' tech for VMs is now based on Rust :-)
secondcoming 6y ago
Cassandra has already been ported to C++ with Scylla. We switched to Scylla a year or so ago, haven't looked back. As for the others, if it can be shown that porting them comes with a decrease in latency and infrastructure costs then they'll be ported. I've often wondered why they haven't been already.
mr_toad 6y ago
Stuff written before Rust existed was not written in Rust? I’m shocked. That’s like going back to the 90’s and pointing to all the software not written in Java. And as for browsers, you know Mozilla created Rust specifically for Firefox?

coddle-hark 6y ago

With this logic there should be no new programming languages ever until there’s a paradigm shift in computing.

johnnycerberus 6y ago
> I always hoped Java would make a comeback, but Rust is looking like the real deal. I was mainly responding to this statement which came across quite aggressive. Java and C++ do not need comebacks and my comment was intended to be a wake-up call to reality. Yes, it is great to learn Rust on the side, but advocating something to be used everywhere is another story.
- jodrellblank 6y ago
  > "but advocating something to be used everywhere is another story." Where is that in the comment you replied to?
  - johnnycerberus 6y ago
    > Where is that in the comment you replied to? I guess you don't know what the real deal is.
    - basicneo 6y ago
      I think you're being downvoted because this is a bit passive aggressive. That said, I sense a bit of Rust bias on HN.

geofft 6y ago

Why do you need to rewrite them? Rust can call C++ code, just like Java and Python can. (I note you list Python libraries like Tensorflow and PyTorch as C++, which is accurate because they're written in C++, but they're certainly heavily used from a non-C++ language.)

johnnycerberus 6y ago
It was more of a proof of what you can build with Java and C++, the infrastructure that provides Python users with capabilities.
jpgvm 6y ago
Rust definitely makes sense anywhere you are currently using C/C++ or Python/Ruby. Java is somewhat of a special case though, especially the Big Data ecosystem. Much of the fastest and most important Java tools are written in pure Java (or other JVM language) and as such can't be called from Rust in a reasonable way. Writing Big Data code in Rust is certainly possible from a performance perspective but is an uphill battle of lack of libraries, integration, etc. Some of this might change in time but I don't see Rust ever displacing Java from Big Data. One of the big reasons for this is the main downside of Java is it's GC, which is mainly problematic for latency sensitive applications. In the Big Data field this is substantially reduced as a problem as the code is usually throughput bound and executed outside of interactive contexts, hiding the cost of the GC where it might be more apparent in other applications. I love Rust and I think it has a lot of potential to replace some of the databases and networked services used in data infrastructure, just don't see it displacing Java as the top dog.
- proverbialbunny 6y ago
  You've got it backwards. Rust is lower level than Java, so Java would be calling Rust, not the other way around. Rust isn't aiming to displace Java. They're two different levels of abstraction. It's the same with Python. Rust isn't aiming to displace Python just as Java isn't aiming to displace Python either. Rust one day could displace C++, but the odds are low, because in response to Rust C++ has stepped up its game and has changed as a language drastically. Most modern C++ features are concepts from Rust.

jjice 6y ago

I don't think rewrites should be the primary talking point, but rather using it for new applications, especially in memory safety critical environments like kernel modules.

g4nt1 6y ago

In my opinion data processing / ML / database development is one of the best use case for rust. You want stable memory management and performance . This is still something than often times is done in C/C++. There are several promising technologies being developer in rust in that space; tikv (key-value store), materialize (steaming DB), arrow/parquet has a rust implementation, Toshi (Elasticsearch), Tantivy (Lucene), etc

stagger87 6y ago

There will be people who downvote you because... - You are hijacking a top comment to insert your off-topic opinion. - You appear to be more territorial than presenting some sort of argument. - You are overly negative and somewhat dismissive/disrespectful. - You are commenting on downvotes, which is generally frowned upon

johnnycerberus 6y ago
My comment in on point, not off-topic. I had no intention to "hijack" the top comment, didn't even think about that until I've seen your comment. I don't see Rust as a replacement for Java and I also pointed out that Rust has to win the mindshare of C++ programmers first, before even reaching other ecosystems. Rust is mainly a replacement for C++.
- danudey 6y ago
  Your comment that Java and Rust don't overlap is valid, but going on to discuss how you're going to keep using C++ and Python, and how Unreal Engine uses C++, is wildly irrelevant. Rust isn't going to replace Java, but it may supplant Java in a lot of fields where one wants both memory safety and an OO language with high performance. No one is (rationally) suggesting that Rust will replace every language. The tone of your post, however, makes it sound as though you have something against Rust, and that you're more interested in pointing out all the things it's not good at, or shouldn't be used for, rather than contributing to a constructive discussion. > I also pointed out that Rust has to win the mindshare of C++ programmers first, before even reaching other ecosystems Not entirely true; Rust is an alternative to C++, but it's vastly more accessible to people who've used other languages like Java, Python, Ruby, and so on, where you get types, OO, and memory safety. Instead of simply resulting in a 1:1 replacement of C++ with Rust, it may result in a minimal reduction in C++ programming and a notable increase in Rust programming which would have been done in another safe OO language instead, but at dramatically worse performance.
  - chc 6y ago
    Small clarification — Rust very deliberately does not do OO. It has features that look sort of like OO from a syntactic standpoint, but being object oriented is not a design goal for Rust.

saagarjha 6y ago

> the browsers are all C++ You mean one is already shipping Rust and another seems to be on track to shipping it soon?

est31 6y ago
Chrome is "on track"? Last I remember they had a document explaining why it's not a good idea for them. I disagree with it, and would very much wish for Chrome to ship Rust, but they don't seem to be interested for now.
- TheDong 6y ago
  They're referring to this[0] I believe, which is not glowing, but does seem mildly optimistic. Specifically: > For now, Chrome investment in Rust will remain a background investigation (mostly directed towards prototyping these tools and techniques). If we become convinced this sort of interoperability is possible, we’ll revisit widespread use of Rust in Chrome [0]: https://www.chromium.org/Home/chromium-security/memory-safety/rust-and-c-interoperability https://www.chromium.org/Home/chromium-security/memory-safet...
  - MaxBarraclough 6y ago
    Right, and their issues there all pertain to interoperability, not to issues with the Rust language and its implementation. They seem to view Rust as production-ready, and a language that would bring relevant benefits to the Chromium project, were it not for the C++ interoperability difficulties.
  - fabrice_d 6y ago
    Looks like this will not happen soon: https://groups.google.com/a/chromium.org/g/blink-dev/c/gDA2SMPePv8 https://groups.google.com/a/chromium.org/g/blink-dev/c/gDA2S...
  - est31 6y ago
    Yes that's the same document I refer to as well. So nothing has changed. A "mildly optimistic" rejection is a rejection of Rust nonetheless. Rust is definitely not "on track" of being adopted in Chromium. For that you'd need more. Right now it's rather a "we are interested, maybe we'll add it". Also note that Chromium engineers also built some languages themselves that solve some of the same problems that Rust solves: https://github.com/google/wuffs https://github.com/google/wuffs

augusto-moura 6y ago

You are saying different things as one argument. Yes, Java applications and Rust mostly don't overlap, but there are exceptions in both ecosystems. Rust might offer some advantages for Data Engineers (the same way as CUDA applications in C++), not all of them as with every language else, and yes you can write Data centric libraries in Rust for Python (easier than C/C++ IIRC). And no, there's no need to rewrite existing services to Rust, it can interface with all the examples you cited. To me Rust is just a simpler and safer C++, if you said you would program C++, you probably will want to program Rust, writing C++ bindings for Rust is basic stuff nowadays

bbatha 6y ago
There are a lot of database-like services written Java (Kafka, Cassandra, ElasticSearch, Neo4J, HBase) that would likely be better in Rust. Yes the common denominator is Apache but they seem to have bought into Rust as well https://arrow.apache.org/blog/2019/02/04/datafusion-donation/ https://arrow.apache.org/blog/2019/02/04/datafusion-donation...
- StreamBright 6y ago
  DataFusion is a tiny project maintained by few people. I am really hopeful that it takes off though.

deleted 6y ago

[deleted]

throwaway894345 6y ago

Your comment seems to assume that your use case “big data engineer” is the only use case or the only Java use case and that the only context in which Rust can grow is to rewrite the existing tools that you use today (as opposed to greenfield projects like firecracker or bottle rocket). Your edit also writes Rust off as “idealistic” and not “pragmatic” without support, which is very far from the impression I have (based on my usage and seeing it used in real applications). In general, your comment seems like generic programming language flamebait.

johnnycerberus 6y ago
Firecracker and Bottlerocket are fine pieces of engineering. If Rust wants to get a share of the pie when it comes to data engineering, then it needs to have at least a killer app with an ecosystem around it. We can talk all day and all night for why some languages are more successful than others but it seems like ecosystems are giving them a boost. Hopefully, Rust will be the language for microVMs and kernel-level modules. I was strictly referring to people that think Rust would displace Java as idealists.
- throwaway894345 6y ago
  I generally agree--languages do better when they become the language of a particular ecosystem and that ecosystem becomes relevant. Go had this with containers and network software a bit more broadly. That said, I don't think anyone was suggesting Rust would displace Java. The OP was pretty clear IMO that his firm had experienced exponential growth in interest in Rust skills while he had been hoping that interest in Java would grow instead. This doesn't mean that Rust will replace Java for data engineering or any other domain in particular.
- mr_toad 6y ago
  > it needs to have at least a killer app with an ecosystem around it Give it 20 years and I can see it doing to Python what Python did to R.
  - abenga 6y ago

chc 6y ago

20 years ago, almost none of that stuff existed. It's entirely possible that whatever exists 20 years from now will all be Rust. I don't think that's likely, but I think it's more likely than the landscape you describe still existing in any recognizable form.

hundchenkatze 6y ago

The part you seem to be replying to doesn't imply that Rust is or should replace Java, at least not to me. > I always hoped Java would make a comeback, but Rust is looking like the real deal. To me it seems like SoSoRoCoCo is hoping their preferred language would make a comeback in popularity while also acknowledging that Rust is gaining popularity and real-world demand.

twic 6y ago

I write Java and Rust at work, and i more or less agree. More or less because there is some overlap: for example, people write low-latency trading systems in both. But it's very hard to get Java to perform well at the kind of work where Rust shines naturally. There is also very little overlap between Rust and Go. There's far more between Java and Go than Rust and either!

woah 6y ago

Being as experienced as you are, couldn't you just learn Rust like any other language if you need it in your work (or just to try it out). Your post has a weird negative vibe, like it would cost you greatly to try it.

basicneo 6y ago
The cost to learning Rust might be low but the opportunity cost might be high, since there's many other ways someone can improve their career.

sjtindell 6y ago

A good point and a long list of tools. But imo this is not the industry to say “good luck rewriting X” in. It is inevitable someone will come along to rewrite or write a new and improved tool for everything on that list. And it might be in Rust.

patrec 6y ago

Debatable. Java's dominance in the data engineering space rests a bit on clay feet: it's unsuitable for scientific computing in a number of ways: for example you can neither write fast nor clean numerical code in java and it does not interface naturally with the languages that people do write numerical code in. This is not an absolute deal breaker, but it adds friction. Also, serveral big data tools like cassandra (scylladb), kafka (redpanda) etc. have basically already been rewritten in C++ not least because they suffer from java-typical performance problems (even if those problems are not necessarily inherent in the choice of java). But C++ is not a very attractive language to write code in if there are viable alternatives, because it is objectively unpleasant and expensive to use (long compile times, terrible build and packaging eco-system, high cognitive load, difficulty of avoiding undefined behavior, etc.). Rust shares some of these defects and has some of its own, but it also offers substantial benefits for some types of problems that seem relevant to data engineering and crucially it interfaces more naturally with Fortran, C++ and Python than Java does. So I think it has a chance to gradually make inroads.

twsted 6y ago
> C++ is not a very attractive language to write code in if there are viable alternatives, because it is objectively unpleasant and expensive to use From a current perspective I may agree, but in the ‘90 a number of programmers like me was really excited to be able to develop at a much higher abstraction level than C using classes and, later, templates, preserving efficiency and performances.

tutfbhuf 6y ago

> databases (MySQL, MongoDB) We are heading towards distributed high available SQL and NoSQL databases (with Redis, Postgres and MySQL compatibility for easier migration) designed to work exceptionally well in cloud environments like CockroachDB (Go), TiKV (Rust), Vitess (Go) to name a few. That is because SRE teams want to be able to scale horizontally and make a node, rack or even region failure to be a normal thing, their company don't need to worry about.

StreamBright 6y ago

> As a big data engineer, Rust does not provide any advantage to me. Good luck rewritting all the big data tools from Java (Elasticsearch, Spark, Kafka, Hadoop, Neo4j, Deeplearning4j, Cassandra, Solr, Arrow, OrientDB) Yep, this is true. Java has an upper hand in data engineering. There are better languages for targeting the Java libraries and JVM than Java itself. I always use Clojure instead of Java when doing data crunching because it is much easier to create parallel programs with it. Kotlin is also coming up in this space. Having said that. I like Rust for some reasons: - targeting multiple libc (musl, no libc at all) - targeting multiple platforms (ppc, mips, etc., even though some of those are tierN) - having ML features Why I do not like Rust: - accidental complexity and convoluted syntax (maybe it is only me, but it is very hard to follow what is happening) For me the ideal language would be something like F# with the good parts of Rust.

caffeine 6y ago

I am a major fan of Rust. I upvoted your comment because it is mostly factual, and correct! I agree that Rust is still early for most adopters. There are a subset of people who need to solve problems that would normally be solved in C or C++, but they don't know or don't want to use those languages. Those people generally love Rust (I am in this camp). Then there is everyone else, who either like Rust because it's a cool language, or don't for any number of reasons all of which are fine!

mynameisash 6y ago

> As a big data engineer, Rust does not provide any advantage to me I assume that this is meant to imply that Rust doesn't provide advantages in the data engineering space. So I'll add in here -- as a data engineer myself -- that Rust is definitely providing an advantage for me. Combining it with Python (using PyO3 bindings) lets me write performant, safe code that interops with Python in a project for which Scala+Spark isn't feasible. So while Rust may not provide an advantage for you, that doesn't mean it doesn't have a potential niche for data engineering.

dmit 6y ago

> Applications of Java and Rust do not overlap. Both are general purpose languages, so of course they overlap. > Good luck rewritting all the big data tools from Java (Elasticsearch, Spark, Kafka, Hadoop, Neo4j, Deeplearning4j, Cassandra, Solr, Arrow, OrientDB) Cassandra (https://scylladb.com https://scylladb.com) and Kafka (https://vectorized.io https://vectorized.io) have already been rewritten once in C++, with massive latency and throughput improvements. No reason why they couldn't get their superior Rust clones in the observable future. Materialize (https://materialize.com https://materialize.com), Noria (https://github.com/mit-pdos/noria https://github.com/mit-pdos/noria), and Sled (https://github.com/spacejam/sled https://github.com/spacejam/sled) are just some of the Rust database projects that are aiming at unseating the de facto standard implementations in the space. InfluxDB (https://www.influxdata.com https://www.influxdata.com) is now doing major Rust development as well. The future is almost here. It's just not evenly distributed yet.

StreamBright 6y ago
> Cassandra (https://scylladb.com https://scylladb.com) and Kafka (https://vectorized.io https://vectorized.io) have already been rewritten once in C++, with massive latency and throughput improvements. It is not a single dimension problem as they try to frame it. Performance is a single dimension of systems. Java with GC is a memory safe solution compare to C++ (hello segmentation faults). The reason why the C++ implementation is faster is not that it is written in C++ but it is written using a different approach. There are plenty of Java HFT systems that run circles around C++ systems yet we cannot draw a conclusion that Java is better suited than C++ for HFT. Implementation details matter, code quality matters and performance does not come purely because you pick a particular programming language. Btw. most of the systems which I was working on using Cassandra instead of ScyllaDB because the companies were not ok switch up for some performance gains (because Cassandra was performant enough for their business use case). I think what you receive as superiority is not as clear cut as you try to make it.

oblio 6y ago

About microVMs, I'm not sure what this is referring to, but this literal post says that AWS' tech for VMs is now based on Rust :-)

secondcoming 6y ago

Cassandra has already been ported to C++ with Scylla. We switched to Scylla a year or so ago, haven't looked back. As for the others, if it can be shown that porting them comes with a decrease in latency and infrastructure costs then they'll be ported. I've often wondered why they haven't been already.

mr_toad 6y ago

Stuff written before Rust existed was not written in Rust? I’m shocked. That’s like going back to the 90’s and pointing to all the software not written in Java. And as for browsers, you know Mozilla created Rust specifically for Firefox?