7 ms·

This is for people who want to live in a future where we use efficient and safe system level languages for massively scalable distributed data processing. IMHO,

This is for people who want to live in a future where we use efficient and safe system level languages for massively scalable distributed data processing. IMHO, Python and Java are not ideal language choices for these purposes. Rust offers much lower TCO compared to current industry best practices in this area.

geezerjay 7y ago
But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety. It's just meaningless marketting piggy-backing on popular buzzwords. And claims about "the future" are simply absurd. Oddly enough, this link appears right next to another story on how Cobol powers the world's economy. Frankly, I'm surprised blockchain wasn't shoved somewhere in the announcement.
- andygrove 7y ago
  Check out my previous benchmarks on Rust-based DataFusion vs JVM-based Apache Spark workloads. It isn't just about raw performance numbers but also about resource requirements (especially RAM requirements when using JVM or similar GC-based languages). There are order-of-magnitude differences.
  - geezerjay 7y ago
    If you have tangible results then present your benchmarks. If you limit your marketing to empty claims regarding "the future" and vague assertions on performance then you're actually actively working to lower your credibility.
    - andygrove 7y ago
      This is a personal open source project. I'm not sure I'm "marketing" it since I make zero dollars from this work. For benchmarks, check out the past 18 months of posts on my blog. Here is the most recent: https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/
      - geezerjay 7y ago
        From your blog post: > "The benchmarks are not 100% fair" I doubt anyone has time to rummage through 18 months of blog posts to look for data on a blog, specially as you're not willing to compile it and the one you cherry-pick is also euphemistically described in your own words as "not 100% fair". But hey, at least you have your marketing buzzwords on point.
        andygrove 7y ago
        Benchmarks are rarely fair. They are often biased in favor of those producing the benchmarks, whether intentionally or not. Whether you want to explore this more or not is your choice. Nobody is asking you to engage in this conversation. If you are not interested and/or are not enjoying the discourse here, feel free to move along.
        i4t 7y ago
        please, stop it.
    - eklavya 7y ago
      Don't you think you are being unreasonably harsh? Why are you attacking somebody's open source project with such passion?
      - andygrove 7y ago
        Thanks eklavya but it's completely normal on HN to have trolls criticizing other people's work. Best thing is to just ignore them. They can move on to criticize other posts. Much easier than contributing.
- andygrove 7y ago
  I admit I haven't benchmarked against COBOL blockchains.
- andygrove 7y ago
  "But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety" Except for the fact that it does in some cases. Rust is definitely safer than C or C++ (unless the Rust code is explicitly using the unsafe keyword).
  - geezerjay 7y ago
    The key fact that you're ignoring is that, as you implicitly acknowledge, "in some cases" it says absolutely nothing about efficiency or safety. Software isn't magic. Your software design practices, and consequentlg yout software, doesn't magically become safe and efficient just because you switch language/framework. The days where the myth that using language X bought you more speed are long gone, along with the credibility kf those who keep making those baseless claims.
    - andygrove 7y ago
      I'm sensing that you are not familiar with Rust and the safety that it introduces compared to C/C++.
    - hellofunk 7y ago
      Some things definitely do become magically safe when the language you are using tightly restricts you from doing very common and unsafe things!
hellofunk 7y ago
> Rust offers much lower TCO Just to satisfy my curiosity, since I’m not that familiar with distributed computing, how is tail call optimization relevant in this area?
- ohnoesjmr 7y ago
  It's total cost of ownership. Spark is a beast, half the resources is taken up by the platform itself.
  - matteuan 7y ago
    Really? Can you explain better "half the resources is taken up by the platform itself". Do you have some numbers, experiments?
    - ohnoesjmr 7y ago
      Running hadoop nodes, name nodes, yarn masters. These are massive complicated processes, that cumulatively are usually more complicated and expensive than the spark jobs themselves. At least from the cases I've seen.
      - liancheng 7y ago
        Actually, none of these processes is essential for running Spark unless you have to read data from HDFS and have to run Spark on YARN. Spark supports reading data from different sources, e.g., cloud blob storage, relational databases, NoSQL systems like C* and HBase. NameNode is only required if your data is stored on HDFS, and that is not an essential problem of Spark. As for scheduling, Spark can run in standalone mode without any YARN components. Actually, that is how Spark clusters run in Databricks.
m0zg 7y ago
I get that Rust is exciting, but you aren't really building this for engineers, right? When embarking on a non-trivial project, it's good to know who the prospective end user is. Helps shape the product requirements. Looking at your feature list for V1, it can easily take a team of 5-7 excellent engineers with strong, relevant domain expertise a couple of years to do most of the items on it well if your end user is "anybody". Some items on it (such as distributed query optimization) are open research problems. Some (distributed query execution, particularly if you want to support high performance multi-way joins) can take years all on their own. If it's something more specialized, it could take a lot less time and effort. I say this as someone who's "been there done that" (at Google). We had to support a broad range of use cases, but we also had some of the best engineers in the world, some with decades of experience in the problem domain, and it still took years to get the product (BigQuery) out the door. This was, to a large extent, because the requirements were so amorphous. First version of what later became BigQuery was built in a few months by a much smaller team, because it was targeted at a much more specialized use case: querying structured application logs in (almost) realtime. It's much easier to not try to be all things to all people.
- andygrove 7y ago
  Some great points there. I have multiple goals here. Goal #1 is to demonstrate how Rust can be effective at data science / analytics and evangelize a bit. Goal #2 is to have fun trying to build some of this myself and improve my skills as a software engineer. Goal #3 (getting a little more vague now) is to ensure that in some hypothetical future I can get a job working with Rust instead of JVM/Apache Spark.
  - m0zg 7y ago
    Got it. Fun side project then! Good luck. Rust does seem like a good fit for something like this. I think the hard-ass responses in the thread (all unwarranted) simply misunderstand the goals.
  - cauterize 7y ago
    FWIW, I'm keeping your name in the back of my head for when my current company inevitably fails at their big data revolution. Biggest product produces 10-15 PB/day that must be processed. Our existing solution is 20 years old and relies on navigating C code with function line counts on the orders of 10k. Rust would've saved me a lot of time from living in gdb. Really enjoyed the article. Keep it up!
kitd 7y ago
> Rust offers much lower TCO compared to current industry best practices in this area. The more commoditised the compute resources are, the more significant the cost of human expertise as part of TCO. Does Rust really offer an easier path for the coder to model data analytics? Eg, for a given model, is the time to implement x cost/hr for a Rust coder less than that of a Java one?
- fnord123 7y ago
  >Eg, for a given model, is the time to implement x cost/hr for a Rust coder less than that of a Java one? time_to_implement * cost/hr + cost_of_bug_damage * P(bug) + time_to_fix_bugs * cost/hr + cost_of_deployment_hw The dream of Rust is that P(bug) and cost_of_deployment_hw are less.
matteuan 7y ago
Assuming what you're saying is correct, then the best thing would be to port Apache Spark from Scala to Rust (or Go?).
- andygrove 7y ago
  That's kinda what I'm trying to do here. Ballista is definitely very much inspired by Apache Spark but not a direct port. Spark has some design choices that were very much driven from being a JVM-first platform e.g. the way lambdas are serialized. When I eventually get to support custom user code in distributed execution I don't want it to be limited to Rust, but that's a topic for another blog post later this year.
  - diroussel 7y ago
    Andy, I see Spark as essentially a monad that lifts functions into it’s execution context, then sechedules the execution on remote nodes, and gathers the results. How can you do that in an AOT language? Do you just express your functions at a higher level. Above the layer of containers?
    - blaenk 7y ago
      Consider Just-in-Time compilation which is very much used in AOT languages [0]. That could be one avenue, as long as the compiled code works on each machine (common instruction set), or maybe something slightly higher-level like LLVM IR can be produced and transmitted and then compiled on the driver/executor. Another approach could simply be to transmit the function source to the driver, compile it, then distribute it to the executors assuming they're the same architecture, or compiling it on executors if not. There are certainly options. [0]: For example see https://llvm.org/docs/tutorial/BuildingAJIT3.html https://llvm.org/docs/tutorial/BuildingAJIT3.html
- diroussel 7y ago
  It’s not possible to do a direct port. Since Spark relies on java serialisation to send executable code and data (closures) over the wire to workers. This relies on byte code. Rust is compiled ahead of time. So you can sort of see how the essence of spark can be ported. It’s essentially a monad. But a direct port to Rust is not possible because Rust uses AOT and the JVM was built to load code dynamically.
pjmlp 7y ago
Rust still has to do some work regarding language ergonomics. Currently I rather wait for Java to get its value types story properly done. For me an ergonomic future is a tracing GC, value types and manual memory management in hot code paths, like D, Eiffel and Modula-3 are capable of, improved with ownership rules.
- 0815test 7y ago
  Conventional tracing GC is antithetical to Rust's zero-cost goals. The community is currently working on things like deterministic collection of reference cycles, e.g. https://github.com/lopopolo/ferrocarril/blob/master/cactusref/README.md https://github.com/lopopolo/ferrocarril/blob/master/cactusre... which should address the same use cases perhaps even more effectively.
  - pjmlp 7y ago
    Right, but if productivity suffers, vs what tracing GC + value types + manual memory in unsafe code, are capable of, then Rust will mostly take over device drivers and kernel code, not userspace apps. Which is what is currently happening to C++ across all major OS SDKs, being driven down the OS stack.
    - twic 7y ago
      I hope so! The Rust leadership's quixotic quest to turn it into a language for the server-side may delay this, though.

geezerjay 7y ago

But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety. It's just meaningless marketting piggy-backing on popular buzzwords. And claims about "the future" are simply absurd. Oddly enough, this link appears right next to another story on how Cobol powers the world's economy. Frankly, I'm surprised blockchain wasn't shoved somewhere in the announcement.

andygrove 7y ago
Check out my previous benchmarks on Rust-based DataFusion vs JVM-based Apache Spark workloads. It isn't just about raw performance numbers but also about resource requirements (especially RAM requirements when using JVM or similar GC-based languages). There are order-of-magnitude differences.
- geezerjay 7y ago
  If you have tangible results then present your benchmarks. If you limit your marketing to empty claims regarding "the future" and vague assertions on performance then you're actually actively working to lower your credibility.
  - andygrove 7y ago
    This is a personal open source project. I'm not sure I'm "marketing" it since I make zero dollars from this work. For benchmarks, check out the past 18 months of posts on my blog. Here is the most recent: https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/ https://andygrove.io/2019/04/datafusion-0.13.0-benchmarks/
    - geezerjay 7y ago
      From your blog post: > "The benchmarks are not 100% fair" I doubt anyone has time to rummage through 18 months of blog posts to look for data on a blog, specially as you're not willing to compile it and the one you cherry-pick is also euphemistically described in your own words as "not 100% fair". But hey, at least you have your marketing buzzwords on point.
      - andygrove 7y ago
        Benchmarks are rarely fair. They are often biased in favor of those producing the benchmarks, whether intentionally or not. Whether you want to explore this more or not is your choice. Nobody is asking you to engage in this conversation. If you are not interested and/or are not enjoying the discourse here, feel free to move along.
      - i4t 7y ago
        please, stop it.
  - eklavya 7y ago
    Don't you think you are being unreasonably harsh? Why are you attacking somebody's open source project with such passion?
    - andygrove 7y ago
      Thanks eklavya but it's completely normal on HN to have trolls criticizing other people's work. Best thing is to just ignore them. They can move on to criticize other posts. Much easier than contributing.
andygrove 7y ago
I admit I haven't benchmarked against COBOL blockchains.
andygrove 7y ago
"But claiming that "X software is written in Y language/framework" says nothing about efficiency or safety" Except for the fact that it does in some cases. Rust is definitely safer than C or C++ (unless the Rust code is explicitly using the unsafe keyword).
- geezerjay 7y ago
  The key fact that you're ignoring is that, as you implicitly acknowledge, "in some cases" it says absolutely nothing about efficiency or safety. Software isn't magic. Your software design practices, and consequentlg yout software, doesn't magically become safe and efficient just because you switch language/framework. The days where the myth that using language X bought you more speed are long gone, along with the credibility kf those who keep making those baseless claims.
  - andygrove 7y ago
    I'm sensing that you are not familiar with Rust and the safety that it introduces compared to C/C++.
  - hellofunk 7y ago
    Some things definitely do become magically safe when the language you are using tightly restricts you from doing very common and unsafe things!

hellofunk 7y ago

> Rust offers much lower TCO Just to satisfy my curiosity, since I’m not that familiar with distributed computing, how is tail call optimization relevant in this area?

ohnoesjmr 7y ago
It's total cost of ownership. Spark is a beast, half the resources is taken up by the platform itself.
- matteuan 7y ago
  Really? Can you explain better "half the resources is taken up by the platform itself". Do you have some numbers, experiments?
  - ohnoesjmr 7y ago
    Running hadoop nodes, name nodes, yarn masters. These are massive complicated processes, that cumulatively are usually more complicated and expensive than the spark jobs themselves. At least from the cases I've seen.
    - liancheng 7y ago
      Actually, none of these processes is essential for running Spark unless you have to read data from HDFS and have to run Spark on YARN. Spark supports reading data from different sources, e.g., cloud blob storage, relational databases, NoSQL systems like C* and HBase. NameNode is only required if your data is stored on HDFS, and that is not an essential problem of Spark. As for scheduling, Spark can run in standalone mode without any YARN components. Actually, that is how Spark clusters run in Databricks.

m0zg 7y ago

I get that Rust is exciting, but you aren't really building this for engineers, right? When embarking on a non-trivial project, it's good to know who the prospective end user is. Helps shape the product requirements. Looking at your feature list for V1, it can easily take a team of 5-7 excellent engineers with strong, relevant domain expertise a couple of years to do most of the items on it well if your end user is "anybody". Some items on it (such as distributed query optimization) are open research problems. Some (distributed query execution, particularly if you want to support high performance multi-way joins) can take years all on their own. If it's something more specialized, it could take a lot less time and effort. I say this as someone who's "been there done that" (at Google). We had to support a broad range of use cases, but we also had some of the best engineers in the world, some with decades of experience in the problem domain, and it still took years to get the product (BigQuery) out the door. This was, to a large extent, because the requirements were so amorphous. First version of what later became BigQuery was built in a few months by a much smaller team, because it was targeted at a much more specialized use case: querying structured application logs in (almost) realtime. It's much easier to not try to be all things to all people.

andygrove 7y ago
Some great points there. I have multiple goals here. Goal #1 is to demonstrate how Rust can be effective at data science / analytics and evangelize a bit. Goal #2 is to have fun trying to build some of this myself and improve my skills as a software engineer. Goal #3 (getting a little more vague now) is to ensure that in some hypothetical future I can get a job working with Rust instead of JVM/Apache Spark.
- m0zg 7y ago
  Got it. Fun side project then! Good luck. Rust does seem like a good fit for something like this. I think the hard-ass responses in the thread (all unwarranted) simply misunderstand the goals.
- cauterize 7y ago
  FWIW, I'm keeping your name in the back of my head for when my current company inevitably fails at their big data revolution. Biggest product produces 10-15 PB/day that must be processed. Our existing solution is 20 years old and relies on navigating C code with function line counts on the orders of 10k. Rust would've saved me a lot of time from living in gdb. Really enjoyed the article. Keep it up!

kitd 7y ago

> Rust offers much lower TCO compared to current industry best practices in this area. The more commoditised the compute resources are, the more significant the cost of human expertise as part of TCO. Does Rust really offer an easier path for the coder to model data analytics? Eg, for a given model, is the time to implement x cost/hr for a Rust coder less than that of a Java one?

fnord123 7y ago
>Eg, for a given model, is the time to implement x cost/hr for a Rust coder less than that of a Java one? time_to_implement * cost/hr + cost_of_bug_damage * P(bug) + time_to_fix_bugs * cost/hr + cost_of_deployment_hw The dream of Rust is that P(bug) and cost_of_deployment_hw are less.

matteuan 7y ago

Assuming what you're saying is correct, then the best thing would be to port Apache Spark from Scala to Rust (or Go?).

andygrove 7y ago
That's kinda what I'm trying to do here. Ballista is definitely very much inspired by Apache Spark but not a direct port. Spark has some design choices that were very much driven from being a JVM-first platform e.g. the way lambdas are serialized. When I eventually get to support custom user code in distributed execution I don't want it to be limited to Rust, but that's a topic for another blog post later this year.
- diroussel 7y ago
  Andy, I see Spark as essentially a monad that lifts functions into it’s execution context, then sechedules the execution on remote nodes, and gathers the results. How can you do that in an AOT language? Do you just express your functions at a higher level. Above the layer of containers?
  - blaenk 7y ago
    Consider Just-in-Time compilation which is very much used in AOT languages [0]. That could be one avenue, as long as the compiled code works on each machine (common instruction set), or maybe something slightly higher-level like LLVM IR can be produced and transmitted and then compiled on the driver/executor. Another approach could simply be to transmit the function source to the driver, compile it, then distribute it to the executors assuming they're the same architecture, or compiling it on executors if not. There are certainly options. [0]: For example see https://llvm.org/docs/tutorial/BuildingAJIT3.html https://llvm.org/docs/tutorial/BuildingAJIT3.html
diroussel 7y ago
It’s not possible to do a direct port. Since Spark relies on java serialisation to send executable code and data (closures) over the wire to workers. This relies on byte code. Rust is compiled ahead of time. So you can sort of see how the essence of spark can be ported. It’s essentially a monad. But a direct port to Rust is not possible because Rust uses AOT and the JVM was built to load code dynamically.

pjmlp 7y ago

Rust still has to do some work regarding language ergonomics. Currently I rather wait for Java to get its value types story properly done. For me an ergonomic future is a tracing GC, value types and manual memory management in hot code paths, like D, Eiffel and Modula-3 are capable of, improved with ownership rules.

0815test 7y ago
Conventional tracing GC is antithetical to Rust's zero-cost goals. The community is currently working on things like deterministic collection of reference cycles, e.g. https://github.com/lopopolo/ferrocarril/blob/master/cactusref/README.md https://github.com/lopopolo/ferrocarril/blob/master/cactusre... which should address the same use cases perhaps even more effectively.
- pjmlp 7y ago
  Right, but if productivity suffers, vs what tracing GC + value types + manual memory in unsafe code, are capable of, then Rust will mostly take over device drivers and kernel code, not userspace apps. Which is what is currently happening to C++ across all major OS SDKs, being driven down the OS stack.
  - twic 7y ago
    I hope so! The Rust leadership's quixotic quest to turn it into a language for the server-side may delay this, though.