5 ms·

Why Rust is taking the data engineering world by storm

33 points by zishha1b0njq 2y ago

deleted 2y ago
[deleted]
jph 2y ago
> All the most-impactful projects in the data engineering world are now written in Rust. Is this true? I love Rust and use it daily, yet I often hear from data engineering peers that Rust has too much friction for most data engineering projects, because of the borrow checker, lifetimes, compile times, inability to express functions and traits in some ways, async futures, and ABI/FFI issues with external data processing code. Happy to be corrected about all the above, though.
- b112 2y ago
  The "article" is just a book ad with a paragraph or two of contentious stuff to hook a buy. Don't read too much into it. (Pun intended)
- dijksterhuis 2y ago
  if you mentally rewrite the title to “lower level data systems” instead of “data engineering” i feel like it starts making a bit more sense. spark et al have massive continuous momentum. they’re not going anyway any time soon.
JBorrow 2y ago
Calling PostgreSQL a 'legacy C codebase' is funny.
- cgh 2y ago
  Was just about to type the same thing. It's hard to escape the feeling that this article belongs somewhere around the Peak of Inflated Expectations on the hype cycle chart.
- znpy 2y ago
  Indeed. The whole discourse around rust is getting more ideological than technical day after day. It's like some people are getting irreasonable intolerant to non-rust languages. It's... Puzzling, to say it politely.
daft_pink 2y ago
The real reason is that it’s not that hard for a python coder to switchover to rust when you tell them about the performance benefits and enhanced reliability.
airstrike 2y ago
> Rust has not only a bright future in the data engineering world, it's already the Looks like you accidentally a whole word there
- znpy 2y ago
  took me a bit to get the joke :)
  - airstrike 2y ago
    It's an older meme, sir, but it checks out ;-) https://knowyourmeme.com/memes/i-accidentally https://knowyourmeme.com/memes/i-accidentally
tayo42 2y ago
Save you a read It's memory safe and compatible with C code Then it concludes with a pitch to read the authors learn rust book for 35 dollars
- deleted 2y ago
  [deleted]
dijksterhuis 2y ago
eh, yes, and no. like, i see the authors point about backend data systems things like databases, query engines and storage. for the *low level* systems side of data engineering, yeah i see what you’re saying. for the higher level *take super hacky code from data scientists and turn it into a production grade pipeline* i just don’t see it right now. the thing is, right now at least, i would take the data science python code, clean it up then make it run via pyspark (or just multiprocessing). it’s still the fast happy path in my experience. this means i don’t have to rewrite the whole thing in rust from scratch, and the data science team can continue to contribute to it with a bit of hand holding (i have zero confidence in an average data scientist learning rust). also, i can run it on a single machine, or over a cluster when we start getting more data throughput. and i can set up autoscaling so i don’t have to pay attention when that’s happened. (caveat: unless there’s some super simple python->rust processing library that basically mimics pyspark that i probably don’t know about?) so, like, yeah, i see the point from the low level side of things. but at the higher level… i don’t think it’s there… yet. better title might have been “why rust is taking the low level data systems world by storm” addendum: i am actually learning rust at the moment (not for data stuff) and it has struck me as a decent language for data engineering pipeline work because of the emphasis on well defined structures (strong typing). is basically defining a concrete data model. i will check out the authors autumn of rust thingy blog post series at some point. might be helpful.
chrstr 2y ago
Well, Rust is great and everything... but calling DuckDB "the previous generation of data engineering" seems a bit funny. And if anything, DuckDB will probably have more impact than all the other mentioned tools (or maybe it even already has).
rwaksmunski 2y ago
I'm not sure about enterprise use cases, but for my project I was able to process 60TB of Common Crawl data with Rust at line rate on a budget server without any issues. Code was easy enough to write and it ran great on second try with resources to spare. I would pick Rust for big data projects again.
perrygeo 2y ago
I wrote a data ingest process, a simple web crawler that parsed and processed data from html pages and inserted to a database. This is a side project for me - no resources to maintain a k8s cluster or batch processing system or orchestration layer. Just a good old unix process running on a cheapo VPS. The process would periodically wake up and get to work, sleep, repeat. I used Python with polars for the first iteration. It worked but I was plagued by downtime, constantly hitting runtime exceptions that I hadn't considered. I rewrote it in Rust (also using polars) and it wasn't necessarily faster but it was rock solid. Every code path was explicitly handled and it would loop forever; there was literally no way to cause a panic. And it's been running for over 2 years without a single bug. Rust is the language if you have a well-defined data crunching problem to automate. It's not so great for exploratory work but for production code that needs to be rock solid, consider it. Being able to sleep at night without babysitting your data pipeline is nice!
- jochem9 2y ago
  This x10. When working with high variety data there are so many things that can go wrong (esp when working with external data sources), so you want a strongly typed compiled and memory safe language. I really believe that as data engineers we need to adopt a language like Rust. It helps a lot that it has polars, so we can keep working with the dataframe paradigm.
bjornnn 2y ago
Stop.

33 points by zishha1b0njq 2y ago

deleted 2y ago

[deleted]

jph 2y ago

> All the most-impactful projects in the data engineering world are now written in Rust. Is this true? I love Rust and use it daily, yet I often hear from data engineering peers that Rust has too much friction for most data engineering projects, because of the borrow checker, lifetimes, compile times, inability to express functions and traits in some ways, async futures, and ABI/FFI issues with external data processing code. Happy to be corrected about all the above, though.

b112 2y ago
The "article" is just a book ad with a paragraph or two of contentious stuff to hook a buy. Don't read too much into it. (Pun intended)
dijksterhuis 2y ago
if you mentally rewrite the title to “lower level data systems” instead of “data engineering” i feel like it starts making a bit more sense. spark et al have massive continuous momentum. they’re not going anyway any time soon.

JBorrow 2y ago

Calling PostgreSQL a 'legacy C codebase' is funny.

cgh 2y ago
Was just about to type the same thing. It's hard to escape the feeling that this article belongs somewhere around the Peak of Inflated Expectations on the hype cycle chart.
znpy 2y ago
Indeed. The whole discourse around rust is getting more ideological than technical day after day. It's like some people are getting irreasonable intolerant to non-rust languages. It's... Puzzling, to say it politely.

daft_pink 2y ago

The real reason is that it’s not that hard for a python coder to switchover to rust when you tell them about the performance benefits and enhanced reliability.

airstrike 2y ago

> Rust has not only a bright future in the data engineering world, it's already the Looks like you accidentally a whole word there

znpy 2y ago
took me a bit to get the joke :)
- airstrike 2y ago
  It's an older meme, sir, but it checks out ;-) https://knowyourmeme.com/memes/i-accidentally https://knowyourmeme.com/memes/i-accidentally

tayo42 2y ago

Save you a read It's memory safe and compatible with C code Then it concludes with a pitch to read the authors learn rust book for 35 dollars

deleted 2y ago
[deleted]

dijksterhuis 2y ago

eh, yes, and no. like, i see the authors point about backend data systems things like databases, query engines and storage. for the *low level* systems side of data engineering, yeah i see what you’re saying. for the higher level *take super hacky code from data scientists and turn it into a production grade pipeline* i just don’t see it right now. the thing is, right now at least, i would take the data science python code, clean it up then make it run via pyspark (or just multiprocessing). it’s still the fast happy path in my experience. this means i don’t have to rewrite the whole thing in rust from scratch, and the data science team can continue to contribute to it with a bit of hand holding (i have zero confidence in an average data scientist learning rust). also, i can run it on a single machine, or over a cluster when we start getting more data throughput. and i can set up autoscaling so i don’t have to pay attention when that’s happened. (caveat: unless there’s some super simple python->rust processing library that basically mimics pyspark that i probably don’t know about?) so, like, yeah, i see the point from the low level side of things. but at the higher level… i don’t think it’s there… yet. better title might have been “why rust is taking the low level data systems world by storm” addendum: i am actually learning rust at the moment (not for data stuff) and it has struck me as a decent language for data engineering pipeline work because of the emphasis on well defined structures (strong typing). is basically defining a concrete data model. i will check out the authors autumn of rust thingy blog post series at some point. might be helpful.

chrstr 2y ago

Well, Rust is great and everything... but calling DuckDB "the previous generation of data engineering" seems a bit funny. And if anything, DuckDB will probably have more impact than all the other mentioned tools (or maybe it even already has).

rwaksmunski 2y ago

I'm not sure about enterprise use cases, but for my project I was able to process 60TB of Common Crawl data with Rust at line rate on a budget server without any issues. Code was easy enough to write and it ran great on second try with resources to spare. I would pick Rust for big data projects again.

perrygeo 2y ago

I wrote a data ingest process, a simple web crawler that parsed and processed data from html pages and inserted to a database. This is a side project for me - no resources to maintain a k8s cluster or batch processing system or orchestration layer. Just a good old unix process running on a cheapo VPS. The process would periodically wake up and get to work, sleep, repeat. I used Python with polars for the first iteration. It worked but I was plagued by downtime, constantly hitting runtime exceptions that I hadn't considered. I rewrote it in Rust (also using polars) and it wasn't necessarily faster but it was rock solid. Every code path was explicitly handled and it would loop forever; there was literally no way to cause a panic. And it's been running for over 2 years without a single bug. Rust is the language if you have a well-defined data crunching problem to automate. It's not so great for exploratory work but for production code that needs to be rock solid, consider it. Being able to sleep at night without babysitting your data pipeline is nice!

jochem9 2y ago
This x10. When working with high variety data there are so many things that can go wrong (esp when working with external data sources), so you want a strongly typed compiled and memory safe language. I really believe that as data engineers we need to adopt a language like Rust. It helps a lot that it has polars, so we can keep working with the dataframe paradigm.

bjornnn 2y ago

Stop.