3 ms·
I’ve been writing data ETL pipelines in Rust and having a fantastic time. Most of my data, cost and performance requirements aren’t suited to the likes of Five
by FridgeSeal 4y ago
I’ve been writing data ETL pipelines in Rust and having a fantastic time.
Most of my data, cost and performance requirements aren’t suited to the likes of FiveTran/etc, and whilst I could theoretically use Python, it’s far too slow, and the propensity to crash unexpectedly (meaning I have to spend extra time debugging) means the development velocity is actually worse than just writing it in Rust. Plus, most of the libs and tooling I’d use from Python are available in Rust.
I’ve also written Rest and GRPC API’s in Rust (mostly for serving up data and being a query layer), and found it a far more pleasant experience than Python/Typescript/C#. Admittedly a somewhat ”smaller” use-case than a lot of what people would consider web-API’s but still.
- gregw2 4y agoHow much of your ETL is rust vs SQL? I agree pandas for ETL isn’t the fastest but why not isn’t SQL faster still? Or don’t you find yourself in the situation where there is so little logic in the ETL that rust performance/safety isn’t that useful?