4 ms·
> Why would you create a special data type to represent a string split by Whitespace? Big chunks of Rust are build around iterators, and `split_whitespace` ret
by ekidd 9y ago
> Why would you create a special data type to represent a string split by Whitespace?
Big chunks of Rust are build around iterators, and `split_whitespace` returns an iterator of type `SplitWhitespace`. Because this is a concrete type, the Rust compiler and LLVM will then work to together to completely inline it, and they will generate code that looks like a hand-rolled loop.
There are some downsides to this system—it usually takes me about 10 minutes to write custom iterators for a new data structure—but iterators are very nice to program with and they go fast.
There's a new 'impl Trait' feature scheduled for later this year which will eliminate the need to export a custom struct like this. And it will eliminate the 10 minutes I spend writing iterators. Of course, it adds a new language feature. Nothing's free.
> Rust gives me a headache. I want to like it, but it just seems so... overengineered
I admit, Rust does sometimes have a "heavy industry" feeling to it. But this has some nice benefits, too:
1. I can write cross-platform CLI tools that Just Work on Linux, MacOS and Windows, because Rust has good abstractions for paths, files, threads, etc.
2. I can write multi-threaded code that does things like, "Read an arbitrary stream of bytes in a background thread, compress it, break it into 5MB chunks, and upload each of those chunks to S3 in parallel, using no more than N worker threads and applying backpressure, and do all this in the background while I work on something else." And thanks to Rust's threading rules, all this will work on the first try, with no nightmarish threading bugs.
3. In general, if my Rust code actually compiles, there's about an 85% chance that it will work flawlessly on the first try.
Personally, these are benefits that I'm willing to pay for. And Rust does require some familiarity both with how processors work, and with functional programming. And of course, everybody has different tradeoffs. But for certain kinds of work, Rust really hits the sweet spot.
- romwell 9y ago>and upload each of those chunks to S3 in parallel, using no more than N worker threads Do you use a crate like Rayon for this? I'm just starting with Rust, and my current understanding is that without using a library, one can only spawn threads and distribute load by hand (as opposed to automatic scheduling a-la OpenMP). Is this correct?
- ekidd 9y agoWe're hoping to open source our streaming S3 uploader. Behind the scenes, it uses the `crossbeam` crate (for scoped threads) and BurntSushi's `chan` crate (for bounded, blocking mpmc queues, which is how we get backpressure from a worker pool). One of the cool things about Rust is that it's possible for third-party crates to implement threading primitives. `rayon` is awesome, and it uses `crossbeam_dequeue` internally. But `rayon` is really best-suited to computational parallelism on many small pieces of data, and less suited to I/O parallelism. So it may be worth using `crossbeam` directly in that case. All the threads in our S3 uploader run on a single machine, because multi-part uploads are mostly limited by how much memory we want to use for buffering S3 object parts, not by CPU. (Of course, I'll probably overhaul a lot of this code later this year once `#[async]` and `futures` stabilize, so I can also stop paying for the memory used by thread stacks.) But the cool part is that we can already do streaming input, compression, chunking and parallel uploads without ever creating a temp file. (And we can upload multiple data streams using the same worker pool.) This took about three days to build.
- romwell 9y agoSounds great, thanks! Computational parallelism is what I'm after, so I'll look more into Crayon.