We're hoping to open source our streaming S3 uploader. Behind the scenes, it uses the `crossbeam` crate (for scoped threads) and BurntSushi's `chan` crate (for

We're hoping to open source our streaming S3 uploader. Behind the scenes, it uses the `crossbeam` crate (for scoped threads) and BurntSushi's `chan` crate (for bounded, blocking mpmc queues, which is how we get backpressure from a worker pool). One of the cool things about Rust is that it's possible for third-party crates to implement threading primitives. `rayon` is awesome, and it uses `crossbeam_dequeue` internally. But `rayon` is really best-suited to computational parallelism on many small pieces of data, and less suited to I/O parallelism. So it may be worth using `crossbeam` directly in that case. All the threads in our S3 uploader run on a single machine, because multi-part uploads are mostly limited by how much memory we want to use for buffering S3 object parts, not by CPU. (Of course, I'll probably overhaul a lot of this code later this year once `#[async]` and `futures` stabilize, so I can also stop paying for the memory used by thread stacks.) But the cool part is that we can already do streaming input, compression, chunking and parallel uploads without ever creating a temp file. (And we can upload multiple data streams using the same worker pool.) This took about three days to build.