5 ms·

Fivefold Slower Compared to Go? Optimizing Rust's Protobuf Decoding Performance

85 points by signa11 3y ago

KAdot 3y ago
The benchmark is not comparing apples to apples. prost is the most widely used Protobuf implementation in Rust, maintained by the Tokio organization. prost generates structs and serialization/deserialization code for you. easyproto according to GitHib Search is used only by two projects. easyproto provides primitives for serializing and deserializing Protobuf, and requires hand writing code to do both. A fair comparison would be prost vs google.golang.org/protobuf, or easyproto vs parts of quick-protobuf. In most cases you can make Go as fast as Rust, but from my experience writing performance-sensitive code in Go requires significantly larger time investment and overall requires deeper language expertise. Pebble (RocksDB replacement in Go by CockroachDB) is a good example of this, the codebase is littered with hand-inlined[1] functions, hand-unrolled loops and it's not[2] even using Go memory management for performance critical parts, it's using the C memory allocator and manual memory management. [prost]: https://github.com/tokio-rs/prost https://github.com/tokio-rs/prost [easyproto]: https://github.com/VictoriaMetrics/easyproto https://github.com/VictoriaMetrics/easyproto [google.golang.org/protobuf]: https://github.com/protocolbuffers/protobuf-go https://github.com/protocolbuffers/protobuf-go [quick-protobuf]: https://github.com/tafia/quick-protobuf https://github.com/tafia/quick-protobuf [1]: https://github.com/cockroachdb/pebble/blob/c34894c46703fd823ce06e994c92a8b71571e2ce/sstable/block_iter.go#L287-L350 https://github.com/cockroachdb/pebble/blob/c34894c46703fd823... [2]: https://github.com/cockroachdb/pebble/blob/master/docs/memory.md#manual-memory-management https://github.com/cockroachdb/pebble/blob/master/docs/memor...
netprole 3y ago
In the image at the top of the article, why is the Rust crab altered to have "angry" eyes and holding a knife aimed at the Go gopher? Aside from the joke of "don't bring a knife to a protobuf fight" the inference of violence sucks and lessens the spirit of friendly competition and "all in good fun." I don't know if Rust has a code-of-conduct or rules for use of their mascot, but I bet this doesn't follow it.
- steveklabnik 3y ago
  Ferrous (the crab) is deliberately not an official mascot, and is in the public domain. There is no code of conduct or rules.
- reubenmorais 3y ago
  It's the logo of CrabLang: https://crablang.org/ https://crablang.org/
  - netprole 3y ago
    Thanks for that, good to know
- signa11 3y ago
  aren’t you reading a bit too much into it?
  - netprole 3y ago
    Maybe you're not thinking about it enough? Do you know anyone who has been almost fatally stabbed (attacked by gangs) with knives? I do! It likely violates a code-of-conduct and is unprofessional.
    - reacharavindh 3y ago
      Isn’t it tiring to think about every possible offence in the world and how someone somewhere will be offended by (knives, tires, cars, pens, saws, sharks, planes, needles, ropes, speakers, games, … you get the point) while writing an article about a programming language? At some point one needs to exercise common sense and learn to live in a public society where people speak and not everyone is out to offend you.
      - netprole 3y ago
        What does a weapon (a knife) have to do with an article about a programming language?
secondcoming 3y ago
> In Go, a string is just a simple wrapper around []byte, and deserializing a string field can be done by simply assigning the original buffer's pointer and length to the string field. However, Rust's PROST, when deserializing String type fields, needs to copy the data from the original buffer into the String It's interesting that the protobuf code generator for Go seems to allow this direct access. For C++, you also need a copy (and potentially a heap allocation) since `string` fields are returned via `const std::string&`. Protobuf support for `std::string_view` has been years in the making. Maybe just switch to FlatBuffers?
- v0y4g3r 3y ago
  FlatBuffers does the trick as a better replacement to ProtoBuf when running in some resource-critical devices. We were working on a project that leverage Arrow IPC (internally FlatBuffers) and shared memory to collect metrics on edge devices with limit CPU and memory, hopefully we can open source it soon. But in this case the format is defined by [Prometheus](https://github.com/prometheus/prometheus/blob/main/prompb/remote.proto#L22-L28 https://github.com/prometheus/prometheus/blob/main/prompb/re...) and we just adopt that.
MrBuddyCasino 3y ago
So in this case, with some modest performance engineering, Golang is surprisingly fast out of the box, and Rust requires more effort and increasingly less idiomatic code to reach the same result. Is this typical?
- wokwokwok 3y ago
  While it’s a nice bed time story if you like go, the reality is no, it’s not typical. Go has a good perf story, but typically rust or c++ would be faster after heavy optimisation; and should be more or less on par with typical applications. This isn’t a critique of go, and shouldn’t surprise anyone. Typically go also has unexpected optimisation hoops to jump through and problems related to the heavy use of channels (see the well documented answer here: https://stackoverflow.com/questions/47312029/when-should-you-use-a-mutex-over-a-channel https://stackoverflow.com/questions/47312029/when-should-you...), so you would generally expect it to be slower… …but, naive implementations are always slower, and really, it’s probably much of a muchness out the box for most day to day uses. In almost all situations (even python or Java) you can get great performance if you invest time and effort in it. But idiomatic code typically faster than rust? No, not really.
  - riku_iki 3y ago
    > but typically rust or c++ would be faster after heavy optimisation; it is concerning how much digging was required to optimize rust code in this case.
    - fredrikholm 3y ago
      Whilst still being 1/3rd slower than the Go version.
      - Filligree 3y ago
        They could have used lifetimes. Instead they used elaborate workarounds, which turned out to be slower.
      - 3y ago
sebstefan 3y ago
I got confused through it ; is the resulting code still technically safe?
- v0y4g3r 3y ago
  It's technically `safe` as long as you never access the decoded struct once the original bytes is dropped. I believe that how `unsafe` works, the programmer, instead of compiler, ensures safety.
  - pflanze 3y ago
    Yes, but their code appears to actually be `unsafe` (in the Rust terminology sense) without specifying that in their function declarations. They use `unsafe` inside their `slice` function, but return a value that is unsafe to use, hence `slice` should be marked `unsafe`, as should `copy_to_bytes` and then `merge_bytes`. Same for PromLabel::merge_field and PromTimeSeries::merge_field as far as I can see, and maybe higher up in their actual app. This is definitely not how Rust code is supposed to work, if a function isn't marked unsafe, it should not be allowed to introduce UB; they violate that. This approach is on par security wise with C/C++ code iff programmers are aware of the pitfalls, which normally isn't the case since Rust programmers expect non-unsafe functions to be safe (i.e. not require additional care to avoid undefined behaviour). They either need to mark their functions `unsafe`, or use lifetimes (which may require changes in some APIs, which may be the reason they didn't).
    - pflanze 3y ago
      I was looking at the main branch, and described the situation there. They have a different branch for the optimization work; in that branch, they do mark those functions as `unsafe` (and already did when I posted).
K0nserv 3y ago
About step 2: You still need to drop items, that's required for Vec<T> to be sound. By not calling `Vec::clear` and instead using pub struct RepeatedField<T> { vec: Vec<T>, len: usize, } with indexing patterns like `self.vec[..self.len]` and setting `self.len = 0` to clear you avoid the cost of dropping all items at once(like Vec::clear does). However you still have to drop items, so with this solution the cost is amortised in `RepeatedField::push` and other methods that do `self.vec[i] = new_item`. > It's designed to avoid the drop overhead Isn't true. You don't avoid the the overhead, at most you delay/amortise it.
- v0y4g3r 3y ago
  WriteRequest::timeseries is a vector (https://github.com/prometheus/prometheus/blob/main/prompb/remote.proto#L23 https://github.com/prometheus/prometheus/blob/main/prompb/re...) and the repeated file `Timeseries::labels` and `Timeseries::samples` are reused across different timeseries. You don't have to alloc a new vector for the lables and samples for each new timeseries instance.
  - K0nserv 3y ago
    That would be true if you used `Vec::clear` too, it doesn't allocate a new vector. My point was that you still end up running Drop implementations with RepeatedField<T>, just not all at once. See https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=75621151b0a3e81da16741a24f8f9e96 https://play.rust-lang.org/?version=stable&mode=debug&editio...

85 points by signa11 3y ago

The benchmark is not comparing apples to apples. prost is the most widely used Protobuf implementation in Rust, maintained by the Tokio organization. prost generates structs and serialization/deserialization code for you. easyproto according to GitHib Search is used only by two projects. easyproto provides primitives for serializing and deserializing Protobuf, and requires hand writing code to do both. A fair comparison would be prost vs google.golang.org/protobuf, or easyproto vs parts of quick-protobuf. In most cases you can make Go as fast as Rust, but from my experience writing performance-sensitive code in Go requires significantly larger time investment and overall requires deeper language expertise. Pebble (RocksDB replacement in Go by CockroachDB) is a good example of this, the codebase is littered with hand-inlined[1] functions, hand-unrolled loops and it's not[2] even using Go memory management for performance critical parts, it's using the C memory allocator and manual memory management. [prost]: https://github.com/tokio-rs/prost https://github.com/tokio-rs/prost [easyproto]: https://github.com/VictoriaMetrics/easyproto https://github.com/VictoriaMetrics/easyproto [google.golang.org/protobuf]: https://github.com/protocolbuffers/protobuf-go https://github.com/protocolbuffers/protobuf-go [quick-protobuf]: https://github.com/tafia/quick-protobuf https://github.com/tafia/quick-protobuf [1]: https://github.com/cockroachdb/pebble/blob/c34894c46703fd823ce06e994c92a8b71571e2ce/sstable/block_iter.go#L287-L350 https://github.com/cockroachdb/pebble/blob/c34894c46703fd823... [2]: https://github.com/cockroachdb/pebble/blob/master/docs/memory.md#manual-memory-management https://github.com/cockroachdb/pebble/blob/master/docs/memor...

netprole 3y ago

In the image at the top of the article, why is the Rust crab altered to have "angry" eyes and holding a knife aimed at the Go gopher? Aside from the joke of "don't bring a knife to a protobuf fight" the inference of violence sucks and lessens the spirit of friendly competition and "all in good fun." I don't know if Rust has a code-of-conduct or rules for use of their mascot, but I bet this doesn't follow it.

steveklabnik 3y ago
Ferrous (the crab) is deliberately not an official mascot, and is in the public domain. There is no code of conduct or rules.
reubenmorais 3y ago
It's the logo of CrabLang: https://crablang.org/ https://crablang.org/
- netprole 3y ago
  Thanks for that, good to know
signa11 3y ago
aren’t you reading a bit too much into it?
- netprole 3y ago
  Maybe you're not thinking about it enough? Do you know anyone who has been almost fatally stabbed (attacked by gangs) with knives? I do! It likely violates a code-of-conduct and is unprofessional.
  - reacharavindh 3y ago
    Isn’t it tiring to think about every possible offence in the world and how someone somewhere will be offended by (knives, tires, cars, pens, saws, sharks, planes, needles, ropes, speakers, games, … you get the point) while writing an article about a programming language? At some point one needs to exercise common sense and learn to live in a public society where people speak and not everyone is out to offend you.
    - netprole 3y ago
      What does a weapon (a knife) have to do with an article about a programming language?

secondcoming 3y ago

> In Go, a string is just a simple wrapper around []byte, and deserializing a string field can be done by simply assigning the original buffer's pointer and length to the string field. However, Rust's PROST, when deserializing String type fields, needs to copy the data from the original buffer into the String It's interesting that the protobuf code generator for Go seems to allow this direct access. For C++, you also need a copy (and potentially a heap allocation) since `string` fields are returned via `const std::string&`. Protobuf support for `std::string_view` has been years in the making. Maybe just switch to FlatBuffers?

v0y4g3r 3y ago
FlatBuffers does the trick as a better replacement to ProtoBuf when running in some resource-critical devices. We were working on a project that leverage Arrow IPC (internally FlatBuffers) and shared memory to collect metrics on edge devices with limit CPU and memory, hopefully we can open source it soon. But in this case the format is defined by [Prometheus](https://github.com/prometheus/prometheus/blob/main/prompb/remote.proto#L22-L28 https://github.com/prometheus/prometheus/blob/main/prompb/re...) and we just adopt that.

MrBuddyCasino 3y ago

So in this case, with some modest performance engineering, Golang is surprisingly fast out of the box, and Rust requires more effort and increasingly less idiomatic code to reach the same result. Is this typical?

wokwokwok 3y ago
While it’s a nice bed time story if you like go, the reality is no, it’s not typical. Go has a good perf story, but typically rust or c++ would be faster after heavy optimisation; and should be more or less on par with typical applications. This isn’t a critique of go, and shouldn’t surprise anyone. Typically go also has unexpected optimisation hoops to jump through and problems related to the heavy use of channels (see the well documented answer here: https://stackoverflow.com/questions/47312029/when-should-you-use-a-mutex-over-a-channel https://stackoverflow.com/questions/47312029/when-should-you...), so you would generally expect it to be slower… …but, naive implementations are always slower, and really, it’s probably much of a muchness out the box for most day to day uses. In almost all situations (even python or Java) you can get great performance if you invest time and effort in it. But idiomatic code typically faster than rust? No, not really.
- riku_iki 3y ago
  > but typically rust or c++ would be faster after heavy optimisation; it is concerning how much digging was required to optimize rust code in this case.
  - fredrikholm 3y ago
    Whilst still being 1/3rd slower than the Go version.
    - Filligree 3y ago
      They could have used lifetimes. Instead they used elaborate workarounds, which turned out to be slower.
    - 3y ago

sebstefan 3y ago

I got confused through it ; is the resulting code still technically safe?

v0y4g3r 3y ago
It's technically `safe` as long as you never access the decoded struct once the original bytes is dropped. I believe that how `unsafe` works, the programmer, instead of compiler, ensures safety.
- pflanze 3y ago
  Yes, but their code appears to actually be `unsafe` (in the Rust terminology sense) without specifying that in their function declarations. They use `unsafe` inside their `slice` function, but return a value that is unsafe to use, hence `slice` should be marked `unsafe`, as should `copy_to_bytes` and then `merge_bytes`. Same for PromLabel::merge_field and PromTimeSeries::merge_field as far as I can see, and maybe higher up in their actual app. This is definitely not how Rust code is supposed to work, if a function isn't marked unsafe, it should not be allowed to introduce UB; they violate that. This approach is on par security wise with C/C++ code iff programmers are aware of the pitfalls, which normally isn't the case since Rust programmers expect non-unsafe functions to be safe (i.e. not require additional care to avoid undefined behaviour). They either need to mark their functions `unsafe`, or use lifetimes (which may require changes in some APIs, which may be the reason they didn't).
  - pflanze 3y ago
    I was looking at the main branch, and described the situation there. They have a different branch for the optimization work; in that branch, they do mark those functions as `unsafe` (and already did when I posted).

K0nserv 3y ago

About step 2: You still need to drop items, that's required for Vec<T> to be sound. By not calling `Vec::clear` and instead using pub struct RepeatedField<T> { vec: Vec<T>, len: usize, } with indexing patterns like `self.vec[..self.len]` and setting `self.len = 0` to clear you avoid the cost of dropping all items at once(like Vec::clear does). However you still have to drop items, so with this solution the cost is amortised in `RepeatedField::push` and other methods that do `self.vec[i] = new_item`. > It's designed to avoid the drop overhead Isn't true. You don't avoid the the overhead, at most you delay/amortise it.

v0y4g3r 3y ago
WriteRequest::timeseries is a vector (https://github.com/prometheus/prometheus/blob/main/prompb/remote.proto#L23 https://github.com/prometheus/prometheus/blob/main/prompb/re...) and the repeated file `Timeseries::labels` and `Timeseries::samples` are reused across different timeseries. You don't have to alloc a new vector for the lables and samples for each new timeseries instance.
- K0nserv 3y ago
  That would be true if you used `Vec::clear` too, it doesn't allocate a new vector. My point was that you still end up running Drop implementations with RepeatedField<T>, just not all at once. See https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=75621151b0a3e81da16741a24f8f9e96 https://play.rust-lang.org/?version=stable&mode=debug&editio...