10 ms·

Rewrite the VP9 codec library in Rust

200 points by marcodiego 3y ago

vladimirralev 3y ago
The VP9 codec I imagine would heavily benefit from SIMD? Just SIMD should massively outperform any advantage gained by putting this into the kernel. Why does the kernel need an unoptimized codec?
- ZeroCool2u 3y ago
  SIMD can be pretty architecture specific. For example, does the CPU support AVX-512 or SSE3? So, you have to have a few code paths if you're going to support a wide variety of hardware. I don't have an answer to your question it just occurs to me that maybe the Linux kernel doesn't allow a lot of SIMD for this reason or maybe they require a fall back/slow code path to be available if you're submitting a patch that includes SIMD operations? Curious if anyone knows the answer to this.
  - __s 3y ago
    Kernel avoids SIMD because it doesn't have a kernel keeping its registers coherent https://stackoverflow.com/a/46677815 https://stackoverflow.com/a/46677815
    - deleted 3y ago
      [deleted]
    - ZeroCool2u 3y ago
      Thank you for the answer! Very interesting.
  - brundolf 3y ago
    Fwiw, Rust's standard library has a cross-platform Simd abstraction: https://doc.rust-lang.org/std/simd/index.html https://doc.rust-lang.org/std/simd/index.html It's nightly-only for now, but I've used it and it's lovely. I've added Simd to projects that I never otherwise would have, just because this made it so easy and accessible
    - ZeroCool2u 3y ago
      That is metal af ;)
- pavon 3y ago
  I don't think that the driver implements VP9 in software. It uses hardware acceleration to perform the actual encode/decode, but is managing preparing of coefficients and DMA of data buffers in the driver.
briantakita 3y ago
[flagged]
- megous 3y ago
  No. It's all C code in the stacktrace. And kernel devs still don't have to deal with Rust, if they don't want to.
  - briantakita 3y ago
    > No. It's all C code in the stacktrace. And kernel devs still don't have to deal with Rust, if they don't want to. Thank you for the answer. I was wondering since the conversion to Rust is a large undertaking & rewrites can have surprises. Part of the question was if this was an argument to use Rust since NULL pointers are something that Rust goes through great lengths to guard against. And wow...my original post got flagged? It was a sincere question that I was curious about & didn't have an answer for. Either everyone needs to chill...or is mentioning NULL pointer references in the Linux Kernel some sort of unspoken mortal sin or was I somehow offensive? Please, someone let me know if I'm out of line cause I just see zealotry of some type that I cannot begin to imagine. I want to avoid such land mines in the future...for the love of all that is holy.
    - nextaccountic 3y ago
      > I was wondering since the conversion to Rust is a large undertaking & rewrites can have surprises. There is no concerted effort to rewrite Linux in Rust, so there is no large undertaking. In spite of this VP9 rewrite, things are not being rewritten in Rust for the most part. Some individuals are rewriting some things but more importantly, some new stuff (such as drivers) are being written in Rust. It's this new stuff written in Rust (and not previously written in C) that is the focus IMO. And most C stuff isn't going to be rewritten. Many maintainers don't know Rust and couldn't review Rust code anyway. The impact of Rust in the kernel so far is pretty modest (the most important code being the driver for Apple silicon GPUs, I think) > And wow...my original post got flagged? It was a sincere question that I was curious about & didn't have an answer for. Either everyone needs to chill...or is mentioning NULL pointer references in the Linux Kernel some sort of unspoken mortal sin or was I somehow offensive? I didn't flag your post, but I would guess it was flagged because you immediately assumed that Rust was to blame.
renewiltord 3y ago
Interesting. Use case is so that there is some code exercising this path. While the algorithm is cool, of course, the code itself is quite straightforward as it is. An interesting thing I didn’t know is that the kernel code avoids recursion (this one has a depth parameter to prevent recursion past some point). I can see why this was picked as a candidate. Straightforward implementation. Good test suite. Self-contained and not a moving target. Lots of the code is using the coefficients array against the framecontext but I don’t know how to enforce the bounds invariant that the two are the same. For the fixed size arrays I could see how it’s done, but otherwise it seems like the bounds checker will trigger. But I’m reading on my phone so maybe that’s just a misread. Not a big performance hit even if it does. But perhaps it doesn’t, and I’d be curious why.
revskill 3y ago
I think the inferior of Rust vs C could be "over checking" ?
lionkor 3y ago
> These algorithms use the data received from userspace in order to index into a lot of arrays and thus benefit from Rust's memory safety. Yeah, thats not a good argument for Rust specifically. Maybe Zig would be interesting for array-specific things like sentinel-terminated arrays ans their amazing C interop. Further, Zig code is unmatched in how explicit everything is. There is no abstraction hiding complexity, its built on the idea of custom allocators from the ground up. This seems like someone looking for a usecase for Rust over C, rather than someone looking for a better solution altogether. Thats okay, I guess, but Rust does not do array bounds checking very well and the borrowchecker doesnt help there at all. The large amount of hidden complexity in a lot of the language dont make it more safe by any means.
dureuill 3y ago
> Much has been spoken at various occasions about drivers and I feel that the consensus is to wait for now. Interesting, I did not follow that development. I thought the plan was to use Rust for some out-of-tree/optional drivers. What changed?
- Macha 3y ago
  There was opposition to building interfaces for toy drivers and the last thread had suggestions at a rust filesystem interface rebuffed by saying they should try rewrite the ext2 driver in rust to prove that it was usable for real filesystems rather than toy ones. I'd guess similar thought processes fuelled this decision.
- charcircuit 3y ago
  It's already used for an intree driver, but a lot of the infrastructure for making Rust drivers was not be upstreamed.
rust_is_dead 3y ago
[dead]
Havoc 3y ago
Seems like a reasonable way forward.
- kramerger 3y ago
  Android started this way too with media libraries being updated first.
cyber_kinetist 3y ago
> These algorithms use the data received from userspace in order to index into a lot of arrays and thus benefit from Rust's memory safety. If everything is done with arrays and indices (apparently from looking at the code: (https://gitlab.collabora.com/dwlsalmeida/for-upstream/-/blob/7681609641fb9c94a0b5d827dc2e7290401689c3/rust/kernel/media/v4l2_core/vp9.rs https://gitlab.collabora.com/dwlsalmeida/for-upstream/-/blob...) it seems like Rust's borrow checker doesn't really help at all, and the only thing Rust really does for you is just bounds checks on arrays (with additional runtime overhead)... So I'm not sure how this can really improve the state of things compared to C with the equivalent bounds checks.
- woodruffw 3y ago
  > So I'm not sure how this can really improve the state of things compared to C with the equivalent bounds checks. The simplest answer here is that a compiler-introduced bounds check is almost always better than a human one. Humans make bounds errors, compilers generally don't. The longer answer is that the current state of the code does not guarantee its future state. The fact that bounds checks constitute the current majority of safety guardrails does not mean that future refactors won't benefit from Rust's temporal memory safety guarantees. Or more abstractly: it's easier to perform safe refactors when your safety properties compose natively, rather than having to bolt another layer of checks onto pre-existing language that doesn't support them natively. Edit: Forgot to mention: another benefit of bounds checking in the language itself is optimization: when humans bounds-check, the compiler needs to recognize human patterns to safety remove or merge redundant bounds checks. When the language specifies its own bounds checks, the compiler knows exactly what they'll look like and can optimize accordingly. Modern optimizing compilers are very good at detecting human-written bounds, but a fully compiler-controlled optimization is going to beat a human-augmented optimization >95% of the time.
  - hulitu 3y ago
    > Humans make bounds errors, compilers generally don't. I think i saw this in a movie. It didn't turned well. /s
- 3y ago
xmichael909 3y ago
if only there were a standalone player for linux, osx or windows...
trimbo 3y ago
> This patch ports the VP9 library written by Andrzej into Rust as a proof-of-concept ... > this library will not need any further updates for the same reason we have never touched its C counterpart What was being proof-of-concepted? What's the metric of success for introducing Rust in a case where no one was doing any sort of active work anyway?
- uo21tp5hoyg 3y ago
  From what I understand the need for a "proof-of-concept" comes from the fact these codecs/drivers often using memory unsafe "tricks" to increase performance and therefore need to be properly tested on a myriad of hardware to make sure the conversion to memory safe code isn't a significant performance impact.

200 points by marcodiego 3y ago

The VP9 codec I imagine would heavily benefit from SIMD? Just SIMD should massively outperform any advantage gained by putting this into the kernel. Why does the kernel need an unoptimized codec?

ZeroCool2u 3y ago
SIMD can be pretty architecture specific. For example, does the CPU support AVX-512 or SSE3? So, you have to have a few code paths if you're going to support a wide variety of hardware. I don't have an answer to your question it just occurs to me that maybe the Linux kernel doesn't allow a lot of SIMD for this reason or maybe they require a fall back/slow code path to be available if you're submitting a patch that includes SIMD operations? Curious if anyone knows the answer to this.
- __s 3y ago
  Kernel avoids SIMD because it doesn't have a kernel keeping its registers coherent https://stackoverflow.com/a/46677815 https://stackoverflow.com/a/46677815
  - deleted 3y ago
    [deleted]
  - ZeroCool2u 3y ago
    Thank you for the answer! Very interesting.
- brundolf 3y ago
  Fwiw, Rust's standard library has a cross-platform Simd abstraction: https://doc.rust-lang.org/std/simd/index.html https://doc.rust-lang.org/std/simd/index.html It's nightly-only for now, but I've used it and it's lovely. I've added Simd to projects that I never otherwise would have, just because this made it so easy and accessible
  - ZeroCool2u 3y ago
    That is metal af ;)
pavon 3y ago
I don't think that the driver implements VP9 in software. It uses hardware acceleration to perform the actual encode/decode, but is managing preparing of coefficients and DMA of data buffers in the driver.

briantakita 3y ago

[flagged]

megous 3y ago
No. It's all C code in the stacktrace. And kernel devs still don't have to deal with Rust, if they don't want to.
- briantakita 3y ago
  > No. It's all C code in the stacktrace. And kernel devs still don't have to deal with Rust, if they don't want to. Thank you for the answer. I was wondering since the conversion to Rust is a large undertaking & rewrites can have surprises. Part of the question was if this was an argument to use Rust since NULL pointers are something that Rust goes through great lengths to guard against. And wow...my original post got flagged? It was a sincere question that I was curious about & didn't have an answer for. Either everyone needs to chill...or is mentioning NULL pointer references in the Linux Kernel some sort of unspoken mortal sin or was I somehow offensive? Please, someone let me know if I'm out of line cause I just see zealotry of some type that I cannot begin to imagine. I want to avoid such land mines in the future...for the love of all that is holy.
  - nextaccountic 3y ago
    > I was wondering since the conversion to Rust is a large undertaking & rewrites can have surprises. There is no concerted effort to rewrite Linux in Rust, so there is no large undertaking. In spite of this VP9 rewrite, things are not being rewritten in Rust for the most part. Some individuals are rewriting some things but more importantly, some new stuff (such as drivers) are being written in Rust. It's this new stuff written in Rust (and not previously written in C) that is the focus IMO. And most C stuff isn't going to be rewritten. Many maintainers don't know Rust and couldn't review Rust code anyway. The impact of Rust in the kernel so far is pretty modest (the most important code being the driver for Apple silicon GPUs, I think) > And wow...my original post got flagged? It was a sincere question that I was curious about & didn't have an answer for. Either everyone needs to chill...or is mentioning NULL pointer references in the Linux Kernel some sort of unspoken mortal sin or was I somehow offensive? I didn't flag your post, but I would guess it was flagged because you immediately assumed that Rust was to blame.

renewiltord 3y ago

Interesting. Use case is so that there is some code exercising this path. While the algorithm is cool, of course, the code itself is quite straightforward as it is. An interesting thing I didn’t know is that the kernel code avoids recursion (this one has a depth parameter to prevent recursion past some point). I can see why this was picked as a candidate. Straightforward implementation. Good test suite. Self-contained and not a moving target. Lots of the code is using the coefficients array against the framecontext but I don’t know how to enforce the bounds invariant that the two are the same. For the fixed size arrays I could see how it’s done, but otherwise it seems like the bounds checker will trigger. But I’m reading on my phone so maybe that’s just a misread. Not a big performance hit even if it does. But perhaps it doesn’t, and I’d be curious why.

revskill 3y ago

I think the inferior of Rust vs C could be "over checking" ?

lionkor 3y ago

> These algorithms use the data received from userspace in order to index into a lot of arrays and thus benefit from Rust's memory safety. Yeah, thats not a good argument for Rust specifically. Maybe Zig would be interesting for array-specific things like sentinel-terminated arrays ans their amazing C interop. Further, Zig code is unmatched in how explicit everything is. There is no abstraction hiding complexity, its built on the idea of custom allocators from the ground up. This seems like someone looking for a usecase for Rust over C, rather than someone looking for a better solution altogether. Thats okay, I guess, but Rust does not do array bounds checking very well and the borrowchecker doesnt help there at all. The large amount of hidden complexity in a lot of the language dont make it more safe by any means.

dureuill 3y ago

> Much has been spoken at various occasions about drivers and I feel that the consensus is to wait for now. Interesting, I did not follow that development. I thought the plan was to use Rust for some out-of-tree/optional drivers. What changed?

Macha 3y ago
There was opposition to building interfaces for toy drivers and the last thread had suggestions at a rust filesystem interface rebuffed by saying they should try rewrite the ext2 driver in rust to prove that it was usable for real filesystems rather than toy ones. I'd guess similar thought processes fuelled this decision.
charcircuit 3y ago
It's already used for an intree driver, but a lot of the infrastructure for making Rust drivers was not be upstreamed.

rust_is_dead 3y ago

[dead]

Havoc 3y ago

Seems like a reasonable way forward.

kramerger 3y ago
Android started this way too with media libraries being updated first.

cyber_kinetist 3y ago

> These algorithms use the data received from userspace in order to index into a lot of arrays and thus benefit from Rust's memory safety. If everything is done with arrays and indices (apparently from looking at the code: (https://gitlab.collabora.com/dwlsalmeida/for-upstream/-/blob/7681609641fb9c94a0b5d827dc2e7290401689c3/rust/kernel/media/v4l2_core/vp9.rs https://gitlab.collabora.com/dwlsalmeida/for-upstream/-/blob...) it seems like Rust's borrow checker doesn't really help at all, and the only thing Rust really does for you is just bounds checks on arrays (with additional runtime overhead)... So I'm not sure how this can really improve the state of things compared to C with the equivalent bounds checks.

woodruffw 3y ago
> So I'm not sure how this can really improve the state of things compared to C with the equivalent bounds checks. The simplest answer here is that a compiler-introduced bounds check is almost always better than a human one. Humans make bounds errors, compilers generally don't. The longer answer is that the current state of the code does not guarantee its future state. The fact that bounds checks constitute the current majority of safety guardrails does not mean that future refactors won't benefit from Rust's temporal memory safety guarantees. Or more abstractly: it's easier to perform safe refactors when your safety properties compose natively, rather than having to bolt another layer of checks onto pre-existing language that doesn't support them natively. Edit: Forgot to mention: another benefit of bounds checking in the language itself is optimization: when humans bounds-check, the compiler needs to recognize human patterns to safety remove or merge redundant bounds checks. When the language specifies its own bounds checks, the compiler knows exactly what they'll look like and can optimize accordingly. Modern optimizing compilers are very good at detecting human-written bounds, but a fully compiler-controlled optimization is going to beat a human-augmented optimization >95% of the time.
- hulitu 3y ago
  > Humans make bounds errors, compilers generally don't. I think i saw this in a movie. It didn't turned well. /s
3y ago

xmichael909 3y ago

if only there were a standalone player for linux, osx or windows...

trimbo 3y ago

> This patch ports the VP9 library written by Andrzej into Rust as a proof-of-concept ... > this library will not need any further updates for the same reason we have never touched its C counterpart What was being proof-of-concepted? What's the metric of success for introducing Rust in a case where no one was doing any sort of active work anyway?

uo21tp5hoyg 3y ago
From what I understand the need for a "proof-of-concept" comes from the fact these codecs/drivers often using memory unsafe "tricks" to increase performance and therefore need to be properly tested on a myriad of hardware to make sure the conversion to memory safe code isn't a significant performance impact.