4 ms·

For my workload it's optimal to keep working nodes deserialized anyway. YMMV but what I'm doing gets me over a million operations per second -- 1 Gbps of reque

For my workload it's optimal to keep working nodes deserialized anyway. YMMV but what I'm doing gets me over a million operations per second -- 1 Gbps of requests -- on a single CPU so I don't think I'm going far wrong. EDIT: I just noticed you're the bcachefs author. I'm guessing you're dealing with much larger key/value pairs than I am (40+40)?

7e 3y ago
Well, bcachefs is currently the slowest of all Linux filesystems.
- 7e 3y ago
  See https://www.phoronix.com/review/bcachefs-linux-67 https://www.phoronix.com/review/bcachefs-linux-67.
  - magicalhippo 3y ago
    Probably more interesting to consider the follow-up article[1] after some debug code was disabled and such. [1]: https://www.phoronix.com/review/bcachefs-benchmarks-linux67 https://www.phoronix.com/review/bcachefs-benchmarks-linux67
    - 7e 3y ago
      Still dead last on a number of benchmarks, including FIO random reads, Dbench, and Postgres. And by a lot.
      - psd1 3y ago
        If I created the slowest mainstream Linux filesystem I'd be quite proud. Sneer all you want.
koverstreet 3y ago
No, that's a typical size for bcachefs as well. But I've had to optimize for memory usage, performance when working set does not fit into ram - we also don't have a serialize/deserialize step, that will really hurt you when working set doesn't fit in ram. You want to validate when you're reading in a btree node, but you want to keep that as cheap as possible. For avoiding the serialize/deserialize, the thing to use now would be Cap'n Proto.
- cperciva 3y ago
  Ah, that would explain it. Yeah, I have a total datastore size which is out of core, but the working set is generally in core... partly because I can use all the RAM on the system if I want to, whereas filesystems are (usually!) expected to occupy just a portion of the system memory. FWIW, my "deserialize" doesn't involve copying all the data -- I just construct pointers into the (serialized) page. Serializing has to copy data though.
  - valenterry 3y ago
    Just wanted to say, I really enjoyed reading your guys conversation, since I barely never do low-level optimizations like that. Very interesting, thanks for sharing your thoughts!

7e 3y ago

Well, bcachefs is currently the slowest of all Linux filesystems.

7e 3y ago
See https://www.phoronix.com/review/bcachefs-linux-67 https://www.phoronix.com/review/bcachefs-linux-67.
- magicalhippo 3y ago
  Probably more interesting to consider the follow-up article[1] after some debug code was disabled and such. [1]: https://www.phoronix.com/review/bcachefs-benchmarks-linux67 https://www.phoronix.com/review/bcachefs-benchmarks-linux67
  - 7e 3y ago
    Still dead last on a number of benchmarks, including FIO random reads, Dbench, and Postgres. And by a lot.
    - psd1 3y ago
      If I created the slowest mainstream Linux filesystem I'd be quite proud. Sneer all you want.

koverstreet 3y ago

No, that's a typical size for bcachefs as well. But I've had to optimize for memory usage, performance when working set does not fit into ram - we also don't have a serialize/deserialize step, that will really hurt you when working set doesn't fit in ram. You want to validate when you're reading in a btree node, but you want to keep that as cheap as possible. For avoiding the serialize/deserialize, the thing to use now would be Cap'n Proto.

cperciva 3y ago
Ah, that would explain it. Yeah, I have a total datastore size which is out of core, but the working set is generally in core... partly because I can use all the RAM on the system if I want to, whereas filesystems are (usually!) expected to occupy just a portion of the system memory. FWIW, my "deserialize" doesn't involve copying all the data -- I just construct pointers into the (serialized) page. Serializing has to copy data though.
- valenterry 3y ago
  Just wanted to say, I really enjoyed reading your guys conversation, since I barely never do low-level optimizations like that. Very interesting, thanks for sharing your thoughts!