5 ms·

Dictionary encoding is already used in fury. We will encode same string as an varint when it's seen later. The thing here is that many cases the string don't ha

Dictionary encoding is already used in fury. We will encode same string as an varint when it's seen later. The thing here is that many cases the string don't have repeat. Imagine you send `record Point(into x, int y)` in an rpc. You have one 'Point' to send. No dictionary encoding will be used in such cases

vlovich123 2y ago
You seem to be fundamentally misunderstanding how the zstd dictionary works. It’s a dictionary at the statistical level. Your `record Point(int x, int y)` RPC would statistically at the bit level have to look like no other message for it to have no help. That seems unlikely since there’s all sorts of framing. And I question the mindset of ad-hoc compression schemes that are trying to shrink the long tail of “sent only once” messages. zstd’s approach is fundamentally very different from a basic interning dictionary. Seriously, there’s been tons of research on this. It’s also interesting that Fury claims to be 0-copy and then does all these ad-hoc field compression which is the literal definition of not 0-copy (& yes, varint encoding is not 0-copy and neither is this custom string compression). Anyway, unless you actually do a proper comparison against zstd with a trained dictionary against a representative corpus, we’re just going in circles with me saying “you need to evaluate your ad-hoc compression against zstd” and you trying to argue from first principles that the custom compression scheme wins.
- chaokunyang 2y ago
  Hi, it's not “sent only once” . It's millions of “sent only once” . The thing here are that all those RPC are stateless, so the context for compression are the one message itself. i.e. compress object like `Point(1,2)` only without priori knowledge. Users may use train static zstd. But Fury can't, Fury doesn't know which data will be serialized. And meta string encoding are not statistical, it won't beats zstd. It's just for cases zstd is not suitable.

vlovich123 2y ago

You seem to be fundamentally misunderstanding how the zstd dictionary works. It’s a dictionary at the statistical level. Your `record Point(int x, int y)` RPC would statistically at the bit level have to look like no other message for it to have no help. That seems unlikely since there’s all sorts of framing. And I question the mindset of ad-hoc compression schemes that are trying to shrink the long tail of “sent only once” messages. zstd’s approach is fundamentally very different from a basic interning dictionary. Seriously, there’s been tons of research on this. It’s also interesting that Fury claims to be 0-copy and then does all these ad-hoc field compression which is the literal definition of not 0-copy (& yes, varint encoding is not 0-copy and neither is this custom string compression). Anyway, unless you actually do a proper comparison against zstd with a trained dictionary against a representative corpus, we’re just going in circles with me saying “you need to evaluate your ad-hoc compression against zstd” and you trying to argue from first principles that the custom compression scheme wins.

chaokunyang 2y ago
Hi, it's not “sent only once” . It's millions of “sent only once” . The thing here are that all those RPC are stateless, so the context for compression are the one message itself. i.e. compress object like `Point(1,2)` only without priori knowledge. Users may use train static zstd. But Fury can't, Fury doesn't know which data will be serialized. And meta string encoding are not statistical, it won't beats zstd. It's just for cases zstd is not suitable.