8 ms·

I made this thing! AMA :)

excerionsforte 6y ago
When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ https://serde.rs/, which seems to be a recent addition. I was looking for an alternative to messagepack.
- Aardappel 6y ago
  "market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.
  - webmaven 6y ago
    Rather than "market", you could "promote" or "evangelize" instead...
  - excerionsforte 6y ago
    Marketing is an umbrella term for spreading awareness. Less people that know about it then the less likely your product gets experimented with and then chosen. I don't think any recent serialization benchmarks include flexbuffers. There's different ways to make people aware of alternatives. I chose to spread word of this product on HN :)
    - Aardappel 6y ago
      Even that kind of "marketing" takes time :) But thanks for helping out.
mavam 6y ago
Unlike MsgPack, FlexBuffers doesn’t support custom/extension types. Is this something that is planned for the future?
- Aardappel 6y ago
  I searched for "msgpack extension types" and "msgpack custom types" and nothing conclusive came up, so you'll have to explain what you mean. FlexBuffers is a schemaless format much like JSON, so is naturally extensible, and typeless, so I don't see how it would need any such thing.
  - kubelsmieci 6y ago
    I think it is called polymorphism, at least on msgpack libraries I used.
  - mavam 6y ago
    Here's the link to the spec about extension types: https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types https://github.com/msgpack/msgpack/blob/master/spec.md#exten... From there: MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data. Applications can assign 0 to 127 to store application-specific type information. An example usage is that application defines type = 0 as the application's unique type system, and stores name of a type and values of the type at the payload.
    - Aardappel 6y ago
      Ah, thanks! I presume that is done for compactness, because I otherwise see no reason to pack an application type in with the built-in types. FlexBuffers could support a similar scheme if we wanted to, though putting custom data in the format is already easy enough using the "blob" type, which is arbitrary bytes much like the MessagePack feature.
      - 6y ago
bsaul 6y ago
there is a comment up thread from people at google claiming flatbuffers aren't faster than protobuf in practice. that surprises me a lot, as i was interested in flat buffers for performance gains. do you have an opinion ?
- dreinhardt 6y ago
  As i understood it, this is not about being faster, but about being more memory efficient.
  - bsaul 6y ago
    zero copy supposedly means you're saving on memory allocation when reading. You're saving the decoding step, so my guess would have been that in practice you're gaining both on memory consumption and cpu.
- Aardappel 6y ago
  I've worked with people internally, and it is indeed a challenge to speed up services with FlatBuffers. This is because 99.9% of internal communications flow over Protobuf, and many such services are simply: receive Protobuf, modify a few things, send it on the next service. So to deploy FlatBuffers there, you usually have to translate from Protobuf to FlatBuffers, do some high intensity stretch of communication in FlatBuffers only, then convert back. Given the cost of conversions, it would require replacing fairly large amounts of infrastructure for it to be a clear win, and given that those conversions have to be written entirely by hand, that is often not feasible. So translation of what the other comment said: "we gave up". FlatBuffers is used to great effect, but often with new teams building new things, that are not yet shackled in Protobuf everywhere, like many of the AI related teams.
  - bsaul 6y ago
    Ok, now that gives a very different picture from what the original comment explained. Of course, protobuf to flatbuffer is slower than just protobuf... Thanks a lot for your time (and work!). FYI, i'm planning on using flatbuffer for communication between a mobile app native code and a cross-platform library, hoping to save on data decoding time. A bit similar to what Xi ide is doing. If you have any advice, you're welcome ! :D
XCSme 6y ago
What do you think is the best buffer protocol to use for multiplayer games? We used Protobuf for a fast-paced .io game, but encoding-decoding turned out to be pretty slow and generated a lot of garbage in JS. We were in the process to switch to FlatBuffers (before the company went bankrupt), but the syntax made it feel harder to use compared to Protobuf, not sure about the performance though (we expected it to be faster and less garbage created because of the zero-copy). So, would you recommend Protobuf, Cap'n'Proto, FlatBuffers or FlexBuffers for multiplayer games? The usual packets are game states or user input sent at high frequency.
- mintyc 6y ago
  My experience with capn proto was a mixed bag. Quite apart from its neglect for years as Kenton was working on sandstorm.io, I found the low level access interfaces to extract best performance to be both fiddly and often rely on an intimate knowledge of the internals. As is often the case, maximum performance for your use case requires more intimate detail. I can't remember, but some of these zero copy protocols may still require byte order swapping on mixed endian systems. (But cache access times probably dwarf the byte swapping)
- Aardappel 6y ago
  I originally designed FlatBuffers for games (though admittedly more for things like level data or save game data than network packets), so I'd think it is pretty suitable. I had actually used Protobuf on a game project just before, and its performance problems led directly to the no-unpacking no-allocation design that FlatBuffers has. So FlatBuffers will make an incoming packet waaay faster to work with than Protobuf. On the downside, Protobuf tends to be a little smaller, so if bandwidth is a greater concern than (de-)serialization speed, you might still prefer it. Additionally, receiving data over the network raises the question of how you handle packets that have been corrupted (or intentionally malformatted by an attacker), and in the case of FlatBuffers you'd need to at least run the "verifier" over the packet before accessing it if you don't want your game servers to crash when this happens. That slows it down a little bit, but is still fast, i.e. still doesn't allocate etc. Cap'n Proto will perform similarly, though does have the downside that all fields take space on the wire, regardless of whether they're set or not. So which is better depends on the kind of data you want to send and how it is likely to evolve. Frankly, for the absolute highest performance (and lowest bandwidth) game networking you still need a custom encoding.

When I had searched HN for this, there was none. Curious, why didn't you market it? Seems production ready or maybe it was planned in the future. I'm glad to have seen this, which was on https://serde.rs/ https://serde.rs/, which seems to be a recent addition. I was looking for an alternative to messagepack.

Aardappel 6y ago
"market" it? This is an open source project, with zero budget. I'd rather spend my time writing more code than trying to market anything, which seems hard in todays climate :) Good systems will eventually reach people organically, even if it takes longer that way.
- webmaven 6y ago
  Rather than "market", you could "promote" or "evangelize" instead...
- excerionsforte 6y ago
  Marketing is an umbrella term for spreading awareness. Less people that know about it then the less likely your product gets experimented with and then chosen. I don't think any recent serialization benchmarks include flexbuffers. There's different ways to make people aware of alternatives. I chose to spread word of this product on HN :)
  - Aardappel 6y ago
    Even that kind of "marketing" takes time :) But thanks for helping out.

mavam 6y ago

Unlike MsgPack, FlexBuffers doesn’t support custom/extension types. Is this something that is planned for the future?

Aardappel 6y ago
I searched for "msgpack extension types" and "msgpack custom types" and nothing conclusive came up, so you'll have to explain what you mean. FlexBuffers is a schemaless format much like JSON, so is naturally extensible, and typeless, so I don't see how it would need any such thing.
- kubelsmieci 6y ago
  I think it is called polymorphism, at least on msgpack libraries I used.
- mavam 6y ago
  Here's the link to the spec about extension types: https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types https://github.com/msgpack/msgpack/blob/master/spec.md#exten... From there: MessagePack allows applications to define application-specific types using the Extension type. Extension type consists of an integer and a byte array where the integer represents a kind of types and the byte array represents data. Applications can assign 0 to 127 to store application-specific type information. An example usage is that application defines type = 0 as the application's unique type system, and stores name of a type and values of the type at the payload.
  - Aardappel 6y ago
    Ah, thanks! I presume that is done for compactness, because I otherwise see no reason to pack an application type in with the built-in types. FlexBuffers could support a similar scheme if we wanted to, though putting custom data in the format is already easy enough using the "blob" type, which is arbitrary bytes much like the MessagePack feature.
    - 6y ago

bsaul 6y ago

there is a comment up thread from people at google claiming flatbuffers aren't faster than protobuf in practice. that surprises me a lot, as i was interested in flat buffers for performance gains. do you have an opinion ?

dreinhardt 6y ago
As i understood it, this is not about being faster, but about being more memory efficient.
- bsaul 6y ago
  zero copy supposedly means you're saving on memory allocation when reading. You're saving the decoding step, so my guess would have been that in practice you're gaining both on memory consumption and cpu.
Aardappel 6y ago
I've worked with people internally, and it is indeed a challenge to speed up services with FlatBuffers. This is because 99.9% of internal communications flow over Protobuf, and many such services are simply: receive Protobuf, modify a few things, send it on the next service. So to deploy FlatBuffers there, you usually have to translate from Protobuf to FlatBuffers, do some high intensity stretch of communication in FlatBuffers only, then convert back. Given the cost of conversions, it would require replacing fairly large amounts of infrastructure for it to be a clear win, and given that those conversions have to be written entirely by hand, that is often not feasible. So translation of what the other comment said: "we gave up". FlatBuffers is used to great effect, but often with new teams building new things, that are not yet shackled in Protobuf everywhere, like many of the AI related teams.
- bsaul 6y ago
  Ok, now that gives a very different picture from what the original comment explained. Of course, protobuf to flatbuffer is slower than just protobuf... Thanks a lot for your time (and work!). FYI, i'm planning on using flatbuffer for communication between a mobile app native code and a cross-platform library, hoping to save on data decoding time. A bit similar to what Xi ide is doing. If you have any advice, you're welcome ! :D

XCSme 6y ago

What do you think is the best buffer protocol to use for multiplayer games? We used Protobuf for a fast-paced .io game, but encoding-decoding turned out to be pretty slow and generated a lot of garbage in JS. We were in the process to switch to FlatBuffers (before the company went bankrupt), but the syntax made it feel harder to use compared to Protobuf, not sure about the performance though (we expected it to be faster and less garbage created because of the zero-copy). So, would you recommend Protobuf, Cap'n'Proto, FlatBuffers or FlexBuffers for multiplayer games? The usual packets are game states or user input sent at high frequency.

mintyc 6y ago
My experience with capn proto was a mixed bag. Quite apart from its neglect for years as Kenton was working on sandstorm.io, I found the low level access interfaces to extract best performance to be both fiddly and often rely on an intimate knowledge of the internals. As is often the case, maximum performance for your use case requires more intimate detail. I can't remember, but some of these zero copy protocols may still require byte order swapping on mixed endian systems. (But cache access times probably dwarf the byte swapping)
Aardappel 6y ago
I originally designed FlatBuffers for games (though admittedly more for things like level data or save game data than network packets), so I'd think it is pretty suitable. I had actually used Protobuf on a game project just before, and its performance problems led directly to the no-unpacking no-allocation design that FlatBuffers has. So FlatBuffers will make an incoming packet waaay faster to work with than Protobuf. On the downside, Protobuf tends to be a little smaller, so if bandwidth is a greater concern than (de-)serialization speed, you might still prefer it. Additionally, receiving data over the network raises the question of how you handle packets that have been corrupted (or intentionally malformatted by an attacker), and in the case of FlatBuffers you'd need to at least run the "verifier" over the packet before accessing it if you don't want your game servers to crash when this happens. That slows it down a little bit, but is still fast, i.e. still doesn't allocate etc. Cap'n Proto will perform similarly, though does have the downside that all fields take space on the wire, regardless of whether they're set or not. So which is better depends on the kind of data you want to send and how it is likely to evolve. Frankly, for the absolute highest performance (and lowest bandwidth) game networking you still need a custom encoding.