4 ms·

Why? Zero copy state transfer is a viable and high performance alternative. Security and integrity can (should?) be implemented at a different layer.

chaokunyang 3y ago
Not exactly. For java serialization, you can serialize java native object directly without define dsl and compile the schema, which are mush more easy to use. But it also means the deserialization will need to create the user-defined class, which may contains malicious coode in `constructor/queals/hashCode`. So the security can't be done at a different layer unless you are in a intra-net which no attack will happen which may be implemented at a different layer but we can't ensure that. The
chaokunyang 3y ago
We must realize there always a tradeoff here. If you define a dsl for serialization data and generate the code like protobuf, the security issue will be much less. But it comes with the cost. Protobuf generated class are not the domain class, and can't be used for domain-driven application developemnt, and it doesn't support circular references too. What fury does it provide better performance and provide better usability.
mumblemumble 3y ago
> Zero copy state transfer is a viable and high performance alternative. Completely agreed, but also, adding my own emphasis there. The technique has has enough gotchas, edge cases, and additional security considerations that it should really be an alternative that people can opt for when they need it, and never be the default approach.
- chaokunyang 3y ago
  Yes, it should be an alternative. In fury, we disabled it by default, and all those benchmarks doesn't enable it. Most objects are value bounded, not binary data bounded. Zero-copy won't have a big speed up too. But if the objects graph has many ByteBuffer/Tensor/DataFrame/ArrowTable, then the zero-copy will have a big leap. see python pickle5 out-of-band serialization to speed up pandas/numpy as an example
pron 3y ago
But the only reason core serialization is considered a "horrible mistake" has nothing to do with performance or with any aspect of the technical implementation. It is considered a mistake only because serialising arbitrary classes breaks invariants in a way that has serious correctness and security implications.
- mkleczek 3y ago
  The point is that security and bypassing constructors are completely orthogonal as serialisation by memcpy is completely viable strategy and is no less secure than calling constructors on objects.
  - pron 3y ago
    Whether or not they're orthogonal, it is that aspect that we talk about when we talk about the mistake of core serialization in the JDK. Also, whether calling constructors is necessary for security depends on many details. When we talk about fixing serialization in Java, we talk about a design that is independent of the format used for the serialized data. I understand you're that this is a different layer from the one you're interested in, but that is the layer where the mistake was made, and so that is the layer that requires fixing.
    - mkleczek 3y ago
      The premise of the sentence you cited is that it is necessary to call application code (constructors) to maintain security/integrity. What I am saying is that it is NOT necessary to implement secure serialisation and deserialisation because you CAN (and should?) implement it at a different layer and transparently to the application code.
      - pron 3y ago
        It is necessary to call application code to maintain integrity (i.e. the guarantee of global program invariants), if you want to design classes for serialization that is agnostic to serialized format and that accounts for cases where not both sides of the communication are trusted. If program objects can be constructed from bits that arrive from external input that might not be trusted, some code needs to run to ensure that the objects maintain their invariants. Of course, you could say that objects cannot maintain any of their own invariants (that is the approach taken by Zig, where, unlike in Java, it is impossible to explicitly define, at the language level, an object that represents only prime integers), but that requires a very different language design, one that has not yet been proven to scale well (and one that isn't taken by Java specifically).

chaokunyang 3y ago

Not exactly. For java serialization, you can serialize java native object directly without define dsl and compile the schema, which are mush more easy to use. But it also means the deserialization will need to create the user-defined class, which may contains malicious coode in `constructor/queals/hashCode`. So the security can't be done at a different layer unless you are in a intra-net which no attack will happen which may be implemented at a different layer but we can't ensure that. The

chaokunyang 3y ago

We must realize there always a tradeoff here. If you define a dsl for serialization data and generate the code like protobuf, the security issue will be much less. But it comes with the cost. Protobuf generated class are not the domain class, and can't be used for domain-driven application developemnt, and it doesn't support circular references too. What fury does it provide better performance and provide better usability.

mumblemumble 3y ago

> Zero copy state transfer is a viable and high performance alternative. Completely agreed, but also, adding my own emphasis there. The technique has has enough gotchas, edge cases, and additional security considerations that it should really be an alternative that people can opt for when they need it, and never be the default approach.

chaokunyang 3y ago
Yes, it should be an alternative. In fury, we disabled it by default, and all those benchmarks doesn't enable it. Most objects are value bounded, not binary data bounded. Zero-copy won't have a big speed up too. But if the objects graph has many ByteBuffer/Tensor/DataFrame/ArrowTable, then the zero-copy will have a big leap. see python pickle5 out-of-band serialization to speed up pandas/numpy as an example

pron 3y ago

But the only reason core serialization is considered a "horrible mistake" has nothing to do with performance or with any aspect of the technical implementation. It is considered a mistake only because serialising arbitrary classes breaks invariants in a way that has serious correctness and security implications.

mkleczek 3y ago
The point is that security and bypassing constructors are completely orthogonal as serialisation by memcpy is completely viable strategy and is no less secure than calling constructors on objects.
- pron 3y ago
  Whether or not they're orthogonal, it is that aspect that we talk about when we talk about the mistake of core serialization in the JDK. Also, whether calling constructors is necessary for security depends on many details. When we talk about fixing serialization in Java, we talk about a design that is independent of the format used for the serialized data. I understand you're that this is a different layer from the one you're interested in, but that is the layer where the mistake was made, and so that is the layer that requires fixing.
  - mkleczek 3y ago
    The premise of the sentence you cited is that it is necessary to call application code (constructors) to maintain security/integrity. What I am saying is that it is NOT necessary to implement secure serialisation and deserialisation because you CAN (and should?) implement it at a different layer and transparently to the application code.
    - pron 3y ago
      It is necessary to call application code to maintain integrity (i.e. the guarantee of global program invariants), if you want to design classes for serialization that is agnostic to serialized format and that accounts for cases where not both sides of the communication are trusted. If program objects can be constructed from bits that arrive from external input that might not be trusted, some code needs to run to ensure that the objects maintain their invariants. Of course, you could say that objects cannot maintain any of their own invariants (that is the approach taken by Zig, where, unlike in Java, it is impossible to explicitly define, at the language level, an object that represents only prime integers), but that requires a very different language design, one that has not yet been proven to scale well (and one that isn't taken by Java specifically).