4 ms·

The comparison to JDK serialization is surprising me. Noone should be using it for anything serious in production. Even the Chief Architect of the Java platform

by npstr 3y ago

The comparison to JDK serialization is surprising me. Noone should be using it for anything serious in production. Even the Chief Architect of the Java platform has called it a "horrible mistake". https://www.infoworld.com/article/3275924/oracle-plans-to-dump-risky-java-serialization.html https://www.infoworld.com/article/3275924/oracle-plans-to-du...

chaokunyang 3y ago
There are too many java serialization libraries, I compared fury with jdk/kryo/fst/protostuff/protobuf/flatbuffers/thrift/msgpack/canproto/acro/jackson/json. Fury are fastest too. See https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki for detailed benchmark results. It's a hard choice to select one for the title, so I use JDK for it, which may be not a good choice. BTW, fury support jit serialization for jdk17 record, which is super fast compared to other serialization frameworks such as kryo
pron 3y ago
Just to be clear, though, the mistake that Mark Reinhold refers to isn't the particular implementation of the JDK's core serialization, but the design that allows arbitrary objects to be deserialized while bypassing their constructors (and so their established invariants). Unfortunately, many other serialization libraries — faster or slower — are repeating the same mistake. I.e., any serialization library that can serialize any class that implements Serializable suffers from all the same flaws of core serialization. Correct serialization can only be done for classes that are designed for it by having a well-known construction protocol, which are currently basic collections, enums, Strings, records, and classes that register specific serialization code.
- chaokunyang 3y ago
  I see, totally agreed! If any objects can be serialized, the deserialization will introduce security issue too. For example, `constructor/equals/hashCode` may contains malicious code, which introduce the deserialization risks. Fury put much work on this to avoid the open dynamic deserialization risks. But although `basic collections, enums, Strings, records, and classes that register specific serialization code` are the only objects should be allowed for serialization, and they are serialized by the construction protocol in fury already. There are so many applications has used the existing serialization protocol assumption, we have to keep compatible, otherwise most of application can't use fury.
- mkleczek 3y ago
  Why? Zero copy state transfer is a viable and high performance alternative. Security and integrity can (should?) be implemented at a different layer.
  - chaokunyang 3y ago
    Not exactly. For java serialization, you can serialize java native object directly without define dsl and compile the schema, which are mush more easy to use. But it also means the deserialization will need to create the user-defined class, which may contains malicious coode in `constructor/queals/hashCode`. So the security can't be done at a different layer unless you are in a intra-net which no attack will happen which may be implemented at a different layer but we can't ensure that. The
  - chaokunyang 3y ago
    We must realize there always a tradeoff here. If you define a dsl for serialization data and generate the code like protobuf, the security issue will be much less. But it comes with the cost. Protobuf generated class are not the domain class, and can't be used for domain-driven application developemnt, and it doesn't support circular references too. What fury does it provide better performance and provide better usability.
  - mumblemumble 3y ago
    > Zero copy state transfer is a viable and high performance alternative. Completely agreed, but also, adding my own emphasis there. The technique has has enough gotchas, edge cases, and additional security considerations that it should really be an alternative that people can opt for when they need it, and never be the default approach.
    - chaokunyang 3y ago
      Yes, it should be an alternative. In fury, we disabled it by default, and all those benchmarks doesn't enable it. Most objects are value bounded, not binary data bounded. Zero-copy won't have a big speed up too. But if the objects graph has many ByteBuffer/Tensor/DataFrame/ArrowTable, then the zero-copy will have a big leap. see python pickle5 out-of-band serialization to speed up pandas/numpy as an example
LispSporks22 3y ago
There's also a chapter on JDK serialized in Bloch's "Effective Java" worth a read: Item 85: Prefer alternatives to Java serialization Item 86: Implement Serializable with great caution Item 87: Consider using a custom serialized form Item 88: Write readObject methods defensively Item 89: For instance control, prefer enum types to readResolve Item 90: Consider serialization proxies instead of serialized instances
- chaokunyang 3y ago
  Those pattern are all supported in fury. And it seems fury are the only framework which implement the jdk `writeObject/readObject/writeReplace/readResolve/readObjectNoData` methods except jdk iteself. Other java serialization will jsut ignore those methods, and get incorect results. But today I'll suggest to avoid to use JDK `riteObject/readObject/`. Since to be compatible with JDK serialization API behaviour, it will introduce performance and space overhead. We can register custom Serializer by fury.registerSerializer(xxx.class, XXXSerializer.class). `writeReplace/readResolve` are an useful pattern, and can be used when needed.

by npstr 3y ago

chaokunyang 3y ago

There are too many java serialization libraries, I compared fury with jdk/kryo/fst/protostuff/protobuf/flatbuffers/thrift/msgpack/canproto/acro/jackson/json. Fury are fastest too. See https://github.com/eishay/jvm-serializers/wiki https://github.com/eishay/jvm-serializers/wiki for detailed benchmark results. It's a hard choice to select one for the title, so I use JDK for it, which may be not a good choice. BTW, fury support jit serialization for jdk17 record, which is super fast compared to other serialization frameworks such as kryo

pron 3y ago

Just to be clear, though, the mistake that Mark Reinhold refers to isn't the particular implementation of the JDK's core serialization, but the design that allows arbitrary objects to be deserialized while bypassing their constructors (and so their established invariants). Unfortunately, many other serialization libraries — faster or slower — are repeating the same mistake. I.e., any serialization library that can serialize any class that implements Serializable suffers from all the same flaws of core serialization. Correct serialization can only be done for classes that are designed for it by having a well-known construction protocol, which are currently basic collections, enums, Strings, records, and classes that register specific serialization code.

chaokunyang 3y ago
I see, totally agreed! If any objects can be serialized, the deserialization will introduce security issue too. For example, `constructor/equals/hashCode` may contains malicious code, which introduce the deserialization risks. Fury put much work on this to avoid the open dynamic deserialization risks. But although `basic collections, enums, Strings, records, and classes that register specific serialization code` are the only objects should be allowed for serialization, and they are serialized by the construction protocol in fury already. There are so many applications has used the existing serialization protocol assumption, we have to keep compatible, otherwise most of application can't use fury.
mkleczek 3y ago
Why? Zero copy state transfer is a viable and high performance alternative. Security and integrity can (should?) be implemented at a different layer.
- chaokunyang 3y ago
  Not exactly. For java serialization, you can serialize java native object directly without define dsl and compile the schema, which are mush more easy to use. But it also means the deserialization will need to create the user-defined class, which may contains malicious coode in `constructor/queals/hashCode`. So the security can't be done at a different layer unless you are in a intra-net which no attack will happen which may be implemented at a different layer but we can't ensure that. The
- chaokunyang 3y ago
  We must realize there always a tradeoff here. If you define a dsl for serialization data and generate the code like protobuf, the security issue will be much less. But it comes with the cost. Protobuf generated class are not the domain class, and can't be used for domain-driven application developemnt, and it doesn't support circular references too. What fury does it provide better performance and provide better usability.
- mumblemumble 3y ago
  > Zero copy state transfer is a viable and high performance alternative. Completely agreed, but also, adding my own emphasis there. The technique has has enough gotchas, edge cases, and additional security considerations that it should really be an alternative that people can opt for when they need it, and never be the default approach.
  - chaokunyang 3y ago
    Yes, it should be an alternative. In fury, we disabled it by default, and all those benchmarks doesn't enable it. Most objects are value bounded, not binary data bounded. Zero-copy won't have a big speed up too. But if the objects graph has many ByteBuffer/Tensor/DataFrame/ArrowTable, then the zero-copy will have a big leap. see python pickle5 out-of-band serialization to speed up pandas/numpy as an example

LispSporks22 3y ago

There's also a chapter on JDK serialized in Bloch's "Effective Java" worth a read: Item 85: Prefer alternatives to Java serialization Item 86: Implement Serializable with great caution Item 87: Consider using a custom serialized form Item 88: Write readObject methods defensively Item 89: For instance control, prefer enum types to readResolve Item 90: Consider serialization proxies instead of serialized instances

chaokunyang 3y ago
Those pattern are all supported in fury. And it seems fury are the only framework which implement the jdk `writeObject/readObject/writeReplace/readResolve/readObjectNoData` methods except jdk iteself. Other java serialization will jsut ignore those methods, and get incorect results. But today I'll suggest to avoid to use JDK `riteObject/readObject/`. Since to be compatible with JDK serialization API behaviour, it will introduce performance and space overhead. We can register custom Serializer by fury.registerSerializer(xxx.class, XXXSerializer.class). `writeReplace/readResolve` are an useful pattern, and can be used when needed.