3 ms·
AFAIK, Hadoop uses protocol buffers for message passing, not Java serialization.
by zeroxfe 8y ago
AFAIK, Hadoop uses protocol buffers for message passing, not Java serialization.
- yzmtf2008 8y agoHadoop uses Avro actually, but the point still stands :)
- erik_seaberg 8y agoHadoop expects your keys and values to implement Writable and serialize themselves (a lot of these are actually hand-written expecting the instance to get reused for each input tuple). There's optional and fairly clumsy glue that makes Avro work in a key or value.
- sitkack 8y agoLast time I dived into details, the communication protocol for HDFS was serialized java objects using the built in serialization mechanism. Edit, they switched to Protocol Buffers in 0.23 https://wiki.apache.org/hadoop/HadoopRpc https://wiki.apache.org/hadoop/HadoopRpc