5 ms·

This article is misguided, at least wrt Java. For "distributed systems" object identity needs to be created and ensured programmatically (e.g. by an object id).

This article is misguided, at least wrt Java. For "distributed systems" object identity needs to be created and ensured programmatically (e.g. by an object id). equals() and hashCode() must be derived from an object's immutable properties and they are as safe or unsafe as you implement them.

scott_s 14y ago
I'm not sure if you understood the author correctly - your points don't disagree with him. He is not talking about using hashes for identity, but for routing. And he is advocating something more general than what you said. You said, basically, "implement your own hash functions for your objects." He said, basically, "don't use Object.hashCode(); you can easily just hash the serialization of the object, or implement a hash function yourself."
- rickmode 14y ago
  Stated another way: The contract Object.hashCode() does not require a stable results between processes; it _only_ requires a stable result during a single run of an application. So if you need a stable distributed hash, the article recommends using a different method than Object.hashCode(), for instance DistributedObject.stableHash(). Using Object.hashCode() can bite you since not all implementations will have stable over multiple runs.
  - cbsmith 14y ago
    ...except that there is nothing that prevents you from having a stricter contract on hashCode() with your hashed objects... like EVERY SYSTEM does.
    - lmm 14y ago
      But not e.g. protobuf. So don't write a distributed routing framework that assumes hashCode() is stable - if you do, it will work fine for Strings and HashMaps and all the standard java objects, and then fail when someone tries to use it with protobuf objects.
      - cbsmith 14y ago
        I ended up posting something much longer on this: http://news.ycombinator.com/item?id=4131326 http://news.ycombinator.com/item?id=4131326 Your comment qualifies as being "wrong", because of one part: "...it will work fine for Strings and HashMaps and all the standard java objects...". In fact hashCode() is stable for a very small subset of the types that ship with standard Java, and even in some of those cases there are caveats. Arrays come pretty standard with Java, but... boolean alwaysFalse(int[] a) { return a.hashCode() == a.clone().hashCode(); }
        lmm 14y ago
        a.clone().equals(a) is false too; you can't use Array as a thing to route on any more than you can use Object. The point is that in standard java objects with value semantics have stable hashCode implementations, and this is not true in general (And FWIW arrays are a weird corner of Java as far as I'm concerned; every codebase I've worked on avoids them as much as possible)
        cbsmith 14y ago
        > a.clone().equals(a) is false too. a.clone().equals(a) HAS to be false in that case. It's more than a bit of a given if a.hashCode() != a.clone().hashCode(). > you can't use Array as a thing to route on any more than you can use Object Oh, sure you can. You just need to use java.util.Arrays's methods for "equals" and "hashCode" (or potentially deepHashCode). Also, ArrayList does work. Yes, that is totally F'd up. > The point is that in standard java objects with value semantics have stable hashCode implementations, and this is not true in general Value semantics are pretty rare in Java. I'm not even sure what you really mean there. Pass-by-value is doesn't happen with Java objects. I could see you referring to immutable types, but that doesn't include any of the collection classes. I'm guessing you mean "primitive type wrappers and collections". It'd make more sense if you said something like, "pretty much anything that overrides Object.equals(Object)"... because that's the way it is supposed to work. They are rare in standard class library, because there is little business logic there. In practice, anything that resembles an identifier, and therefore all keys, tends to do the override though. Indeed, most of what people tend to call "business objects" tend to do the override. That's why the convention is there. Most importantly: almost all overrides do so in a fashion that is stable across processes. That's also why distributed frameworks can and should employ the convention/protocol. That equals/hashCode methods in Object are following the Smalltalk trick of having protocol defined in Object even though you shouldn't really use it without subclassing. The Object method isn't a "reference implementation" of the protocol, but rather a placeholder (one they forgot to override in Array objects, and then tried to backdoor in with java.util.Arrays).
- cbsmith 14y ago
  Except he cited specific examples of distributed systems (Hadoop, Storm) that use hashCode() but also make it quite clear you need to ensure you provide a hashCode() implementation that is entirely derived from the state of the object which is preserved in serialized form. Seriously, it's just a bad article.
  - scott_s 14y ago
    No, not need to. He said it was convenient if you're already serializing the object anyway. Which, if you're routing an object in a distributed system, is probably true.
    - cbsmith 14y ago
      Yes, it is terribly convenient because if Type.hashCode() has problems with copies of objects, this will help: int computeHash(byte[] serializedBytes) { return serializeBytes.hashCode(); } Now you've gone from a hashCode() method that won't work if the programmer screwed the pooch, to a hashCode() method that won't work period. ;-) [Yes, it won't be screwed up if also you use your own hash.] The problem with serialization is you lose the semantics of the object. The "same values to the same sequence of bytes" is a tighter constraint than you might think. The same "values" might have multiple representations which are logically identical but might still be relevant in any number of ways (for example, additional information that has nothing to do with identity).
      - scott_s 14y ago
        [Yes, it won't be screwed up if also you use your own hash.] It's strange that you disregard that in an aside, as it's his entire point. Personally, I prefer to implement hashes on the values inside the object, and not its serialization. They are unrelated concepts, and it helps to keep the concepts separate in the implementation. But for the purposes of routing in a distributed system - which is what the post is about - it's an acceptable solution.
        14y ago
- zmb_ 14y ago
  Object.hashCode() typically returns the memory address of the object, obviously it won't be the same across processes. This should be common knowledge for all Java programmers. The only way to do what the author wants is to define the serialization rules and the hash function the serialized object is fed to. Again, this should be obvious to every Java programmer.
  - scott_s 14y ago
    And yet, the author claims that real distributed systems in production use make the mistake.
    - zmb_ 14y ago
      And isn't that scary? It's not like this is any kind of secret or obscure detail, it's something you would learn in any half-competent Java 101 class or if you bothered to read the docs to the methods you're calling.

scott_s 14y ago

I'm not sure if you understood the author correctly - your points don't disagree with him. He is not talking about using hashes for identity, but for routing. And he is advocating something more general than what you said. You said, basically, "implement your own hash functions for your objects." He said, basically, "don't use Object.hashCode(); you can easily just hash the serialization of the object, or implement a hash function yourself."

rickmode 14y ago
Stated another way: The contract Object.hashCode() does not require a stable results between processes; it _only_ requires a stable result during a single run of an application. So if you need a stable distributed hash, the article recommends using a different method than Object.hashCode(), for instance DistributedObject.stableHash(). Using Object.hashCode() can bite you since not all implementations will have stable over multiple runs.
- cbsmith 14y ago
  ...except that there is nothing that prevents you from having a stricter contract on hashCode() with your hashed objects... like EVERY SYSTEM does.
  - lmm 14y ago
    But not e.g. protobuf. So don't write a distributed routing framework that assumes hashCode() is stable - if you do, it will work fine for Strings and HashMaps and all the standard java objects, and then fail when someone tries to use it with protobuf objects.
    - cbsmith 14y ago
      I ended up posting something much longer on this: http://news.ycombinator.com/item?id=4131326 http://news.ycombinator.com/item?id=4131326 Your comment qualifies as being "wrong", because of one part: "...it will work fine for Strings and HashMaps and all the standard java objects...". In fact hashCode() is stable for a very small subset of the types that ship with standard Java, and even in some of those cases there are caveats. Arrays come pretty standard with Java, but... boolean alwaysFalse(int[] a) { return a.hashCode() == a.clone().hashCode(); }
      - lmm 14y ago
        a.clone().equals(a) is false too; you can't use Array as a thing to route on any more than you can use Object. The point is that in standard java objects with value semantics have stable hashCode implementations, and this is not true in general (And FWIW arrays are a weird corner of Java as far as I'm concerned; every codebase I've worked on avoids them as much as possible)
        cbsmith 14y ago
        > a.clone().equals(a) is false too. a.clone().equals(a) HAS to be false in that case. It's more than a bit of a given if a.hashCode() != a.clone().hashCode(). > you can't use Array as a thing to route on any more than you can use Object Oh, sure you can. You just need to use java.util.Arrays's methods for "equals" and "hashCode" (or potentially deepHashCode). Also, ArrayList does work. Yes, that is totally F'd up. > The point is that in standard java objects with value semantics have stable hashCode implementations, and this is not true in general Value semantics are pretty rare in Java. I'm not even sure what you really mean there. Pass-by-value is doesn't happen with Java objects. I could see you referring to immutable types, but that doesn't include any of the collection classes. I'm guessing you mean "primitive type wrappers and collections". It'd make more sense if you said something like, "pretty much anything that overrides Object.equals(Object)"... because that's the way it is supposed to work. They are rare in standard class library, because there is little business logic there. In practice, anything that resembles an identifier, and therefore all keys, tends to do the override though. Indeed, most of what people tend to call "business objects" tend to do the override. That's why the convention is there. Most importantly: almost all overrides do so in a fashion that is stable across processes. That's also why distributed frameworks can and should employ the convention/protocol. That equals/hashCode methods in Object are following the Smalltalk trick of having protocol defined in Object even though you shouldn't really use it without subclassing. The Object method isn't a "reference implementation" of the protocol, but rather a placeholder (one they forgot to override in Array objects, and then tried to backdoor in with java.util.Arrays).
cbsmith 14y ago
Except he cited specific examples of distributed systems (Hadoop, Storm) that use hashCode() but also make it quite clear you need to ensure you provide a hashCode() implementation that is entirely derived from the state of the object which is preserved in serialized form. Seriously, it's just a bad article.
- scott_s 14y ago
  No, not need to. He said it was convenient if you're already serializing the object anyway. Which, if you're routing an object in a distributed system, is probably true.
  - cbsmith 14y ago
    Yes, it is terribly convenient because if Type.hashCode() has problems with copies of objects, this will help: int computeHash(byte[] serializedBytes) { return serializeBytes.hashCode(); } Now you've gone from a hashCode() method that won't work if the programmer screwed the pooch, to a hashCode() method that won't work period. ;-) [Yes, it won't be screwed up if also you use your own hash.] The problem with serialization is you lose the semantics of the object. The "same values to the same sequence of bytes" is a tighter constraint than you might think. The same "values" might have multiple representations which are logically identical but might still be relevant in any number of ways (for example, additional information that has nothing to do with identity).
    - scott_s 14y ago
      [Yes, it won't be screwed up if also you use your own hash.] It's strange that you disregard that in an aside, as it's his entire point. Personally, I prefer to implement hashes on the values inside the object, and not its serialization. They are unrelated concepts, and it helps to keep the concepts separate in the implementation. But for the purposes of routing in a distributed system - which is what the post is about - it's an acceptable solution.
      - 14y ago
zmb_ 14y ago
Object.hashCode() typically returns the memory address of the object, obviously it won't be the same across processes. This should be common knowledge for all Java programmers. The only way to do what the author wants is to define the serialization rules and the hash function the serialized object is fed to. Again, this should be obvious to every Java programmer.
- scott_s 14y ago
  And yet, the author claims that real distributed systems in production use make the mistake.
  - zmb_ 14y ago
    And isn't that scary? It's not like this is any kind of secret or obscure detail, it's something you would learn in any half-competent Java 101 class or if you bothered to read the docs to the methods you're calling.