4 ms·

Considering that 43% of the words (in my dictionary at least) include "s" it's not so strange: $ cat /usr/share/dict/words | wc -l 235976 $ cat /

Considering that 43% of the words (in my dictionary at least) include "s" it's not so strange: $ cat /usr/share/dict/words | wc -l 235976 $ cat /usr/share/dict/words | grep s | wc -l 101882

filleokus 2y ago
GP meant that all collisions contained 's - a possessive apostrophe (if I remember my grammar correctly?). oncogene's, toadstool's, Carib's. The /usr/share/dict/words on my macOS machine doesn't seem to have any 's words in it.
- vlovich123 2y ago
  counterclaiming Doesn’t have any s and stemming doesn’t have any possessive. I think drawing conclusions from a 32bit hash against a 123k word list is fruitless. You’d expect at least 1 collision due simply to birthday paradox.
  - airstrike 2y ago
    oncogene's <> counterclaiming ingrates <> toadstool's Carib's <> stemming all 3 pairs have one word ending in "'s", which I think is what the GGGP (?) was pointing out
    - bravura 2y ago
      Yes. But I didn't realize that 43% of the words in that dictionary contain "'s". In that case, the probability that, for 3 collisions, there is at least one element in every collision contains "'s" is 31%. That seems reasonable to me.
      - airstrike 2y ago
        but 43% of the words contain s, not 's
  - zimpenfish 2y ago
    > counterclaiming Doesn’t have any s and stemming doesn’t have any possessive. But "oncogene's" (collides with counterclaiming) and "Carib's" (collides with stemming) do. Also I did point out that the 64 and 128 bit variants have no collisions.
    - vlovich123 2y ago
      Yeah, but it's an uninteresting test. As others point out, 32-bit XXH3 has the same issue. It wouldn't surprise me if 32-bit AES did too. It has nothing to do with the quality of the hash algorithm & more to do with the fact that you're not building a perfect hash with such a small hash size.
      - zimpenfish 2y ago
        > Yeah, but it's an uninteresting test. On its own, yes, fair, but it was combined with the 64 and 128 bit variants which aren't uninteresting. > As others point out, 32-bit XXH3 has the same issue. Yes, me. > It has nothing to do with the quality of the hash algorithm The original post asked "I would like to see the collision distribution on /usr/share/dict/words." and that is what they got a demonstration of - some collisions on 32 bit, none on 64 or 128 bit. What else could be done to satisfy their original request in a way that would be interesting to you?
        vlovich123 2y ago
        Point to the fact that it passes smasher in addition to that 64-bit and 128-bit variants don’t have collisions and ignore 32-bit altogether. I don’t know why libraries even bother producing 32-bit variants at this point because of how easy a collision is.

Considering that 43% of the words (in my dictionary at least) include "s" it's not so strange: $ cat /usr/share/dict/words | wc -l 235976 $ cat /usr/share/dict/words | grep s | wc -l 101882

filleokus 2y ago

GP meant that all collisions contained 's - a possessive apostrophe (if I remember my grammar correctly?). oncogene's, toadstool's, Carib's. The /usr/share/dict/words on my macOS machine doesn't seem to have any 's words in it.

vlovich123 2y ago
counterclaiming Doesn’t have any s and stemming doesn’t have any possessive. I think drawing conclusions from a 32bit hash against a 123k word list is fruitless. You’d expect at least 1 collision due simply to birthday paradox.
- airstrike 2y ago
  oncogene's <> counterclaiming ingrates <> toadstool's Carib's <> stemming all 3 pairs have one word ending in "'s", which I think is what the GGGP (?) was pointing out
  - bravura 2y ago
    Yes. But I didn't realize that 43% of the words in that dictionary contain "'s". In that case, the probability that, for 3 collisions, there is at least one element in every collision contains "'s" is 31%. That seems reasonable to me.
    - airstrike 2y ago
      but 43% of the words contain s, not 's
- zimpenfish 2y ago
  > counterclaiming Doesn’t have any s and stemming doesn’t have any possessive. But "oncogene's" (collides with counterclaiming) and "Carib's" (collides with stemming) do. Also I did point out that the 64 and 128 bit variants have no collisions.
  - vlovich123 2y ago
    Yeah, but it's an uninteresting test. As others point out, 32-bit XXH3 has the same issue. It wouldn't surprise me if 32-bit AES did too. It has nothing to do with the quality of the hash algorithm & more to do with the fact that you're not building a perfect hash with such a small hash size.
    - zimpenfish 2y ago
      > Yeah, but it's an uninteresting test. On its own, yes, fair, but it was combined with the 64 and 128 bit variants which aren't uninteresting. > As others point out, 32-bit XXH3 has the same issue. Yes, me. > It has nothing to do with the quality of the hash algorithm The original post asked "I would like to see the collision distribution on /usr/share/dict/words." and that is what they got a demonstration of - some collisions on 32 bit, none on 64 or 128 bit. What else could be done to satisfy their original request in a way that would be interesting to you?
      - vlovich123 2y ago
        Point to the fact that it passes smasher in addition to that 64-bit and 128-bit variants don’t have collisions and ignore 32-bit altogether. I don’t know why libraries even bother producing 32-bit variants at this point because of how easy a collision is.