7 ms·

What's Hashing? (2022)

40 points by ctxc 3y ago

vlmutolo 3y ago
> Examples of popular cryptographic hash algorithms: SHA, BCrypt, MD5. It's worth noting that MD5 is no longer considered a cryptographically secure hash function.
- trevithick 3y ago
  From later in the article: > MD5 is considered a weak hashing algorithm these days; we use BCrypt, which is stronger.
- dspillett 3y ago
  SHA1 too for that matter. Other members of the SHA family are still fine to use.
- Genbox 3y ago
  Also, technically bcrypt is not a general hash algorithm, but rather a password-hash construction. Where SHA1 and MD5 are primitives that can be used in arbitrary crypt constructions, bcrypt is already a construction specialized for passwords. You wouldn't (or at least shouldn't) hash a file with bcrypt.
vore 3y ago
The example is… not the best: if you’re checking if two pages are identical, you should match up each byte and compare them byte for byte. Hashing just adds an extra step: you’re going to need to read every byte anyway!
- RichieAHB 3y ago
  Given this is for a non-tech audience I think this is reasonable. It basically talks about it being a lossy encoding of things that represent the original. Also, if the words / chars were stored as an array or string already you wouldn’t need to read every byte.
- js2 3y ago
  Yup. A better example might be something like assigning all the letters a number value (a 1, b 2, ...) and then summing all of them. It's important that the analogy be simple to comprehend but it should still be, well, analogous to the real process.
  - ctxc 3y ago
    Hi js2! I do it in the immediate next section, "Improving the algorithm" (take all the letters and assign a number value). Do you think there's a way I can improve that part or say that better? I try to ramp it up from a loose anology to slowly model the actual working (although I'm no expert on that!)
- Kalium 3y ago
  The webpage example does a good job of setting the stage for passwords. You definitely do not want to compare passwords that way unless you are trying to create a textbook timing vulnerability.
  - ctxc 3y ago
    Hi Kalium, could you elaborate? Did you mean my password hashing example?
    - dspillett 3y ago
      Your password example is fine - Kalium was commenting on vore's suggestion that for small data a byte-for-byte compare would work just as well.
      - ctxc 3y ago
        Ahh, got it, thanks :D
- ctxc 3y ago
  Hello Vore! Yes, of course. I wrote this for a few non technical friends - a simple example they can relate to (like we skim docs) seemed like the best way to start off without scaring them off the article :P
- dspillett 3y ago
  > you’re going to need to read every byte anyway! But you might not need to read them at your current location, though that consideration might be overkill for the level this article is aimed at. For instance reading a full remote file to compare it to a local one byte-for-byte will be slower than sha256sum some.file; ssh user@host sha256sum some.file or cmp <(sha256sum some.file) <(ssh user@host sha256sum some.file) unless the file is very small (in which case a simple download by HTTP(S) or read via network filesystem may be faster than negotiating a SSH connection and running the remote hash command). In either case accessing a pre-computed hash of the remote content will likely be faster than both options, assuming a stored hash exists and you are looking to verify your local copy.
heeton 3y ago
If you're an alcoholic brit, hashing can also mean something else. https://guildfordh3.org.uk/what-is-hashing/ https://guildfordh3.org.uk/what-is-hashing/
- ctxc 3y ago
  Wow, TIL xD
- headwayoldest 3y ago
  There are kennels in the states as well, if you've ever wondered what it would be like to run through a swamp with a bag of Franzia
  - pjot 3y ago
    While also wearing a red dress in New Orleans right?
- Bloating 3y ago
  On, On! Directory of HHH, including the US https://www.gotothehash.net/ https://www.gotothehash.net/
ctxc 3y ago
Hello there! Author here. Just a small guide for non-technical people about hashing - a small seemingly scary word, but understanding that could potentially make so many other things click. Please let me know what you think and check out other articles as well! One to get you started: https://blog.dvsj.in/obfuscation/ https://blog.dvsj.in/obfuscation/
kazinator 3y ago
This is misleading because it only covers message digest hashing, presenting that as being the definition of hashing "what hashing is". The other important hashing is the reduction of an object of any kind (including, but not limited to texts) being used as a key, to a numeric code (usually fitting into a single machine word) to be used as an index for fast look up in a "hash table" data structure. This is related to testing whether two things are the same, but the security aspect isn't important. More important is the aspect that the hash code is exploited as an integer in order to jump to the right bucket or starting point in the hash table. Hash tables can be on disk, not only in memory.
kazinator 3y ago
By the way, the credit for hashing (the indexing technique for quickly finding items in a data set, not message digests) belongs to some person in China. I never see this mentioned. For instance, look at the Wikipedia page on hashing; mainly some 20th century white dudes are mentioned there, and one A. D. Linh (Vietnamese-looking name) is credited with open addressing. https://en.wikipedia.org/wiki/Hash_table#History https://en.wikipedia.org/wiki/Hash_table#History However, decades before that, Chinese character dictionaries used step-by-step algorithms for finding chinese characters in a dictionary, which are identical to hashing. First you examine the character according to a list of precise rules, which have various cases. The cases assign numbers to parts of the character according to structural features, and those numbers are then combined to form a code, such as the "four corner code". From the code, you can proceed directly to a page of the dictionary, where you search through a short list of characters that have the same code. Unless there is prior art for that, that is where the credit lies. https://en.wikipedia.org/wiki/Four-Corner_Method#Origin https://en.wikipedia.org/wiki/Four-Corner_Method#Origin "The Four-Corner Method was invented in the 1920s by Wang Yunwu, the editor in chief at Commercial Press Ltd., China. It was based on experiments by Lin Yutang and others." The significant thing about the prior art also is that it was hashing non-textual, but rather graphical objects. A chinese character is text, of course, but the hashing method treats the features graphically. It's not looking at the etymologically correct radicals and components and using their individual code values, but purely visual features like crossings, boxes and dots. Before the Four Corner method, there were existing numeric lookup methods for dictionaries, like using the 214 Kangxi radicals. That also a kind of proto-hashing method. However, the hashing function is difficult (you have to already know which part of the character is considered the radical and which radical that is under variation). The number of buckets is pretty large, resulting in long searches. The Four Corner Method is a refinement: you can hash any character without knowing what it its radical, and the code space is decently large to reduce the searching through collisons.
- bux93 3y ago
  Digital roots were known to Pythagoras. According to https://medium.com/@beeezy/pythagoras-hacked-your-password-802a027796ed https://medium.com/@beeezy/pythagoras-hacked-your-password-8... they're also mentioned in "Indian Vedic math". Seems like prior art for a numeric hash.
unethical_ban 3y ago
What is the simplest hash algorithm to understand conceptually/mathematically? The equivalent of XOR with a one time pad level of math, or a replacement scheme for encryption? (In other words, doesn't have to be secure, but covers the concept)
- fluoridation 3y ago
  The first thing that comes to mind is adding all the characters in the input. It's easy to understand how if the values for two inputs are different then necessarily the two inputs are different.
- pestatije 3y ago
  in some cases the identity hash(tat is, no hash) is good enough, eg. integers, fixed-length strings, etc
dartvox 3y ago
[dead]
WaitWaitWha 3y ago
checksum =/= hash They are arrived to them differently, and provide different purpose.
jazzyjackson 3y ago
I've been hashing files for years but just recently heard it used by an old radio geek use it as slang for noise, which suddenly made a lot of sense etymologically - a hash function takes structure and turns it into noise. Onions and potatoes optional.

40 points by ctxc 3y ago

> Examples of popular cryptographic hash algorithms: SHA, BCrypt, MD5. It's worth noting that MD5 is no longer considered a cryptographically secure hash function.

trevithick 3y ago
From later in the article: > MD5 is considered a weak hashing algorithm these days; we use BCrypt, which is stronger.
dspillett 3y ago
SHA1 too for that matter. Other members of the SHA family are still fine to use.
Genbox 3y ago
Also, technically bcrypt is not a general hash algorithm, but rather a password-hash construction. Where SHA1 and MD5 are primitives that can be used in arbitrary crypt constructions, bcrypt is already a construction specialized for passwords. You wouldn't (or at least shouldn't) hash a file with bcrypt.

vore 3y ago

The example is… not the best: if you’re checking if two pages are identical, you should match up each byte and compare them byte for byte. Hashing just adds an extra step: you’re going to need to read every byte anyway!

RichieAHB 3y ago
Given this is for a non-tech audience I think this is reasonable. It basically talks about it being a lossy encoding of things that represent the original. Also, if the words / chars were stored as an array or string already you wouldn’t need to read every byte.
js2 3y ago
Yup. A better example might be something like assigning all the letters a number value (a 1, b 2, ...) and then summing all of them. It's important that the analogy be simple to comprehend but it should still be, well, analogous to the real process.
- ctxc 3y ago
  Hi js2! I do it in the immediate next section, "Improving the algorithm" (take all the letters and assign a number value). Do you think there's a way I can improve that part or say that better? I try to ramp it up from a loose anology to slowly model the actual working (although I'm no expert on that!)
Kalium 3y ago
The webpage example does a good job of setting the stage for passwords. You definitely do not want to compare passwords that way unless you are trying to create a textbook timing vulnerability.
- ctxc 3y ago
  Hi Kalium, could you elaborate? Did you mean my password hashing example?
  - dspillett 3y ago
    Your password example is fine - Kalium was commenting on vore's suggestion that for small data a byte-for-byte compare would work just as well.
    - ctxc 3y ago
      Ahh, got it, thanks :D
ctxc 3y ago
Hello Vore! Yes, of course. I wrote this for a few non technical friends - a simple example they can relate to (like we skim docs) seemed like the best way to start off without scaring them off the article :P
dspillett 3y ago
> you’re going to need to read every byte anyway! But you might not need to read them at your current location, though that consideration might be overkill for the level this article is aimed at. For instance reading a full remote file to compare it to a local one byte-for-byte will be slower than sha256sum some.file; ssh user@host sha256sum some.file or cmp <(sha256sum some.file) <(ssh user@host sha256sum some.file) unless the file is very small (in which case a simple download by HTTP(S) or read via network filesystem may be faster than negotiating a SSH connection and running the remote hash command). In either case accessing a pre-computed hash of the remote content will likely be faster than both options, assuming a stored hash exists and you are looking to verify your local copy.

heeton 3y ago

If you're an alcoholic brit, hashing can also mean something else. https://guildfordh3.org.uk/what-is-hashing/ https://guildfordh3.org.uk/what-is-hashing/

ctxc 3y ago
Wow, TIL xD
headwayoldest 3y ago
There are kennels in the states as well, if you've ever wondered what it would be like to run through a swamp with a bag of Franzia
- pjot 3y ago
  While also wearing a red dress in New Orleans right?
Bloating 3y ago
On, On! Directory of HHH, including the US https://www.gotothehash.net/ https://www.gotothehash.net/

ctxc 3y ago

Hello there! Author here. Just a small guide for non-technical people about hashing - a small seemingly scary word, but understanding that could potentially make so many other things click. Please let me know what you think and check out other articles as well! One to get you started: https://blog.dvsj.in/obfuscation/ https://blog.dvsj.in/obfuscation/

kazinator 3y ago

This is misleading because it only covers message digest hashing, presenting that as being the definition of hashing "what hashing is". The other important hashing is the reduction of an object of any kind (including, but not limited to texts) being used as a key, to a numeric code (usually fitting into a single machine word) to be used as an index for fast look up in a "hash table" data structure. This is related to testing whether two things are the same, but the security aspect isn't important. More important is the aspect that the hash code is exploited as an integer in order to jump to the right bucket or starting point in the hash table. Hash tables can be on disk, not only in memory.

kazinator 3y ago

By the way, the credit for hashing (the indexing technique for quickly finding items in a data set, not message digests) belongs to some person in China. I never see this mentioned. For instance, look at the Wikipedia page on hashing; mainly some 20th century white dudes are mentioned there, and one A. D. Linh (Vietnamese-looking name) is credited with open addressing. https://en.wikipedia.org/wiki/Hash_table#History https://en.wikipedia.org/wiki/Hash_table#History However, decades before that, Chinese character dictionaries used step-by-step algorithms for finding chinese characters in a dictionary, which are identical to hashing. First you examine the character according to a list of precise rules, which have various cases. The cases assign numbers to parts of the character according to structural features, and those numbers are then combined to form a code, such as the "four corner code". From the code, you can proceed directly to a page of the dictionary, where you search through a short list of characters that have the same code. Unless there is prior art for that, that is where the credit lies. https://en.wikipedia.org/wiki/Four-Corner_Method#Origin https://en.wikipedia.org/wiki/Four-Corner_Method#Origin "The Four-Corner Method was invented in the 1920s by Wang Yunwu, the editor in chief at Commercial Press Ltd., China. It was based on experiments by Lin Yutang and others." The significant thing about the prior art also is that it was hashing non-textual, but rather graphical objects. A chinese character is text, of course, but the hashing method treats the features graphically. It's not looking at the etymologically correct radicals and components and using their individual code values, but purely visual features like crossings, boxes and dots. Before the Four Corner method, there were existing numeric lookup methods for dictionaries, like using the 214 Kangxi radicals. That also a kind of proto-hashing method. However, the hashing function is difficult (you have to already know which part of the character is considered the radical and which radical that is under variation). The number of buckets is pretty large, resulting in long searches. The Four Corner Method is a refinement: you can hash any character without knowing what it its radical, and the code space is decently large to reduce the searching through collisons.

bux93 3y ago
Digital roots were known to Pythagoras. According to https://medium.com/@beeezy/pythagoras-hacked-your-password-802a027796ed https://medium.com/@beeezy/pythagoras-hacked-your-password-8... they're also mentioned in "Indian Vedic math". Seems like prior art for a numeric hash.

unethical_ban 3y ago

What is the simplest hash algorithm to understand conceptually/mathematically? The equivalent of XOR with a one time pad level of math, or a replacement scheme for encryption? (In other words, doesn't have to be secure, but covers the concept)

fluoridation 3y ago
The first thing that comes to mind is adding all the characters in the input. It's easy to understand how if the values for two inputs are different then necessarily the two inputs are different.
pestatije 3y ago
in some cases the identity hash(tat is, no hash) is good enough, eg. integers, fixed-length strings, etc

dartvox 3y ago

[dead]

WaitWaitWha 3y ago

checksum =/= hash They are arrived to them differently, and provide different purpose.

jazzyjackson 3y ago

I've been hashing files for years but just recently heard it used by an old radio geek use it as slang for noise, which suddenly made a lot of sense etymologically - a hash function takes structure and turns it into noise. Onions and potatoes optional.