7 ms·
Show HN: Tifuhash - Tiny Fast Universal Hash, using 64-bit continued fractions
- johnklos 9y ago"Uses two 64 bit floating point integers for calculation"? Floating point integers?
- 19eightyfour 9y agoI'll correct it, thank you.
- bhaak 9y agoMy snarky self woke up and said "That's an appropriate name for the integer simulation of JavaScript".
- lloeki 9y agoI read it as "JS has no ints"
- rjeli 9y agoSorry, I can only read this as "Today I Fucked Up hash"
- AstralStorm 9y agoIt probably is. So what is wrong with say MurmurHash that you have to hack your own?
- jimktrains2 9y agoYeah, it's one thing to create it for fun, but why publish it as a library for general consumption?
- rjeli 9y agoI strongly disagree with this idea - code should always be released! If it's good, we can avoid rewriting code and have more options for hashing, and if some people point out problems with the code, the author can learn from their mistakes and fix them if possible - they wouldn't want to use it in their private code with the mistakes. I think this thread is more negative than the author deserves. The method of hashing is novel and could possibly lead to exploration in the area, and if not that, where else could the author post to get feedback from the author of murmurhash?? I think the only problem here is that they advertise it as production ready/universal without evidence or rigorous testing, rather than "hey, here's a cool hash I came up with."
- AstralStorm 9y agoIndeed. That claim requires hard statistics on performance on multiple platforms and statistics of the hash function itself. The floating point hash might be useful on a CPU for example.
- jimktrains2 9y ago> I strongly disagree with this idea - code should always be released! There is a difference between releasing code, say on github, and putting it out there like it's ready for use in a production system, like in a package manager. I only said the later shouldn't be done, not the former. Rereading, I guess that wasn't exactly what I said. I can't seem to edit now, though.
- 19eightyfour 9y agoThanks for sticking up for me, rjeli, I really appreciate that!
- 19eightyfour 9y agoI like making things. I want to have an impact. I want to do something new. I want people to try and use my stuff and tell me about it. So I'm achieving that today. Make your own hash function. It's not that hard. Most non crypto hash functions are just something similar to ax + b mod p. And then there's plenty of room left for exotic ones. There's room for yours, too. Personally I've for a long time really liked making RNGs. Many people like making them actually. For me, I always liked the feeling of getting something simple that i made create so much uncompressible apparent entropy out of so little. And i enjoy the challenge of passing the test suites for randomness. And i enjoy the feeling of making tiny mixing functions. I like the diagrams of mixing functions and ciphers. I like the aesthetics of writing that code, both that which is similar to the main habits of many ciphers, the xors, the shifts, as well as coming up with something new. A lot of people like making hashes, and RNGs, and there are a lot of them, and a lot of good, useful and interesting ones. All made by people no smarter than me. I'm fine to get up in front of you all here and try and do something i like, and fall and succeed, and learn. I just enjoy what I'm doing. I'm creative and can come up with new things. I like doing it. Coding is one way i can express that creativity in a way that satisfies me. And i feel more satisfied when i share what i make it with others, and i have more fun when i take it more seriously. I like a challenge. And i think i can do something good with this space. And i want to. And I'm fine to fail too. I like to succeed. And I'm fine to fail. I just like doing it. I like doing more than i like sitting on the sidelines and commenting on other people doing, and secretly wishing i was on the field doing something too, instead of in the stands just having opinions. That always felt fake to me... Because if i secretly wanted to do what someone else was doing, I wouldn't get jealous, I'd just say to myself, how do I get that, too? Because, why not? So other peoples achievements inspired me to do, too, if i liked the same thing. If didn't like it, I wouldn't do it, but i could still enjoy it vicariously. So that's why I do stuff and share it with others and take it seriously. Hope that answers your question.
- 19eightyfour 9y agoI answered the why below. Also, i didn't feel there was anything really "wrong" with others. One motivation i had specifically apart from that paper i linked asking for a different way, was i was bored of seeing the same hash and type of code in hashes and ciphers again and again. And i had made a successful RNG, using a very simple construction, in a similar genre to this same regular genre, using the traditional operations like shifts and xors, and i felt i pretty much had nailed that type of construction to my satisfaction. So i wanted to do something different. A hash with a different aesthetic. A hash that, when you look at the code, it looks different to others. I didn't like that i just had success in this one genre of hash or rng code, and i wanted to do something just using arithmetic operations. It annoyed me that i had never really been able to find, or make, a good hash or rng without using shifts or xors or the same boring multiply stuff. When you see enough of them, it's all just derivative... Changing constants and so on...And the choices, this xor instead of that plus, are basically arbitrary because the space of possible algorithms using similar constructions is so huge, they're really not so different to each other, just more points in the same hyperplane, and there exist still so many other algorithms to find that way, a computer could basically fill in the rest of the dots. So instead of trying to discover a new point in parameter space of constructions like that, i wanted to get back to something more pure, that was more about the essence of a different construction, rather than a few more tweaks to the same type. Also, not just pure, but short. So many lines of shifts and xors are unnecessary...i know from deep and long experience, you don't need that much code to make so much good apparent entropy. I proved that by making a good rng, with a very simple construction. And I've written code like the sea of hashes and mixing functions hundreds of times... And really... It is like, change this shift, change this constant, change the construction, tweak until it works... But it's ask basically the same structure... It's just so boring after a while. I'm trying to convey to you the feeling that motivated me to do it, since that is what you asked. Hope it comes across clearly. Finally, i don't want this to come across as only a condemnation. I draw a lot of inspiration, ideas and information reading other people's ciphers, hashes, and papers. It's just, I don't want to copy the same style always. I want to diversify. I have a good mixing function in the traditional genre ( dosy, also on npm ), and I wanted to see if i could construct a good mixing function in a new aesthetic. And i have. Hooray! Tl;dr - aesthetics. I felt bored with what i had seen and done. I wanted to discover, make and see something new.
- StavrosK 9y agoYou aren't alone at all, that's exactly what I thought it was too.
- 19eightyfour 9y agoYou got me. There are some pretty serious issues with this.
- krisdol 9y agoI think they're only saying it because TIFU is a colloquial acronym for "Today I Fucked Up" (this is how I read it too), not intending to insult your abilities.
- rjeli 9y agodefinitely not a commentary on the work itself!
- 19eightyfour 9y agoActually "the today i ffff up hash" is a pretty cool name for tifuhash, like a backronym. It's probably more memorable / mimetic than "tiny fast universal". And memorable wad one of my original goals, but i was referring to the source code! As well as so many people naming it "today i ..." hash, there's the other comment saying i ought not use the technical term universal.... Maybe it works for me to take the hint and just change the name to today i fucked up hash. Somebody open a PR! I guess future versions can be named like "Wednesday i fucked up hash"... Or maybe now I'm just getting distracted by this name.
- 19eightyfour 9y agoThanks for saying so :)
- StavrosK 9y agoOh, sorry, I only understood what this comment meant after I read the child comment. Yeah, I got "Today I fucked up" from reddit's TIFU, not as a commentary on your abilities.
- pouta 9y agoCame here to say exactly this.
- dchest 9y ago> t.hash(0.00000000000000001) '0000000000000000' > t.hash(0.00000000000000002) '00000000ffffffff' > t.hash(0.00000000000000003) '00000000ffffffff' > t.hash(0.00000000000000004) '00000000ffffffff' > t.hash(0.00000000000000005) '00000000fdffffff' > 0.00000000000000001 > t.hash(1e17) '5555555597d44646' > t.hash(1e18) '55555555ac43d2d1' > t.hash(1e19) '55555555acd2b64f' > t.hash('\000') '0000000000000000' > t.hash('\000\000') '5555555592244992' > t.hash('\000\000\000') '0000000000000000' > t.hash('\001') '9224499200000000' > t.hash('\002') '33333333dedddddd'
- speps 9y agoSo your point is that the name of the hash is appropriate? Today I fucked up a hash function...?
- 19eightyfour 9y agoHaha. Well the name is good then. Since it will work everyday.
- aappleby 9y agoI was going to ask if he'd run it through SMHasher, but... yeah, clearly not. -Austin (author of murmurhash/smhasher)
- 19eightyfour 9y agoThank you, Austin. I'll look into SMHasher. Really appreciate you pointing me in that direction.
- 19eightyfour 9y agoAustin, I've done a first test of a slightly modified CPP version. Thanks very much for this library. It really is so useful. SMHasher fork testing tifuhash is here: https://github.com/dosaygo-coder-0/smhasher https://github.com/dosaygo-coder-0/smhasher And I posted the results in the repo: https://github.com/dosaygo-coder-0/tifuhash/blob/master/tifuhash.smhasher.results.txt https://github.com/dosaygo-coder-0/tifuhash/blob/master/tifu...
- davman 9y agoThis is connected with a truly amazing website that does not make my eyes bleed in any way. https://dosaygo.com/ https://dosaygo.com/
- 19eightyfour 9y agoThat could definitely be improved, too. Thanks
- Asooka 9y agoI'll say this though - the site is absolutely 1000% more usable than the single-page-app-that's-actually-a-document style du jour. Just tone down the colours and remove the horizontal scrollbar (put the two things one below the other). Other than that, this is definitely a motherfucking website :) ( http://motherfuckingwebsite.com/ http://motherfuckingwebsite.com/ )
- 19eightyfour 9y agoThank you for saying this. I am fond of my robust as nails site. I like that it can work on lynx. I shall take your improvement ideas under advisement.
- i336_ 9y agoI just grabbed the horizontal scrollbar (ThinkPad X61, no trackpad) to see what it was hiding, and slowly scrolled that part of the page over. It felt like I'd just hand cranked a CSS animation. Kinda steampunk. Note that I'm not dissing the author here. They appreciate the feedback they've gotten, which means they're receptive, and that's honestly a very large percentage of "this'll work out okay," which in this case means that this project will iterate until it nails something new, or the author will go join one of the other hashing efforts out there.
- 19eightyfour 9y agoThank you. Steampunk. I'm not gonna forget that! Sort of like those elaborate pop up books where you move the paper slider to move the figure across some background. :)
- lorenzhs 9y agoIn addition to the things others have pointed out, note that "universality" is a well-defined concept w.r.t. hash functions: https://en.wikipedia.org/wiki/Universal_hashing https://en.wikipedia.org/wiki/Universal_hashing - you should probably not use it to describe your hash function if you can't show universality.
- 19eightyfour 9y agoI think it is probably universal, but I have to show it to know for sure. I'm going to keep calling it universal until I know more. I think I note it in the README, but some reasons I think it is probably universal are because tifuhash can be parameterized, and it has good independence properties ( passes PractRand ). Showing it is universal or not is another step. I suppose I could experiment, to see if the collision probabilities match the criteria for universal or k-universal. Or maybe I could show it. For showing it, my next step is to read over this paper[1] to see if I can use its methods to show universality. I think it is using properties of the bits under multiplication. And I believe one avenue would be to show it using properties of the bits under division. I don't know how to approach the bits under division right now. [1]: https://arxiv.org/abs/1504.06804 https://arxiv.org/abs/1504.06804
- lorenzhs 9y agoI admire your confidence but I don't share it - showing universality for a complex construct like yours is rather nontrivial. You should really start by testing with SMHasher, though. Best of luck.
- 19eightyfour 9y agoOkay, I'll start with that next time, thanks
- 19eightyfour 9y agoMaybe i ought to have just said that the title includes the qualifier, " - that's the aim anyway," reinforcing the notion that even if we don't know if tifuhash is universal, it aims to be. So work is ongoing in that area, and you're welcome to contribute code or work on an approach to prove it is universal at some point. If you have more you'd like to contribute, you could become co author potentially. Please try to see it like that. That's the idea I'm going for here. Thanks
- deleted 9y ago[deleted]
- richdougherty 9y ago> Novel: using division, or using floating point division, and discarding the high-order-bits ( as we do here ) is not used nor studied so much, if at all, in hash construction Is that because division is slow compared to bit operations?
- 19eightyfour 9y agoI think it's something like that, that division is slower than multiplication in the chip. Also, a lot of hashing had to do with prime fields, and polynomials in them, and people seem to be able to do most of the maths without division. Or when they do divide, then are using powers of two, so thru can just use bit shifts, which are faster. So there's a lot of "discrete" type math in hashing...i don't see why we couldn't have more continuous or factional math. Afterall, Huffman coding is good, but arithmetic coding, which i believe uses fractional math, is better. Probably more hashing can be done the way that I'm doing here. So happy to find this new thing. I'm thinking FPGA... They do floating point right? GPU? Same right? I think continued Egyptian fraction hashing could be on the up.