krka

4 ms·

1.
by krka 3y ago
Now when I'm at an actual computer, I see that it was implemented, at least in the Java version. Removing that made for a 30% speed increase: https://github.com/spotify/sparkey-java/pull/65
2.
by krka 3y ago
True, if the lookup actually implemented this, it would be more divisions. This might partly be why it did not get implemented at all. With a sparse index (25% empty capacity) we should quickly run into empty slots anyway in most cases, and
3.
by krka 3y ago
I haven't worked on this code in a long time but I will try to answer based on what I remember. (Also on phone without easy access to the code) The idea is that all entries have a natural ideal position in the table and the displacemen
4.
by krka 13y ago
Could be due to a bug related to reading uninitialized data on the stack. That could lead to using the wrong number of bits for the hash, causing an unnecessarily high number of hash collisions, which makes it more expensive due to false p
5.
by krka 13y ago
Thanks, that's interesting data! I am not sure why you changed the key format to "key_%09d" - is that an optimization for lmdb, to make sure the insertion order is the same as the internal tree ordering? If so, why is that ne
6.
by krka 13y ago
Well, some DB's "load" the value in some way before giving it to the user, so that time is implicitly measured for those types of DB's but not for others, so I don't think it's a particularly fair comparison. I
7.
by krka 13y ago
Yes, this post was handwaving, and I tried to make that clear ("preliminary", "extremely biased"). On monday I added some slightly more proper benchmark code, you can find it on https://github.com/spotify
8.
by krka 13y ago
I have now created a very simple benchmark suite to give you some rough performance numbers, and updated the README to include some sample numbers for one specific machine.
9.
by krka 13y ago
Bam looks really interesting, definitely a lot simpler than Sparkey, and the basic principle is the same. I have been hesitant to use perfect hashing for Sparkey since I wasn't sure how well it holds up for really large data sets (clos
10.
by krka 13y ago
Here are some preliminary performance benchmarks from my regular workstation (Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz, 8 GB RAM): http://pastebin.com/7buZVgdu The sparkey usage is fairly optimized, but I just randomly put s
11.
by krka 13y ago
There are a bunch of comments related to performance data - the code is available so nothing is stopping anyone from making an unbiased comparison. :) That said, I intend to publish some sort of performance comparison code / results. T
12.
by krka 13y ago
Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.