4 ms·

The author seems to imply he reimplemented hash tables. I was under the impression there was a fairly good number of high quality implementations of hashes tab

The author seems to imply he reimplemented hash tables. I was under the impression there was a fairly good number of high quality implementations of hashes tables in the wild and that it was a thoroughly studied subject. "Not invented here" syndrome?

deleted 15y ago
[deleted]
ig1 15y ago
It's pretty common for banks to implement their own hashtables, the standard ones often make trade-offs that aren't the best choice for high-performance. For example with Java library hash tables if you have a high insert/removal rate you can end up with frequent GCs. Another common example is to do with synchronization code, unless your synchronization requirements are exactly those provided by the library then the overhead added by the synchronization code can have a significant performance impact.
nandemo 15y ago
Well, he didn't write about hash tables in this post... Anyway, even if there are good hash table implementations, sometimes you have to provide a custom hash function. A default hash function will not necessarily give you a table without collisions.
- shin_lao 15y ago
  There's a teaser on the matter in the end. You're right, generally the work revolves about writing the right hash function. We'll see in the next article.
veyron 15y ago
For components not in the critical path, third party implementations are acceptable. However, it is not surprising to see every piece in the critical path to be written in-house (feed handlers, for which there are many companies selling solutions) or rebuilt with heavy modifications (kernel). I had intended to say hash function, but it was very late by the time I posted.
- orenmazor 15y ago
  I can fully understand it. it's the same in the technical startups that I've been involved with (i.e. hardware or science): - build a quick mvp with off the shelf stuff - start optimizing. by the time you're done, most of your parts are no longer generic and have optimizations where you need them.
- shin_lao 15y ago
  Looking forward to reading your next post then. We've worked a lot on fast data access and I wonder if you outperformed us.
  - veyron 15y ago
    Do you have any latency numbers you could share? :)
    - shin_lao 15y ago
      We don't directly do HFT, we work with banks to help them do that well. All the numbers I could give you is what our data engine (which is what we sell) gives on our test environment, I cannot share what our customers actually get once deployed and configured on their system. Generally, they are happy. :) We're going to post benchmarks soon as we're going out of stealth mode.

deleted 15y ago

[deleted]

ig1 15y ago

It's pretty common for banks to implement their own hashtables, the standard ones often make trade-offs that aren't the best choice for high-performance. For example with Java library hash tables if you have a high insert/removal rate you can end up with frequent GCs. Another common example is to do with synchronization code, unless your synchronization requirements are exactly those provided by the library then the overhead added by the synchronization code can have a significant performance impact.

nandemo 15y ago

Well, he didn't write about hash tables in this post... Anyway, even if there are good hash table implementations, sometimes you have to provide a custom hash function. A default hash function will not necessarily give you a table without collisions.

shin_lao 15y ago
There's a teaser on the matter in the end. You're right, generally the work revolves about writing the right hash function. We'll see in the next article.

veyron 15y ago

For components not in the critical path, third party implementations are acceptable. However, it is not surprising to see every piece in the critical path to be written in-house (feed handlers, for which there are many companies selling solutions) or rebuilt with heavy modifications (kernel). I had intended to say hash function, but it was very late by the time I posted.

orenmazor 15y ago
I can fully understand it. it's the same in the technical startups that I've been involved with (i.e. hardware or science): - build a quick mvp with off the shelf stuff - start optimizing. by the time you're done, most of your parts are no longer generic and have optimizations where you need them.
shin_lao 15y ago
Looking forward to reading your next post then. We've worked a lot on fast data access and I wonder if you outperformed us.
- veyron 15y ago
  Do you have any latency numbers you could share? :)
  - shin_lao 15y ago
    We don't directly do HFT, we work with banks to help them do that well. All the numbers I could give you is what our data engine (which is what we sell) gives on our test environment, I cannot share what our customers actually get once deployed and configured on their system. Generally, they are happy. :) We're going to post benchmarks soon as we're going out of stealth mode.