5 ms·
I've never quite understood why banks still opt for kdb when (a) hiring staff is ridiculously difficult/expensive and (b) there are so many other cheaper altern
by osrec 9y ago
I've never quite understood why banks still opt for kdb when (a) hiring staff is ridiculously difficult/expensive and (b) there are so many other cheaper alternatives available. They literally have a strangle hold on every eFX desk in London, and I cannot really figure out why!
- geocar 9y agoMaybe you're wrong about (b)? Sure, I could cobble something about as fast and functional out of postgres, C, awk, and a bunch of other things, but it'd certainly cost more just in developer time than kdb.
- osrec 9y agoYour comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are many open source offerings, which are now matching it both in terms of performance and flexibility. Plus hiring good people is significantly easier and cheaper.
- dogruck 9y agoAs an ex-MD you also know that banks used to not be too price sensitive, so they didn't mind paying a lot for some Kx experts. Of course, these days, with revenues being squeezed, that's changed.
- osrec 9y agoAgree, price is a much bigger factor now.
- oper8or 9y agoWhat are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.
- alfiedotwtf 9y agoEveryone is hanging on for OP to deliver
- kthielen 9y agoThe one I linked to in my previous post also produces code that's fast enough to use in the critical path in low-latency trading systems. As you may know, q's "scalar performance" is not great; similar to Scheme or python due to boxing overhead (as you can see in the linked k.h file below). Also, the fact that q is untyped has a severe impact on its safe use in large and complex projects. (ETA: https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes)
- osrec 9y agoThere are a few. I've used a modified LMDB source in the past with success, employing similar tricks to kdb for performance (i.e. store daily data as contiguous arrays so that reads are quick etc). Either way, implementing a memory mapped store of arrays and operating on it is not too challenging a problem for any good C++ dev.
- oper8or 9y agoThank you for your answers. To summarize, we have 1) Hobbes 2) hacked LMDB 3) C++ memory-mapped store of arrays. Given that options #2 and #3 require some (non-trivial) work, they are not really options. We left with #1,-- hobbes, which was uploaded to GitHub about 5 months ago and has a whopping team of 2 contributors, both employed by Morgan Stanley. This is more than nothing, but not much. I do not have experience with KDB, and looking at the language syntax, not a fan. Integration with Python (depending on implementation) may push KDB towards larger acceptance. So far I was mostly relying on a variation of the option #3.
- kthielen 9y ago
- deleted 9y ago[deleted]
- yalph 9y agoSo what are those alternatives?
- kthielen 9y agoHow about hobbes?
- yalph 9y agoI will check thanks but wnated to learn from him. I do not think there are many alternatives by the way. Is this one a full alternative? Does it provide all functionalities of kdb?
- kthielen 9y agoIt’s better in a lot of ways. It has a type system, can produce much better code, can do cross-process compilation (for multi-process IPC). The FFI/binding process is much simpler, there are more options to record precisely-typed data from applications. And hobbes is used in major high-volume and low-latency systems at Morgan Stanley (where q originated, as you may know).
- yalph 9y agoHow come kx is ok with this one? I heard they sue every moving object around.
- kthielen 9y agoKx does seem to bully people with threats of lawsuits, not sure how often they actually follow through. hobbes is not a k/q clone, it's much more like Haskell actually. The features that make hobbes especially compelling for its production use-cases, like its complex type system, are features that kdb has never had and probably never will have.
- yalph 9y agoHi thank you very much, I work in the industry and I will definitely take a look at it. You seem to be one of the developers behind this project. Is there any way I can contact you in the future?
- kthielen 9y agohttps://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes
- dilap 9y agohuh, so what do i need to learn to be one of these ridiculously expensive people working w/ kdb?
- ah- 9y agoI'd start with https://code.kx.com/q4m3/ https://code.kx.com/q4m3/
- osrec 9y agoAlso, you need to be lucky enough to somehow move into a team that uses kdb heavily (this usually happens via an internal move within an investment bank). Learning it using the tutorials is not the same as implementing it in a live environment, and the tutorials probably won't be enough to get you a job/contract. If you could say something like "I've worked on the FX desk at JP Morgan, where we used kdb daily", you'll be hired everywhere. Basically, you need a bit of luck in the first instance to get into a kdb focused team that can train you - after that, you're sorted!
- beagle3 9y agoHistorically, an APL background was a good indication that you grok the apl/k/j/q mindset, which is very different from the Algol (c/c#/Java/pascal) world and also from the Lisp world.