6 ms·

Kdb+ and Python: EmbedPy and PyQ

99 points by gricardo99 9y ago

mclovinit345 9y ago
the idea that I could seamlessly move data between python and kdb+ is making me salivate. Please, make this work well and lower the cost-barrier to using kdb+ and I think you'd see this become the defacto "stack" for many database setups.
- ah- 9y ago
  All this needs now is a free (at least commercial use allowed 64bit version, ideally OSS) kdb+. I'd love to use this, many things are so much more straight forward in q than they are in pandas. But if it means noone can run my software unless they pay for kdb+ it's a non-starter. I wonder if this would work with Kona.
osrec 9y ago
I've never quite understood why banks still opt for kdb when (a) hiring staff is ridiculously difficult/expensive and (b) there are so many other cheaper alternatives available. They literally have a strangle hold on every eFX desk in London, and I cannot really figure out why!
- geocar 9y ago
  Maybe you're wrong about (b)? Sure, I could cobble something about as fast and functional out of postgres, C, awk, and a bunch of other things, but it'd certainly cost more just in developer time than kdb.
  - osrec 9y ago
    Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are many open source offerings, which are now matching it both in terms of performance and flexibility. Plus hiring good people is significantly easier and cheaper.
    - dogruck 9y ago
      As an ex-MD you also know that banks used to not be too price sensitive, so they didn't mind paying a lot for some Kx experts. Of course, these days, with revenues being squeezed, that's changed.
      - osrec 9y ago
        Agree, price is a much bigger factor now.
    - oper8or 9y ago
      What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.
      - alfiedotwtf 9y ago
        Everyone is hanging on for OP to deliver
      - kthielen 9y ago
        The one I linked to in my previous post also produces code that's fast enough to use in the critical path in low-latency trading systems. As you may know, q's "scalar performance" is not great; similar to Scheme or python due to boxing overhead (as you can see in the linked k.h file below). Also, the fact that q is untyped has a severe impact on its safe use in large and complex projects. (ETA: https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes)
      - osrec 9y ago
        There are a few. I've used a modified LMDB source in the past with success, employing similar tricks to kdb for performance (i.e. store daily data as contiguous arrays so that reads are quick etc). Either way, implementing a memory mapped store of arrays and operating on it is not too challenging a problem for any good C++ dev.
      - oper8or 9y ago
  - kthielen 9y ago
    https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes
- dilap 9y ago
  huh, so what do i need to learn to be one of these ridiculously expensive people working w/ kdb?
  - ah- 9y ago
    I'd start with https://code.kx.com/q4m3/ https://code.kx.com/q4m3/
    - osrec 9y ago
      Also, you need to be lucky enough to somehow move into a team that uses kdb heavily (this usually happens via an internal move within an investment bank). Learning it using the tutorials is not the same as implementing it in a live environment, and the tutorials probably won't be enough to get you a job/contract. If you could say something like "I've worked on the FX desk at JP Morgan, where we used kdb daily", you'll be hired everywhere. Basically, you need a bit of luck in the first instance to get into a kdb focused team that can train you - after that, you're sorted!
  - beagle3 9y ago
    Historically, an APL background was a good indication that you grok the apl/k/j/q mindset, which is very different from the Algol (c/c#/Java/pascal) world and also from the Lisp world.
bkeroack 9y ago
I like it when kdb is mentioned because then I can post this link: https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 It's one of my all-time favorites. A window into a certain type of mind.
- zie 9y ago
  Oh my. that's just.. wow. Obviously done before code-review became a standard!
  - hakanderyal 9y ago
    This thread might give some insights about the coding style: https://news.ycombinator.com/item?id=13565743 https://news.ycombinator.com/item?id=13565743
  - pjmlp 9y ago
    Code-review is a cool thing in SV, trendy startups and the big industry related companies, sadly on most companies whose main business is completely unrelated to IT, just like unit tests it gets a spot just behind writing documentation.
- chc4 9y ago
  I'm surprised you didn't link an actual c file; everything related to k seems to trigger an exorbitantly high number of "what the fuck"s. https://github.com/kevinlawler/kona/blob/master/src/0.c https://github.com/kevinlawler/kona/blob/master/src/0.c for example. http://archive.vector.org.uk/art10501320 http://archive.vector.org.uk/art10501320 is one of my favorite articles, though
  - RodgerTheGreat 9y ago
    That article was enough to inspire me to write a K interpreter, and eventually land me a job working with K. If I ever meet Stephen Taylor in person I imagine it'll be an interesting story to tell.

99 points by gricardo99 9y ago

the idea that I could seamlessly move data between python and kdb+ is making me salivate. Please, make this work well and lower the cost-barrier to using kdb+ and I think you'd see this become the defacto "stack" for many database setups.

ah- 9y ago
All this needs now is a free (at least commercial use allowed 64bit version, ideally OSS) kdb+. I'd love to use this, many things are so much more straight forward in q than they are in pandas. But if it means noone can run my software unless they pay for kdb+ it's a non-starter. I wonder if this would work with Kona.

osrec 9y ago

I've never quite understood why banks still opt for kdb when (a) hiring staff is ridiculously difficult/expensive and (b) there are so many other cheaper alternatives available. They literally have a strangle hold on every eFX desk in London, and I cannot really figure out why!

geocar 9y ago
Maybe you're wrong about (b)? Sure, I could cobble something about as fast and functional out of postgres, C, awk, and a bunch of other things, but it'd certainly cost more just in developer time than kdb.
- osrec 9y ago
  Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are many open source offerings, which are now matching it both in terms of performance and flexibility. Plus hiring good people is significantly easier and cheaper.
  - dogruck 9y ago
    As an ex-MD you also know that banks used to not be too price sensitive, so they didn't mind paying a lot for some Kx experts. Of course, these days, with revenues being squeezed, that's changed.
    - osrec 9y ago
      Agree, price is a much bigger factor now.
  - oper8or 9y ago
    What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.
    - alfiedotwtf 9y ago
      Everyone is hanging on for OP to deliver
    - kthielen 9y ago
      The one I linked to in my previous post also produces code that's fast enough to use in the critical path in low-latency trading systems. As you may know, q's "scalar performance" is not great; similar to Scheme or python due to boxing overhead (as you can see in the linked k.h file below). Also, the fact that q is untyped has a severe impact on its safe use in large and complex projects. (ETA: https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes)
    - osrec 9y ago
      There are a few. I've used a modified LMDB source in the past with success, employing similar tricks to kdb for performance (i.e. store daily data as contiguous arrays so that reads are quick etc). Either way, implementing a memory mapped store of arrays and operating on it is not too challenging a problem for any good C++ dev.
    - oper8or 9y ago
- kthielen 9y ago
  https://github.com/Morgan-Stanley/hobbes https://github.com/Morgan-Stanley/hobbes
dilap 9y ago
huh, so what do i need to learn to be one of these ridiculously expensive people working w/ kdb?
- ah- 9y ago
  I'd start with https://code.kx.com/q4m3/ https://code.kx.com/q4m3/
  - osrec 9y ago
    Also, you need to be lucky enough to somehow move into a team that uses kdb heavily (this usually happens via an internal move within an investment bank). Learning it using the tutorials is not the same as implementing it in a live environment, and the tutorials probably won't be enough to get you a job/contract. If you could say something like "I've worked on the FX desk at JP Morgan, where we used kdb daily", you'll be hired everywhere. Basically, you need a bit of luck in the first instance to get into a kdb focused team that can train you - after that, you're sorted!
- beagle3 9y ago
  Historically, an APL background was a good indication that you grok the apl/k/j/q mindset, which is very different from the Algol (c/c#/Java/pascal) world and also from the Lisp world.

bkeroack 9y ago

I like it when kdb is mentioned because then I can post this link: https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 It's one of my all-time favorites. A window into a certain type of mind.

zie 9y ago
Oh my. that's just.. wow. Obviously done before code-review became a standard!
- hakanderyal 9y ago
  This thread might give some insights about the coding style: https://news.ycombinator.com/item?id=13565743 https://news.ycombinator.com/item?id=13565743
- pjmlp 9y ago
  Code-review is a cool thing in SV, trendy startups and the big industry related companies, sadly on most companies whose main business is completely unrelated to IT, just like unit tests it gets a spot just behind writing documentation.
chc4 9y ago
I'm surprised you didn't link an actual c file; everything related to k seems to trigger an exorbitantly high number of "what the fuck"s. https://github.com/kevinlawler/kona/blob/master/src/0.c https://github.com/kevinlawler/kona/blob/master/src/0.c for example. http://archive.vector.org.uk/art10501320 http://archive.vector.org.uk/art10501320 is one of my favorite articles, though
- RodgerTheGreat 9y ago
  That article was enough to inspire me to write a K interpreter, and eventually land me a job working with K. If I ever meet Stephen Taylor in person I imagine it'll be an interesting story to tell.