4 ms·

Programming in K

67 points by tosh 9mo ago

chrisaycock 8mo ago
I first encountered q/kdb+ at a quant job in 2007. I learned so much from the array semantics about how to concisely represent time-series logic that I can't imagine ever using a scalar language for research. Fun fact: the aj (asof join) function was my inspiration for pandas.merge_asof. I added the extra parameters (direction, tolerance, allow_exact_matches) because of the limitations I kept hitting in kdb. https://code.kx.com/q/ref/aj/ https://code.kx.com/q/ref/aj/ https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html https://pandas.pydata.org/docs/reference/api/pandas.merge_as...
- zX41ZdbW 8mo ago
  Similarly, this is how it was introduced in ClickHouse in 2019: https://github.com/ClickHouse/ClickHouse/pull/4774 https://github.com/ClickHouse/ClickHouse/pull/4774
- leprechaun1066 8mo ago
  The aj function at its heart is a bin (https://code.kx.com/q/ref/bin/ https://code.kx.com/q/ref/bin/) search between the two tables, on the requested columns, to find the indices of the right table to zip onto the left table. aj[`sym`time;t;q] becomes t,'(`sym`time _q)(`sym`time#q)bin`sym`time#t The rest of the aj function internals are there to handle edge cases, handling missing columns and options for filling nulls. A lot of the joins can be distilled to the core operators/functions in a similar manner. For example the plus-join is x+0i^y(cols key y)#x
  - chrisaycock 8mo ago
    Indeed, my very first attempt used numpy.searchsorted: https://numpy.org/doc/2.2/reference/generated/numpy.searchsorted.html https://numpy.org/doc/2.2/reference/generated/numpy.searchso... I couldn't figure-out how Arthur's bin matched on symbol though, so I switched to a linear scan on the right table to record the last-seen index for each "by" element. While it worked, my hash table was messy because I relied on Python to handle a whole tuple as a key, which had some issues during initial testing. The asof join I wrote for Empirical properly categorizes the keys before they are matched. That approach worked far better. https://www.empirical-soft.com/tutorial.html#dataframes https://www.empirical-soft.com/tutorial.html#dataframes
- sceadu 8mo ago
  dang I had no idea you wrote the asof join for pandas. thank you for that
koolala 8mo ago
I wish there was a language like K that worked with single precision floats. Would be great to use with graphics.
- ksherlock 8mo ago
  q is like k and has single precision floats.
  - koolala 8mo ago
    q is closed source so it isn't really viable to build anything out of it unfortunately. I wonder how they made floats work since it is based on k.
jjtheblunt 8mo ago
the github repo linked, under "manual.md", has a link which redirects to malware, such as that ostensibly to "k6".
- tosh 8mo ago
  ty, I made a pull request: https://github.com/JohnEarnest/ok/pull/111 https://github.com/JohnEarnest/ok/pull/111
monster_truck 8mo ago
K fucking rules if you're trying to minmax a game or balance your own. Figuring out things like "what percentage of the time will my starting hand contain the cards I need for my deck to function, and if it doesn't, how many mulligans will it take" will basically ruin competitive MTG for you. I used to buy physical verisons of the decks I made it to top 500 with in Arena, stopped because it wasn't a fun challenge anymore (and lost all interest when they started dragging other IPs in, they'll never get another cent from me)
Koshkin 8mo ago
K[ryptic!]* *I appreciate mathematical notation, but this is not it.

67 points by tosh 9mo ago

I first encountered q/kdb+ at a quant job in 2007. I learned so much from the array semantics about how to concisely represent time-series logic that I can't imagine ever using a scalar language for research. Fun fact: the aj (asof join) function was my inspiration for pandas.merge_asof. I added the extra parameters (direction, tolerance, allow_exact_matches) because of the limitations I kept hitting in kdb. https://code.kx.com/q/ref/aj/ https://code.kx.com/q/ref/aj/ https://pandas.pydata.org/docs/reference/api/pandas.merge_asof.html https://pandas.pydata.org/docs/reference/api/pandas.merge_as...

zX41ZdbW 8mo ago
Similarly, this is how it was introduced in ClickHouse in 2019: https://github.com/ClickHouse/ClickHouse/pull/4774 https://github.com/ClickHouse/ClickHouse/pull/4774
leprechaun1066 8mo ago
The aj function at its heart is a bin (https://code.kx.com/q/ref/bin/ https://code.kx.com/q/ref/bin/) search between the two tables, on the requested columns, to find the indices of the right table to zip onto the left table. aj[`sym`time;t;q] becomes t,'(`sym`time _q)(`sym`time#q)bin`sym`time#t The rest of the aj function internals are there to handle edge cases, handling missing columns and options for filling nulls. A lot of the joins can be distilled to the core operators/functions in a similar manner. For example the plus-join is x+0i^y(cols key y)#x
- chrisaycock 8mo ago
  Indeed, my very first attempt used numpy.searchsorted: https://numpy.org/doc/2.2/reference/generated/numpy.searchsorted.html https://numpy.org/doc/2.2/reference/generated/numpy.searchso... I couldn't figure-out how Arthur's bin matched on symbol though, so I switched to a linear scan on the right table to record the last-seen index for each "by" element. While it worked, my hash table was messy because I relied on Python to handle a whole tuple as a key, which had some issues during initial testing. The asof join I wrote for Empirical properly categorizes the keys before they are matched. That approach worked far better. https://www.empirical-soft.com/tutorial.html#dataframes https://www.empirical-soft.com/tutorial.html#dataframes
sceadu 8mo ago
dang I had no idea you wrote the asof join for pandas. thank you for that

koolala 8mo ago

I wish there was a language like K that worked with single precision floats. Would be great to use with graphics.

ksherlock 8mo ago
q is like k and has single precision floats.
- koolala 8mo ago
  q is closed source so it isn't really viable to build anything out of it unfortunately. I wonder how they made floats work since it is based on k.

jjtheblunt 8mo ago

the github repo linked, under "manual.md", has a link which redirects to malware, such as that ostensibly to "k6".

tosh 8mo ago
ty, I made a pull request: https://github.com/JohnEarnest/ok/pull/111 https://github.com/JohnEarnest/ok/pull/111

monster_truck 8mo ago

K fucking rules if you're trying to minmax a game or balance your own. Figuring out things like "what percentage of the time will my starting hand contain the cards I need for my deck to function, and if it doesn't, how many mulligans will it take" will basically ruin competitive MTG for you. I used to buy physical verisons of the decks I made it to top 500 with in Arena, stopped because it wasn't a fun challenge anymore (and lost all interest when they started dragging other IPs in, they'll never get another cent from me)

Koshkin 8mo ago

K[ryptic!]* *I appreciate mathematical notation, but this is not it.