3 ms·

Was working at a quant pod at Millennium for a bit where they used it. I was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas

Was working at a quant pod at Millennium for a bit where they used it. I was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas. The irony was that the Python code was shorter because there were so many more library functions and better abstractions and syntax. So it was slow and unintuitive for zero benefit whatsoever.

wood_spirit 2y ago
But how did your perf compare to the best of the K kicking quants around you? Were they too being less productive than they would have been in python? I’m not saying they were right or better. Horses of courses. Array languages do my head in and my choice is sql.
- eigenvalue 2y ago
  I was able to explore new ideas much much faster using Python than the experienced k people could. But creativity is more important anyway. Ultimately, having good ideas/data/signals trumps fancy or fast data wrangling. Glad I’m doing other things now in any case.
  - gitonthescene 2y ago
    Any detail whatsoever would make this a more credible claim. I haven’t met many people, including those skeptical of the performance claims, who have called K _slow_. Maybe for particular domains but I’d doubt that includes the kind of quant work that gets done at Millennium.
    - spopejoy 2y ago
      I've heard plenty of complaints over the years, and only within quant, unsurprisingly since that is the only field you'll get paid to use K. A brilliant programmer I met who came from DE Shaw said he reimplemented a K-based portfolio optimization pipeline because the performance hit a wall once the dataset got large enough. He was able to beat K with Java of all things. Columnar and timeseries dbs have continued to evolve, K is the same tech it was in the 2000s. The only reason it gets used at a Millennium is that whatever trade is still printing money, not any tech advantage.
      - gitonthescene 2y ago
        I appreciate it's probably not possible to share too many details but I wouldn't be surprised if the choice of Java wasn't simply preference. It may have been a problem with the pipeline rather than with K. I.e. a fix might have been available using K but it can be easier (and harder) to just use some out-of-the box solution. I agree that columnar and time series dbs may have caught up with K over the years, but most of the complaints I've heard about K aren't technical.
Qem 2y ago
> was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas. You can try klongpy, a K-like array language implementation that runs atop numpy: https://pypi.org/project/klongpy/ https://pypi.org/project/klongpy/
eggy 2y ago
Geometric mean in Numpy vs. J: (Copied from some forum, since I don't use Python much) import numpy as np def geo_mean_overflow(iterable): return np.exp(np.log(iterable).mean()) Or, from statistics import. geometric_mean geometric_mean([1.0, 0.00001, 10000000000.]) # 46.415888336127786 In J, since I don't know K: gm=:#%:*/ Even shorter than Python whether it's a canned lib routine or created from composing simple functions. And I don't need to format code on HN in J because it's so short anyway, besides I don't know how!

wood_spirit 2y ago

But how did your perf compare to the best of the K kicking quants around you? Were they too being less productive than they would have been in python? I’m not saying they were right or better. Horses of courses. Array languages do my head in and my choice is sql.

eigenvalue 2y ago
I was able to explore new ideas much much faster using Python than the experienced k people could. But creativity is more important anyway. Ultimately, having good ideas/data/signals trumps fancy or fast data wrangling. Glad I’m doing other things now in any case.
- gitonthescene 2y ago
  Any detail whatsoever would make this a more credible claim. I haven’t met many people, including those skeptical of the performance claims, who have called K _slow_. Maybe for particular domains but I’d doubt that includes the kind of quant work that gets done at Millennium.
  - spopejoy 2y ago
    I've heard plenty of complaints over the years, and only within quant, unsurprisingly since that is the only field you'll get paid to use K. A brilliant programmer I met who came from DE Shaw said he reimplemented a K-based portfolio optimization pipeline because the performance hit a wall once the dataset got large enough. He was able to beat K with Java of all things. Columnar and timeseries dbs have continued to evolve, K is the same tech it was in the 2000s. The only reason it gets used at a Millennium is that whatever trade is still printing money, not any tech advantage.
    - gitonthescene 2y ago
      I appreciate it's probably not possible to share too many details but I wouldn't be surprised if the choice of Java wasn't simply preference. It may have been a problem with the pipeline rather than with K. I.e. a fix might have been available using K but it can be easier (and harder) to just use some out-of-the box solution. I agree that columnar and time series dbs may have caught up with K over the years, but most of the complaints I've heard about K aren't technical.

Qem 2y ago

> was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas. You can try klongpy, a K-like array language implementation that runs atop numpy: https://pypi.org/project/klongpy/ https://pypi.org/project/klongpy/

eggy 2y ago

Geometric mean in Numpy vs. J: (Copied from some forum, since I don't use Python much) import numpy as np def geo_mean_overflow(iterable): return np.exp(np.log(iterable).mean()) Or, from statistics import. geometric_mean geometric_mean([1.0, 0.00001, 10000000000.]) # 46.415888336127786 In J, since I don't know K: gm=:#%:*/ Even shorter than Python whether it's a canned lib routine or created from composing simple functions. And I don't need to format code on HN in J because it's so short anyway, besides I don't know how!