5 ms·

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal l

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented (no comments, single letter parameters, arcane function names). "We understand it after some time and this way other teams cannot use our ideas." Overall, their whole stack was outdated and ofc they could not do very interesting things with a tool such as Q. For example, they plotted graphs by copying data from qStudio to Excel... The only good thing was they did not buy the docker / k8s bs and were deploying directly on servers. It makes sense that quants should be able to fix things in production very quickly but I think it would also make sense for web app developers not to wait 10 minutes (and that's when you have good infra) to see a fix in production. I have a theory on why quants actually like kdb: it's a good *weapon*. It serves some purpose but I would not call it a *tool* as building with it is tedious. People like that it just works out of the box. But although you can use a sword to drive nails, it is not its purpose. Continuing on that theory, LISP (especially Racket) would be the best *tool* available as it is not the most powerful language out of the box but allows to build a lot of abstractions with features to modify the language itself. C++ and Python are just great programming languages as you can build good software on them, Python being also a fairly good weapon. Q might give the illusion of being the best language to explore quant data, but that's just because quants do not invest enough time into building good software and using good tools. When you actually master a Python IDE, you are definitely more productive than any Q programmer. And don't get me started on performance (the link covers it anyway even though the prose is bad).

keithalewis 2y ago
[flagged]
- eru 2y ago
  It's not a good filter in that case. I can learn obscure languages just fine, but that doesn't make me any more pleasant to hang out with.
  - socksy 2y ago
    I'm not sure that was ever a requirement in these industries
    - keithalewis 2y ago
      It is not a requirement. Just a way to weed out people who think they are special snowflakes.
      - eru 2y ago
        I'm perfectly capable of learning obscure language _and_ thinking I'm a special snowflake. (In fact, I'm a special snowflake _because_ I am into weird languages.)
Jorge1o1 2y ago
Their pykx integration is going a long way to fix some of the gaps in: - charting - machine learning/statsmodels - html processing/webscrapes Because for example you can just open a Jupyter Notebook and do: import pykx as kx df = kx.q(“select from foo where bar”) plt.plot(df[“x”], df[“y”]) It’s truly an incredibly seamless and powerful integration. You get the best of both worlds and it may be the saving feature of the product in the next 10 years
- nivertech 2y ago
  I think this will only work with regular qSQL on a specific database node, i.e. RDB, IDB, HDB[1]. It will be much harder for a mortal Python developer to use Functional qSQL[2] which will join/merge/aggregate data from all these nodes. The join/merge/aggregation is usually application-specific and done on some kind of gateway node(s). Querying each of them is slightly different, with different keys and secondary indices, and requires using a parse tree (AST) of a query. --- [1] RDB - RAM DB (recent in-memory data), IDB (Intraday DB - recent data which doesn't fit into RAM), HDB - Historical DB (usually partitioned by date or other time-based or integral column). [2] https://code.kx.com/q/basics/funsql/ https://code.kx.com/q/basics/funsql/
  - Jorge1o1 2y ago
    That’s accurate enough. I think the workflow was more built for a q dev occasionally dipping into python rather than the other way around. I think you touch on something really interesting which is the kink in the kdb+ learning curve when you go from really simple functions,tables, etc. to actually building a performant kdb architecture.
  - qkdb1 2y ago
    Will be interesting to see what comes of some of the things that are being put on their roadmap https://code.kx.com/pykx/2.5/roadmap.html#upcoming-changes https://code.kx.com/pykx/2.5/roadmap.html#upcoming-changes seems to be moving in a direction of an API similar to Polars
wenc 2y ago
The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing today with Python and DuckDB and Jupyter on Parquet files. DuckDB not only parallelizes, it vectorizes. I’m not sure how it benchmarks against kdb+ but the responsiveness of DuckDB at least feels as fast as kdb+ on large datasets. (Though I’m sure kdb+ is vastly more optimized). The difference? DuckDB is free.
- singhrac 2y ago
  We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time. It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).
  - wenc 2y ago
    Yes Wes McKinney was involved in both Pandas and Parquet and Arrow.
    - 7thaccount 2y ago
      I remember reading something awhile back that when building pandas he was getting a lot of inspiration from things like APL and I assume Kdb+.
- cout 2y ago
  Do you use duckdb for real-time queries or just historical? You mentioned parquet but afaik it's not well suited for appending data.
  - wenc 2y ago
dumah 2y ago
You didn’t learn Q in two weeks to the extent that you are qualified to assert that someone who knows how to use a Python IDE is more productive than a quant dev with decades of experience. I find it much more likely that you couldn’t understand their code and quit out of frustration. If you were a highly skilled quant dev and this was a good seat, quitting after two weeks would have been a disaster to manage the next transition given the terms these contracts always have.
belfthrow 2y ago
Not surviving more than 2 weeks in a QF role because of kdb, and then suggesting they should rewrite everything to LISP is one of the more HN level recidivous comments I think I have ever seen.
hysteria2024 2y ago
[dead]

keithalewis 2y ago

[flagged]

eru 2y ago
It's not a good filter in that case. I can learn obscure languages just fine, but that doesn't make me any more pleasant to hang out with.
- socksy 2y ago
  I'm not sure that was ever a requirement in these industries
  - keithalewis 2y ago
    It is not a requirement. Just a way to weed out people who think they are special snowflakes.
    - eru 2y ago
      I'm perfectly capable of learning obscure language _and_ thinking I'm a special snowflake. (In fact, I'm a special snowflake _because_ I am into weird languages.)

Jorge1o1 2y ago

Their pykx integration is going a long way to fix some of the gaps in: - charting - machine learning/statsmodels - html processing/webscrapes Because for example you can just open a Jupyter Notebook and do: import pykx as kx df = kx.q(“select from foo where bar”) plt.plot(df[“x”], df[“y”]) It’s truly an incredibly seamless and powerful integration. You get the best of both worlds and it may be the saving feature of the product in the next 10 years

nivertech 2y ago
I think this will only work with regular qSQL on a specific database node, i.e. RDB, IDB, HDB[1]. It will be much harder for a mortal Python developer to use Functional qSQL[2] which will join/merge/aggregate data from all these nodes. The join/merge/aggregation is usually application-specific and done on some kind of gateway node(s). Querying each of them is slightly different, with different keys and secondary indices, and requires using a parse tree (AST) of a query. --- [1] RDB - RAM DB (recent in-memory data), IDB (Intraday DB - recent data which doesn't fit into RAM), HDB - Historical DB (usually partitioned by date or other time-based or integral column). [2] https://code.kx.com/q/basics/funsql/ https://code.kx.com/q/basics/funsql/
- Jorge1o1 2y ago
  That’s accurate enough. I think the workflow was more built for a q dev occasionally dipping into python rather than the other way around. I think you touch on something really interesting which is the kink in the kdb+ learning curve when you go from really simple functions,tables, etc. to actually building a performant kdb architecture.
- qkdb1 2y ago
  Will be interesting to see what comes of some of the things that are being put on their roadmap https://code.kx.com/pykx/2.5/roadmap.html#upcoming-changes https://code.kx.com/pykx/2.5/roadmap.html#upcoming-changes seems to be moving in a direction of an API similar to Polars

wenc 2y ago

The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing today with Python and DuckDB and Jupyter on Parquet files. DuckDB not only parallelizes, it vectorizes. I’m not sure how it benchmarks against kdb+ but the responsiveness of DuckDB at least feels as fast as kdb+ on large datasets. (Though I’m sure kdb+ is vastly more optimized). The difference? DuckDB is free.

singhrac 2y ago
We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time. It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).
- wenc 2y ago
  Yes Wes McKinney was involved in both Pandas and Parquet and Arrow.
  - 7thaccount 2y ago
    I remember reading something awhile back that when building pandas he was getting a lot of inspiration from things like APL and I assume Kdb+.
cout 2y ago
Do you use duckdb for real-time queries or just historical? You mentioned parquet but afaik it's not well suited for appending data.
- wenc 2y ago

dumah 2y ago

You didn’t learn Q in two weeks to the extent that you are qualified to assert that someone who knows how to use a Python IDE is more productive than a quant dev with decades of experience. I find it much more likely that you couldn’t understand their code and quit out of frustration. If you were a highly skilled quant dev and this was a good seat, quitting after two weeks would have been a disaster to manage the next transition given the terms these contracts always have.

belfthrow 2y ago

Not surviving more than 2 weeks in a QF role because of kdb, and then suggesting they should rewrite everything to LISP is one of the more HN level recidivous comments I think I have ever seen.

hysteria2024 2y ago

[dead]