6 ms·

I remember reading the manual for arthur whitney's shakti/k9 and seeing this normative claim: > Learning k9 means code like {x@(!#x)+\!#y} is clear and actuall

by jd3 4y ago

I remember reading the manual for arthur whitney's shakti/k9 and seeing this normative claim: > Learning k9 means code like {x@(!#x)+\!#y} is clear and actually prefered to something much longer and verbose. https://estradajke.github.io/k9-simples/k9/index.html https://estradajke.github.io/k9-simples/k9/index.html All I could think of was Abelson/Sussman's evergreen quote from the preface of SICP: > Programs are meant to be read by humans, and only incidentally for computers to execute.

shin_lao 4y ago
Lots of respect for Arthur, KDB+ was transformative when it arrived but I think now we need to move beyond that. Will take one or two decades for banks to migrate though lol.
- steveBK123 4y ago
  It's also a misunderstood product from the outside. Unfortunately banks had a tendency to pay a bunch of senior engineers to do a build out for 3 years and then turn it over to cheap nearshore consultants to operate for 10 years. The core of most big bank code you see was written before 2010. Where I have seen bakeoffs and largely failed attempts to replace, the challenge is the number of vendor & open solutions you need to cobble together to replace all the use cases that KDB supports. Every year it was a slightly different combination that didn't really cover all the bases. To replace KDB in many of these large use cases, you need - a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations, an event bus, a stream processor, and a programming language to write more complex applications close to the data. A lot of the newer cloud-centric offerings are more about massive parallelism/scale than about pure single thread/individual request-response performance. This is great for the vast majority of CRUD apps with millions/billions of users. It's even great for some financial "backtesting" type use cases where you can kick off large It generally isn't sufficient for many of the real-time financial use cases of doing ad-hoc analytics on billions/day datasets with expected responses in ms. The biggest complaints are its expensive, and the devs are expensive. On the other hand you usually don't need as much hardware or dev staff to support it.
  - timeserious 4y ago
    Hi Steve. See my reply in above thread to Shin which calls out alot of the operational improvements we've made. Just an FYI, there's also been significant changes in how we price so it's now alot easier to start small, get started quickly and decrease the time to value with more mainstream skillsets. With the new product strategy we're seeing alot more buy side clients (hedge funds/asset managers) use kdb+/q for their workloads with KX Insights.
  - zX41ZdbW 4y ago
    > a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations Sounds like ClickHouse. And we see kdb slowly start being replaced...
- timeserious 4y ago
  Hi Shin - KX employee here (the creators of q/kdb+). We launched a new product to take our time-series tech to new markets and usecases with high performance analytic needs. Over the past 2-3 years we've listened closely to the community and have made huge improvements to make kdb+ easier to develop with using options like ANSI-SQL/Python/Stream Processing DSL/microservices/Open API etc. The banks are adopting all of the new kit quickly and leverage the cloud native capabilities (like kubernetes operators/object storage) to help scale and take their deployments to new heights without deep technical kdb+ expertise. Thought it was worthwhile calling out the adoption and product strategy here at KX.. lots more to come! https://code.kx.com/insights/ https://code.kx.com/insights/
  - shin_lao 4y ago
    Hey there thanks for your answer, know about this product, think you need to do better :-) Best of luck!
avmich 4y ago
I think you might prefer notation y = a * x^2 + b * x + c to notation where those things are described some other ways. Same here, APL languages ask you to learn some notation - and then effectively use it, so the programs with that could be "meant to be read by humans".
- gjulianm 4y ago
  If APL actually followed mathematical notation it would be extremely verbose. Most math books, articles and explanations use quite a lot of words, far more than symbols, precisely to allow them to be “read by humans”. Those are mostly used for computations and symbolic manipulations.
  - avmich 4y ago
    Nobody holds you from adding as much comments to APL code as you wish. I once wanted to write a parser generator in J, a very incomplete version is here - https://code.jsoftware.com/wiki/User:Alex_Mikhailov/Parsing https://code.jsoftware.com/wiki/User:Alex_Mikhailov/Parsing . This is practically text, as you'd expect from a math article, with embedded code which allows computer execution to have those ideas working.
    - gjulianm 4y ago
      But there is a difference between “verbose code” and “adding comments”, isn’t it? There’s always the issue with comment-code disagreement. But mainly the point is about the language itself and the prioritization of terseness over unambiguity and intuition. Good mathematical notation strives for the latter, not the former.
      - avmich 4y ago
        > There’s always the issue with comment-code disagreement. You just mentioned mathematical articles where the math expressions are a minority of the text, right? > But mainly the point is about the language itself and the prioritization of terseness over unambiguity and intuition. Math notation had - and keeps having - reason to be terser than plain text. Same for APL - which started life as "Iverson notation". > Good mathematical notation strives for the latter, not the former. Don't you think APL is a pretty good mathematical notation? If not, why?

by jd3 4y ago

shin_lao 4y ago

Lots of respect for Arthur, KDB+ was transformative when it arrived but I think now we need to move beyond that. Will take one or two decades for banks to migrate though lol.

steveBK123 4y ago
It's also a misunderstood product from the outside. Unfortunately banks had a tendency to pay a bunch of senior engineers to do a build out for 3 years and then turn it over to cheap nearshore consultants to operate for 10 years. The core of most big bank code you see was written before 2010. Where I have seen bakeoffs and largely failed attempts to replace, the challenge is the number of vendor & open solutions you need to cobble together to replace all the use cases that KDB supports. Every year it was a slightly different combination that didn't really cover all the bases. To replace KDB in many of these large use cases, you need - a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations, an event bus, a stream processor, and a programming language to write more complex applications close to the data. A lot of the newer cloud-centric offerings are more about massive parallelism/scale than about pure single thread/individual request-response performance. This is great for the vast majority of CRUD apps with millions/billions of users. It's even great for some financial "backtesting" type use cases where you can kick off large It generally isn't sufficient for many of the real-time financial use cases of doing ad-hoc analytics on billions/day datasets with expected responses in ms. The biggest complaints are its expensive, and the devs are expensive. On the other hand you usually don't need as much hardware or dev staff to support it.
- timeserious 4y ago
  Hi Steve. See my reply in above thread to Shin which calls out alot of the operational improvements we've made. Just an FYI, there's also been significant changes in how we price so it's now alot easier to start small, get started quickly and decrease the time to value with more mainstream skillsets. With the new product strategy we're seeing alot more buy side clients (hedge funds/asset managers) use kdb+/q for their workloads with KX Insights.
- zX41ZdbW 4y ago
  > a fast columnar PB-scale database, an efficient on-disk store, a fast in-memory cache, SQL support plus with expressive query language additions for time series operations Sounds like ClickHouse. And we see kdb slowly start being replaced...
timeserious 4y ago
Hi Shin - KX employee here (the creators of q/kdb+). We launched a new product to take our time-series tech to new markets and usecases with high performance analytic needs. Over the past 2-3 years we've listened closely to the community and have made huge improvements to make kdb+ easier to develop with using options like ANSI-SQL/Python/Stream Processing DSL/microservices/Open API etc. The banks are adopting all of the new kit quickly and leverage the cloud native capabilities (like kubernetes operators/object storage) to help scale and take their deployments to new heights without deep technical kdb+ expertise. Thought it was worthwhile calling out the adoption and product strategy here at KX.. lots more to come! https://code.kx.com/insights/ https://code.kx.com/insights/
- shin_lao 4y ago
  Hey there thanks for your answer, know about this product, think you need to do better :-) Best of luck!

avmich 4y ago

I think you might prefer notation y = a * x^2 + b * x + c to notation where those things are described some other ways. Same here, APL languages ask you to learn some notation - and then effectively use it, so the programs with that could be "meant to be read by humans".

gjulianm 4y ago
If APL actually followed mathematical notation it would be extremely verbose. Most math books, articles and explanations use quite a lot of words, far more than symbols, precisely to allow them to be “read by humans”. Those are mostly used for computations and symbolic manipulations.
- avmich 4y ago
  Nobody holds you from adding as much comments to APL code as you wish. I once wanted to write a parser generator in J, a very incomplete version is here - https://code.jsoftware.com/wiki/User:Alex_Mikhailov/Parsing https://code.jsoftware.com/wiki/User:Alex_Mikhailov/Parsing . This is practically text, as you'd expect from a math article, with embedded code which allows computer execution to have those ideas working.
  - gjulianm 4y ago
    But there is a difference between “verbose code” and “adding comments”, isn’t it? There’s always the issue with comment-code disagreement. But mainly the point is about the language itself and the prioritization of terseness over unambiguity and intuition. Good mathematical notation strives for the latter, not the former.
    - avmich 4y ago
      > There’s always the issue with comment-code disagreement. You just mentioned mathematical articles where the math expressions are a minority of the text, right? > But mainly the point is about the language itself and the prioritization of terseness over unambiguity and intuition. Math notation had - and keeps having - reason to be terser than plain text. Same for APL - which started life as "Iverson notation". > Good mathematical notation strives for the latter, not the former. Don't you think APL is a pretty good mathematical notation? If not, why?