4 ms·

The linked document only contains a warning about how versioning is weird, and a description of the syntax. No examples beyond trivial one-liners. What problem

The linked document only contains a warning about how versioning is weird, and a description of the syntax. No examples beyond trivial one-liners. What problem is K trying to solve? What does a K program look like?

sz4kerto 2y ago
Absolutely not being sarcastic: one problem it solves is that it is very hard to read as a beginner, so it can be intimidating (although it becomes much easier to read a bit later). This, coupled with the general arrogance of k/q practitioners (again, not really saying this in a negative way) and that k, kdb, etc. deliberately doesn't give you guardrails makes people who write k/q seem a bit 'mythical' and make them feel very clever. So I think k, q and kdb are fun to work with, but one of the major components of its success is that it allowed a community (in finance) to evolve that can earn 50-150% more than their peer groups who do the same work in Java or C++. 10 years ago a kx course cost $1500 per person per day.
- pjmlp 2y ago
  To note that those are typical prices for enterprise level certifications, including some products that some Java or C++ devs might need to interact with, when working on those kind of environments.
- cppandjava 2y ago
  Hmm. I work in finance writing C++ and Java and I doubt other people in finance make 50-150% more than me because they know `q`.
  - gitonthescene 2y ago
    I don't know. If you're writing Java you may not be working on the same types of problems.
bear8642 2y ago
K is a fast vector language, used (primarily) for time series data analysis. >What does a K program look like? You might want to check out https://news.ycombinator.com/item?id=40335921 https://news.ycombinator.com/item?id=40335921 beagle3 and geocar both have various comments you might want to search for.
- mananaysiempre 2y ago
  > a fast vector language With an Oracle-style DeWitt clause[1] prohibiting public benchmarks. [1] https://mlochbaum.github.io/BQN/implementation/kclaims.html https://mlochbaum.github.io/BQN/implementation/kclaims.html
  - rustc 2y ago
    Shakti (the latest K implementation by the author of K) claims [1] to load a 50gb csv in 1.6 seconds which according to them takes 265 seconds with Polars. Has anyone independently verified these claims? Is Polars really leaving 2 orders of magnitude performance on the table? [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k
    - bear8642 2y ago
      > [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k You can link to the subsections: https://shakti.com/compare/h2o.k https://shakti.com/compare/h2o.k
    - orlp 2y ago
      Disclaimer: I work for Polars inc. As a sanity check I just cloned https://github.com/h2oai/db-benchmark https://github.com/h2oai/db-benchmark, ran the data generation script and ran on a 64 core AMD EPYC (AWS c7a.16xlarge): import polars as pl lf = pl.scan_csv("G1_1e9_1e2_0_0.csv") print(lf.select(pl.col.v1.sum()).collect()) The above script ran in 7.58 seconds. If I change the collect() to collect(new_streaming=True) to use the new streaming engine I've been working on, it runs in 6.90 seconds. I can't realistically time the full "read CSV to memory" with this 50 GB file on this machine as we start swapping (this machine has 128GiB memory) and/or evicting data from disk cache (this machine has a slow EC2 SSD attached to it), so we do have a blow-up of memory usage (which could be as simple as loading small integers into an 8-byte Uint64 column). I think it's likely that on K's machine the "read full CSV to memory" approach also started swapping, giving the large runtime. However, in Polars you'd typically write your query using LazyFrames, which means we don't actually have to load the full CSV into memory. EDIT: running on a m7a.16xlarge with twice the memory (256GiB) once the CSV file is in disk cache Polars can parse the full CSV file into an in-memory dataframe in 7.68 seconds. K's claim that it parses the full 50GB CSV in 1.6 seconds if true is very impressive regardless.
swiftcoder 2y ago
This is kind of the problem with every introductory text to an APL-family language. I get the idea that one either already knows one needs an array programming language, or doesn't grok why anyone would need one
reedf1 2y ago
K solves the problem of bank account for two groups of people, kX Systems and quants.
FjordWarden 2y ago
I've only played around with k and APL in my spare time so I can't speak to real world problems. It is a ridiculously powerful query language, where in SQL you have only started writing `SELECT ...`, in k you are already done. But you need to have very good tacit knowledge of algorithms and the weird syntax to be productive, like oh I need to calculate an integral-image of this time-series, but that just a pre-scan over addition, boom and you are done. The theory of array programming with a focus in combinators is also an interesting perspective on functional programming. IMHO not something you should write full program in, but that hasn't stopped some from trying.
- bee_rider 2y ago
  This was a helpful comment. After the article, the question that popped into my head was… so ok should I try and compare this to like BLAS or something like Jax? But, this sort of language is more about writing and reading from the disk efficiently, right? I guess SIMD type optimizations would be less of a thing.
  - FjordWarden 2y ago
    I think that array languages have historically used memory mapped files for IO, and treat them like a big data frame, but other versions also support streaming IO. Its up to the implementers of the runtime to use SIMD instructions if they deem this optimal but not something you would use yourself.
  - rak1507 2y ago
    Personally I think the best comparison would be Python+Pandas/polars+... or R+tidyverse+..., the key thing being there's less need for the "..." in a language with good table manipulation etc built in.
- Pet_Ant 2y ago
  I feel like measuring things in characters is not meaningful, but only in tokens. Replacing "SELECT" with "SEL" would not improve SQL in the slightest.
Thorrez 2y ago
A one-liner in k tends to be equivalent to a much larger program in another language. Here's a program in k. I'm not sure exactly what it does. I think it might be a json encoder/decoder: https://github.com/KxSystems/kdb/blob/master/e/json.k https://github.com/KxSystems/kdb/blob/master/e/json.k
- cubefox 2y ago
  It appears you accidentally linked to log where someone fell on his keyboard.
  - andai 2y ago
    I think Whitney's greatest achievement isn't even any of his languages—though they are very impressive—but that he convinced banks to pay him millions of dollars to write IOCCC style code!
- bregma 2y ago
  Dialup modems on a bad connection used to generate more readable code.
- saghm 2y ago
  It says a lot that the name of the file for is more informative about what the code does than the entirety of the file itself. "Readability is a property of the reader" indeed, but also the writer...
poulpy123 2y ago
The problem solved by K is the long-term employment of people writing K. You can't be fired if you're the only one understanding more or less the codebase
- dboreham 2y ago
  This is true about more software development than you realize.
deleted 2y ago
[deleted]
vessenes 2y ago
Some snark in here, I'll try and give it a fair shake. Whitney's site mentions '300 spartans' as the rough number of people using k, although it's probably more than that. Two reasons k folks like k: first, if you believe that programmer working memory, as in the number of chars or lines of code you personally can hold in your head is limited, then it might make sense to be as terse as possible -- this will significantly increase the range of things you can reason about. Second, if such a language were to focus more on array and vector-level manipulation, then for certain sorts of math tasks, you might be pretty close to grad student nirvana -- programming looks like using a chalkboard to work out a strategy for some processing, and then straightforwardly translating this strategy without mucking around with all the 100s of lines of weird shit say python or java make you do to process something in bulk and in parallel. On top of this, whitney is a mad genius, and his k interpreters tend to be SCREAMING fast, and, like a couple of hundred kilobytes compiled. Over time the language has built connections to large-scale data processing jobs (as in, you run a microsend-or-shorter-timeframe strategy based on realtime depth data from 500 different stocks, say), and it has benefitted from the path dependence you get there. Anyway back to the top - it exists as both a rallying cry for and a great tool for a certain sort of engineer that wants to make millions of dollars and refer to him/herself as a "Spartan" of coders.
skruger 2y ago
Yeah—true. I wrote it as “the missing manual” for ngn/k, enough to get someone over the initial hump. It’s not a “Mastering k” tome.
BoiledCabbage 2y ago
It's the first page of a 5 page post/book. Make sure to check out the other 4 pages linked at the footer.

sz4kerto 2y ago

Absolutely not being sarcastic: one problem it solves is that it is very hard to read as a beginner, so it can be intimidating (although it becomes much easier to read a bit later). This, coupled with the general arrogance of k/q practitioners (again, not really saying this in a negative way) and that k, kdb, etc. deliberately doesn't give you guardrails makes people who write k/q seem a bit 'mythical' and make them feel very clever. So I think k, q and kdb are fun to work with, but one of the major components of its success is that it allowed a community (in finance) to evolve that can earn 50-150% more than their peer groups who do the same work in Java or C++. 10 years ago a kx course cost $1500 per person per day.

pjmlp 2y ago
To note that those are typical prices for enterprise level certifications, including some products that some Java or C++ devs might need to interact with, when working on those kind of environments.
cppandjava 2y ago
Hmm. I work in finance writing C++ and Java and I doubt other people in finance make 50-150% more than me because they know `q`.
- gitonthescene 2y ago
  I don't know. If you're writing Java you may not be working on the same types of problems.

bear8642 2y ago

K is a fast vector language, used (primarily) for time series data analysis. >What does a K program look like? You might want to check out https://news.ycombinator.com/item?id=40335921 https://news.ycombinator.com/item?id=40335921 beagle3 and geocar both have various comments you might want to search for.

mananaysiempre 2y ago
> a fast vector language With an Oracle-style DeWitt clause[1] prohibiting public benchmarks. [1] https://mlochbaum.github.io/BQN/implementation/kclaims.html https://mlochbaum.github.io/BQN/implementation/kclaims.html
- rustc 2y ago
  Shakti (the latest K implementation by the author of K) claims [1] to load a 50gb csv in 1.6 seconds which according to them takes 265 seconds with Polars. Has anyone independently verified these claims? Is Polars really leaving 2 orders of magnitude performance on the table? [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k
  - bear8642 2y ago
    > [1]: https://shakti.com/ https://shakti.com/ -> Compare -> h2o.k You can link to the subsections: https://shakti.com/compare/h2o.k https://shakti.com/compare/h2o.k
  - orlp 2y ago
    Disclaimer: I work for Polars inc. As a sanity check I just cloned https://github.com/h2oai/db-benchmark https://github.com/h2oai/db-benchmark, ran the data generation script and ran on a 64 core AMD EPYC (AWS c7a.16xlarge): import polars as pl lf = pl.scan_csv("G1_1e9_1e2_0_0.csv") print(lf.select(pl.col.v1.sum()).collect()) The above script ran in 7.58 seconds. If I change the collect() to collect(new_streaming=True) to use the new streaming engine I've been working on, it runs in 6.90 seconds. I can't realistically time the full "read CSV to memory" with this 50 GB file on this machine as we start swapping (this machine has 128GiB memory) and/or evicting data from disk cache (this machine has a slow EC2 SSD attached to it), so we do have a blow-up of memory usage (which could be as simple as loading small integers into an 8-byte Uint64 column). I think it's likely that on K's machine the "read full CSV to memory" approach also started swapping, giving the large runtime. However, in Polars you'd typically write your query using LazyFrames, which means we don't actually have to load the full CSV into memory. EDIT: running on a m7a.16xlarge with twice the memory (256GiB) once the CSV file is in disk cache Polars can parse the full CSV file into an in-memory dataframe in 7.68 seconds. K's claim that it parses the full 50GB CSV in 1.6 seconds if true is very impressive regardless.

swiftcoder 2y ago

This is kind of the problem with every introductory text to an APL-family language. I get the idea that one either already knows one needs an array programming language, or doesn't grok why anyone would need one

reedf1 2y ago

K solves the problem of bank account for two groups of people, kX Systems and quants.

FjordWarden 2y ago

I've only played around with k and APL in my spare time so I can't speak to real world problems. It is a ridiculously powerful query language, where in SQL you have only started writing `SELECT ...`, in k you are already done. But you need to have very good tacit knowledge of algorithms and the weird syntax to be productive, like oh I need to calculate an integral-image of this time-series, but that just a pre-scan over addition, boom and you are done. The theory of array programming with a focus in combinators is also an interesting perspective on functional programming. IMHO not something you should write full program in, but that hasn't stopped some from trying.

bee_rider 2y ago
This was a helpful comment. After the article, the question that popped into my head was… so ok should I try and compare this to like BLAS or something like Jax? But, this sort of language is more about writing and reading from the disk efficiently, right? I guess SIMD type optimizations would be less of a thing.
- FjordWarden 2y ago
  I think that array languages have historically used memory mapped files for IO, and treat them like a big data frame, but other versions also support streaming IO. Its up to the implementers of the runtime to use SIMD instructions if they deem this optimal but not something you would use yourself.
- rak1507 2y ago
  Personally I think the best comparison would be Python+Pandas/polars+... or R+tidyverse+..., the key thing being there's less need for the "..." in a language with good table manipulation etc built in.
Pet_Ant 2y ago
I feel like measuring things in characters is not meaningful, but only in tokens. Replacing "SELECT" with "SEL" would not improve SQL in the slightest.

Thorrez 2y ago

A one-liner in k tends to be equivalent to a much larger program in another language. Here's a program in k. I'm not sure exactly what it does. I think it might be a json encoder/decoder: https://github.com/KxSystems/kdb/blob/master/e/json.k https://github.com/KxSystems/kdb/blob/master/e/json.k

cubefox 2y ago
It appears you accidentally linked to log where someone fell on his keyboard.
- andai 2y ago
  I think Whitney's greatest achievement isn't even any of his languages—though they are very impressive—but that he convinced banks to pay him millions of dollars to write IOCCC style code!
bregma 2y ago
Dialup modems on a bad connection used to generate more readable code.
saghm 2y ago
It says a lot that the name of the file for is more informative about what the code does than the entirety of the file itself. "Readability is a property of the reader" indeed, but also the writer...

poulpy123 2y ago

The problem solved by K is the long-term employment of people writing K. You can't be fired if you're the only one understanding more or less the codebase

dboreham 2y ago
This is true about more software development than you realize.

deleted 2y ago

[deleted]

vessenes 2y ago

Some snark in here, I'll try and give it a fair shake. Whitney's site mentions '300 spartans' as the rough number of people using k, although it's probably more than that. Two reasons k folks like k: first, if you believe that programmer working memory, as in the number of chars or lines of code you personally can hold in your head is limited, then it might make sense to be as terse as possible -- this will significantly increase the range of things you can reason about. Second, if such a language were to focus more on array and vector-level manipulation, then for certain sorts of math tasks, you might be pretty close to grad student nirvana -- programming looks like using a chalkboard to work out a strategy for some processing, and then straightforwardly translating this strategy without mucking around with all the 100s of lines of weird shit say python or java make you do to process something in bulk and in parallel. On top of this, whitney is a mad genius, and his k interpreters tend to be SCREAMING fast, and, like a couple of hundred kilobytes compiled. Over time the language has built connections to large-scale data processing jobs (as in, you run a microsend-or-shorter-timeframe strategy based on realtime depth data from 500 different stocks, say), and it has benefitted from the path dependence you get there. Anyway back to the top - it exists as both a rallying cry for and a great tool for a certain sort of engineer that wants to make millions of dollars and refer to him/herself as a "Spartan" of coders.

skruger 2y ago

Yeah—true. I wrote it as “the missing manual” for ngn/k, enough to get someone over the initial hump. It’s not a “Mastering k” tome.

BoiledCabbage 2y ago

It's the first page of a 5 page post/book. Make sure to check out the other 4 pages linked at the footer.