4 ms·

Calling anything from Arthur "vaporware" is really laughable. This article isn't great, but it's from a finance site, not a technical site: of course it's goin

by kick 7y ago

Calling anything from Arthur "vaporware" is really laughable. This article isn't great, but it's from a finance site, not a technical site: of course it's going to be low on technical details. You would think that people who want technical details would know how to use a search engine. Further, kdb/kdb+ (which Shakti is the successor of) is a columnar-based database. "How many results can it return [...] from a table with how many rows?" is completely irrelevant. There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled.

deleted 7y ago
[deleted]
mhd 7y ago
> Calling anything from Arthur "vaporware" is really laughable. kOS?
- beagle3 7y ago
  It was a research project, and it actually booted on bare metal and ran what it was supposed to (milspec bare bones os with gui, filesysten, db, etc in a few hundred KB)
  - sansnomme 7y ago
    Do you have a link? This sounds fascinating.
    - chrispsn 7y ago
      https://gist.github.com/chrispsn/da00835bb122c42f429a084df83fd746 https://gist.github.com/chrispsn/da00835bb122c42f429a084df83...
- kick 7y ago
  kOS exists, at least one HN user has used it ('geocar). As a research operating system, seeing it released into the public was never a likely goal, and as far as I know no one claimed it would be..
iskander 7y ago
>There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled. Eh, I worked on array languages in grad school and the speed of k was significantly over-hyped. It's trivial to beet with Python and no contest if you want to work in a compiled language. Wall Street develops weird religions around tech.
- beagle3 7y ago
  In my experience it is mostly trivial to beat with C if you adopt columnar storage (which isn’t idiomatic), but even numexpr doesn’t beat k (though it seems like it should), and numpy is still way behind; pandas and plain python not even in the same ballpark. What approach did you use? The one thing nothing beats k on is functionality per time spent (if you are proficient). It is often 100 times shorter but only takes 10 times as long to write.
  - iskander 7y ago
    I'm going to sidestep arguments about its terseness since some people are into that and it's really an aesthetic choice. But for machine learning or numerical algorithms, I found it quite hard to make k nearly as fast as vanilla NumPy code. Overall there's really no good reason for it to be faster. It didn't have (as of ~2011) a JIT or any mechanism for operator fusion, so you end up allocating many intermediates, sometimes using quadratic space which gets immediately reduced along some axis. Yeah, I know "Arthur Whitney is really smart"...but that's not actually a technical reason why k would be faster. I don't remember the specifics but I think that many of the array operators weren't even multi-threaded and I remember only a few of the array operators getting FLOPs that would correspond to SIMD acceleration. So, for the most part, it seemed like a lot of vanilla single threaded C code behind the operators. It's possible that the implementation got smarter over the past decade, but when I was working in the space it seemed like the hagiography and actual performance were worlds apart.
    - chrispsn 7y ago
      At the least, 'range' (!3 -> 0 1 2) is now deferred (including with offsets eg 1+!3).
anon1m0us 7y ago
Shakti themselves used rows as the comparison metric in the original link: https://shakti.com/database-software-history/ https://shakti.com/database-software-history/ Again, saying the speed of k is unrivaled is hype speak. You're adding negative value to the discussion.
- zerozerocool 7y ago
  To give you an idea, here are some benchmarks comparing Python (with pandas/numpy) vs k (aka q/kdb+). https://www.linkedin.com/pulse/data-analysis-example-python-q-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/data-analysis-example-python-... https://www.linkedin.com/pulse/lists-python-q-side-by-side-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/lists-python-q-side-by-side-f... I think that software history page is meant to convey general workloads achievable, rather than performance. -(not the author of the benchmarks nor an employee of shakti FWIW)
typon 7y ago
It really depends on what specific benchmarks you're talking about, but I would be really interested in seeing benchmarks in which k beats C with AVX

by kick 7y ago

deleted 7y ago

[deleted]

mhd 7y ago

> Calling anything from Arthur "vaporware" is really laughable. kOS?

beagle3 7y ago
It was a research project, and it actually booted on bare metal and ran what it was supposed to (milspec bare bones os with gui, filesysten, db, etc in a few hundred KB)
- sansnomme 7y ago
  Do you have a link? This sounds fascinating.
  - chrispsn 7y ago
    https://gist.github.com/chrispsn/da00835bb122c42f429a084df83fd746 https://gist.github.com/chrispsn/da00835bb122c42f429a084df83...
kick 7y ago
kOS exists, at least one HN user has used it ('geocar). As a research operating system, seeing it released into the public was never a likely goal, and as far as I know no one claimed it would be..

iskander 7y ago

>There are hundreds of benchmarks for kdb+, and the speed of k is unrivaled. Eh, I worked on array languages in grad school and the speed of k was significantly over-hyped. It's trivial to beet with Python and no contest if you want to work in a compiled language. Wall Street develops weird religions around tech.

beagle3 7y ago
In my experience it is mostly trivial to beat with C if you adopt columnar storage (which isn’t idiomatic), but even numexpr doesn’t beat k (though it seems like it should), and numpy is still way behind; pandas and plain python not even in the same ballpark. What approach did you use? The one thing nothing beats k on is functionality per time spent (if you are proficient). It is often 100 times shorter but only takes 10 times as long to write.
- iskander 7y ago
  I'm going to sidestep arguments about its terseness since some people are into that and it's really an aesthetic choice. But for machine learning or numerical algorithms, I found it quite hard to make k nearly as fast as vanilla NumPy code. Overall there's really no good reason for it to be faster. It didn't have (as of ~2011) a JIT or any mechanism for operator fusion, so you end up allocating many intermediates, sometimes using quadratic space which gets immediately reduced along some axis. Yeah, I know "Arthur Whitney is really smart"...but that's not actually a technical reason why k would be faster. I don't remember the specifics but I think that many of the array operators weren't even multi-threaded and I remember only a few of the array operators getting FLOPs that would correspond to SIMD acceleration. So, for the most part, it seemed like a lot of vanilla single threaded C code behind the operators. It's possible that the implementation got smarter over the past decade, but when I was working in the space it seemed like the hagiography and actual performance were worlds apart.
  - chrispsn 7y ago
    At the least, 'range' (!3 -> 0 1 2) is now deferred (including with offsets eg 1+!3).

anon1m0us 7y ago

Shakti themselves used rows as the comparison metric in the original link: https://shakti.com/database-software-history/ https://shakti.com/database-software-history/ Again, saying the speed of k is unrivaled is hype speak. You're adding negative value to the discussion.

zerozerocool 7y ago
To give you an idea, here are some benchmarks comparing Python (with pandas/numpy) vs k (aka q/kdb+). https://www.linkedin.com/pulse/data-analysis-example-python-q-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/data-analysis-example-python-... https://www.linkedin.com/pulse/lists-python-q-side-by-side-ferenc-bodon-ph-d-/ https://www.linkedin.com/pulse/lists-python-q-side-by-side-f... I think that software history page is meant to convey general workloads achievable, rather than performance. -(not the author of the benchmarks nor an employee of shakti FWIW)

typon 7y ago

It really depends on what specific benchmarks you're talking about, but I would be really interested in seeing benchmarks in which k beats C with AVX