6 ms·

>kdb+ is so fast and powerful you'd think it impossible. I suspect there is a religious/mythological aspect to K/Q's reputation for speed. During the year that

>kdb+ is so fast and powerful you'd think it impossible. I suspect there is a religious/mythological aspect to K/Q's reputation for speed. During the year that I was using Q I often found it to be slower than the equivalent Matlab code (which benefits from a JIT) and or even NumPy (which has many naively implemented operations).

bladecatcher 10y ago
IMHO, I feel Matlab is too clunky and inelegant for data transformation compared to APL-inspired languages, which needs a slightly mathematically inclined user to appreciate and unleash it's true power, with 10 times less code. The biggest advantage of a language like Q is that it's inherently parallel. Map-reduce is naturally built into the language, so this allows the users to think in terms of breaking problems down into sub-problems and parallelizing them via the "parallel apply" construct[1] Also another very powerful feature is the braindead simplicity of Inter process communication between KDB+ instances [2] [1] http://code.kx.com/wiki/Reference/peach http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Startingkdbplus/ipc http://code.kx.com/wiki/Startingkdbplus/ipc This makes a real difference in ad-hoc environments where you want to setup and tear down complex trading software in real-time
- iskander 10y ago
  > Map-reduce is naturally built into the language I hear this often about both array and functional languages but I think this kind of slogan leads to a misunderstanding of the MapReduce framework (and what's special about it). In a functional or array language, map and reduce have the following signatures: map : key list -> (key -> value) -> value list reduce : value list -> (value, value -> value) -> value These are strictly less powerful than the functions used in MapReduce: read : block -> (input_key, input_value) list map : (input_key, input_value) -> (output_key, output_value) list reduce : (output_key, (output_value list)) -> (output_value, output_value -> output_value) -> (output_key, output_value) Combining this API with a (1) a distributed file system which lets you co-locate computation with data across many computers and (2) a parallel partitioning algorithm enables petabyte-scale computation. It's qualitatively different than in-memory array processing.
  - icsa 10y ago
    kdb+ map/reduce implementation operates across data in memory or on machines in a cluster.
- Athas 10y ago
  Does the Q/K implementation actually execute in parallel? I've always had a hard time figuring out exactly why K is supposed to be so fast (except for column stores, interpreter fitting in L1 cache, and fast hand-written primitives).
  - bladecatcher 10y ago
    Yes, you can start the server with any number of slave processors. As long as you're not writing data to global namespace (i.e, if you can keep your code purely functional), then as pointed out above, one can use the "peach" (parallel each) operator [1] to parallelize execution. Further you also get the "distributed each" to distribute load amongst multiple processes instead of threads [2] [1] http://code.kx.com/wiki/Reference/peach http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Reference/peach#Peach_using_multiple_processes_.28Distributed_each.29 http://code.kx.com/wiki/Reference/peach#Peach_using_multiple...
easytiger 10y ago
It's fast compared to other DBs because most datasets are in RAM
- rodionos 10y ago
  Exactly. I'd say it closest comparable is CEP.
- icsa 10y ago
  kdb+ also supports Petabyte distributed databases. i.e. much larger than RAM. E.g. http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt (Trillion Row Benchmarks)
gricardo99 10y ago
>I suspect there is a religious/mythological aspect to K/Q's reputation for speed It depends what you're doing, but the vast superiority of kdb+ versus its competitors has been demonstrated: http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt
- iskander 10y ago
  >machine: 16 core 256GB (in all cases: date partition, sym index. all queries in RAM.) ... >all query data is cached in RAM (no disk access). 1) What happens when you run these benchmarks on a machine with only 16GB of RAM? 2) How does the KDB performance compare to doing the equivalent operations on a Pandas DataFrame (which, since these are simple in-memory operations, seems like the only fair comparison).
  - icsa 10y ago
    > 1) What happens when you run these benchmarks on a machine with only 16GB of RAM? kdb+ uses ~ 1.2 MB of Resident RAM at startup. In addition, kdb+ storage model in memory and on disk have very little overhead (a few hundred bytes) per column over the raw binary data. The above, combined with memory mapping, allow for large databases to be queried with great performance. I query billion row futures databases on a Macbook Pro with 16 GB RAM with kdb+. > 2) How does the KDB performance compare to doing the equivalent operations on a Pandas DataFrame (which, since these are simple in-memory operations, seems like the only fair comparison). Could someone provide the equivalent Pandas code running against similar TAQ data for the queries in the following benchmark? http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt

bladecatcher 10y ago

IMHO, I feel Matlab is too clunky and inelegant for data transformation compared to APL-inspired languages, which needs a slightly mathematically inclined user to appreciate and unleash it's true power, with 10 times less code. The biggest advantage of a language like Q is that it's inherently parallel. Map-reduce is naturally built into the language, so this allows the users to think in terms of breaking problems down into sub-problems and parallelizing them via the "parallel apply" construct[1] Also another very powerful feature is the braindead simplicity of Inter process communication between KDB+ instances [2] [1] http://code.kx.com/wiki/Reference/peach http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Startingkdbplus/ipc http://code.kx.com/wiki/Startingkdbplus/ipc This makes a real difference in ad-hoc environments where you want to setup and tear down complex trading software in real-time

iskander 10y ago
> Map-reduce is naturally built into the language I hear this often about both array and functional languages but I think this kind of slogan leads to a misunderstanding of the MapReduce framework (and what's special about it). In a functional or array language, map and reduce have the following signatures: map : key list -> (key -> value) -> value list reduce : value list -> (value, value -> value) -> value These are strictly less powerful than the functions used in MapReduce: read : block -> (input_key, input_value) list map : (input_key, input_value) -> (output_key, output_value) list reduce : (output_key, (output_value list)) -> (output_value, output_value -> output_value) -> (output_key, output_value) Combining this API with a (1) a distributed file system which lets you co-locate computation with data across many computers and (2) a parallel partitioning algorithm enables petabyte-scale computation. It's qualitatively different than in-memory array processing.
- icsa 10y ago
  kdb+ map/reduce implementation operates across data in memory or on machines in a cluster.
Athas 10y ago
Does the Q/K implementation actually execute in parallel? I've always had a hard time figuring out exactly why K is supposed to be so fast (except for column stores, interpreter fitting in L1 cache, and fast hand-written primitives).
- bladecatcher 10y ago
  Yes, you can start the server with any number of slave processors. As long as you're not writing data to global namespace (i.e, if you can keep your code purely functional), then as pointed out above, one can use the "peach" (parallel each) operator [1] to parallelize execution. Further you also get the "distributed each" to distribute load amongst multiple processes instead of threads [2] [1] http://code.kx.com/wiki/Reference/peach http://code.kx.com/wiki/Reference/peach [2] http://code.kx.com/wiki/Reference/peach#Peach_using_multiple_processes_.28Distributed_each.29 http://code.kx.com/wiki/Reference/peach#Peach_using_multiple...

easytiger 10y ago

It's fast compared to other DBs because most datasets are in RAM

rodionos 10y ago
Exactly. I'd say it closest comparable is CEP.
icsa 10y ago
kdb+ also supports Petabyte distributed databases. i.e. much larger than RAM. E.g. http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt (Trillion Row Benchmarks)

gricardo99 10y ago

>I suspect there is a religious/mythological aspect to K/Q's reputation for speed It depends what you're doing, but the vast superiority of kdb+ versus its competitors has been demonstrated: http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt

iskander 10y ago
>machine: 16 core 256GB (in all cases: date partition, sym index. all queries in RAM.) ... >all query data is cached in RAM (no disk access). 1) What happens when you run these benchmarks on a machine with only 16GB of RAM? 2) How does the KDB performance compare to doing the equivalent operations on a Pandas DataFrame (which, since these are simple in-memory operations, seems like the only fair comparison).
- icsa 10y ago
  > 1) What happens when you run these benchmarks on a machine with only 16GB of RAM? kdb+ uses ~ 1.2 MB of Resident RAM at startup. In addition, kdb+ storage model in memory and on disk have very little overhead (a few hundred bytes) per column over the raw binary data. The above, combined with memory mapping, allow for large databases to be queried with great performance. I query billion row futures databases on a Macbook Pro with 16 GB RAM with kdb+. > 2) How does the KDB performance compare to doing the equivalent operations on a Pandas DataFrame (which, since these are simple in-memory operations, seems like the only fair comparison). Could someone provide the equivalent Pandas code running against similar TAQ data for the queries in the following benchmark? http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt