4 ms·

Kdb+ [0], a very high-performance and very expensive proprietary time-series database, was written in a dialect of APL. I've seen demos of Kdb+ and it is very

by wenc 9y ago

Kdb+ [0], a very high-performance and very expensive proprietary time-series database, was written in a dialect of APL. I've seen demos of Kdb+ and it is very impressive. It is the top ranking non-GPU database in this benchmark [1]. It is however not the easiest of databases to use -- it uses Q (another APL dialect) as a query language. There was an HN discussion a while back about its maintainability and long-term prospects. [2] [0] https://en.wikipedia.org/wiki/Kdb%2B https://en.wikipedia.org/wiki/Kdb%2B [1] http://tech.marksblogg.com/benchmarks.html http://tech.marksblogg.com/benchmarks.html [2] https://news.ycombinator.com/item?id=11561573 https://news.ycombinator.com/item?id=11561573

arnon 9y ago
I wouldn't trust Marksblogg's tests. They focus on single-table "NYC Taxi" rides which don't represent most database use-cases. He also claims he takes the fastest runtime of the queries over several runs, which is not how real tests are performed.
- coldtea 9y ago
  >I wouldn't trust Marksblogg's tests. They focus on single-table "NYC Taxi" rides which don't represent most database use-cases. No, but they represent a hell of a lot of big data use cases. >He also claims he takes the fastest runtime of the queries over several runs, which is not how real tests are performed. Assuming he already checks for data not being on fs cache, what else would you take? You don't want to measure other load going on in your system and impacting your queries.
- minitoar 9y ago
  It is common practice to take the fastest runtime. This will get you as close to the actual runtime of your code as possible, assuming it's deterministic.
- jerf 9y ago
  "He also claims he takes the fastest runtime of the queries over several runs, which is not how real tests are performed." It isn't, but it should be. For any non-randomized run of the exact same work [1], by definition of non-randomized, any noise that appears in the results of multiple test runs can only be due to factors unrelated to the test being run, such as the OS decisions and such. Taking the minimum is the most honest way to factor those out. A possible exception for the first run, before CPU caches are warmed up. [1]: Ponder that closely; a lot of the knee-jerk reactions to what I just said is handled by that; e.g., if the second time something is cached by the algorithm itself that was not cached the first time around, that's not the same work. Conceptually you can imagine that I'm shooting for the exact same sequence of CPU instructions being run on the exact same data values, though in practice that can be a tall bar to leap even before considering concurrency; as with almost everything in life if you look closely enough the boundaries get fuzzy.
geocar 9y ago
You can use SQL92 as a query language (this is how people usually use ODBC with kdb), simply write "s)" in front: s)SELECT a, b, SUM(c), SUM(d), AVG(e) FROM table WHERE f > 69 GROUP BY a, b but q is much easier: select sum c, sum d, avg e by a, b from table where f > 69 Having used both q/kdb and other databases, I prefer q/kdb. Things I often hate about writing in SQL: - Those unnecessary GROUP BY statements - Too much set theory: ORDER BY - Not enough join operators. asof queries are very difficult to write. - Difficult to be performant Things I generally hate about SQL servers: - Very difficult to use cores strategically (in q I just run taskset+q on e.g. a cron job; in Pg setup is much harder requiring multiple data directories) - Very difficult to keep things in ram (tablespace + unlogged) - Very difficult to implement incremental async builds (triggers + replication is the best I've figured out in Pg; in q I just use IPC and let the kernel queue) - Switching to algorithms to make a new function is ugly and almost a different language (PL/PSQL in the best cases is a horrible language, and C is not so bad but sometimes requires root).

by wenc 9y ago

arnon 9y ago

I wouldn't trust Marksblogg's tests. They focus on single-table "NYC Taxi" rides which don't represent most database use-cases. He also claims he takes the fastest runtime of the queries over several runs, which is not how real tests are performed.

coldtea 9y ago
>I wouldn't trust Marksblogg's tests. They focus on single-table "NYC Taxi" rides which don't represent most database use-cases. No, but they represent a hell of a lot of big data use cases. >He also claims he takes the fastest runtime of the queries over several runs, which is not how real tests are performed. Assuming he already checks for data not being on fs cache, what else would you take? You don't want to measure other load going on in your system and impacting your queries.
minitoar 9y ago
It is common practice to take the fastest runtime. This will get you as close to the actual runtime of your code as possible, assuming it's deterministic.
jerf 9y ago
"He also claims he takes the fastest runtime of the queries over several runs, which is not how real tests are performed." It isn't, but it should be. For any non-randomized run of the exact same work [1], by definition of non-randomized, any noise that appears in the results of multiple test runs can only be due to factors unrelated to the test being run, such as the OS decisions and such. Taking the minimum is the most honest way to factor those out. A possible exception for the first run, before CPU caches are warmed up. [1]: Ponder that closely; a lot of the knee-jerk reactions to what I just said is handled by that; e.g., if the second time something is cached by the algorithm itself that was not cached the first time around, that's not the same work. Conceptually you can imagine that I'm shooting for the exact same sequence of CPU instructions being run on the exact same data values, though in practice that can be a tall bar to leap even before considering concurrency; as with almost everything in life if you look closely enough the boundaries get fuzzy.

geocar 9y ago

You can use SQL92 as a query language (this is how people usually use ODBC with kdb), simply write "s)" in front: s)SELECT a, b, SUM(c), SUM(d), AVG(e) FROM table WHERE f > 69 GROUP BY a, b but q is much easier: select sum c, sum d, avg e by a, b from table where f > 69 Having used both q/kdb and other databases, I prefer q/kdb. Things I often hate about writing in SQL: - Those unnecessary GROUP BY statements - Too much set theory: ORDER BY - Not enough join operators. asof queries are very difficult to write. - Difficult to be performant Things I generally hate about SQL servers: - Very difficult to use cores strategically (in q I just run taskset+q on e.g. a cron job; in Pg setup is much harder requiring multiple data directories) - Very difficult to keep things in ram (tablespace + unlogged) - Very difficult to implement incremental async builds (triggers + replication is the best I've figured out in Pg; in q I just use IPC and let the kernel queue) - Switching to algorithms to make a new function is ugly and almost a different language (PL/PSQL in the best cases is a horrible language, and C is not so bad but sometimes requires root).