6 ms·

KDB+/Q/K is all about analytics too. It is used in more than just financial time series data. Such as power utility usage and resourcing for one. Hadoop is als

by eggy 10y ago

KDB+/Q/K is all about analytics too. It is used in more than just financial time series data. Such as power utility usage and resourcing for one. Hadoop is also about distributed computing, but if the process requires 250x as much RAM, you're going to have a very high TCO regardless of how cheap RAM or servers can be. J is an opensource APL-derived language, but different in some important ways, but it is not as fast as KDB+/Q/K. J is truly an array-based language, whereas K is list based, so it has more in common with Lisp in that singular respect. Kerf is a new language being worked on by one of the creators of the Kona language, an opensource version of the K3 language [1]. It seems these type of articles ignore non-opensource solutions even if they are more efficient in many ways including TCO. How many man-years need to be spent to try and duplicate an existing solution, and still not be a better solution? [1] https://github.com/kevinlawler/kerf https://github.com/kevinlawler/kerf

scottlocklin 10y ago
I often feel like the one eyed man trying to describe color and shape to a world of blind people when describing the power of these programming systems. I realize everyone wants free stuff, but folks seem to find it difficult to comprehend how much more powerful and performant some of the pay-for systems, in particular the array(ish) systems are than stuff like Spark. In principle, peer review should make all open source systems superior. In practice, the people most capable of making something like Spark fast work for Oracle, Microsoft or Kx on database and machine learning engines people pay money for. While much software development is just hooking components together, there is still such a thing as software engineering, and those skills are extremely rare. Even looking at humble stuff like file systems... I thought EXT4 was pretty good until reading of the wonders of ZFS. Thanks for mentioning Kerf. I'll also mention Nial, which has had some exciting developments lately.
srean 10y ago
I completely agree with you. Compared to the bloatware of the Hadoop stack K looks and feels like lightning. But when evaluated on its own K is less magical. Impenetrable code not-withstanding its design is fairly simple (good thing TM) and not that hard to beat. If one could delay code-gen even further, it would be possible to inline and compose even bigger chunks of operations into code giving more speed up. A well JIT'ed numpy or Judy should (famous last words) be up there with K. Since it has not been mentioned yet, let me point out that Barry Jay's Fish should be a part of this conversation. Sadly its quite hard to find. I don't know why those pages have disappeared.

by eggy 10y ago

scottlocklin 10y ago

I often feel like the one eyed man trying to describe color and shape to a world of blind people when describing the power of these programming systems. I realize everyone wants free stuff, but folks seem to find it difficult to comprehend how much more powerful and performant some of the pay-for systems, in particular the array(ish) systems are than stuff like Spark. In principle, peer review should make all open source systems superior. In practice, the people most capable of making something like Spark fast work for Oracle, Microsoft or Kx on database and machine learning engines people pay money for. While much software development is just hooking components together, there is still such a thing as software engineering, and those skills are extremely rare. Even looking at humble stuff like file systems... I thought EXT4 was pretty good until reading of the wonders of ZFS. Thanks for mentioning Kerf. I'll also mention Nial, which has had some exciting developments lately.

srean 10y ago

I completely agree with you. Compared to the bloatware of the Hadoop stack K looks and feels like lightning. But when evaluated on its own K is less magical. Impenetrable code not-withstanding its design is fairly simple (good thing TM) and not that hard to beat. If one could delay code-gen even further, it would be possible to inline and compose even bigger chunks of operations into code giving more speed up. A well JIT'ed numpy or Judy should (famous last words) be up there with K. Since it has not been mentioned yet, let me point out that Barry Jay's Fish should be a part of this conversation. Sadly its quite hard to find. I don't know why those pages have disappeared.