4 ms·

>> This style of processing, invented by columnar database systems such as MonetDB and C-Store True, since MonetDB came of out of the Netherlands around 1993,

by eggy 10y ago

>> This style of processing, invented by columnar database systems such as MonetDB and C-Store True, since MonetDB came of out of the Netherlands around 1993, but KDB+ came out in 1998, well over 8 years before C-Store. K4 the language used in KDB is 230 times faster than Spark/shark and uses 0.2GB of RAM vs. 50GB of RAM for Spark/shark, yet no mention in the article. It seems a strange omission for such a sensational sounding title [1]. I don't understand why big data startups don't try and remake the success of KDB instead of reinventing bits and pieces of the same tech with a result in slower DB operations and more RAM usage. [1] http://kparc.com/q4/readme.txt http://kparc.com/q4/readme.txt

threeseed 10y ago
Because Spark is far, far more than just about storing data and doing basic queries. It is also an analytics e.g. machine learning/modelling platform and a framework for building complex applications on top of Hadoop.
- scottlocklin 10y ago
  You obviously have no idea what you're talking about. You can do all of this in Kx systems as well, and it runs a lot faster than spark. I beat a databricks stack (hand tuned by one of the authors of big-DF) running on a large cluster using one jd node by factors of "a lot." And K is faster than J.
- eggy 10y ago
  KDB+/Q/K is all about analytics too. It is used in more than just financial time series data. Such as power utility usage and resourcing for one. Hadoop is also about distributed computing, but if the process requires 250x as much RAM, you're going to have a very high TCO regardless of how cheap RAM or servers can be. J is an opensource APL-derived language, but different in some important ways, but it is not as fast as KDB+/Q/K. J is truly an array-based language, whereas K is list based, so it has more in common with Lisp in that singular respect. Kerf is a new language being worked on by one of the creators of the Kona language, an opensource version of the K3 language [1]. It seems these type of articles ignore non-opensource solutions even if they are more efficient in many ways including TCO. How many man-years need to be spent to try and duplicate an existing solution, and still not be a better solution? [1] https://github.com/kevinlawler/kerf https://github.com/kevinlawler/kerf
  - scottlocklin 10y ago
    I often feel like the one eyed man trying to describe color and shape to a world of blind people when describing the power of these programming systems. I realize everyone wants free stuff, but folks seem to find it difficult to comprehend how much more powerful and performant some of the pay-for systems, in particular the array(ish) systems are than stuff like Spark. In principle, peer review should make all open source systems superior. In practice, the people most capable of making something like Spark fast work for Oracle, Microsoft or Kx on database and machine learning engines people pay money for. While much software development is just hooking components together, there is still such a thing as software engineering, and those skills are extremely rare. Even looking at humble stuff like file systems... I thought EXT4 was pretty good until reading of the wonders of ZFS. Thanks for mentioning Kerf. I'll also mention Nial, which has had some exciting developments lately.
  - srean 10y ago
    I completely agree with you. Compared to the bloatware of the Hadoop stack K looks and feels like lightning. But when evaluated on its own K is less magical. Impenetrable code not-withstanding its design is fairly simple (good thing TM) and not that hard to beat. If one could delay code-gen even further, it would be possible to inline and compose even bigger chunks of operations into code giving more speed up. A well JIT'ed numpy or Judy should (famous last words) be up there with K. Since it has not been mentioned yet, let me point out that Barry Jay's Fish should be a part of this conversation. Sadly its quite hard to find. I don't know why those pages have disappeared.
turk- 10y ago
Interesting, I've never heard of kdb. Probably because the pricing is absurd from the little research that I have done so far: http://quant.stackexchange.com/questions/3156/is-there-any-thing-out-there-as-a-substitute-for-kdb http://quant.stackexchange.com/questions/3156/is-there-any-t... $25k for 2 core setup is a little crazy. They should definitely add a free tier if they want more adoption and more people investigating their tech. Spark is open source and free.
- eggy 10y ago
  Regardless of price, the tech has been around for quite a few years, and it beats the tech mentioned in this article and in most similar articles making similar claims. As I replied below, I am just surprised that similar technology has not been created by the opensource community that is closer in performance to KDB+. $25k may seem absurd to you, but if it delivers results 230x faster using 1/250th of the RAM, decision makers in business see the TCO (total cost of ownership) as being worth it. It all depends on your needs and how they are met. Jd is the commercial database for the opensource language J, that is also fast, and I believe it costs less than KDB, so there are other faster options. If free as in opensource is your criteria, and not actual cost-benefits analysis, you only have those free choices, but in business time is money, so the 'free' option would be more costly.

by eggy 10y ago

threeseed 10y ago

Because Spark is far, far more than just about storing data and doing basic queries. It is also an analytics e.g. machine learning/modelling platform and a framework for building complex applications on top of Hadoop.

scottlocklin 10y ago
You obviously have no idea what you're talking about. You can do all of this in Kx systems as well, and it runs a lot faster than spark. I beat a databricks stack (hand tuned by one of the authors of big-DF) running on a large cluster using one jd node by factors of "a lot." And K is faster than J.
eggy 10y ago
KDB+/Q/K is all about analytics too. It is used in more than just financial time series data. Such as power utility usage and resourcing for one. Hadoop is also about distributed computing, but if the process requires 250x as much RAM, you're going to have a very high TCO regardless of how cheap RAM or servers can be. J is an opensource APL-derived language, but different in some important ways, but it is not as fast as KDB+/Q/K. J is truly an array-based language, whereas K is list based, so it has more in common with Lisp in that singular respect. Kerf is a new language being worked on by one of the creators of the Kona language, an opensource version of the K3 language [1]. It seems these type of articles ignore non-opensource solutions even if they are more efficient in many ways including TCO. How many man-years need to be spent to try and duplicate an existing solution, and still not be a better solution? [1] https://github.com/kevinlawler/kerf https://github.com/kevinlawler/kerf
- scottlocklin 10y ago
  I often feel like the one eyed man trying to describe color and shape to a world of blind people when describing the power of these programming systems. I realize everyone wants free stuff, but folks seem to find it difficult to comprehend how much more powerful and performant some of the pay-for systems, in particular the array(ish) systems are than stuff like Spark. In principle, peer review should make all open source systems superior. In practice, the people most capable of making something like Spark fast work for Oracle, Microsoft or Kx on database and machine learning engines people pay money for. While much software development is just hooking components together, there is still such a thing as software engineering, and those skills are extremely rare. Even looking at humble stuff like file systems... I thought EXT4 was pretty good until reading of the wonders of ZFS. Thanks for mentioning Kerf. I'll also mention Nial, which has had some exciting developments lately.
- srean 10y ago
  I completely agree with you. Compared to the bloatware of the Hadoop stack K looks and feels like lightning. But when evaluated on its own K is less magical. Impenetrable code not-withstanding its design is fairly simple (good thing TM) and not that hard to beat. If one could delay code-gen even further, it would be possible to inline and compose even bigger chunks of operations into code giving more speed up. A well JIT'ed numpy or Judy should (famous last words) be up there with K. Since it has not been mentioned yet, let me point out that Barry Jay's Fish should be a part of this conversation. Sadly its quite hard to find. I don't know why those pages have disappeared.

turk- 10y ago

Interesting, I've never heard of kdb. Probably because the pricing is absurd from the little research that I have done so far: http://quant.stackexchange.com/questions/3156/is-there-any-thing-out-there-as-a-substitute-for-kdb http://quant.stackexchange.com/questions/3156/is-there-any-t... $25k for 2 core setup is a little crazy. They should definitely add a free tier if they want more adoption and more people investigating their tech. Spark is open source and free.

eggy 10y ago
Regardless of price, the tech has been around for quite a few years, and it beats the tech mentioned in this article and in most similar articles making similar claims. As I replied below, I am just surprised that similar technology has not been created by the opensource community that is closer in performance to KDB+. $25k may seem absurd to you, but if it delivers results 230x faster using 1/250th of the RAM, decision makers in business see the TCO (total cost of ownership) as being worth it. It all depends on your needs and how they are met. Jd is the commercial database for the opensource language J, that is also fast, and I believe it costs less than KDB, so there are other faster options. If free as in opensource is your criteria, and not actual cost-benefits analysis, you only have those free choices, but in business time is money, so the 'free' option would be more costly.