4 ms·
I don't think the authors claim that "The only thing comparable to it from an engineering standpoint are the other APL based databases such as KDB and Jd" is co
by yummyfajitas 11y ago
I don't think the authors claim that "The only thing comparable to it from an engineering standpoint are the other APL based databases such as KDB and Jd" is correct, at least not based on this blog post. Numpy+pandas also seems pretty similar, as is breeze, and Julia has similar features also.
Let me emphasize that while I'm about to implement most of his blog post in numpy/pandas, I don't mean to come off as critical or dismiss building a new one. Kerf looks pretty cool. I just don't think this post gives much reason to believe there is nothing comparable to it.
Idiomatic numpy:
def wavg(x,y):
return sum(x*y) / sum(x)
Less idiomatic, but analogous to kerf or J:
def wavg(x,y):
return add.reduce(x*y) / add.reduce(x)
arange(1,6) is equivalent to his range(5), and full((5,), 1) is equivalent to repeat(5,1).
The SQL-ish engine looks fairly similar to pandas:
df = pandas.DataFrame({'a' : arange(5), 'b' : ['a', 'b', 'c', 'd', 'e']})
#Result is:
a b
0 0 a
1 1 b
2 2 c
3 3 d
4 4 e
df[df.a > 2]
#Result is
a b
3 3 d
4 4 e
Pandas similarly has joins, averages and other aggregates, etc. E.g.:
pandas.merge(df1, df2, how='inner', left_on='a', right_on='b')
is equivalent to:
SELECT * FROM df1 INNER JOIN df2 ON df1.a = df2.b;
Syntactically it's a bit different (embedded SQL is pretty cool) but from this blog post I don't see any major semantic differences.
However, if there are important differences under the hood, I'd love to hear about them. Again, I'm really not trying to criticize Locklin for rolling his own, I'm just disputing a minor point.
- scottlocklin 11y agoThere are lots of "table" like solutions out there: R's data.frame and data.table are obvious examples. Speed, the ability to persist data as an actual database system, and the ability to work with data "out of core" would be the advantages over Pandas. I'll be releasing various performance comparisons over coming weeks to demonstrate these advantages.
- yummyfajitas 11y agoCool, any chance you'll write a blog post discussing that? Being a good column oriented DB combined with a decent vectorized language is definitely not something so easy to find - blaze purports to be that system (sort of), but it isn't really living up to the hype.
- scottlocklin 11y agoOf course we'll blog on this topic! Just not today. That's what the APL world brings to the table. FWIIW, you can read about my comparing Jd (another APL related columnar database) to a Blaze like tool (Spark bigDF) on the J list: http://jsoftware.com/pipermail/database/2015-May/000076.html http://jsoftware.com/pipermail/database/2015-May/000076.html