4 ms·
I gave a talk at some kx/kdb usergroups[2], and definitely saw more than 12 people there. There were hundreds. Its not a trivial language to learn, you have t
by hpcjoe 6y ago
I gave a talk at some kx/kdb usergroups[2], and definitely saw more than 12 people there. There were hundreds.
Its not a trivial language to learn, you have to be ok with very terse languages in general, and ok using the language as a way to mentally model the algorithm. Somewhat "mathematically".
I'm not using/working with it any more, haven't in a few years. Most of my recent work is in Julia, by choice.
Two very different philosophies at work here. Julia provides a sane, more or less classical language to work with, inclusive of numerous macros and modules to help drive productivity. k/q provides a fairly modern apl like experience, leveraging Iverson's idea of a programming language being a "tool for thought".
FWIW, the forces behind k/q (Arthur Whitney, et al) are now working on Shakti[1]. Looks like an interesting project, more of a platform than kdb was.
Bringing this back to JuliaDB, I had been hoping for a very julia centric/idiomatic persistence engine as well as an analytical engine. I'm no fan of SQL or its variations. In reading this, I've learned DataFrames is supposed to be this. However, my impression working with DataFrames has been that it is woefully slow on small data sets (100-ish rows). It cannot handle one of the slightly larger 50k line tabular data sets that I am importing via excel sheets.
For that work, I may need to resort to Perl (my fall back programming/data munging language). This is important, as Perl has a capability of tieing (binding) a variable to an underlying data structure/file/... . This enables trivial persistence, in a completely idiomatic manner (update/insert into the tied variable by setting a value). It would be interesting if this capability existed within julia as well.
[1] https://shakti.com/ https://shakti.com/
[2] https://kx.com/media/2018/03/joe_preview.jpeg https://kx.com/media/2018/03/joe_preview.jpeg (actual image of me talking)
- smabie 6y agoMy impression hasn't been that DataFrame are slow at all. What do you mean it can't handle 50k rows? Like it takes a long time to load or what? Because right now I'm using a DataFrame that has 7m rows and 50 columns and it works very well.
- hpcjoe 6y agoHmm ... could be some of the other tooling around it. Right now I can't get XLSX to talk reliably to DataFrame. I rigged up some tests for 10, 100, 1000, ... points and performance above about 500 points fell off a cliff. I should look into this more. Thanks for letting me know it can handle things that size.