3 ms·
I think the problem is that most "big data" problems aren't really that big, or the latency requirements aren't that stringent. So people are OK with much slow
by steveBK123 3y ago
I think the problem is that most "big data" problems aren't really that big, or the latency requirements aren't that stringent. So people are OK with much slower solutions they can throw midlevel python devs & parallelize AWS compute at. With true time series problems you do get many "big data" problems that you can't actually parallelize (think sequential event based operations where the previous results are needed for deciding what to do with the next event).
With data frames you get some of the "close to the data" interactivity you get with a APL/J/K/Q stack. Of course, K is ~100x faster than Polars which is what.. ~10x faster than Pandas. Meanwhile most people are still using Pandas in a Jupyter notebook despite Polars being right there.
You'd think as people look at their AWS bills they'd be considering solutions that use 1/10th, 1/100th and even 1/1000th the compute time.. and yet.
- 7thaccount 3y agoThey may be breaking even after firing all their internal server people...idk. Other solutions like Kdb+ may be way more efficient, but are also crazy expensive from what I've heard. Is there a free lunch in this context?
- BMarkmann 3y agoIf you mean "free" as in open-source/free, there is J, which has its own builtin database. I'm assuming it's similar to k/kdb, but that's just a guess. Learning an APL variant... not a free lunch. Takes a while and commitment to grasp.
- jodrellblank 3y agoJ language is free, J database isn't - "Jd is a commercial database product from Jsoftware" - https://code.jsoftware.com/wiki/Jd/Overview https://code.jsoftware.com/wiki/Jd/Overview
- 7thaccount 3y agoI've played with J and read a book on it. It's expressive and powerful of course, but wasn't nearly as natural to me as APL or Kdb+.
- steveBK123 3y agoArguably "crazy expensive" software that minimizes crazy expensive hardware is something work considering. Firms pay 10s of times more for things like DataDog than they do for a KDB+ license. Also I found the argument that KDB+ devs are expensive to be laughable once I saw how much (same, or even more) we started to pay AWS/Python devs.
- 7thaccount 3y agoCrazy expensive may be affordable to firms like JP Morgan, but other industries just won't pay that when Postgres is free. It's not as good for the kind of analysis Kdb+ does IMO, but free is free and it's easier to just get a VM from IT.