3 ms·
Not analyze. Transform. I'm not talking about a query. But, tell me more. I always like a better way. Is it more efficient to read/write from SSD? Right now,
by afpx 4y ago
Not analyze. Transform. I'm not talking about a query.
But, tell me more. I always like a better way.
Is it more efficient to read/write from SSD? Right now, I need everything in memory.
- berkle4455 4y agoa single node of clickhouse would likely be sufficient for a fraction of the cost and time. use attached nvme disks (ebs on aws) or a physical local disk.
- eskaytwo 4y agoPolars would be worth looking into too. $50 and hours seems orders of magnitude too high vig 10tb
- rnk 4y agoI don't want to pile on, but I do see startup after startup that is focused on improving the cost/perf payoff of pyspark. Pyspark seems to be basically implemented by having the databricks query planner embedded in it, then it generates multiple queries to evaluate it. I think there's nothing inherent in that process that should make it faster than raw sql queries. There's theoretical advantages of maybe looking at the overall giant pyspark query and decomposing it into efficient pieces. Compare that to someone who wrote a series of separate sql queries to accomplish the same thing, and the qp can't take advantage of any reused computation.
- threeseed 4y ago> I think there's nothing inherent in that process that should make it faster than raw sql queries You can choose when to cache the data in memory. And it's the fact that you can write the code in Python, Scala etc instead of SQL.