4 ms·

Servers grow much bigger. 256GB of RAM is pretty much a standard rack server on a major cloud. On AWS, ultra memory with 24TB of RAM is one API call away from y

Servers grow much bigger. 256GB of RAM is pretty much a standard rack server on a major cloud. On AWS, ultra memory with 24TB of RAM is one API call away from you :-). The article took note of that, but tooling on one node puzzles me. So, multi TB can still be a single node with DuckDB. It's also rather a small ClickHouse cluster. It sounds easier to use DuckDB than the proposed tools. In many use cases, if it's not 24/7, you can use a data lake (e.g. Iceberg) and query it if needed. Databricks seems a way to go, since the author uses Spark.

mulmen 2y ago
What do you mean by “not 24/7”? Why would a data lake go offline?
- pantsforbirds 2y ago
  It's pretty common to run analytics workloads on a fixed schedule. I've setup anlaytics workflows that ingest data every day/week/quarter and generate a report once a day. With workflows like that something like DuckDB can be amazing
- jakozaur 2y ago
  It is very common that the majority of work happens during business hours. In some setups it's easy. It's better to get beefy instance during that time and avoid having it running all the time.
- dimaor 2y ago
  not sure, but maybe it's possible only to update the model in a specific time? are there other uses to the data apart from learning and validation?
- creshal 2y ago
  Probably in the sense of "you're only running the really heavy analytics queries once a day/week/month", and thus don't constantly need the 24TB instances.
mritchie712 2y ago
Iceberg is working hard to support pure python[0] / rust[1] workflows without Spark. Following Tabular's acquisition [2], I hope it still moves in this direction at the same clip (very impressed with the speed of development on pyiceberg). We're using iceberg + duckdb to power analytics in our app[3] and I'm really happy with the combo. 0 - https://github.com/apache/iceberg-python https://github.com/apache/iceberg-python 1 - https://github.com/apache/iceberg-rust https://github.com/apache/iceberg-rust 2 - https://x.com/thisritchie/status/1800522255426072647 https://x.com/thisritchie/status/1800522255426072647 3 - https://www.definite.app/ https://www.definite.app/
- alexott 2y ago
  You can already have it in Delta with Delta Rust and Python bindings: https://github.com/delta-io/delta-rs https://github.com/delta-io/delta-rs
  - mritchie712 2y ago
    yes, we're evaluating Delta. We went with Iceberg out of concern that Delta was too closely tied to Databricks. Following the Tabular acquisition, the decision is murkier.

mulmen 2y ago

What do you mean by “not 24/7”? Why would a data lake go offline?

pantsforbirds 2y ago
It's pretty common to run analytics workloads on a fixed schedule. I've setup anlaytics workflows that ingest data every day/week/quarter and generate a report once a day. With workflows like that something like DuckDB can be amazing
jakozaur 2y ago
It is very common that the majority of work happens during business hours. In some setups it's easy. It's better to get beefy instance during that time and avoid having it running all the time.
dimaor 2y ago
not sure, but maybe it's possible only to update the model in a specific time? are there other uses to the data apart from learning and validation?
creshal 2y ago
Probably in the sense of "you're only running the really heavy analytics queries once a day/week/month", and thus don't constantly need the 24TB instances.

mritchie712 2y ago

Iceberg is working hard to support pure python[0] / rust[1] workflows without Spark. Following Tabular's acquisition [2], I hope it still moves in this direction at the same clip (very impressed with the speed of development on pyiceberg). We're using iceberg + duckdb to power analytics in our app[3] and I'm really happy with the combo. 0 - https://github.com/apache/iceberg-python https://github.com/apache/iceberg-python 1 - https://github.com/apache/iceberg-rust https://github.com/apache/iceberg-rust 2 - https://x.com/thisritchie/status/1800522255426072647 https://x.com/thisritchie/status/1800522255426072647 3 - https://www.definite.app/ https://www.definite.app/

alexott 2y ago
You can already have it in Delta with Delta Rust and Python bindings: https://github.com/delta-io/delta-rs https://github.com/delta-io/delta-rs
- mritchie712 2y ago
  yes, we're evaluating Delta. We went with Iceberg out of concern that Delta was too closely tied to Databricks. Following the Tabular acquisition, the decision is murkier.