2 ms·

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looki

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactive clusters down, which results in long wait times. Analyst/developer ergonomics have not traditionally been a major concern in the big data space. 1.5: Dealing with multiple files instead of a data warehouse SQL interface requires work, and can introduce interesting performance issues. Obviously you can put a SQL interface in front of the data, but most RDBMS / Data Warehouses have a lot of functionality around maintaining your data that you may not get with the native file format, so you get soft-locked into the metadata format that comes from your data lake file format. It's all open, so this is only a half issue, but there are switching costs.

sys13 3y ago
(I do some training for Databricks) 1. Yeah, cluster startup time can be not fun. Here are some solutions: - pools (keeps instances around so you don't have to wait for the cloud to provision them - serverless SQL warehouses (viable if you're doing only SQL) - one job with multiple tasks that share the same job cluster. Delta Live Tables does a similar thing but with streaming autoscaling - streaming: cluster never needs to go down. Can share multiple streams on the same cluster so they load balance each other
- datadrivenangel 3y ago
  How do you keep the pool costs manageable? I see a lot of companies that get sold on Databricks and then are surprised by the cost.
  - sys13 3y ago
    Pool costs become more manageable as you have more clusters sharing the same pool. You can also have it have no incremental cost increase by setting the timeout to be 0, but it makes it less useful. You can have more clusters take advantage of it by using the same instance families. If you purchase reserved instances from Azure/AWS, you might as well make a pool with those as well. You may also want to check out fleet instance types.
hodgesrm 3y ago
> The one and a half downsides not mentioned here: Make that 2.5. If you are building new analytic products the most significant issue is that 90% of the information in data lakes is already there and not in an open format like Iceberg. Instead, it's heaps of CSV and Parquet files, maybe in Hive format but maybe just named in some unique way. If your query engine can't read these it's like being all dressed up with no place to go.
- datadrivenangel 3y ago
  CSV and Parquet are open file formats. They're not good structured table formats, but they are open compared to the binary storage you get with Oracle or Snowflake.
nojito 3y ago
The vast majority of analytics needs do not warrant a cluster. There are many other competitive query engines that do not suffer from the startup latency.
- datadrivenangel 3y ago
  100% agree! Most analytics are done in excel, most analytics should be done in a SQLite or DuckDB database!

sys13 3y ago

(I do some training for Databricks) 1. Yeah, cluster startup time can be not fun. Here are some solutions: - pools (keeps instances around so you don't have to wait for the cloud to provision them - serverless SQL warehouses (viable if you're doing only SQL) - one job with multiple tasks that share the same job cluster. Delta Live Tables does a similar thing but with streaming autoscaling - streaming: cluster never needs to go down. Can share multiple streams on the same cluster so they load balance each other

datadrivenangel 3y ago
How do you keep the pool costs manageable? I see a lot of companies that get sold on Databricks and then are surprised by the cost.
- sys13 3y ago
  Pool costs become more manageable as you have more clusters sharing the same pool. You can also have it have no incremental cost increase by setting the timeout to be 0, but it makes it less useful. You can have more clusters take advantage of it by using the same instance families. If you purchase reserved instances from Azure/AWS, you might as well make a pool with those as well. You may also want to check out fleet instance types.

hodgesrm 3y ago

> The one and a half downsides not mentioned here: Make that 2.5. If you are building new analytic products the most significant issue is that 90% of the information in data lakes is already there and not in an open format like Iceberg. Instead, it's heaps of CSV and Parquet files, maybe in Hive format but maybe just named in some unique way. If your query engine can't read these it's like being all dressed up with no place to go.

datadrivenangel 3y ago
CSV and Parquet are open file formats. They're not good structured table formats, but they are open compared to the binary storage you get with Oracle or Snowflake.

nojito 3y ago

The vast majority of analytics needs do not warrant a cluster. There are many other competitive query engines that do not suffer from the startup latency.

datadrivenangel 3y ago
100% agree! Most analytics are done in excel, most analytics should be done in a SQLite or DuckDB database!