9 ms·

depends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out

missosoup 7y ago
What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.
- ai_ja_nai 7y ago
  Very interesting. Can't find references to "Perfect", though; could you please point to a link?
  - missosoup 7y ago
    https://www.prefect.io https://www.prefect.io Not the most SEO-friendly choice of name. Great product though.
    - vamega 7y ago
      Are you using their cloud product? The core/open source product doesn't have a way to persist schedule data.
- sandGorgon 7y ago
  Could you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?
  - missosoup 7y ago
    tl;dr we use it for a similar set of tasks that one would use Airflow for. Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc. I'll say one thing though. The Perfect website used to be a lot more technical and explicit about what it is and isn't. Now it's mostly sales gobbledegook. Maybe not a good sign. I've seen this happen before with dremio.
    - sandGorgon 7y ago
      This is super interesting! Do you run dask on k8s ? I have been concerned that dask does not leverage kubernetes HPA for autoscaling...but instead chooses to run an external scheduler. How has your experience been ?
- maximente 7y ago
  looks like dask is python-only, so it's a nonstarter (loser) for already existing JVM code that runs on spark
  - missosoup 7y ago
    Spark stacks inevitably end up with PySpark though. It's rework for people who already committed to Spark, sure. And for bigger projects that committed to Spark this change isn't justifiable. But for a greenfield project, choosing Spark is just silly today.
truth_seeker 7y ago
Say for an example, I am using PostgreSQL 12 + CitusDB extension Data cleaning -> PL/SQL and various inbuilt functions for the transformation of data (or new UDF if required at all) Processing -> PostgreSQL Parallel processing on the local node and Citus DB extension for distributed computing and sharding Analytics -> Many options here. Materialized views OR Triggers OR Streaming computation with PipelineDB extension OR Using Logical replication for stream computation ML -> PG support variety of statistics functions. It also supports PL/R and PL/Python extension to interface with ML libraries. Also, there are various kinds of Foreign Data Wrappers supported by PG - https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers
- missosoup 7y ago
  Yeah that's not going to work for what people call analytics workloads today. PG is great but it's not suitable to be a feature store and sure as hell not suitable to fan out ML workloads. In a modern ML stack, PG might play the role of the slow but reliable master store that the rest of the ML pipeline feeds off.
  - riku_iki 7y ago
    > hell not suitable to fan out ML workloads depends on the scale? Not everyone processes petabytes of data. > PG might play the role of the slow You have any benchmark in your hand to support this? I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.
    - missosoup 7y ago
      > I believe highly optimized C code in PG can be significantly faster than Scala inside Spark. There's no question about this. If you can express your task in terms of PG on a single instance, then you probably should. When you get to more complex tasks, like running input through GloVe and pushing ngrams to a temporal store, PG offers very little - which is fine, it's not at all what PG is designed for. Inter-node IO eclipses single node perf, which is why Spark is used despite being a terribly inefficient thing (although in the case of Spark, it's so inefficient that for interim sized workloads you'd actually be better off vertically scaling a single node and using something else). PG won't help at all with these tasks. Also, that smorgasbord of extensions GP listed isn't offered by any cloud vendor as a managed service afaik, meaning you must roll and manage your own. Depending on your needs, that might be a show stopper.

missosoup 7y ago

What does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.

ai_ja_nai 7y ago
Very interesting. Can't find references to "Perfect", though; could you please point to a link?
- missosoup 7y ago
  https://www.prefect.io https://www.prefect.io Not the most SEO-friendly choice of name. Great product though.
  - vamega 7y ago
    Are you using their cloud product? The core/open source product doesn't have a way to persist schedule data.
sandGorgon 7y ago
Could you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?
- missosoup 7y ago
  tl;dr we use it for a similar set of tasks that one would use Airflow for. Unlike Airflow, this lends itself to microbatching and streaming. Plus a bunch of housekeeping items ticked off that Airflow never got around to. With a bit of devops engineering time, you can have perfect manage the size of your worker cluster on k8s and scale it up/down with ingest demand, etc. I'll say one thing though. The Perfect website used to be a lot more technical and explicit about what it is and isn't. Now it's mostly sales gobbledegook. Maybe not a good sign. I've seen this happen before with dremio.
  - sandGorgon 7y ago
    This is super interesting! Do you run dask on k8s ? I have been concerned that dask does not leverage kubernetes HPA for autoscaling...but instead chooses to run an external scheduler. How has your experience been ?
maximente 7y ago
looks like dask is python-only, so it's a nonstarter (loser) for already existing JVM code that runs on spark
- missosoup 7y ago
  Spark stacks inevitably end up with PySpark though. It's rework for people who already committed to Spark, sure. And for bigger projects that committed to Spark this change isn't justifiable. But for a greenfield project, choosing Spark is just silly today.

truth_seeker 7y ago

Say for an example, I am using PostgreSQL 12 + CitusDB extension Data cleaning -> PL/SQL and various inbuilt functions for the transformation of data (or new UDF if required at all) Processing -> PostgreSQL Parallel processing on the local node and Citus DB extension for distributed computing and sharding Analytics -> Many options here. Materialized views OR Triggers OR Streaming computation with PipelineDB extension OR Using Logical replication for stream computation ML -> PG support variety of statistics functions. It also supports PL/R and PL/Python extension to interface with ML libraries. Also, there are various kinds of Foreign Data Wrappers supported by PG - https://wiki.postgresql.org/wiki/Foreign_data_wrappers https://wiki.postgresql.org/wiki/Foreign_data_wrappers

missosoup 7y ago
Yeah that's not going to work for what people call analytics workloads today. PG is great but it's not suitable to be a feature store and sure as hell not suitable to fan out ML workloads. In a modern ML stack, PG might play the role of the slow but reliable master store that the rest of the ML pipeline feeds off.
- riku_iki 7y ago
  > hell not suitable to fan out ML workloads depends on the scale? Not everyone processes petabytes of data. > PG might play the role of the slow You have any benchmark in your hand to support this? I believe highly optimized C code in PG can be significantly faster than Scala inside Spark.
  - missosoup 7y ago
    > I believe highly optimized C code in PG can be significantly faster than Scala inside Spark. There's no question about this. If you can express your task in terms of PG on a single instance, then you probably should. When you get to more complex tasks, like running input through GloVe and pushing ngrams to a temporal store, PG offers very little - which is fine, it's not at all what PG is designed for. Inter-node IO eclipses single node perf, which is why Spark is used despite being a terribly inefficient thing (although in the case of Spark, it's so inefficient that for interim sized workloads you'd actually be better off vertically scaling a single node and using something else). PG won't help at all with these tasks. Also, that smorgasbord of extensions GP listed isn't offered by any cloud vendor as a managed service afaik, meaning you must roll and manage your own. Depending on your needs, that might be a show stopper.