4 ms·

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflak

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift

disgruntledphd2 6y ago
Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.
- dominotw 6y ago
  > best distributed ML system out there I was comparing it for "traditional" data engineering stack that used spark for data munging, transformations ect. I don't have much insight into ML systems or how spark fits there. Not all data teams are building 'ml systems' though. Parent comment wasn't referring to any 'ml systems', not sure why that would be automatically inferred when someone mentions data stack .
  - disgruntledphd2 6y ago
    Yeah, I suppose. I kinda think that distributed SQL is a mostly commoditised space, and wondered what replaced Spark for distributed training. For context, I'm a DS who's spent far too much time not being able to run useful models because of hardware limitations, and a Spark cluster is incredibly good for that. Additionally, I'd argue in favour of Spark even for ETL, as the ability to write (and test!) complicated SQL queries in R, Python and Scala was super, super transformative. We don't really use Spark at my current place, and every time I write Snowflake (which is great, to be fair), I'm reminded of the inherent limitations of SQL and how wonderful Spark SQL was. I'm weird though, to be fair.
- sails 6y ago
  Snowflake I suppose for the average ML use case. Not for your high-performance ML, but for your average data scientist, maybe? Edit: I may be wrong[1], would be curious to know what users who've used Spark AND Snowflake would add to the conversation. [1] https://www.snowflake.com/blog/snowflake-and-spark-part-1-why-spark/ https://www.snowflake.com/blog/snowflake-and-spark-part-1-wh...
  - marcinzm 6y ago
    Snowflake hits its limits with complex transformations I feel. Not just due to using SQL. It's "type system" is simpler than Spark's which makes certain operations annoying. There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience.
    - dominotw 6y ago
      > There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience. We load our data into SF in json and do plenty of list/struct manipulation using their inbuilt functions[1]. I guess you might have write a UDF if you are doing something super weird but inbuilt functions should get you pretty far 90% of the time. https://docs.snowflake.com/en/sql-reference/functions-semistructured.html https://docs.snowflake.com/en/sql-reference/functions-semist...
- somurzakov 6y ago
  distributed ML != Distributed DWH. Distributed ML is tough to train because of very little control over train loop. I personally prefer using single server trainkng even on large datasets, or switch to online learning algos that do train/inference/retrain at the same time. as for snowflake, I havent heard of people using snowflake to train ML, but sbnowflake is a killer in managed distribited DWH that you dont have to tinker and tune
  - disgruntledphd2 6y ago
    So do I, theoretically at least. But Spark is super cool and actually has algorithms which complete in a reasonable time frame on hardware I can get access to. Like, I understand that the SQL portion is pretty commoditised (though even there, SparkSQL python and R API's are super nice), but I'm not aware of any other frameworks for doing distributed training of ML models. Have all the hipsters moved to GPUs or something? \s > sbnowflake is a killer in managed distribited DWH that you dont have to tinker and tune It's so very expensive though, and their pricing model is frustratingly annoying (why the hell do I need tickets?). That being said, tuning Spark/Presto or any of the non-managed alternatives is no fun either, so I wonder if it's the right tradeoff. One thing I really, really like about Spark is the ability to write Python/R/Scala code to solve the problems that cannot be usefully expressed in SQL. All the replies to my original comment seem to forget that, or maybe Snowflake has such functionality and I'm unaware of it.
    - marcinzm 6y ago
      >I'm not aware of any other frameworks for doing distributed training of ML models. Tensorflow, PyTorch (not sure if Ray is needed) and Mxnet all support distributed training across CPUs/GPUs in a single machine or multiple machines. So does XGBoost if you don't want deep learning. You can then run them with KubeFlow or on whatever platform your SaaS provider has (GCP AI Platform, AWS Sagemaker, etc.). edit: >All the replies to my original comment seem to forget that, or maybe Snowflake has such functionality and I'm unaware of it. Snowflake has support for custom Javascript UDFs and a lot of built in features (you can do absurd things with window functions). I also found it much faster than Spark.
- victor106 6y ago
  I agree with this. Along with ML it is also a very high performance extract and transformation engine. Would love to hear what other tech that are being used to replace Spark.
- kenhwang 6y ago
  I'd put spark in both lists. Old is spark-sql, new is the programming language interface.
sseppola 6y ago
Can you elaborate more on the "roles" of the "new stack"? To me dbt/dataform and airflow/dagster are quite similar, so why do you need one of each? fivetran/stitch/singer are all new
- Grimm1 6y ago
  Airflow allows for more complex transformations of data that SQL may not be suited for. DBT is largely stuck utilizing the SQL capabilities of the warehouse it sits on, so for instance, with Redshift you have a really bad time working with JSON based data with DBT, Airflow can solve this problem. That's one example, but last I was working with it we found DBT was great for analytical modeling type transformations but from getting whatever munged up data into a useable format in the first place Airflow was king. We also trained our analysts to write the more analytical DBT transformations which was nice, shifted that work onto them. Don't get me wrong though, you can get really far with just DBT + Fivetran, in fact, it removes like 80% of the really tedious, but trivial ETL work. Airflow is just there for the last 20% (Plus you can then utilize airflow as a general job scheduler)
- pjot 6y ago
  I've used all of these so I might be able to offer some perspective here In an ELT/ETL pipeline: Airflow is similar to the "extract" portion of the pipeline and is great for scheduling tasks and provides the high-level view for understanding state changes and status of a given system. I'll typically use airflow to schedule a job that will get raw data from xyz source(s), do something else with it, then drop it into S3. This can then trigger other tasks/workflows/slack notifications as necessary. You can think of dbt as the "transform" part. It really shines with how it enables data teams to write modular, testable, and version controlled SQL - similar to how a more traditional type developer writes code. For example, when modeling a schema in a data warehouse all of the various source tables, transformation and aggregation logic, as well as materialization methods are able to to live in the their own files and be referenced elsewhere through templating. All of the table/view dependencies are handled under the hood by dbt. For my organization, it helped untangle the web of views building views building views and made it simpler to grok exactly what and where might be changing and how something may affect something else downstream. Airflow could do this too in theory, but given you write SQL to interface with dbt, it makes it far more accessible for a wider audience to contribute. Fivetran/Stitch/Singer can serve as both the "extract" and "load" parts of the equation. Fivetran "does it for you" more or less with their range of connectors for various sources and destinations. Singer simply defines a spec for sources (taps) and destinations (targets) to be used as a standard when writing a pipeline. I think the way Singer drew a line in the sand and approached defining a way of doing things is pretty cool - however active development on it really took a hit when the company was acquired. Stitch came up with the singer spec and their offered service is through managing the and scheduling various taps and targets for you.
deleted 6y ago
[deleted]
fetanchaud 6y ago
Just a thought : what about dremio ?

Agree 100% with this comment, Old stack: Hadoop, spark, hive, hdfs. New stack: kafka/kinesis, fivetran/stitch/singer, airflow/dagster, dbt/dataform, snowflake/redshift

disgruntledphd2 6y ago

Huh, what replaces Spark in those lists? For my money, its the best distributed ML system out there, so I'd be interested to know what new hotness I'm missing.

dominotw 6y ago
> best distributed ML system out there I was comparing it for "traditional" data engineering stack that used spark for data munging, transformations ect. I don't have much insight into ML systems or how spark fits there. Not all data teams are building 'ml systems' though. Parent comment wasn't referring to any 'ml systems', not sure why that would be automatically inferred when someone mentions data stack .
- disgruntledphd2 6y ago
  Yeah, I suppose. I kinda think that distributed SQL is a mostly commoditised space, and wondered what replaced Spark for distributed training. For context, I'm a DS who's spent far too much time not being able to run useful models because of hardware limitations, and a Spark cluster is incredibly good for that. Additionally, I'd argue in favour of Spark even for ETL, as the ability to write (and test!) complicated SQL queries in R, Python and Scala was super, super transformative. We don't really use Spark at my current place, and every time I write Snowflake (which is great, to be fair), I'm reminded of the inherent limitations of SQL and how wonderful Spark SQL was. I'm weird though, to be fair.
sails 6y ago
Snowflake I suppose for the average ML use case. Not for your high-performance ML, but for your average data scientist, maybe? Edit: I may be wrong[1], would be curious to know what users who've used Spark AND Snowflake would add to the conversation. [1] https://www.snowflake.com/blog/snowflake-and-spark-part-1-why-spark/ https://www.snowflake.com/blog/snowflake-and-spark-part-1-wh...
- marcinzm 6y ago
  Snowflake hits its limits with complex transformations I feel. Not just due to using SQL. It's "type system" is simpler than Spark's which makes certain operations annoying. There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience.
  - dominotw 6y ago
    > There's a lack of UDFs for working with complex types (lists, structs, etc.). Having to write UDFs in Javascript is also not the greatest experience. We load our data into SF in json and do plenty of list/struct manipulation using their inbuilt functions[1]. I guess you might have write a UDF if you are doing something super weird but inbuilt functions should get you pretty far 90% of the time. https://docs.snowflake.com/en/sql-reference/functions-semistructured.html https://docs.snowflake.com/en/sql-reference/functions-semist...
somurzakov 6y ago
distributed ML != Distributed DWH. Distributed ML is tough to train because of very little control over train loop. I personally prefer using single server trainkng even on large datasets, or switch to online learning algos that do train/inference/retrain at the same time. as for snowflake, I havent heard of people using snowflake to train ML, but sbnowflake is a killer in managed distribited DWH that you dont have to tinker and tune
- disgruntledphd2 6y ago
  So do I, theoretically at least. But Spark is super cool and actually has algorithms which complete in a reasonable time frame on hardware I can get access to. Like, I understand that the SQL portion is pretty commoditised (though even there, SparkSQL python and R API's are super nice), but I'm not aware of any other frameworks for doing distributed training of ML models. Have all the hipsters moved to GPUs or something? \s > sbnowflake is a killer in managed distribited DWH that you dont have to tinker and tune It's so very expensive though, and their pricing model is frustratingly annoying (why the hell do I need tickets?). That being said, tuning Spark/Presto or any of the non-managed alternatives is no fun either, so I wonder if it's the right tradeoff. One thing I really, really like about Spark is the ability to write Python/R/Scala code to solve the problems that cannot be usefully expressed in SQL. All the replies to my original comment seem to forget that, or maybe Snowflake has such functionality and I'm unaware of it.
  - marcinzm 6y ago
    >I'm not aware of any other frameworks for doing distributed training of ML models. Tensorflow, PyTorch (not sure if Ray is needed) and Mxnet all support distributed training across CPUs/GPUs in a single machine or multiple machines. So does XGBoost if you don't want deep learning. You can then run them with KubeFlow or on whatever platform your SaaS provider has (GCP AI Platform, AWS Sagemaker, etc.). edit: >All the replies to my original comment seem to forget that, or maybe Snowflake has such functionality and I'm unaware of it. Snowflake has support for custom Javascript UDFs and a lot of built in features (you can do absurd things with window functions). I also found it much faster than Spark.
victor106 6y ago
I agree with this. Along with ML it is also a very high performance extract and transformation engine. Would love to hear what other tech that are being used to replace Spark.
kenhwang 6y ago
I'd put spark in both lists. Old is spark-sql, new is the programming language interface.

sseppola 6y ago

Can you elaborate more on the "roles" of the "new stack"? To me dbt/dataform and airflow/dagster are quite similar, so why do you need one of each? fivetran/stitch/singer are all new

Grimm1 6y ago
Airflow allows for more complex transformations of data that SQL may not be suited for. DBT is largely stuck utilizing the SQL capabilities of the warehouse it sits on, so for instance, with Redshift you have a really bad time working with JSON based data with DBT, Airflow can solve this problem. That's one example, but last I was working with it we found DBT was great for analytical modeling type transformations but from getting whatever munged up data into a useable format in the first place Airflow was king. We also trained our analysts to write the more analytical DBT transformations which was nice, shifted that work onto them. Don't get me wrong though, you can get really far with just DBT + Fivetran, in fact, it removes like 80% of the really tedious, but trivial ETL work. Airflow is just there for the last 20% (Plus you can then utilize airflow as a general job scheduler)
pjot 6y ago
I've used all of these so I might be able to offer some perspective here In an ELT/ETL pipeline: Airflow is similar to the "extract" portion of the pipeline and is great for scheduling tasks and provides the high-level view for understanding state changes and status of a given system. I'll typically use airflow to schedule a job that will get raw data from xyz source(s), do something else with it, then drop it into S3. This can then trigger other tasks/workflows/slack notifications as necessary. You can think of dbt as the "transform" part. It really shines with how it enables data teams to write modular, testable, and version controlled SQL - similar to how a more traditional type developer writes code. For example, when modeling a schema in a data warehouse all of the various source tables, transformation and aggregation logic, as well as materialization methods are able to to live in the their own files and be referenced elsewhere through templating. All of the table/view dependencies are handled under the hood by dbt. For my organization, it helped untangle the web of views building views building views and made it simpler to grok exactly what and where might be changing and how something may affect something else downstream. Airflow could do this too in theory, but given you write SQL to interface with dbt, it makes it far more accessible for a wider audience to contribute. Fivetran/Stitch/Singer can serve as both the "extract" and "load" parts of the equation. Fivetran "does it for you" more or less with their range of connectors for various sources and destinations. Singer simply defines a spec for sources (taps) and destinations (targets) to be used as a standard when writing a pipeline. I think the way Singer drew a line in the sand and approached defining a way of doing things is pretty cool - however active development on it really took a hit when the company was acquired. Stitch came up with the singer spec and their offered service is through managing the and scheduling various taps and targets for you.

deleted 6y ago

[deleted]

fetanchaud 6y ago

Just a thought : what about dremio ?