3 ms·

Out of personal (and painful) experience, using Spark for general-purpose ETL processes is a bad idea. Spark is meant to be used in highly-distributed systems a

by dvt 7y ago

Out of personal (and painful) experience, using Spark for general-purpose ETL processes is a bad idea. Spark is meant to be used in highly-distributed systems and with tables that have like trillions of rows. RDDs use an optimized distributed programming model that takes a lot of practice and getting used to. Some operations are virtually impossible due to the fact that executors run on separate contexts. Caveat emptor.

EdwardDiego 7y ago
What issues did you encounter? We've been using Spark since 1.0, and the dataframes API and Catalyst query engine have abstracted away RDDs nicely, we rarely have to use them. We recently converted a bunch of legacy ETL jobs that ran against the same dataset in Vertica to Spark in EMR by storing the raw data as Parquet, caching it in memory, and running the 30 or so different SQL queries against it using Spark SQL. Only issue we encountered was one query used three unions which was rather inefficient, but once that was replaced with a query using grouping properly, problem solved.
- dvt 7y ago
  First, like I mentioned, Spark uses separate contexts for all executors (by design). This means that running nested RDD mappings is a nightmare. Just Google "nested for each loop Spark" or something along those lines[1]. This is an extremely common paradigm, especially when dealing with more complex transforms (in my specific case, we needed to hit an API, and then hit another API based on results of the previous API). Second, Spark is not re-entrant. This means that to really get usefulness out of Spark's massive parallelization, you need to use clever and non-obvious tricks to do something that "seems" simple[2]. In some extreme cases, the RDD needs to be fully serialized (danger zone if it's a big one). Finally, and related to the two above, Spark is just simply strange. You really need to have a "parallel" mindset when writing code on a Spark cluster. I constantly had to look stuff up and documentation was spotty at best. Once you throw in the rest of the ensemble (Pandas, Numpy, etc.) you end up with a very domain-specific codebase. In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill. [1] https://stackoverflow.com/questions/34339300/nesting-parallelizations-in-spark-whats-the-right-approach https://stackoverflow.com/questions/34339300/nesting-paralle... [2] https://stackoverflow.com/questions/32619570/can-i-use-spark-dataframe-inside-regular-spark-map-operation https://stackoverflow.com/questions/32619570/can-i-use-spark...
  - ibains 7y ago
    The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large. Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model. In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit. Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.
    - dvt 7y ago
      Just to be clear, I'm not discounting the work you're doing with Prophecy. We were using Databricks, and I can definitely see how a platform like Prophecy is a step up. I have a personal rule that I will always prop up any budding startup or entrepreneur (the golden rule and all that). So huge congrats on your release!
  - jeremy_k 7y ago
    > In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill. I like that you said this and I've seen a similar sentiment before. I was first exposed to it watching an ElixirConf keynote from José Valim[1] about the Flow framework they built in Elixir where he summarized it as "For between 40-80% of the jobs submitted to MapReduce systems, you'd be better off running them on a single machine." which referenced the paper Musketeer: all for one, one for all in data processing systems [2]. While I'm no Data Engineer myself, I do often wonder if distributing the workload is always better? The anecdote above indicates that a powerful single multicore machine may be right solution for many. Now that isn't to discount what Prophecy is trying to do, my company just went through a huge re-platforming moving from on on premises to the cloud and it is not easy; any company trying to tackle that space is on the right track. But I just wonder if its overkill for most use cases? [1] - https://www.youtube.com/watch?v=srtMWzyqdp8 https://www.youtube.com/watch?v=srtMWzyqdp8 [2] - http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketeer.pdf http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketee...
  - deleted 7y ago

by dvt 7y ago

EdwardDiego 7y ago

What issues did you encounter? We've been using Spark since 1.0, and the dataframes API and Catalyst query engine have abstracted away RDDs nicely, we rarely have to use them. We recently converted a bunch of legacy ETL jobs that ran against the same dataset in Vertica to Spark in EMR by storing the raw data as Parquet, caching it in memory, and running the 30 or so different SQL queries against it using Spark SQL. Only issue we encountered was one query used three unions which was rather inefficient, but once that was replaced with a query using grouping properly, problem solved.

dvt 7y ago
First, like I mentioned, Spark uses separate contexts for all executors (by design). This means that running nested RDD mappings is a nightmare. Just Google "nested for each loop Spark" or something along those lines[1]. This is an extremely common paradigm, especially when dealing with more complex transforms (in my specific case, we needed to hit an API, and then hit another API based on results of the previous API). Second, Spark is not re-entrant. This means that to really get usefulness out of Spark's massive parallelization, you need to use clever and non-obvious tricks to do something that "seems" simple[2]. In some extreme cases, the RDD needs to be fully serialized (danger zone if it's a big one). Finally, and related to the two above, Spark is just simply strange. You really need to have a "parallel" mindset when writing code on a Spark cluster. I constantly had to look stuff up and documentation was spotty at best. Once you throw in the rest of the ensemble (Pandas, Numpy, etc.) you end up with a very domain-specific codebase. In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill. [1] https://stackoverflow.com/questions/34339300/nesting-parallelizations-in-spark-whats-the-right-approach https://stackoverflow.com/questions/34339300/nesting-paralle... [2] https://stackoverflow.com/questions/32619570/can-i-use-spark-dataframe-inside-regular-spark-map-operation https://stackoverflow.com/questions/32619570/can-i-use-spark...
- ibains 7y ago
  The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large. Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model. In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit. Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.
  - dvt 7y ago
    Just to be clear, I'm not discounting the work you're doing with Prophecy. We were using Databricks, and I can definitely see how a platform like Prophecy is a step up. I have a personal rule that I will always prop up any budding startup or entrepreneur (the golden rule and all that). So huge congrats on your release!
- jeremy_k 7y ago
  > In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill. I like that you said this and I've seen a similar sentiment before. I was first exposed to it watching an ElixirConf keynote from José Valim[1] about the Flow framework they built in Elixir where he summarized it as "For between 40-80% of the jobs submitted to MapReduce systems, you'd be better off running them on a single machine." which referenced the paper Musketeer: all for one, one for all in data processing systems [2]. While I'm no Data Engineer myself, I do often wonder if distributing the workload is always better? The anecdote above indicates that a powerful single multicore machine may be right solution for many. Now that isn't to discount what Prophecy is trying to do, my company just went through a huge re-platforming moving from on on premises to the cloud and it is not easy; any company trying to tackle that space is on the right track. But I just wonder if its overkill for most use cases? [1] - https://www.youtube.com/watch?v=srtMWzyqdp8 https://www.youtube.com/watch?v=srtMWzyqdp8 [2] - http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketeer.pdf http://www.cs.utexas.edu/users/ncrooks/2015-eurosys-musketee...
- deleted 7y ago