3 ms·

I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala typ

I recently switched from Scala/Spark to the PyData stack and have been loving it. It's been so refreshing to stop having to think about the JVM, SBT, Scala types, and IntelliJ. PyData libraries let you focus on your data. Wes wrote a 10 things I hate about Pandas article back in 2017 that said "pandas rule of thumb: have 5 to 10 times as much RAM as the size of your dataset". Source: https://wesmckinney.com/blog/apache-arrow-pandas-internals/ https://wesmckinney.com/blog/apache-arrow-pandas-internals/ I'm not sure that rule of thumb is up-to-date anymore. Modern versions of Pandas use PyArrow, another project Wes started, and should be more memory efficient. Wes just started a new company that'll continue building Arrow: https://medium.com/@patterson.joshua/joining-forces-for-an-arrow-native-future-cbafe27b6eb7 https://medium.com/@patterson.joshua/joining-forces-for-an-a... Dask makes it easy to scale Pandas, a la Spark. Each partition in a Dask DataFrame is a Pandas DataFrame, so you can indirectly use Pandas on terabytes of data. Disclosure: I work at Coiled, the Dask company founded by Matt Rocklin. The PyData stack makes machine learning / advanced analytics so much easier than Spark in my opinion. I still love Spark for ETL, but think modern PyData is a better stack for ML (for most companies).

leetrout 5y ago
Did you evaluate Julia (or already have experience with it)?
- marcinzm 5y ago
  The issue is that 95% of the ML ecosystem is in Python so if you're doing ML your best options are Python and things that integrated with Python (ie: PySpark).
  - MrPowers 5y ago
    I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function: def input_file_name(): sc = SparkContext._active_spark_context return Column(sc._jvm.functions.input_file_name()) You can see it's just using py4j to invoke a JVM function.
    - marcinzm 5y ago
      You can use PySpark without ever writing or thinking about Java code. The library handles all the interop for you and even does vectorized serialization with PyArrow. The same way machine learning code is actually mostly C++ or CUDA but you almost never think of that when using Python libraries.
- MrPowers 5y ago
  I haven't tried Julia yet, but suspect there is a large class of problems that are best suited for Julia. Your question motivated me to do a bit a research and it looks like a cloud based Julia runtime exists (JuliaHub). It looks like a promising solution, especially if you need to leverage a Julia lib.

leetrout 5y ago

Did you evaluate Julia (or already have experience with it)?

marcinzm 5y ago
The issue is that 95% of the ML ecosystem is in Python so if you're doing ML your best options are Python and things that integrated with Python (ie: PySpark).
- MrPowers 5y ago
  I agree that most of the ML ecosystem libs are in Python, but don't think they're easily accessible via PySpark. PySpark is a thin wrapper on Scala Spark (via py4j). PySpark has better interop with Scala/Java libs. Here's an example PySpark function: def input_file_name(): sc = SparkContext._active_spark_context return Column(sc._jvm.functions.input_file_name()) You can see it's just using py4j to invoke a JVM function.
  - marcinzm 5y ago
    You can use PySpark without ever writing or thinking about Java code. The library handles all the interop for you and even does vectorized serialization with PyArrow. The same way machine learning code is actually mostly C++ or CUDA but you almost never think of that when using Python libraries.
MrPowers 5y ago
I haven't tried Julia yet, but suspect there is a large class of problems that are best suited for Julia. Your question motivated me to do a bit a research and it looks like a cloud based Julia runtime exists (JuliaHub). It looks like a promising solution, especially if you need to leverage a Julia lib.