5 ms·

.Net for Apache Spark Preview

65 points by affogarty 7y ago

kenhwang 7y ago
What's probably more interesting is how similar .net, scala, and python are in query performance. Not sure if that can be attributed to great python performance, or really bad scala/.net performance.
- imbac82 7y ago
  https://devblogs.microsoft.com/dotnet/introducing-net-for-apache-spark/#performance https://devblogs.microsoft.com/dotnet/introducing-net-for-ap...
  - aeroevan 7y ago
    Python 2.7? Also, the new Apache Arrow integration changes the python performance characteristics a lot; I wonder if they are using arrow for their JVM <-> CLR interop, if not that probably would be a good idea.
    - polskibus 7y ago
      It would be interesting to hear about it from MS. Do you know of other settings / configurations / features that could greatly influence the result of such comparison?
      - MichaelRys 7y ago
        Since we started replying to these points on the Github thread at https://github.com/dotnet/spark/issues/45 https://github.com/dotnet/spark/issues/45, I am suggesting to continue the discussion there. As mentioned there, we want to be transparent with the benchmark code and systems we use. We are currently working on arrow support to compare fairly.
    - polskibus 7y ago
      Your post made me curious and I raised the issue with MS at https://github.com/dotnet/spark/issues/45 https://github.com/dotnet/spark/issues/45. I hope that benefits the community and gets MS on the right track (by finally supporting Arrow).
- aeroevan 7y ago
  Most of PySpark is simply telling the JVM what to do, it's not actually running python directly. UDFs are where the real differences are, and they mentioned CLR UDFs serialize the spark Rows 2x faster than Python, but it's not clear if they were using apache arrow enabled pandas UDFs which are 3x-100x faster: https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html https://databricks.com/blog/2017/10/30/introducing-vectorize...
- bunderbunder 7y ago
  A large percentage of Spark code is really just assembling lego blocks. The built-in blocks are themselves all written in Java or Scala, and the performance of the code that stacks them together is negligible. It's mainly when you start writing custom UDFs (IOW, fabricating your own lego blocks) that platform interop and the performance of your language of choice become a big deal.
flowerlad 7y ago
Python is a second class citizen in the world of Spark. Perf issues for UDFs. Some functions are only available through Scala. Python support for new features is always late. And so on. It is good to have .NET support but I will stick with Scala for the same reasons I switched from Python to Scala.
- pjmlp 7y ago
  That is the exact reason that I learned to stay with platform languages for production code, even if there are more interesting ones trying to plug into it. FFI, extra debugging layers and lesser tooling integration don't pay off a couple of language feature bullet points.
vvladymyrov 7y ago
I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug issues. For this you still need to dig into Apache core which is in Scala. This implementation in .Net would be "gateway drug" for moving your production to Scala/JVM. It happened to me with PySpark - majority or tasks at hand can be solved with PySpark. But digging into the issues and stack traces brought me to Scala internals of Apache Spark. As a result in cases when python specific libraries are not needed and high performance needed I would write Spark programs Scala from the beginning.
- bunderbunder 7y ago
  On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.
  - vvladymyrov 7y ago
    For python guys - pyspark is now also installable from pip as a package (it include some .jars so it is ~100Mb size python package). So my team for local unit tests installs pyspark as a package.
  - MichaelRys 7y ago
    These are all good points. Debugability and general support for the development lifecycle are important. We are definitively working on providing first class development experiences for .NET developers. .NET for Apache Spark is already available as a nuget package for local install. We are currently working on adding support to VS Code, Visualstudio etc. Feel free to provide us your preferred dev platform. [Disclaimer: I am Program Manager for the .NET for Apache Spark effort]
tombert 7y ago
How is this different than Mobius [1]? [1] https://github.com/Microsoft/Mobius https://github.com/Microsoft/Mobius
- manigandham 7y ago
  From the github repo: https://github.com/dotnet/spark#inspiration-and-special-thanks https://github.com/dotnet/spark#inspiration-and-special-than... >> Mobius: C# and F# language binding and extensions to Apache Spark, a pre-cursor project to .NET for Apache Spark from the same Microsoft group.
- jongalloway2 7y ago
  Here's my understanding: - Mobius is .NET Framework / Mono based and x-plat isn’t great, .NET for Apache Spark is .NET Core / .NET Standard and built with x-plat as a primary concern - Mobius only targets up to Spark 2.0; while Spark LTS is up to 2.4 now - .NET for Apache Spark is built to take advantage of .NET Core performance improvements, showing big advantages over Python and R bindings, especially when user defined functions are a major factor - .NET for Apache Spark is driven by lessons learned and customer demand, including major big data users inside and outside Microsoft Disclaimer: I know people that worked on this and helped from .NET Foundation side, but the above is my non-official summary from readme's and stuff.
maa5444 7y ago
it would be not cleaner just learning scala ?
deleted 7y ago
[deleted]

65 points by affogarty 7y ago

What's probably more interesting is how similar .net, scala, and python are in query performance. Not sure if that can be attributed to great python performance, or really bad scala/.net performance.

imbac82 7y ago
https://devblogs.microsoft.com/dotnet/introducing-net-for-apache-spark/#performance https://devblogs.microsoft.com/dotnet/introducing-net-for-ap...
- aeroevan 7y ago
  Python 2.7? Also, the new Apache Arrow integration changes the python performance characteristics a lot; I wonder if they are using arrow for their JVM <-> CLR interop, if not that probably would be a good idea.
  - polskibus 7y ago
    It would be interesting to hear about it from MS. Do you know of other settings / configurations / features that could greatly influence the result of such comparison?
    - MichaelRys 7y ago
      Since we started replying to these points on the Github thread at https://github.com/dotnet/spark/issues/45 https://github.com/dotnet/spark/issues/45, I am suggesting to continue the discussion there. As mentioned there, we want to be transparent with the benchmark code and systems we use. We are currently working on arrow support to compare fairly.
  - polskibus 7y ago
    Your post made me curious and I raised the issue with MS at https://github.com/dotnet/spark/issues/45 https://github.com/dotnet/spark/issues/45. I hope that benefits the community and gets MS on the right track (by finally supporting Arrow).
aeroevan 7y ago
Most of PySpark is simply telling the JVM what to do, it's not actually running python directly. UDFs are where the real differences are, and they mentioned CLR UDFs serialize the spark Rows 2x faster than Python, but it's not clear if they were using apache arrow enabled pandas UDFs which are 3x-100x faster: https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html https://databricks.com/blog/2017/10/30/introducing-vectorize...
bunderbunder 7y ago
A large percentage of Spark code is really just assembling lego blocks. The built-in blocks are themselves all written in Java or Scala, and the performance of the code that stacks them together is negligible. It's mainly when you start writing custom UDFs (IOW, fabricating your own lego blocks) that platform interop and the performance of your language of choice become a big deal.

flowerlad 7y ago

Python is a second class citizen in the world of Spark. Perf issues for UDFs. Some functions are only available through Scala. Python support for new features is always late. And so on. It is good to have .NET support but I will stick with Scala for the same reasons I switched from Python to Scala.

pjmlp 7y ago
That is the exact reason that I learned to stay with platform languages for production code, even if there are more interesting ones trying to plug into it. FFI, extra debugging layers and lesser tooling integration don't pay off a couple of language feature bullet points.

vvladymyrov 7y ago

I've seen announcement about .Net interior support in Apache Spark some time ago. The benchmarks are interesting and tell the story - in few cases it is faster than Python, but slower than native (for Spark) Scala/JVM. Maybe with Arrow interchange Python's performance would increase (and for other interpose that would use Array - i.e. for .Net). But performance is not the only thing - there is also ability to debug issues. For this you still need to dig into Apache core which is in Scala. This implementation in .Net would be "gateway drug" for moving your production to Scala/JVM. It happened to me with PySpark - majority or tasks at hand can be solved with PySpark. But digging into the issues and stack traces brought me to Scala internals of Apache Spark. As a result in cases when python specific libraries are not needed and high performance needed I would write Spark programs Scala from the beginning.

bunderbunder 7y ago
On a somewhat related note, for my purposes the real deciding factor in sticking with Scala/JVM for (production) Spark work is testability: With that setup, it's dead easy to fire up a local Spark context, run unit tests against it, and keep the tests running reasonably fast.
- vvladymyrov 7y ago
  For python guys - pyspark is now also installable from pip as a package (it include some .jars so it is ~100Mb size python package). So my team for local unit tests installs pyspark as a package.
- MichaelRys 7y ago
  These are all good points. Debugability and general support for the development lifecycle are important. We are definitively working on providing first class development experiences for .NET developers. .NET for Apache Spark is already available as a nuget package for local install. We are currently working on adding support to VS Code, Visualstudio etc. Feel free to provide us your preferred dev platform. [Disclaimer: I am Program Manager for the .NET for Apache Spark effort]

tombert 7y ago

How is this different than Mobius [1]? [1] https://github.com/Microsoft/Mobius https://github.com/Microsoft/Mobius

manigandham 7y ago
From the github repo: https://github.com/dotnet/spark#inspiration-and-special-thanks https://github.com/dotnet/spark#inspiration-and-special-than... >> Mobius: C# and F# language binding and extensions to Apache Spark, a pre-cursor project to .NET for Apache Spark from the same Microsoft group.
jongalloway2 7y ago
Here's my understanding: - Mobius is .NET Framework / Mono based and x-plat isn’t great, .NET for Apache Spark is .NET Core / .NET Standard and built with x-plat as a primary concern - Mobius only targets up to Spark 2.0; while Spark LTS is up to 2.4 now - .NET for Apache Spark is built to take advantage of .NET Core performance improvements, showing big advantages over Python and R bindings, especially when user defined functions are a major factor - .NET for Apache Spark is driven by lessons learned and customer demand, including major big data users inside and outside Microsoft Disclaimer: I know people that worked on this and helped from .NET Foundation side, but the above is my non-official summary from readme's and stuff.

maa5444 7y ago

it would be not cleaner just learning scala ?

deleted 7y ago

[deleted]