7 ms·
Microsoft announces major commitment to Apache Spark
- insulanian 10y agoUsually when I hear about Apache Sparc, there is Scala being mentioned too. Does Scala have the best support for it and what's the situation on .NET side? (F#?)
- pjmlp 10y agoThere are some people doing ML and data analysis in F#, mostly financial institutions, but Spark is written in Scala so it is all about the JVM eco-system.
- Govannon 10y agoI would be surprised if it didn't end up being C#, but I have no real foundation for that sentiment.
- nchammas 10y agoFirst class language support in Apache Spark: * Scala * Python * Java * R All these languages are equal, but Scala tends to be more equal than others in some areas of the API. I also believe R is mostly restricted to the DataFrame API. Third-party language support: * Clojure [0] To develop on Spark in a new, non-JVM language, you'd need a bridge to Java. That's how PySpark works [1], and I believe R follows a similar pattern. [0] https://github.com/yieldbot/flambo https://github.com/yieldbot/flambo [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals https://cwiki.apache.org/confluence/display/SPARK/PySpark+In...
- baldfat 10y agoI am thinking that R is the reason why this is happening. Now that Microsoft has supported and invested in R (Smart thing on their part) R will get more Spark features. Having better R connections to Spark's API would be amazing.
- _dark_matter_ 10y agoFirst class support for running dplyr pipelines on Spark would be killer, imho.
- pjmlp 10y agoInteresting how the whole R thing happened, about three years ago I wouldn't have any idea what R is about and now see it everywhere, even at customer sites.
- koloron 10y agoHaskell: https://github.com/tweag/sparkle https://github.com/tweag/sparkle
- bunderbunder 10y agoThere's also third-party support for .NET via https://github.com/Microsoft/Mobius https://github.com/Microsoft/Mobius
- cwyers 10y agoThird-party from Spark's perspective, but first-party from C#'s perspective.
- henridf 10y agoThird-party Node.js bindings: https://github.com/henridf/apache-spark-node https://github.com/henridf/apache-spark-node https://github.com/EclairJS/eclairjs-node https://github.com/EclairJS/eclairjs-node
- jbooth 10y agoMore like 3 tiers of support: Scala -- native API Java -- java wrappers for Scala API, slightly clunky but same implementation Python+R -- janky process involving forking a process and feeding strings back and forth between JVM and python/R via pipes
- nchammas 10y agoYou're right about Python and R having to pass data back and forth to the JVM for certain operations, but also keep in mind that native code still runs in the native interpreter. That means you have access to the full ecosystem of the native language. For example, if I want to convert an RDD of JSON strings into Python dictionaries: import json rdd_dict = rdd.map(lambda x: json.loads(x)) Same goes for any external Python libraries I install on the cluster and want to use in my Spark job. You can even run your Python code on PyPy [4]! For me, working in Python generally feels like a first class experience on Spark. There are areas -- like GraphX [0], certain niche features [1] -- where Scala is definitely easier to work with, but with time that is becoming less [2] and less [3] true thanks to the DataFrame API. [0] https://spark.apache.org/graphx/ https://spark.apache.org/graphx/ [1] http://stackoverflow.com/q/23995040/877069 http://stackoverflow.com/q/23995040/877069 [2] https://github.com/graphframes/graphframes https://github.com/graphframes/graphframes [3] http://stackoverflow.com/a/37150604/877069 http://stackoverflow.com/a/37150604/877069 [4] https://github.com/apache/spark/pull/2144 https://github.com/apache/spark/pull/2144
- twic 10y agoHas anyone tried running Jython on top of the native API? Would that make any sense?
- SatvikBeri 10y agoSpark's syntax is basically identical to Scala's, so you can use it as a library with standard Scala functions. In contrast PySpark etc. will still let you get all the same features, but you have to write code that doesn't look like Python, by either using Scala-like code with the RDD API or SQL-like code with the SQL API.
- bunderbunder 10y agoSpark's written in Scala, so Scala will presumably always be the best-supported language. On the inside Spark's also pretty intimately tied to some Java implementation details (e.g., it uses Java serialization to push code around), so any non-JVM language is always going to have to deal with a bit of friction. That said, from my experience the Python integration is still first-class. I think it might lag the Scala APIs a bit, but that isn't a huge deal unless you really need to be on the bleeding edge at all times. The only really big downside in my book is that there's a small performance hit you take by choosing to use the Python API instead of the Scala one, and that might compound into something of practical significance if you're using it for really big data tasks. I assume you'd have similar problems using it with R or a .NET language.
- peterwwillis 10y agoIs it just me, or does any technology that Microsoft supports late in the game give you sort of a queasy feeling?
- gokhan 10y agoEasy to see their motive. This is good for their cloud business, they already know their monopoly days are over, they're making money with Azure and growing that front (127% [1]). Whatever your platform is, you can host it on Azure. And big data will be more lucrative each day. Spark is a cluster computing framework and clusters lead to cloud. [1] http://www.businessinsider.com/microsoft-azure-vs-aws-revenue-2016-1 http://www.businessinsider.com/microsoft-azure-vs-aws-revenu...
- sargun 10y ago"But there's no way to tease out how much of that $5.1 billion is Azure versus the older products. We also don't know if Azure is profitable, althoug the overall business segment earned $2.58 billion in profits — down 1% from a year ago." Servers and tools did $4.5B of revenue in FY12, Q3 (https://www.microsoft.com/Investor/EarningsAndFinancials/Earnings/SegmentResults/ServerAndTools/FY12/Q3/performance.aspx https://www.microsoft.com/Investor/EarningsAndFinancials/Ear...). This was before Azure really had any footprint. You do the math.
- elliotec 10y agoWhy, and what exactly do you mean?
- adventured 10y agoThe parent is probably having flashbacks to a time now long gone, in which Microsoft was widely regarded in tech circles as the evil empire. They would attempt to 'embrace and extend' potentially threatening (to them) technology that they didn't control, as a means to hijack or kill it. Many people - including internally at Microsoft - often called it: embrace, extend, extinguish: https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish https://en.wikipedia.org/wiki/Embrace,_extend_and_extinguish
- elcapitan 10y agoCan't wait until they install it on my computer!
- DoofusOfDeath 10y agoI assume that Microsoft considers the Windows EULA to let them use your computer as an Azure compute node.
- mistermann 10y agoThis is actually a rather interesting idea.....allowing users to voluntarily participate as a compute node would be extremely cool.
- pvelagal 10y agoUsers can contribute idle CPU cycles to a variety of projects http://www.hyper.net/dc-howto.html http://www.hyper.net/dc-howto.html . AFAIK, lot of folks have been doing this for decades.
- frozenport 10y agohttps://lh3.googleusercontent.com/-45xROH3jwF0/VuFHctFcV_I/AAAAAAABOSM/uBevq27hzkw/w506-h337/Windows.png https://lh3.googleusercontent.com/-45xROH3jwF0/VuFHctFcV_I/A...
- ssahoo 10y agoNot a surprise, since https://www-03.ibm.com/press/us/en/pressrelease/47107.wss https://www-03.ibm.com/press/us/en/pressrelease/47107.wss