11 ms·
FastSpark: A New Fast Native Implementation of Spark from Scratch
- mindcrime 7y agoThat's mondo righteous. I think I may have finally found a reason to learn Rust.
- gok 7y agoI'm kind of surprised it took this long for someone to do this. It was clear very early on that the JVM was a bad match for what Spark was trying to do.
- mistrial9 7y agothis sounds far too simplistic, so .. reference?
- bcbrown 7y agoI attended a talk at Strata a few years back by a Spark committer who was talking about how Spark was stretching JVM memory allocations far past how the JVM was originally designed. Do a couple searches for "spark JVM OOM" and you'll see some discussions about similar things.
- gok 7y agoDead on
- splix 7y ago> You’ve reached the end of your free member preview for this month :( I guess I can read Medium posts only during first couple of days in a month
- vojta_letal 7y agoAnonymous windows have found a second use-case.
- missosoup 7y ago"You’ve reached the end of your free member preview for this month" Stop hosting your content on a platform that holds it hostage so that it can make money off it without giving anything back to you.
- ieatpies 7y agoTry incognito mode
- missosoup 7y agoWhy? This is an economy where content competes for clicks, not clicks competing for content. The author of that content wants me to see it, Medium doesn't want me to see it. I don't care enough to try to circumvent their arrangement. Given the number of votes on my root comment, it seems neither do most people.
- tandav 7y agoor (sign out & remove medium cookies)
- jdminhbg 7y agoYou can also use Reader Mode on Safari, which not only avoids the modals and popups but gets rid of the top and bottom bars as well. Long-click on the Reader Mode button and you can set it to always use it on medium.com.
- koolba 7y ago> Long-click on the Reader Mode button and you can set it to always use it on medium.com. Learning this just made my day!
- teej 7y agoThe author has chosen to monetize this article. They have the choice to make it free to everyone.
- 7y ago
- Joeri 7y agoThis sounds too good to be true. If it is this easy to be orders of magnitude faster than spark on JVM, why haven't the spark developers ported spark to native code already?
- raja_sekar 7y agoThe author of the repo here. It is definitely not orders of magnitude faster. I didn't mention it anywhere also I guess. But yeah, JVM is sometimes a problem for in-memory computing for big data processing. Spark itself tried to address this. This is what their tungsten engine does. They circumvent huge Java Objects by using native types through JNI(sun.misc.Unsafe). This is the reason why Dataframes are generally much faster than RDD(which typically uses Java objects). This is the reason only certain native types are allowed in Dataframes. This project was just for exploring the feasibility of implementing itself in the native language. Closure serialization can be a nightmare here. If it actually translated to even 2-4X better performance than Spark which itself is very difficult to achieve considering years of optimizations went into Spark, it can be a good alternative and can reduce cloud costs a bit, especially if the Python APIs remain compatible. Spark Dataframes are already highly optimized. Therefore I just thought of open-sourcing it and if others see the benefits, it will automatically grow with the help of the community. It is still a long, long way to reach Spark level maturity. Spark is indeed a very huge ecosystem built upon an already big Hadoop ecosystem.
- fourthark 7y ago> if others see the benefits, it will automatically grow with the help of the community There’s nothing automatic about it, you or someone else will need to put a lot of work into leading the community, merging pull requests, debugging, etc. (Sad to say, promotion too, in a lot of cases.)
- raja_sekar 7y agoI didn't mean it in that way. Instead of sitting idly on my laptop, it might at least be useful for someone, and if it really proves to be beneficial, then people might contribute to it. Yeah, not denying your point, it does require huge effort from some people to get it into a mature production-ready stage.
- truth_seeker 7y agoNice, but I can't find any reason to choose Spark over modern Distributed SQL databases (CockroachDB, CitusDB, TiDB etc. or cloud vendor-specific SQL DBs)
- madenine 7y agodepends what you’re doing. For querying large datasets? 100% with you. For data cleaning, processing, analytics, ML on decently large datasets? Spark wins out
- missosoup 7y agoWhat does spark win at exactly? Dask+Perfect is a much better experience all round including perf, with virtually none of the cluster management hell involved.
- ai_ja_nai 7y agoVery interesting. Can't find references to "Perfect", though; could you please point to a link?
- sandGorgon 7y agoCould you talk about Prefect ? we are in the process of moving from Spark to Dask. I have never heard of prefect. what do you use it for ?
- latenightcoding 7y agoAlso relevant: https://github.com/thrill/thrill https://github.com/thrill/thrill A Spark inspired framework written in modern C++.
- wiradikusuma 7y agoI wonder if Spark, compiled with Graal, would produce much better performance (compared to plain Spark), so no need to rewrite.
- walkingolof 7y agoI assume you mean Graal native-image mode, i.e. ahead of time compiled. JIT based execution still beats AOT due to the continuous profiling advantage, costs a bit more in terms of resources, but for long-running processes, JIT wins.
- sandGorgon 7y agoA lot of people using PySpark are moving to Dask for significantly faster performance. Dask is also built for kubernetes - which is a huge deployment win. Spark is still in-between yarn and kubernetes.
- walkingolof 7y agohttps://github.com/GoogleCloudPlatform/spark-on-k8s-operator https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
- wenc 7y agoSpark is still potentially faster for SQL-like workloads due to the existence of a query optimizer. Dask works at a different level of abstraction and does not have a query optimizer.
- sandGorgon 7y agoAre you talking about the spark SQL catalyst optimiser ? That's apples to oranges - because dask does not expose a SQL syntax that needs a query optimiser. Also pyspark has the additional issue of serialisation between python and jvm. Turns out that just getting rid of that is a huge performance boost.
- wenc 7y agoIt’s not apples to oranges with respect to my point though. Most operations on dataframe-like objects can be described in SQL operations. Spark supports these operations and Catalyst can optimize query plans for these. You are correct in that Dask does not optimize for this because Dask operations are more primitive hence it does not have the correct level of abstraction to do query optimization, only task graph optimization. Which reinforces my point that if you have a SQL-like workload on Dask.dataframes, chances are Dask may not outperform Spark.
- sandGorgon 7y agoIMHO i dont agree with you. Spark's SQL is a consequence of the need to work across languages - Scala and Python. So SQL gives the abstraction necessary to be usable in both places. It is also the language in which data scientists communicate with Spark production engineers. Every Spark production engineer i know, translates the SQL written by data scientists back into high performance RDD code. That's the advantage of Dask - there is no SQL abstraction needed. Pandas Dataframes are already the lingua franca of data scientists.. in fact, orders of magnitude more than SQL ever will be. TLDR - Dask doesnt need SQL because the people who will push Dask to production already are far more comfortable in Dataframes than they ever will be in SQL. You may still argue that spark RDD is faster than dask (and you may indeed by right)...but not having a SQL engine is not a problem for Dask.
- choppaface 7y agoThe examples are all in Rust, so it’s very hard to make a non-toy demo. Usually if one uses the RDD API, one has some sort of library code that’s already in Java or Python and it’s impractical to port that code to make the job run. Or more likely, somebody will write an initial version using that code and then port / optimize the job later. Native dependencies usually mean you’ll need docker. Spark pre-dates docker, and just relatively recently added the Kubernetes runner, which makes dockerized jobs easy. But historically it hasn’t been easy to run a job in a containerized environment with the native dependencies you need. You can ship native deps with your job, but that’s not easy, especially if you need a rebuild with each job. The main advantage of Spark is flexibility and interoperability. You save time by not having to write something optimized on day 1 (for something you might throw away). And you get SQL support, something Beam / Hadoop don’t have (certainly not for Python). There are lots of benchmarks where Spark SQL is not a winner, but the point is Spark will help you save development time.
- raja_sekar 7y agoThe author of the repo here. It is still very much in the POC stage. It will definitely have python APIs in the future. One of the primary reasons to choose a native language is to have better python integration. I intend to have to APIs almost identical to Spark, so that it will be easy to migrate. It is still very early to assure this, but it is the objective.
- raja_sekar 7y agoAuthor of the article here. Sorry about Medium restriction. I will just use Github to host my content hereafter. For now, you can use this friend link. https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native-implementation-of-spark-from-scratch-368373a29a5c?source=friends_link&sk=4c498ff1410d550048b9871191720997 https://medium.com/@rajasekar3eg/fastspark-a-new-fast-native...
- MaxBarraclough 7y agoFor the confused: this is about Apache Spark, not the Ada-based SPARK language. [0] Perhaps I'm alone here but I'd prefer the title say Apache Spark explicitly. [0] https://en.wikipedia.org/wiki/SPARK_(programming_language) https://en.wikipedia.org/wiki/SPARK_(programming_language)
- mariusae 7y agoSee also bigslice (https://bigslice.io https://bigslice.io) for another take on this.
- krcz 7y agoI'm wondering how much could be gained if one used all possible optimizations: e.g. by analyzing the data flow graph - expressed using DSL - and generating native node programs, using CPU thread pinning and user space network stack (like ScyllaDB does [1]). [1] https://www.scylladb.com/product/technology/ https://www.scylladb.com/product/technology/
- wmf 7y agoA lot. https://www.weld.rs/ https://www.weld.rs/
- aabbcc1241 7y agoI like to see people re-implement things and share their (better) results. Even if the results isn't better than the 'battle-tested' existing solutions, at least we can learn something in the process.
- wenc 7y agoScylla [1] for instance is a C++ rewrite and a drop-in replacement of JVM-based Cassandra, and from what I've read is fairly stable and performs much faster. [1] https://www.scylladb.com/ https://www.scylladb.com/