3 ms·
Does somebody know why these improvements could not be integrated into DataFrames directly? Why add another very similar layer of abstraction? They mention:
by rkrzr 11y ago
Does somebody know why these improvements could not be integrated into DataFrames directly?
Why add another very similar layer of abstraction?
They mention:
"Unification of DataFrames with Datasets – due to compatibility guarantees, DataFrames and Datasets currently cannot share a common parent class. With Spark 2.0, we will be able to unify these abstractions with minor changes to the API, making it easy to build libraries that work with both."
Anybody know what those "compatibility guarantees" are?
- nl 11y agoDataframes aren't strongly typed (while RDDs are). It's difficult to add strong typing to an existing API. I certainly prefer using DataFrames generally, but sometimes you need to switch to RDDs because some operations are easier. Hopefully Datasets bridge the gap well. I haven't used them yet, but the examples look nice.
- JoshRosen 11y agoThe blog is referring to source- and binary-API compatibility guarantees that Spark provides. IIRC, in order for DataFrames and Datasets to share a common parent class certain common methods' return types would need to change in a binary-incompatible way that would break DataFrame code which was compiled against older Spark versions. Due to Spark's API compatibility and versioning policies, this type of compatibility break can only occur in major releases, so it can only be done in Spark 2.0.0 and not Spark 1.6.