3 ms·
is there a way to store multidimensional arrays in spark? e.g pytorch tensors?
by mlevental 8y ago
is there a way to store multidimensional arrays in spark? e.g pytorch tensors?
- mruts 8y agoThe article isn’t about Apache Spark. But no there isn’t. Spark doesn’t have row indices like pandas. It has pretty much the same querying capabilities as a SQL table.
- lern_too_spel 8y agoThis is off-topic, but yes. https://spark.apache.org/docs/latest/mllib-data-types.html https://spark.apache.org/docs/latest/mllib-data-types.html shows some strategies for representing distributed 2d arrays, and it is straightforward to generalize these to more dimensions or to unfold the multidimensional array into a 2d array.