3 ms·

> It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from a

• ianmcook 6y ago
  The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.
  • waynesonfire 6y ago
    That's exactly how I read every data format. The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq. To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?
• zten 6y ago
  The important part to focus on is _between processes_. Consider Spark and PySpark. The Python bits of Spark are in a sidecar process to the JVM running Spark. If you ask PySpark to create a DataFrame from Parquet data, it'll instruct the Java process to load the data. Its in-memory form will be Arrow. Now, if you want to manipulate that data in PySpark using Python-only libraries, prior to the adoption of Arrow it used to serialize and deserialize the data between processes on the same host. With Arrow, this process is simplified -- however, I'm not sure if it's simplified by exchanging bytes that don't require serialization/deserialization between the processes or by literally sharing memory between the processes. The docs do mention zero-copied shared memory.
• deleted 6y ago
  [deleted]