7 ms·

Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote

by jayd16 6y ago

Can someone dig into the pros and cons of the columnar aspect of Arrow? To some degree there are many other data transfer formats but this one seems to promote its columnar orientation. Things like eg. protobuffers support hierarchical data which seems like a superset of columns. Is there a benefit to a column based format? Is it an enforced simplification to ensure greater compatibility or is there some other reason?

tomnipotent 6y ago
This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, which are then written over the network to another Arrow compatible service that just copies the data as-is from the network into local memory and resumes working.
- waynesonfire 6y ago
  > It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.
  - ianmcook 6y ago
    The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.
    - waynesonfire 6y ago
      That's exactly how I read every data format. The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq. To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?
  - zten 6y ago
    The important part to focus on is _between processes_. Consider Spark and PySpark. The Python bits of Spark are in a sidecar process to the JVM running Spark. If you ask PySpark to create a DataFrame from Parquet data, it'll instruct the Java process to load the data. Its in-memory form will be Arrow. Now, if you want to manipulate that data in PySpark using Python-only libraries, prior to the adoption of Arrow it used to serialize and deserialize the data between processes on the same host. With Arrow, this process is simplified -- however, I'm not sure if it's simplified by exchanging bytes that don't require serialization/deserialization between the processes or by literally sharing memory between the processes. The docs do mention zero-copied shared memory.
  - deleted 6y ago
    [deleted]
aldanor 6y ago
- Selective/lazy access (e.g. I have 100 columns but I want to quickly pull out/query just 2 of them). - Improved compression (e.g. a column of timestamps). - Flexible schemas being easy to manage (e.g. adding more columns, or optional columns). - Vectorization/SIMD-friendly.
wging 6y ago
Redshift's explanation is pretty good. Among other things, if you only need a few columns there are entire blocks of data you don't need to touch at all. https://docs.aws.amazon.com/redshift/latest/dg/c_columnar_storage_disk_mem_mgmnt.html https://docs.aws.amazon.com/redshift/latest/dg/c_columnar_st... It's truly magical when you scope down a SELECT to the columns you need and see a query go blazing fast. Or maybe I'm easily impressed.
mumblemumble 6y ago
A columnar format is almost always what you want for analytical workloads, because their access patterns tend to iterate ranges of rows but select only a few columns at random. About the only thing protocol buffers has in common is that it's a standardized binary format. The use case is largely non-overlapping, though. Protobuf is meant for transmitting monolithic datagrams, where the entire thing will be transmitted and then decoded as a monolithic blob. It's also, out of the box, not the best for efficiently transmitting highly repetitive data. Column-oriented formats cut down on some repetition of metadata, and also tend to be more compressible because similar data tends to get clumped together. Coincidentally, Arrow's format for transmitting data over a network, Arrow Flight, uses protocol buffers as its messaging format. Though the payload is still blocks of column-oriented data, for efficiency.
georgewfraser 6y ago
The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe: (A int, B int, C int, D int) And I write: A + B In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory) will be a lot more efficient. In a row-oriented representation like protobuf, all your C and D values are going to get dragged into the CPU registers alongside the A and B values that you actually want. Column-oriented representation is also more friendly to SIMD CPU instructions. You can still use SIMD with a row-oriented representation, but you have to use gather-scatter operations which makes the whole thing less efficient.
- jzer0cool 6y ago
  Could you expand on this more columnar data and row data? I missed something here how the data is organized and what you mean by the C,D values getting dragged along.
  - int_19h 6y ago
    Row data is an array of structs, with each struct representing a record. Columnar data is a struct of arrays, with each array representing a column.
  - repsilat 6y ago
    To rephrase the sibling comment, if you had an array of four {ABCD} structs, there's basically two ways of storing them on disk: 1. AAAA BBBB CCCC DDDD 2. ABCD ABCD ABCD ABCD One major heuristic in how CPUs make your code fast is to assume that if you access some memory, you're probably interested in the memory nearby. So when you access the first "A" bit of memory (common to both sequences above), depending on the memory layout you use, the CPU might also be smart and load the next bits into memory too -- maybe the next "AA", maybe "BC". Depending on your workload, one or the other of those might be faster. If you're only interested in the first ABCD element because you're doing SELECT * FROM users WHERE id=$1 then you'll likely want "row-oriented" data -- the #2 scheme above. But if you're interested in all of the A values and none of the values from B/C/D because you're doing SELECT AVG(age) FROM users then you'll likely want something "column-oriented" -- the #1 scheme above.

by jayd16 6y ago

tomnipotent 6y ago

This is intended for analytical workloads where you're often doing things that can benefit from vectorization (like SIMD). It's much faster to SUM(X) when all values of X are neatly laid out in-memory. It also has the added benefit of eliminating serialization and deserialization of data between processes - a Python process can now write to memory which is read by a C++ process that's doing windowed aggregations, which are then written over the network to another Arrow compatible service that just copies the data as-is from the network into local memory and resumes working.

waynesonfire 6y ago
> It also has the added benefit of eliminating serialization and deserialization of data between processes Is that accurate? It still has to deserialize from apache arrow format to whatever the cpu understands.
- ianmcook 6y ago
  The Arrow Feather format is an on-disk representation of Arrow memory. To read a Feather file, Arrow just copies it byte for byte from disk into memory. Or Arrow can memory-map a Feather file so you can operate on it without reading the whole file into memory.
  - waynesonfire 6y ago
    That's exactly how I read every data format. The advantage you describe is in the operations that can performed against the data. It would be nice to see what this API looks like and how it compares to flatbuffers / pq. To help me understand this benefit, can you talk through what it's like to add 1 to each record and write it back to disk?
- zten 6y ago
  The important part to focus on is _between processes_. Consider Spark and PySpark. The Python bits of Spark are in a sidecar process to the JVM running Spark. If you ask PySpark to create a DataFrame from Parquet data, it'll instruct the Java process to load the data. Its in-memory form will be Arrow. Now, if you want to manipulate that data in PySpark using Python-only libraries, prior to the adoption of Arrow it used to serialize and deserialize the data between processes on the same host. With Arrow, this process is simplified -- however, I'm not sure if it's simplified by exchanging bytes that don't require serialization/deserialization between the processes or by literally sharing memory between the processes. The docs do mention zero-copied shared memory.
- deleted 6y ago
  [deleted]

aldanor 6y ago

- Selective/lazy access (e.g. I have 100 columns but I want to quickly pull out/query just 2 of them). - Improved compression (e.g. a column of timestamps). - Flexible schemas being easy to manage (e.g. adding more columns, or optional columns). - Vectorization/SIMD-friendly.

wging 6y ago

Redshift's explanation is pretty good. Among other things, if you only need a few columns there are entire blocks of data you don't need to touch at all. https://docs.aws.amazon.com/redshift/latest/dg/c_columnar_storage_disk_mem_mgmnt.html https://docs.aws.amazon.com/redshift/latest/dg/c_columnar_st... It's truly magical when you scope down a SELECT to the columns you need and see a query go blazing fast. Or maybe I'm easily impressed.

mumblemumble 6y ago

A columnar format is almost always what you want for analytical workloads, because their access patterns tend to iterate ranges of rows but select only a few columns at random. About the only thing protocol buffers has in common is that it's a standardized binary format. The use case is largely non-overlapping, though. Protobuf is meant for transmitting monolithic datagrams, where the entire thing will be transmitted and then decoded as a monolithic blob. It's also, out of the box, not the best for efficiently transmitting highly repetitive data. Column-oriented formats cut down on some repetition of metadata, and also tend to be more compressible because similar data tends to get clumped together. Coincidentally, Arrow's format for transmitting data over a network, Arrow Flight, uses protocol buffers as its messaging format. Though the payload is still blocks of column-oriented data, for efficiency.

georgewfraser 6y ago

The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe: (A int, B int, C int, D int) And I write: A + B In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory) will be a lot more efficient. In a row-oriented representation like protobuf, all your C and D values are going to get dragged into the CPU registers alongside the A and B values that you actually want. Column-oriented representation is also more friendly to SIMD CPU instructions. You can still use SIMD with a row-oriented representation, but you have to use gather-scatter operations which makes the whole thing less efficient.

jzer0cool 6y ago
Could you expand on this more columnar data and row data? I missed something here how the data is organized and what you mean by the C,D values getting dragged along.
- int_19h 6y ago
  Row data is an array of structs, with each struct representing a record. Columnar data is a struct of arrays, with each array representing a column.
- repsilat 6y ago
  To rephrase the sibling comment, if you had an array of four {ABCD} structs, there's basically two ways of storing them on disk: 1. AAAA BBBB CCCC DDDD 2. ABCD ABCD ABCD ABCD One major heuristic in how CPUs make your code fast is to assume that if you access some memory, you're probably interested in the memory nearby. So when you access the first "A" bit of memory (common to both sequences above), depending on the memory layout you use, the CPU might also be smart and load the next bits into memory too -- maybe the next "AA", maybe "BC". Depending on your workload, one or the other of those might be faster. If you're only interested in the first ABCD element because you're doing SELECT * FROM users WHERE id=$1 then you'll likely want "row-oriented" data -- the #2 scheme above. But if you're interested in all of the A values and none of the values from B/C/D because you're doing SELECT AVG(age) FROM users then you'll likely want something "column-oriented" -- the #1 scheme above.