7 ms·
Uhh.. maybe. It's a serde that's trying to be cross-language / platform. I guess it also offers some APIs to process the data so you can minimize serde operati
by waynesonfire 6y ago
Uhh.. maybe. It's a serde that's trying to be cross-language / platform.
I guess it also offers some APIs to process the data so you can minimize serde operations. But, I dunno. It's been hard to understand the benefit of the libabry and the posts here don't help.
- TuringTest 6y agoIf it works as a universal intermediate exchange language, it could help standardize connections among disparate systems. When you have N systems, it takes N^2 translators to build direct connections to transfer data between them; but it only takes N translators if all them can talk the same exchange language.
- waynesonfire 6y agocan you define what at translator is? I don't understand the complexity you're constructing. I have N systems and they talk protobuf. What's the problem?
- TuringTest 6y agoBy a translator, I mean a library that allows accessing data from different subsystems (either languages or OS processes). In this case, the advantages are that 1) Arrow is language agnostic, so it's likely that it can be used as a native library in your program and 2) it doesn't copy data to make it accessible to another process, so it saves a lot of marshalling / unmarshalling steps (assuming both sides use data in tabular format, which is typical of data analysis contexts).
- deleted 6y ago[deleted]
- mumblemumble 6y agoIt's not just a serde. One of its key use cases is eliminating serde.
- waynesonfire 6y agoI just don't believe you. My CPU doesn't understand Apache Arrow 3.0.
- kristjansson 6y agoNot GP post, but it might have been better stated as 'eliminating serde overhead'. Arrow's RPC serialization [1] is basically Protobuf, with a whole lot of hacks to eliminate copies on both ends of the wire. So it's still 'serde', but markedly more efficient for large blocks of tabular-ish data. [1]: https://arrow.apache.org/docs/format/Flight.html https://arrow.apache.org/docs/format/Flight.html
- wesm 6y ago> Arrow's serialization is Protobuf Incorrect. Only Arrow Flight embeds the Arrow wire format in a Protocol Buffer, but the Arrow protocol itself does not use Protobuf.
- kristjansson 6y agoApologies, off base there. Edited with a pointer to Flight :)
- mumblemumble 6y agoSo, there are several components to Arrow. One of them transfers data using IPC, and naturally needs to serialize. The other uses shared memory, which eliminates the need for serde. Sadly, the latter isn't (yet) well supported anywhere but Python and C++. If you can/do use it, though, data are just kept as as arrays in memory. Which is exactly what the CPU wants to see.
- superdimwit 6y agoShared memory format is supported in Julia too!
- chrisweekly 6y agoFor those wondering what a SerDe is: https://docs.serde.rs/serde/ https://docs.serde.rs/serde/
- waynesonfire 6y agoI meant serialize / deserialize in the literal sense.
- nevi-me 6y agoThe term likely predates the Rust implementation. SerDe is Serializer & Deserializer, which could be any framework or tool that allows the serialization and deserialization of data. I first came across the concept in Apache Hive.
- deleted 6y ago[deleted]
- wesm 6y agoThere's no serde by design (aside from inspecting a tiny piece of metadata indicating the location of each constituent block of memory). So data processing algorithms execute directly against the Arrow wire format without any deserialization.
- waynesonfire 6y agoOf course there is. There is always deserialization. The data format is most definitely not native to the CPU.
- wesm 6y agoI challenge you to have a closer look at the project. Deserialization by definition requires bytes or bits to be relocated from their position in the wire protocol to other data structures which are used for processing. Arrow does not require any bytes or bits to be relocated. So if a "C array of doubles" is not native to the CPU, then I don't know what is.
- throwaway894345 6y agoPerhaps "zero-copy" is a more precise or well-defined term?
- waynesonfire 6y agoCPUs come in many flavors. One area where they differ is in the way that bytes of a word are represented in memory. Two common formats are Big Endian and Little Endian. This is an example where a "C array of doubles" would be incompatible and some form of deserilaziation would be needed. My understanding is that an apache arrow library provides an API to manipulate the format in a platform agnostic way. But to claim that it eliminates deserialization is false.
- johncolanduoni 6y agoBig endian is pretty rare among anything you’d be doing in-memory analytics on. Looks like you can choose the endianess of the format if you need to, but it’s little endian by default: https://arrow.apache.org/docs/format/Columnar.html https://arrow.apache.org/docs/format/Columnar.html. I’d suggest reading up on the format, it covers the properties it provides to be friendly to direct random access.