7 ms·
I don't understand much about this apache/java data streaming ecosystem (ETL, Kafka, Cassandra, they're all buzzword bingo to me and i don't know what it all me
by skrebbel 6y ago
I don't understand much about this apache/java data streaming ecosystem (ETL, Kafka, Cassandra, they're all buzzword bingo to me and i don't know what it all means), but maybe someone here can translate this to simpler application programmer terms?
I read the overview, and I'm not sure yet, but is this like an in-memory database that runs inside your process? Like, sqlite without disk persistence, or Erlang ETS, but then columnar?
I can't completely tell from the overview whether it's about the data format or the querying capability. A columnar ETS alternative would be splendid indeed!
- cookguyruffles 6y agoI'm similarly confused. It seems to be a family of table encodings that sacrifice encoding simplicity for compactness. PyArrow implements parquet files for example, but also I think feather? PR for this project is a mess. Front page should be a bullet point list of deliverables rather than aspirations nobody understands
- vletal 6y agoCome on, it's on the top of the front page https://arrow.apache.org/ https://arrow.apache.org/
- cookguyruffles 6y agoYou don't understand, I'm already a user of PyArrow and it doesn't match that page at all. It handles Parquet and Feather, right? They don't appear on the home page. Clicking the "specifications" link instead starts to talk about Flatbuffers. What's going on?
- papercrane 6y agoArrow is the in-memory format, PyArrow supports loading and saving that data as Parquet and Feather formatted files.
- nerdponx 6y agoParquet and Feather are on-disk file formats. Arrow is an in-memory format. Parquet is not Arrow, but they work well together, in that one can easily be (de)serialized to the other. Feather uses the Arrow IPC format internally.
- Tostino 6y agoTheir FAQ page at least answered some of that for me. Have the same feeling. http://arrow.apache.org/faq http://arrow.apache.org/faq
- vletal 6y agoLong story short: Apache Arrow defines a format for (tabular) data to allow efficient computation and easier interop and sharing data between different frameworks.
- vmind 6y agoArrow itself is a standardised in-memory columnar data representation. The benefit of this is you can then send data between processes without needing to be concerned with serialise/deserialise. There's then a growing ecosystem around this, e.g. Flight for making the sending of data easier, DataFusion for querying, etc.
- aynyc 6y agoCan you persist Arrow-format data to disk? I see a lot of interests in it, but I can't figure out the use cases. For example, let's say I have ton (xx TB) of well-structured objects on S3. I want to run query via Spark/Presto on the data. I still need to deserialize the data from ORC/PARQUET into ARROW right? The advantage with ARROW here is if Spark/Presto can use this format to pass the data between worker nodes, the query would be faster because we don't need to deserialize/serialize when passing data between nodes? If yes, how do I utilize the format in Spark/Presto?
- lmeyerov 6y agoYou can: it is serializable and self-describing. However, unless "disk" is super fast and thus more likely memory, and your data is ephemeral, you probably shouldn't. Instead, we've been happier as parquet/orc: tunable compression, nicer multi-part / parallel readers, and a bit more stable. There is feather for persistence, but you don't need it: just as how you can stream binary arrow buffers to processes, you can write raw arrow to disk. In theory it might give some teams in some setups parallel read/write speedups, but we've been exploring other paths there, e.g., 90+GB/s per node via GDS https://pavilion.io/nvidia https://pavilion.io/nvidia . I'm not aware of feather efforts targeting that kind of perf but would be curious! To utilize w/ spark.. it already does underneath ;-) an increasing flow is something like spark filter -> gpu compute+ai, where the transfer is spark cpu rdd -> arrow (spark-native) -> rapids/tensorflow Edit: Arrow dev does seem more active than parquet/orc (and a lot of their dev is _by_ arrow devs!), so give it another couple of years, and I can see arrow being stable enough that you can persist data with less fear of having to reprocess older files and having most of the compression features you'd want!
- yazaddaruvala 6y agoAs I understand it, Arrow is about the data format, to best optimize query capabilities in any language. Think, “CapnProto” or “Protobufs” but for querying data rather than only data transfer. In theory Arrow will enable users to “trivially” create high performance SQLite type querying in any programming language. Arrow doesn’t help with any other features of SQLite tho. Maybe one day Arrow will also target on-disc analytics/persistence but for now it delegates to Parquet.
- lmeyerov 6y agoProtobufs is a great comparison point and exactly what we upgraded from. It's like protobuf but specialized for rich data tables and, for those living outside google's proprietary layers, modern sensibilities. This brings wins like self-describing (no .proto, it optionally includes a schema), streaming, columnar by default, etc. By standardizing the notion of a table (and chunk within it, a record batch), it plugs into compute/io frameworks that work on tables: spark, pandas, etc. Most compute/db framework that gives you lists/tables will likely have out of the box support for it, and even when not explicit, via tools like turbodc for SQL DBs. Arrow has compute libs, but we don't use them (yet). More as interop for compute engines that are fast (ex: rapids for gpu) or feature rich (ex: pandas for chunks). Likewise, for I/O, interop for good formats there: on-the-fly arrow -> persistent parquet|orc. It's enabling us to do cool stuff like fast interop between our cpu<>gpu code, and in the latest initiative, crossing even process & language runtime boundaries w/ zero-copy.
- T-R 6y agoData science does a lot of SQL-like and linear-algebra-like transformations over a lot of data, and needs it to be reasonably performant. This means you want to do things like minimize overhead of indexing into data, and use things like SIMD instructions/GPU or parallelize work. To do this, you generally want your data in column-major format - organized as objects of arrays, rather than arrays of objects. Dataframe libraries like Pandas (which uses optimized linear algebra libraries like BLAS/LAPACK under the hood, via numpy) and the Spark Dataframe API are for working with columnar data and getting performance via SIMD or parallelization, respectively. Generally people start off by doing these computations in a series of batch jobs (an "ETL pipeline", orchestrated with something like Airflow), to transform data into whatever shape they ultimately want it in; streaming technologies like Spark Streaming and Kafka can help with incrementally adding new rows to your data, rather than recomputing the whole thing every batch-job run. Whenever you want to involve multiple systems or multiple libraries in your dataframe transformations, there's potentially a lot of computational overhead in serializing the dataframes or just converting them between memory representations. Arrow is a standardized format, spearheaded by the person who wrote Pandas, that attempts to match the in-memory representation, so that whether you're passing the data between libraries in-memory or writing a file for some other system to read, no unnecessary transformations need to happen to work on the data.
- 6gvONxR4sf7o 6y ago> linear-algebra-like transformations > To do this, you generally want your data in column-major format I'd argue that the basic element of linear algebra is matrix vector multiplication, which I figured was best done row-major. Column major is great in other data use cases, but 'linear-algebra-like, therefore column major' doesn't feel right.
- BenoitP 6y agoI don't know about linear algebra, but column major lets you compress thus: * Dictionary encoding: US,US,US,US,FR -> US:0,FR:1;0,0,0,0,1 * Run-length encoding: 0,0,0,0,1 -> 4x0,1x1 * Delta encoding: 0,1,2,3,4 -> 5x'+1' * Storing the min and max for a chunk Basically: exploit the data type to compress it. Which enables very fast filtering and projections. (And now that the IO bottleneck has been managed you can do your gigantic logistic regression)
- SatvikBeri 6y agoAs one example, I use Arrow in production to share DataFrames between Python and Julia processes. My previous method was saving to disk and loading, this reduces the time from ~2 minutes to ~5 seconds.
- nerdponx 6y agoYou're actually sharing the memory between the processes? I'd love to know how that works.
- SatvikBeri 6y agoI actually don't know very much about how it works (which is a testament to the usability of the libraries!) I use PyCall to use Python from Julia, and PyJulia to do the reverse. PyCall & PyJulia have functionality to easily share arrays. PyCall is pretty seamless, PyJulia is a bit more work but still solid. To share a DataFrame, I convert it to Arrow, get a bytearray representation of that, use PyCall/PyJulia to access the array from a different process, and reinterpret it as Arrow data within that process. PyCall: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl PyJulia: https://pyjulia.readthedocs.io/en/latest/ https://pyjulia.readthedocs.io/en/latest/
- tracyhenry 6y agoHere's a talk by Arrow's lead maintainer Wes Mckinney: https://youtu.be/OtIU7HsHCE8?t=2731 https://youtu.be/OtIU7HsHCE8?t=2731 which I think gave a good overview on the high-level motivations behind Apache Arrow.
- kthejoker2 6y agoCan I just pushback on this misuse of the term "buzzword bingo"? By all means, admit you're not familiar with them, but don't cast that as some kind of culture jamming point of pride. ETL is a design pattern. Kafka and Cassandra are tools. Arrow is a data format. Sorry, will cop to being completely defensive here, but don't call my baby ugly.
- deleted 6y ago[deleted]
- nxpnsv 6y agoThis is actually a really a useful breakdown.
- deleted 6y ago[deleted]
- zarathustreal 6y ago>don't cast that as some kind of culture jamming point of pride. >Sorry, will cop to being completely defensive here, but don't call my baby ugly. Not trying to be antagonistic here for what it's worth - I see your statement as far more guilty of casting your understanding of this technology as a culture jamming point of pride than the GP. Furthermore the implication that "people who unironically refer to this process as ETL" is the dominant culture (and statements expressing the irony of giving the most common form of computation a specific meaningless acronym are necessarily "culture jamming") is not correct in my experience but YMMV
- kthejoker2 6y agoI don't know your world, but this is like telling a carpenter that hammers, jigsaws, and mahogany are "buzzword bingo", it's condescending and disrespectful. Dominant culture has nothing to do with it, it's about respecting boundaries. You not knowing something isn't an indictment on the something.
- skrebbel 6y ago
- derriz 6y agoNo it's not a database - it's much more low level than that. It's a chunked columnar storage format for large data sets. Think of storage layout for a table of data. With arrow, the table is first split row-wize into chunks (sometimes/often just one chunk), then each column of each chunk is stored as an array. The underlying arrays layout supports vectorized operations. Querying is largely independent of arrow itself which is just the memory format. But the format was designed to support efficient querying. If used as a disk format, for example, you can efficiently load a subset of columns without touching the entire file. And if you're lucky and/or you've chunked the data appropriately you may be able to skip entire chunks. The format is also language agnostic - as long as the language is python or C++ :) - and allows zero-copy passing of data across the language barrier assuming a shared memory model. This zero-copy feature is important when dealing with large in-memory data-sets. Unfortunately, until the entire Python data-science ecosystem is re-written from scratch, the application for arrow will largely for library writers and plumbing. Yes, as a data-scientist, you can easily turn your arrow table into a Pandas Dataframe or a numpy array but you run the risk of expensive copies occurring (actually a copy is guaranteed to happen if the table has more than one chunk) which sort-of defeats the whole purpose. And since to do anything useful with the data you're going to have to perform this conversion - as most of the Python data-science ecosystem is built on numpy and pandas - the format is not particularly interesting to data-science users, I feel. it their storage formats, I believe, will continue to dominate for columnar storage in the Python world for the foreseeable future.
- adrianmonk 6y agoETL: That data over there is in that format, and that other data over in that other place is in that other format, and I want it all over here in this format. So I will extract it (get it out of those other places), transform it (put it in my format), and load it (store it here). The term has been around for decades and traditionally is used by database people, but it can apply to any process that does this. A concrete example might be if you run a business where you sell used books through Amazon and eBay. They each have their own format for info about the status of a product you have listed (whether it has sold, which shipping option the buyer chose, whether payment was received, etc.), but you might want to have that data from both sources in one place so you can see a dashboard or analyze it.
- EdwardDiego 6y agoApache Kafka and Apache Cassandra may be written in JVM hosted languages, but they're widely used outside the JVM ecosystem. Kafka is a distributed append-only log widely used in data streaming. Cassandra is a distributed NoSQL datastore. Not sure why you raised that, though, given they're not mentioned in the linked article.