4 ms·
There is a point which one needs to be aware of with Parquet: if you are not stepping into it with tools well suited for it (that is: Go, C++, Java or Python co
by julik 4y ago
There is a point which one needs to be aware of with Parquet: if you are not stepping into it with tools well suited for it (that is: Go, C++, Java or Python codebase/runtime, mainstream platform...) you are going to have a bad time. For all its advantages, Parquet is _very_ complicated (IMO overcomplicated, on multiple fronts), and uses binary dependencies which you absolutely need to carry around (Thrift?..). Up to a certain data size the advantage of CSV is not that it is a better format (it is decidedly, absolutely worse/more ambiguous than Parquet) but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything. If you ask me, Parquet - for all its advantages - is a product of the Cambrian explosion of the Apache-big-data-projects. Very little in its design warrants (or requires) using things like Thrift, so I won't exclude that something less overcomplicated (and more welcoming, and requiring less dependencies) will appear on the scene at some point.
- cm2187 4y agoTo be honest any file format that would do 1) strong typing/binary data, 2) string deduplication would capture most of the advantages of parquet, except being a column store format which helps in some cases and hurts in others.
- julik 4y agoAnd even using a columnar format but without all the complexity of Parquet would be comparatively easy (when writing using buffers)
- nomadluap 4y agoAt that point just start using an SQLite database as a data format.
- jonstewart 4y agoSQLite is a row-oriented store, not a column-oriented store, ie, there are fundamental tradeoffs between the two.
- cm2187 4y agoBut sqlite has no compression by default.
- bsder 4y agoIn addition, very few of these kind of serialization formats are useful for mutable data. One thing that I have been struggling with is to find an in-memory, MUTABLE data format/data structure that can be used between different processes in shared memory. I'm not sure one exists.
- julik 4y agoI have been using SQLite DBs to that end with some amount of success. If I remember well a lot of software of yore used BDB to the same effect.
- deepsun 4y agoApache Arrow? As far as I understand it, it's more for cross-process in-memory fast access (e.g. numbers crunching), but it's mutable.
- bsder 4y agoAs far as I can tell, Apache Arrow is not very mutable. It basically has to reallocate the universe. I don't need large data between processes. I need short-lived medium data ... large enough and short-lived enough to make serialization painful but mutable enough to make "stores" kind of annoying.
- manigandham 4y agoUse Redis. It's an in-memory data structure server, first created to have the same data structures usable and accessible across processes and languages. Or plenty of other in-memory database products (Apache Ignite) or fast key/value (ScyllaDB) or just use RDBMS (Postgres or SQLite).
- imtringued 4y ago>but the fact that you can write a valid CSV in a couple of lines of anything, and read in a couple of lines of anything. I seriously hope you are implying the usage of a library. Hand rolled CSV implentations are generally wrong and ruin your data.
- lisasays 4y agoThe beauty of CSV is that you don't -need- a library to access the data. It's just there, right in front of you. Hand rolled CSV implentations are generally wrong and ruin your data. Yeah sure - asymptotically. Once you've gotten a handle on your data, you can take care of all the bulletproofing later. None of this is meant to be cheeky, BTW. The learning curve and general cognitive overhead of Parquet is very considerable.