4 ms·

I'm a little confused with Arrow being included in the comparison. What does Arrow have to do with Parquet? We are talking about the file format Parquet, right

• pletnes 5y ago
  This isn’t a comp sci geek doing performance analysis. It’s a guy who wants to do stats, and then throws up 3 options and discusses tradeoffs - memory and time consumption. He got his little workflow from a few minutes to a second, and that is what matters to him.
• wodenokoto 5y ago
  I was also confused at the headline but felt the comparison made sense by the end of the article. > Why not at least benchmark the query execution alone instead of startup and loading of data? Because they don’t intend to have R running with 4gb of data in memory constantly. I’d assume there’s a SQLite /duckdb instance running, but that it is negligible. Maybe the test even includes starting the database server. So the perspective is “I want some data on my hard drive, and I want to query it from R every once in a while, what are my options?” And suddenly a file format becomes comparable to a database. Another way to look at it is to consider the compiled arrow code in the arrow library as a database engine. Then arrow is just another alternative to SQLite.
  • willvarfar 5y ago
    (SQLite and DuckDB are both client-side databases.)
    • loxias 5y ago
      You mean in-process databases :)
    • wodenokoto 5y ago
      Is database daemon the right word, when the database is running locally? I thought the database process was still a server when running locally, like how Wayland is a display server.
      • hueho 5y ago
        There isn't a daemon to speak off, they are shared libraries that you load and call functions on. I don't know much about R, but if it considers the whole memory space of the R process, it probably counted the memory consumption of the libraries themselves as well.
        • kristjansson 5y ago
          The memory profiling only measures allocations on the R heap (i.e. objects managed by R's GC). It doesn't measure allocations by other libraries running in the the same process, unless they use R's allocation machinery.
• seertaak 5y ago
  > What does Arrow have to do with Parquet? We are talking about the file format Parquet, right? Does Arrow use Parquet as its default data storage format? This question comes up quite often. Parquet is a _file_ format, Arrow is a language-independent _in-memory_ format. You can e.g. read a parquet file into a typed Arrow buffer backed by shared memory, allowing code written in Java, Python, or C++ (and many more!) to read from it in a performant way (i.e. without copies). Another way of looking at it, if you have a C++ background, is that (roughly speaking) it makes C++'s coolest feature - templates -, and the performance gains obtained by the concomitant inlinability of the generated code - available in other languages. For example, you can write `pa.array([1, 2], type=pa.uint16())` in python, which translates roughly to `std::vector<uint16_t>{1, 2}` in C++. But it's not quite that; Arrow arrays actually consist of several buffers, one of which is a bit mask indicating whether the next item in the array is valid or missing (what previously was accomplished by NaN). While I'm not a huge fan of Arrow's inheritance-based C++ implementation (it's quite clunky to say the least), it's an important project IMHO. Next, why compare Arrow with SQLite and DuckDB? Because it's what it's being used for already! For example, PySpark uses Arrow to mediate data between Python and Scala (the implementation language), providing access to the data through an SQL-like language.
  • chrisjc 5y ago
    Makes sense. I should have included this functionality in my description of the value Arrow brings: > read ... into a typed Arrow buffer backed by shared memory, allowing code written in Java, Python, or C++ (and many more!) to read from it in a performant way (i.e. without copies). Very powerful indeed. You lost me here though: > Next, why compare Arrow with SQLite and DuckDB? Because it's what it's being used for already! What is already being used for what? The example that follows that describes the advantages of PySpark (Python/Scala) using Arrow makes sense, but I'm having trouble understanding your assertion relating it to SQLite and DuckDB?
    • seertaak 5y ago
      > What is already being used for what? Let's say you have some data. You can choose to store it in a relational DB, like SQLite or DuckDB, or you can store it in a parquet file (and load it into an Arrow buffer). And the point is that if you combine Arrow with, say, Spark, then as a user you can accomplish something similar to what you might accomplish with a relational DB. But you don't need to hassle with setting up a DB server and maintaining it. All you need is a job that outputs a parquet file, and uploads that to S3. And then Spark - through Arrow! - will allow you to execute queries against that DB. Using Arrow + Spark, you get the ability to a dataframe as if it's SQL, but you can still do pandas-style stuff i.e. treat it as a dataframe. OTOH you lose the more esoteric SQL stuff like fancy constraints, triggers, foreign keys.
• FridgeSeal 5y ago
  I assume Parquet has a close relationship with Arrow (project wise) and the parquet libraries I’ve used all supported reading the Arrow representation straight from the Parquet file. I imagine they’re all getting compared because 2 of them are “embedded” relational DB (even though their data structure layout is optimised for different things) and the remaining option appears to present similar benefits and functionality in a different style.