4 ms·

> Why aren't we replacing CSV with it, especially for big data applications? Big data applications tend to use other structured binary formats like parquet and

> Why aren't we replacing CSV with it, especially for big data applications? Big data applications tend to use other structured binary formats like parquet and avro, which any big data tool can typically parse.

wenc 8y ago
Parquet is a great columnar format, and Avro is great for schema evolution. But they're not that easy to work with directly. To this day there isn't a decent Parquet viewer to do adhoc data viewing. SQLite on the other hand has a lightweight SQL REPL that can be invoked from the command line. Spark can work with SQLite via JDBC, though obviously it isn't as native as Parquet. Between SQLite and Parquet, I might pick Parquet under most circumstances. But it seems to me SQLite ought to at least be a better option than CSV for Spark jobs (less work needed to do type inference, predicate pushdowns are trivial, etc.)
- diogofranco 8y ago
  csv is notoriously bad with spark - in fact you couldn't write to csv for the longest time without external libraries like the databricks one, so I don't contest your last point. SQlite is usually a single file representing a database though, I don't know how it would work with partitioning and stuff, and then how to handle the schema evolving across sqlite files.
  - drej 8y ago
    Also, the now native Spaek CSV export doesn’t follow RFC 4180 with default settings, it escapes double quotes with a backlash. So odd.
AutoEngineer 8y ago
Speaking for an Engineer unfortunately often data is simple CSV. Parsing this data robustly and visualizing it without scripting is still something that hasn't been solved. If you have any idea how to do it, I'd be more than glad to hear about it.
- wenc 8y ago
  You can visualize CSVs easily without scripting: use Tableau or some other BI tool [1] (not free though). There are free alternatives, but many require programming. [1] https://www.absentdata.com/tableau-alternatives/ https://www.absentdata.com/tableau-alternatives/

wenc 8y ago

Parquet is a great columnar format, and Avro is great for schema evolution. But they're not that easy to work with directly. To this day there isn't a decent Parquet viewer to do adhoc data viewing. SQLite on the other hand has a lightweight SQL REPL that can be invoked from the command line. Spark can work with SQLite via JDBC, though obviously it isn't as native as Parquet. Between SQLite and Parquet, I might pick Parquet under most circumstances. But it seems to me SQLite ought to at least be a better option than CSV for Spark jobs (less work needed to do type inference, predicate pushdowns are trivial, etc.)

diogofranco 8y ago
csv is notoriously bad with spark - in fact you couldn't write to csv for the longest time without external libraries like the databricks one, so I don't contest your last point. SQlite is usually a single file representing a database though, I don't know how it would work with partitioning and stuff, and then how to handle the schema evolving across sqlite files.
- drej 8y ago
  Also, the now native Spaek CSV export doesn’t follow RFC 4180 with default settings, it escapes double quotes with a backlash. So odd.

AutoEngineer 8y ago

Speaking for an Engineer unfortunately often data is simple CSV. Parsing this data robustly and visualizing it without scripting is still something that hasn't been solved. If you have any idea how to do it, I'd be more than glad to hear about it.

wenc 8y ago
You can visualize CSVs easily without scripting: use Tableau or some other BI tool [1] (not free though). There are free alternatives, but many require programming. [1] https://www.absentdata.com/tableau-alternatives/ https://www.absentdata.com/tableau-alternatives/