yeah we do not push an etl pipeline through json unless we have to (and generally cannot), most etl-scale data engineering we do is almost all arrow/parquet/orc

yeah we do not push an etl pipeline through json unless we have to (and generally cannot), most etl-scale data engineering we do is almost all arrow/parquet/orc/protobuf etc, and slow legacy, odbc/json, which is streams that we turn into typed and compact data. I think json fine for command/metadata layers though, esp early on, but pretty core to what I look for an etl/streaming tool is out-of-the-box foundations for the data plane the good news is the implicitly typed json examples look arrow friendly, so users can to/from_json if they don't care about data speed/quality like when prototyping and not think about it. there may be other data-engineering-friendly formats that'd work too. prefect, dask, and friends solve it by abstracting over it. you can send whatever you want.. and it happens to be friendly to dataframes (pydata) / compact & typed data. but there projects seem to be more about source/sink, so encouraging structure by default would be helpful...