4 ms·
Why not use Parquet?
by zeristor 1y ago
Why not use Parquet?
- constantcrying 1y agoOr HDF5 or any other format which is actually meant to store large amounts of floating point data.
- mcraiha 1y agoExcel does not output Parquet.
- speed_spread 1y agoTrue. But also Excel probably collapses into a black hole going straight to hell trying to handle 21GB of data.
- hermitcrab 1y agoExcel .xlsx files are limited to 1,048,576 rows and 16,384 columns. Excel .xls files are limited to 65,536 rows and 256 columns.
- mihular 1y ago21GB/s, not 21GB ...
- anthk 1y agomawk would handle a 21 GB csv (or maybe one true awk) fast enough.
- buyucu 1y agoExcel often outputs broken csv :)
- hinkley 1y agoI have been privileged in my career to never need to parse Excel output but occasionally feed it input. Especially before Grafana was a household name. Putting something out so manager stops asking you 20 questions about the data is a double edged sword though. Those people can hallucinate more than a pre-Covid AI engine. Grafana is just weird enough that people would rather consume a chart than try to make one, then you have some control over the acid trip.
- _ea1k 1y agoBecause that would be too logical. :) It is an interesting benchmark anyway.