2 ms·
Yeah.. I agree. It's not really that I don't want to use spark, it's more that I don't want to have to use spark just to convert some data files. If I can wri
by justinsaccount 10y ago
Yeah.. I agree. It's not really that I don't want to use spark, it's more that I don't want to have to use spark just to convert some data files.
If I can write a short tool that wraps the
spark.read.json(input paths).write.parquet("/target/path")
As
spark.read.json(argv[1]).write.parquet(argv[2])
and then use it like
json_to_parquet /logs/json/2016-12-28.gz /logs/parquet/2016-12-28
or something, that would work.
I just wish there were some more cli friendly tools for the cases where the grep | cut | sort pipeline works, but you'd like to just make it a bit more efficient.