3 ms·
I wish there was more details on the benchmark. For example, I am interested to know if schema inference was turned on for spark-csv?
by falaki 10y ago
I wish there was more details on the benchmark. For example, I am interested to know if schema inference was turned on for spark-csv?
- gcommer 10y agoThe benchmarking scripts are in their repo: this line seems to answer your question: ("inferschema" was turned on, probably to make it an apples-to-apples comparison since static schemas is still on their TODO) https://github.com/wiseio/paratext/blob/ca347a552a53595b680c5adc3e1373ebc267864f/bench/run_experiment.py#L113 https://github.com/wiseio/paratext/blob/ca347a552a53595b680c...