3 ms·
> Apache Spark Is Very Slow I’m betting that the bottleneck is running ML training. Otherwise I can’t see why an ETL job on that amount of data would take sev
by mr_toad 8y ago
> Apache Spark Is Very Slow
I’m betting that the bottleneck is running ML training. Otherwise I can’t see why an ETL job on that amount of data would take several minutes.
- felipe_aramburu 8y agoAn etl job could take an infinite amount of time. I am not trying to be cute by saying this but the point is that some ETL jobs might be as simple as loading data from a file others could involve millions of transformations. There is no limit to what you can do in ETL. This is not just loading the data from a file and providing it to xgboost.
- felipe_aramburu 8y agoAnd as we state in the post. This does not include ML times since we wanted to show how much faster we were making ETL. The ML timing differences are even starker than ETL.