6 ms·
Spark has improved our ETL jobs by orders of magnitude, both with respect to performance and ability to engage our workforce (mostly Python programmers). Previ
by fixxer 11y ago
Spark has improved our ETL jobs by orders of magnitude, both with respect to performance and ability to engage our workforce (mostly Python programmers).
Previous tools that improved workflow: docker, nginx.
- mfincham 11y agoWhat is Spark? Google turns up a bunch of projects called that.
- tuckerman 11y agoI assume they are talking about Apache Spark http://spark.apache.org/ http://spark.apache.org/
- xfax 11y agoMind if I ask how you use Spark for your ETL jobs?
- fixxer 11y agoFeature engineering. Transfers about 3.5b records into features that go into a variety of models. Previously was a hadoop streaming job (~40 hours); now about 6.