2 ms·
I've done some fairly intense Spark turning. We were able to get a query that took 16 hours to 4 seconds. This took a long time, and included changing how the
by nl 8y ago
I've done some fairly intense Spark turning.
We were able to get a query that took 16 hours to 4 seconds. This took a long time, and included changing how the data was stored etc.
Anyway, fast Spark performance is possible but it is very sensitive to tuning (this was in ~2016, so maybe things have improved).
- scarejunba 8y agoInteresting. This is worth writing a blog post on.
- nl 8y agoI did a talk on it at Hadoop World or some HortonWorks conference one year. Don't know where the slides when though.
- scarejunba 8y agoDo you remember the title? I can try finding it on the Internet if you happen to be able to find the title (maybe from your submission email?). Thanks for doing the talk.
- nl 8y agoThis was the conference: https://dataworkssummit.com/sydney-2017/ https://dataworkssummit.com/sydney-2017/, but they don't seem to have uploaded any of the sessions from Sydney to the SlideShare. The one I did the previous year is nicely linked from http://hadoopsummit.org/melbourne/agenda/ http://hadoopsummit.org/melbourne/agenda/