3 ms·
Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any idea
by zaptheimpaler 10y ago
Looks awesome, the work on performance especially is much appreciated :D I would love to be able to get flame graphs on my spark cluster per node too - any ideas on how exactly FB does this?
- threeseed 10y agoLooks they are using the same library from Presto. You would just add the Java agent line to your Spark config i.e. spark.executor.extraJavaOptions. https://github.com/prestodb/presto/issues/4004 https://github.com/prestodb/presto/issues/4004 Although you have to distribute the library across to all of the data nodes (if using YARN) or Spark nodes (if in standalone).
- coredog64 10y agoThanks! I was going to ask where I might find it since Google results were a little thin on the ground.
- peterstjohn 10y agoI knocked up a quick Ansible script this morning. It is _very_ rough right now and is more like 'here's what you could do' (and you have to compile perf-map-agent beforehand), but I got it working on a small cluster: https://github.com/falloutdurham/spark-flame https://github.com/falloutdurham/spark-flame