3 ms·

My company is considering "upgrading" our Storm topologies to Heron. Given that Storm is Heron's predecessor, and Storm itself is maintained by the ASF, I wonde

by cozos 9y ago

My company is considering "upgrading" our Storm topologies to Heron. Given that Storm is Heron's predecessor, and Storm itself is maintained by the ASF, I wonder what this donation means for Heron. p.s. does anybody have experience with Dhalion, Heron's autoscaling thing (is it a module? or a plugin, idk). All I could find on it was a research paper and a barren github repository; how do you use this thing?

kramasamy 9y ago
Great! If you could join the heron slack - we can help you with upgrading and how to use Dhalion - http://heronstreaming.herokuapp.com/ http://heronstreaming.herokuapp.com/
avflor 9y ago
Great! Thanks for the questions. The Dhalion repo (https://github.com/Microsoft/Dhalion https://github.com/Microsoft/Dhalion) contains only the general Dhalion APIs. These are not Heron specific, that's why they are not part of Heron. In Heron, the healthmgr module essentially integrates these Dhalion APIs with Heron. Currently, Heron contains two Dhalion policies. The first one automatically restarts instances that exhibit backpressure. The second one scales up or restarts instances depending on some topology characteristics. Please join slack and we can show you how to use/modify the appropriate Heron yaml files so that you can use Dhalion. Dhalion does not require a separate installation -- it is already integrated with Heron and just needs to be configured depending on your use case. We are currently working on more aggressive autoscaling policies which should be part of the code base soon. Please let us know if you have other questions.
- cozos 9y ago
  Thanks for the reply! For us, the ability to self-tune is one of Heron's most attractive features. Tuning components/executors in Storm was a big pain point. Have you guys ever tried using the Dhalion autoscaling combined with hardware autoscaling from a cloud provider (i.e. Azure, AWS EC2)?
  - avflor 9y ago
    No, we have not used auto-scaling with automatic hardware provisioning. We assume that the hardware is there and we just request more containers (e.g, YARN containers). Theoretically, it should be possible to do that if the autoscaling policy is extended to request for additional hardware before changing the parallelism of a stage. But this functionality is not currently there.

by cozos 9y ago

kramasamy 9y ago

Great! If you could join the heron slack - we can help you with upgrading and how to use Dhalion - http://heronstreaming.herokuapp.com/ http://heronstreaming.herokuapp.com/

avflor 9y ago

Great! Thanks for the questions. The Dhalion repo (https://github.com/Microsoft/Dhalion https://github.com/Microsoft/Dhalion) contains only the general Dhalion APIs. These are not Heron specific, that's why they are not part of Heron. In Heron, the healthmgr module essentially integrates these Dhalion APIs with Heron. Currently, Heron contains two Dhalion policies. The first one automatically restarts instances that exhibit backpressure. The second one scales up or restarts instances depending on some topology characteristics. Please join slack and we can show you how to use/modify the appropriate Heron yaml files so that you can use Dhalion. Dhalion does not require a separate installation -- it is already integrated with Heron and just needs to be configured depending on your use case. We are currently working on more aggressive autoscaling policies which should be part of the code base soon. Please let us know if you have other questions.

cozos 9y ago
Thanks for the reply! For us, the ability to self-tune is one of Heron's most attractive features. Tuning components/executors in Storm was a big pain point. Have you guys ever tried using the Dhalion autoscaling combined with hardware autoscaling from a cloud provider (i.e. Azure, AWS EC2)?
- avflor 9y ago
  No, we have not used auto-scaling with automatic hardware provisioning. We assume that the hardware is there and we just request more containers (e.g, YARN containers). Theoretically, it should be possible to do that if the autoscaling policy is extended to request for additional hardware before changing the parallelism of a stage. But this functionality is not currently there.