Using ECMP to balance the traffic across the load balancers is definitely the way to go. You can couple this with a dynamic routing protocol (BGP/OSPF/ISIS) to

Using ECMP to balance the traffic across the load balancers is definitely the way to go. You can couple this with a dynamic routing protocol (BGP/OSPF/ISIS) to achieve fast failure detection of load-balancers too. We used this strategy at a previous company I worked at deploying large Openstack based clouds, worked really well. Though if you want to to be easy on the backends you need to use some sort of stick table or consistent hashing mechanism as described in the paper.