Honestly this entire system is a mess. But it's working so nobody is going to change it, plus going "over the top" and building a better, more idealistic soluti

Honestly this entire system is a mess. But it's working so nobody is going to change it, plus going "over the top" and building a better, more idealistic solution will have the same set of problems. Assuming you end up building something dramatically better or easier, getting market share means addressing more and more use cases, and you more or less end up as the n+1th protocol or standard. To me the only long term in dealing with this mess is going to be some shift in how we actually do computing on data. Leave the data at rest / in-situ and move more and more compute capacity to where it sits, then merge results together later. We're getting to the point where containers are common place, and FaaS is becoming comfortable.