Here is how some of it worked. Some reasonably smart people build a system that works well enough to make progress, but then it has problems (like, world ending

Here is how some of it worked. Some reasonably smart people build a system that works well enough to make progress, but then it has problems (like, world ending problems requiring constant care). These problems manifest in requiring the attention of really smart people under a decent manager to go forth and bring the system under control. This was what I experienced having re-architected the real-time system not once, but twice. That's right, I redesigned the system twice for a variety of reasons. The first time was to handle even more scale and reliability, and I talked about it here: https://www.usenix.org/conference/srecon17americas/program/presentation/erlich https://www.usenix.org/conference/srecon17americas/program/p... That approach worked well, but wow, did it become unwieldy as the new features, massive scale, and much better reliability resulted in more use-cases. We stuff so much into until it became a problem. That problem required solving, and that was the basis for BladeRunner: https://dl.acm.org/doi/10.1145/3477132.3483572 https://dl.acm.org/doi/10.1145/3477132.3483572 So, to answer your question, the research generally manifests from a need. Sometimes it is accidental, but it is often driven by some kind of essential need. In FB's case, it is improving reliability, massive scale, and driving engineering pain down. Edit: to add onto the XKCD, there is a tower of babel effect for many things. An area that interests me is protocol design for streams since we are very much in the dark ages.