4 ms·

You're missing the point, which is that the current limitations of the system may not be the limitations in the future.

• jandrewrogers 13y ago
  Current limitations do imply future limitations. Database engines (and similar software) necessarily embed a large number of tradeoffs and assumptions in their design at the most fundamental levels. Every line of code is written to support the target workload to the exclusion of others because of the tradeoffs required. Design decisions are extremely sticky over the long-term because they are tacitly embedded in every piece of code. The point you are missing is that (1) significantly altering the basic architectural characteristics is tantamount to a complete rewrite from scratch and (2) existing users design their applications around the design assumptions of the platform so fundamentally changing the architecture abandons the user base as well. This is why in practice almost everyone starts from a blank slate if they need new architectural capabilities. And in the few cases where they did manage to re-architect an existing system, not only did it cost more than doing it from scratch but they lost their user base anyway. I've designed these types of systems for a long time. When a database engine is first designed, its capabilities, limitations, and future performance ceiling are essentially set in stone. All potential modifications have to fit within those constraints so you need to be cognizant of the choices you are making but not really thinking about. An experienced database engine designer can look at a system design and tell you what types of data models and workloads the system will always do poorly on. And every database engine design has weaknesses designed into them. Hadoop just happens to be a particularly weak engine with a low ceiling in terms of its expressiveness.