I agree. While data can always outgrow any single machine's ability to manage and process it in a optimal manner; the right software can allow larger data sets

I agree. While data can always outgrow any single machine's ability to manage and process it in a optimal manner; the right software can allow larger data sets to be performant without resorting to distributed systems. If the software takes full advantage of all the CPU cores and has an intelligent caching strategy then some amazing things can happen on affordable hardware. I am building a new kind of data management system. It can manage unstructured data like a file system as an object store. It can build very large relational tables and query them quickly. Its flexible schema allows the management of semi-structured data like NoSql systems. (https://didgets.com/ https://didgets.com/) It has distributed features built into the architecture, but I have only implemented the single node so far. On a $2000 desktop computer it can easily manage 200 million files and tables with 100M+ rows or thousands of columns.