3 ms·
From about 2008/2009/2010 or so on there was perhaps an over-emphasis on specialized tools for the mass acquisition of streams of data. Maybe in large part due
by cmrdporcupine 4y ago
From about 2008/2009/2010 or so on there was perhaps an over-emphasis on specialized tools for the mass acquisition of streams of data. Maybe in large part due to the explosion of $$ in ad-tech. Some people had legitimately insane click/impression streams -- I worked at a couple companies like that. Development of DBs based on LSM trees or other write-specialized storage structures became important. Existing relational databases weren't particularly well built for this stuff. This was part of, but not the whole story with the whole NoSQL thing. People were willing to go completely denormalized in order to gain some advantage or ability here. It helped that much of the data looked at was of perhaps little structural complexity.
In the meantime SSD storage took off, so the IOPS from a stock drive have skyrocketed, business domains for large data sets have broadened beyond click/impression streams, and the challenge now is not "can I store all this data" it's "WTH do I do with it?"
Regardless of quantity of data, structuring and analysis and querying of said data remains paramount. The challenge for anybody working with data is to represent and extract knowledge. I remain convinced that logic -- first order logic and its offshoot in the relational model -- remains the best tool for reasoning about knowledge. Codd's prognostications on data from the 1970s are still profound.
I think we're in a space now where we can turn our attention to knowledge management, not just accumulating streams of unstructured data. The challenge in a business is to discover and capture the rules and relationship in data. SQL is an existing but poor tool for this, based on some of the concepts in the relational model but tossing them together in a relatively uncomposable and awkward way (though it remains better than the dogs breakfast of "NoSQL" alternatives that were tossed together for a while there.)
My employer is working in this space, I think they have a really good product: https://relational.ai/ https://relational.ai/