3 ms·
What matters most is what you want to still be there when the power goes off: data. No amount of processing power matters if you don't have the data. Everyone
by fmjrey 7y ago
What matters most is what you want to still be there when the power goes off: data.
No amount of processing power matters if you don't have the data.
Everyone in the industry focuses too much on the processing side: objects, functions, containers, VMs, k8s, etc. but nobody really gives proper attention to data, its provenance, where it stays and where it goes, etc. I'm not saying engineers don't think about these things, they obviously have to think about it at some point. It's just that data is always accessory to the story.
It's like processing is the cool kid and data is the stinky one nobody wants to approach unless you have to.
Look at the 12 factor principles for example, where is data in there? How easy is it to take data from one place/cloud/database to another?
Data is the raw material, it needs to be the primary concern in programming languages and architectures, not objects or functions, containers or whatever, those come after, not first.
- Areading314 7y agoSQL and its continued success is a testament to how valuable this approach can be. Could be something here though, as I have found maintaining databases and datasets in a sane way to be difficult, even today.
- pesfandiar 7y agoThe problems around data are in general harder to address than the processing part. Most people dabbling in software engineering don't have the skills or attention span to work on those, and most of the issues are already solved by extremely complicated systems (e.g. DBs, Apache projects, ...). As for 12 factor principles, it was first touted by a PaaS provider. The whole idea is that they take care of the nitty gritty, while you can focus on the exciting (and technically easier) parts of software systems.
- fmjrey 7y agoI'd be tempted to say something similar: data is harder because it sticks, in the sense that it ties us back to the physical world because it's always tied to a place somewhere, it has to be moved, copied, synchronised, etc. However when I look at the accidental complexity we have created on the processing side, I wonder if it does not surpass the essential complexity of dealing with data. In other words: maybe we have yet to create the proper concepts and tools to deal with data, all this time the industry created a tower of babel (and of overspent $$$) with our languages, frameworks, containers, etc.
- Rafuino 7y agoAs someone in the hardware world where CPUs and fancy accelerators are all the rage, it's the same story when it comes to thinking about non-volatile memory (AKA NAND, Optane, PMem, etc.)
- pmlnr 7y agoI would add the problem of long term archiving - both as the physical medium and file formats into this. Will my jpgs be accessible 50 years dpwn the line? If yes, who will remember and have the equipment to read 100gb blu-ray mdisc?