6 ms·
I want to commend you for asking this question. Once you open your mind to systems thinking (in a general, broad or abstract sense) it will make you a far bette
by whalesalad 6y ago
I want to commend you for asking this question. Once you open your mind to systems thinking (in a general, broad or abstract sense) it will make you a far better engineer.
When it comes to data, you are ultimately worried about 1. storing it and making sure it stays there and 2. retrieving it or asking questions about that data with certain guarantees. Speed? Consistency? Local access? Grabbing a ton of rows at once? Grabbing really old data quick? The old adage is true here: nothing in life is free. If you want fast writes you might sacrifice read performance, or vice versa. If you dial one knob up, one knob needs to get dailed down (usually). All of the tools you listed have various trade offs and were designed or optimized for specific workloads. Some are more general (PSQL is a great example) but looking at them all spread out on a table the differences become more clear.
Choosing your tool will depend on how well it will meet your requierments and how it is going to play nice with all your other systems. Systems thinking is a lot bigger than choosing a performant tool that has the right libraries. You gotta think about long term support: how do I do backups of my data? How do I restore data? How do I perform upgrades down the road? How do I deal with downtime, can I throw more resources at it?
Long story short: I am very glad to hear more people thinking about systems engineering but make sure you don't get too caught up in the specific tooling and libraries. Learning and practicing the concepts and fundamentals and making sure to pause to think in the abstract boxes-and-lines sense is very important, too.
Learning about 'Clean architecture' and 'hexagonal architecture' will help to reinforce good systems design patterns.
- techsin101 6y agoMy thought process is that I don't want to neccessarily learn all the tools but rather understand them so I'd know which one to pick and when. Right now everything seems redududant from my perspective. i.e. Why have time series database? you can save that in postgres, just have two columns time, data.???
- whalesalad 6y agoStoring it is only half the problem, but what happens when you have 1 billion rows? How quickly will you be able to query the exact data you want? I would encourage you to stand up a PSQL instance, pack it with a hundred million rows of simulated data, and experiment :)