3 ms·
Well, that is only true (and helpful) if it is a dataset that benefits from being in a tabular format. Consider, for instance, particle-based simulation data (
by JBorrow 2y ago
Well, that is only true (and helpful) if it is a dataset that benefits from being in a tabular format.
Consider, for instance, particle-based simulation data (loading that into any database is a waste of time...), or a set of images that you need to perform non-standard individualized processing on (again, a waste of time).
Stuffing stuff into postgres or clickhouse is great for your typical 'data science' workflows where data is consistent and the problem is 'oh no our transaction volume has increased 100x'. But in other (some would claim more interesting) cases, using database systems is unhelpful.
- v2thegreat 2y agoHi! I'm the author. You're right! The context that I was talking about was actually in the GIS space! Depending on what you're trying to do, it's not uncommon to have a single dataset that's 50TB large (think: 100-500m resolution global raster with daily data for 30 years) And that's not even considering any memory overhead when performing operations on top of that data. That type of stuff wouldn't fit into Postgres (we tried once, a long time ago), and usually, you might be trying to use multiple datasets simultaneously, so that data adds up pretty quickly! It's my first post (and writing) ever, so I have room for improvement. Thanks for taking the time to read!