4 ms·
Data analysis workloads more often run into problems solved by partitioning rather than indexes.
by tomnipotent 6y ago
Data analysis workloads more often run into problems solved by partitioning rather than indexes.
- mistrial9 6y agoyes and no - I trained on "larger than RAM datasets" intentionally, and subsequently took on projects that require "larger than RAM datasets". Two things happened on my way, companies led by Google invented and deployed datasets previously impossible e.g. BigTable, and secondly the equipment I worked on went from modest to mid-sized RAM (hard to define that). Granted that lots of very useful (and common?) tasks are not "larger than RAM datasets", and then have very different characteristics.. which is starting to sound like "Excel problems look like this, they always fit in RAM, they need partitions not indexes" and the like.. There is a blind-man-and-the-Elephant drift here, which is not terrible, but might need calling out to improve..
- tomnipotent 6y ago> companies led by Google invented and deployed datasets previously impossible e.g. BigTable BigTable, Redshift, Snowflake and every other "big data" storage system rely heavily on partitioning to achieve the scales they're able to achieve. Some of these systems don't even support indexes.