3 ms·
I am building my own database engine using some data objects (key-value stores) I invented to form columnar store tables. It has some really fast query speeds a
by didgetmaster 4y ago
I am building my own database engine using some data objects (key-value stores) I invented to form columnar store tables. It has some really fast query speeds and analytic features (e.g. pivot tables) that test favorably compared to Postgres and other RDBMS offerings. (https://www.youtube.com/watch?v=OVICKCkWMZE https://www.youtube.com/watch?v=OVICKCkWMZE)
Partitioning a big table is definitely on my TODO list. How big does a typical table need to grow before partitioning is seen as a 'necessity'? What are some ways current partitioning strategies have made things too difficult?
- eatonphil 4y agoThat's awesome. Link to your repo or any posts you've written about your progress?
- didgetmaster 4y agoI haven't decided yet which parts I am going to open source or which license I am going to use; but the binaries are available for free download on our website https://www.Didgets.com https://www.Didgets.com and I have a newsletter at Didgets.substack.com
- nine_k 4y agoTo my mind, the right maximum size of a partition is such that your important queries are still fast. This often means that your most important indexes for the partition that serves most queries fit in RAM. So it's highly specific to a particular schema. Often partitions are made for time-related data, and these partitions may have smaller granularity. Say, you want to keep last 6 months of data, but make the partitions week-sized, or even day-sized, to make transitions more smooth.