5 ms·
We do exactly this, with billions of rows each day of statistical data right into MSSQL server instances. For data like this, the only difference is that we par
by endianswap 13y ago
We do exactly this, with billions of rows each day of statistical data right into MSSQL server instances. For data like this, the only difference is that we partition the data into daily tables and provide views into the data as necessary.
What experience or expertise can you share as to why this doesn't work?
- belluchan 13y agoI mean if it works for you I cannot say it doesn't work. It doesn't seem like a palatable solution however. Does your SQL database also have other types of information? Like account data? Does having billions of rows of statistical data make it more difficult to manage, replicate and secure non-statistical data? Like your logs must just be useless. You can bring down trees with dynamite, but just because that "works" doesn't mean it's the optimal solution.
- jules 13y agoWhat's the reason for partitioning into daily tables? Why not add a day column and index it on that column?
- endianswap 13y agoEasier to move old historical data onto larger, slower drives if it's stored independently.
- RyanZAG 13y agoIndex would become bigger than ram. Removing old data would be more difficult. They likely only use todays data. This is probably a trading data system of some kind.