4 ms·
1. 10 million records is not big data. 2. If the rate of data acquisition is 1 million records a year, then 9 million records are the same as last year. There'
by brudgers 10y ago
1. 10 million records is not big data.
2. If the rate of data acquisition is 1 million records a year, then 9 million records are the same as last year. There's no reason to hit the database again, because time series data [to be time series data] does not change.
3. This suggests that processing the data from the Oracle database is not a requirement...i.e. it could be moved to another system and processed there. Again, it's small data and doesn't change so duplication presents neither a storage issue nor a consistency issue.
4. The size and static nature of the data suggest that it might fit into memory on a single, moderately speced PC. An AWS type approach is also possible.
5. The right data storage format depends on the workload...maybe something column oriented?
Good luck.