3 ms·
For versioning numeric data complications include: - changing dimensions - handling changing types e.g. A column of strings becoming a compound of floats - eff
by pablobaz 7y ago
For versioning numeric data complications include:
- changing dimensions
- handling changing types e.g. A column of strings becoming a compound of floats
- efficiently handling appends
- efficiently handling corrections of previous data
- how to cleanup, remove old versions
The version store storage engine in Arctic provides fast and efficient versioning for numeric data. The code is quite readable so it might be worth having a look at its implementation too.
https://github.com/man-group/arctic/blob/master/README.md https://github.com/man-group/arctic/blob/master/README.md