4 ms·
The article mentions having a long history of data, but I want to stress a corollary of this: if you think you might eventually want to sell data, either put yo
by paulgb 6y ago
The article mentions having a long history of data, but I want to stress a corollary of this: if you think you might eventually want to sell data, either put your data in an append-only data format now so that history is preserved, or at least take regular snapshots.
Otherwise, any time you update the data in place, you make it impossible to reconstruct what the data "looked like at the time" for a backtest.
- robswc 6y agoDefinitely agree. I'm not on the level of these hedge funds but I do love using data for trading.
- fractionalhare 6y agoYep. You typically need to incubate your data for several quarters to demonstrate a real correlation. And you have to balance how you prove this against the fact that some firms will try to reproduce the data collection internally once they learn how you get it.
- conformist 6y agoAbsolutely. Ideally, you'd want both, "append only" and a vintage of the corrected historical data at every point in time. And good metadata.
- jkinudsjknds 6y agoWhat do you mean by this? I'm imagining, say, rainfall per day, with a column for date, and a column for rainfall. What's useful about getting the data as it was vs. just having a total dataset with accurate dates.
- paulgb 6y agoFor something as simple as a sensor reading, you're right, the most natural way to store it is basically an append-only time-series database so it doesn't require much special care. But say the Rainfall table had a foreign key to a Sensor table that had the coordinates of the sensor. If the physical sensor were moved, the most straightforward data schema would call for the coordinates for the Sensor table to be updated, but then the historic data would have the wrong coordinates if you did a join. The ideal solution is to design a bitemporal schema so that rows are only ever inserted, not updated, but failing that, regular automatic database snapshots are a good start.