3 ms·
For finance in particular, I'd say we're drowning in a massive volume of shitty data. A client of mine purchases several fundamental feeds from Quandl, and I e
by claytonjy 8y ago
For finance in particular, I'd say we're drowning in a massive volume of shitty data.
A client of mine purchases several fundamental feeds from Quandl, and I email them regularly to point out errors. Not weird, hard, tricky errors, but stuff like "why are all these volumes missing" or "there's a 1-day closing price increase of 1200%" or "you divided when you should have multiplied". This tells me neither Quandl nor the original provider (e.g. Zacks) do any serious data validation, despite claiming to.
If the companies people have been paying for decades for this data get it wrong this often, how can I trust any weirder data they're trying to sell me? I thought the point of buying these feeds was to let the seller worry about quality assurance.
- vasilipupkin 8y agothis doesn't matter - any sophisticated user will have their own software to clean the data anyway. Their concern is getting the data, they know how to clean it once they have it.
- claytonjy 8y agoWe're not talking about data cleaning, but about data validation. I can fix a weirdly formatted field (cleaning), but I can't reliably impute most kinds of missing data. I can detect errors, but can't fix most of them without additional information...which is exactly what I'm paying the provider for.
- vasilipupkin 8y agoyou can, there are ways to do it. interpolation, etc. Sometimes the data is missing just because it's not available, you still have to handle that case. proper way of filling in this missing data will depend on what you are using it for - so for provider to do it would be kind of wrong actually.
- claytonjy 8y agoI think we're talking past eachother here. I don't expect the provider to do imputation for me, but I shouldn't have to bug them to get the best version of the data they have. Sure, sometimes missing is missing, but in my experience with Quandl/Zacks, its usually an error on their end. The price jumps are sometimes because they conflated two different tickers. If they divided instead of multiplying (split factors), I have to have external information to even detect the error! Same goes if they get a date wrong somewhere.
- thwy12321 8y agothis is what people in this thread dont really understand, investors want the raw feed. Theres nothing to be gained from an aggregated, cleaned feed that everyone has