3 ms·
I come from the non-technical side of things. Do you know of any resources that would cover this issue, but for someone on the business side? Not an engineer,
by anongraddebt 8y ago
I come from the non-technical side of things. Do you know of any resources that would cover this issue, but for someone on the business side?
Not an engineer, so while I understand this in a general/abstract sense, my understanding is limited to, "Cleaning/transformation is messy and a time sink due to non-standardization of data."
- pmart123 8y agoOne good example I uncovered a while back was that Bloomberg timestamped its crude oil futures data by finding the last trade to occur in a given second and rounding down. This means that the user of the data had no idea if the price used on the 10:30:30 AM print occurred at 10:30:00.999 or 10:30:00.001. Obviously, this could create problems if thought you found a lead/lag relationship between say oil and oil stocks. Similarly, say a vendor aggregated website visits/pageviews but didn't account for the fact that 1/3 of the traffic was coming click-bots in developing countries. If they presented you with the raw data you could figure it out and filter those countries out, but if it is aggregated, you might not discover the issue. Then, there could be even simpler ones like determining the opening price for a stock. If say the first print of stock XYZ trades 10 shares at a price of $20, but a millisecond later, 100k shares trade at $20.11, which print should you use in your simulation algorithms as the opening print?