3 ms·
> The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns. That's not asymptotic theory
by rv-de 8y ago
> The results were consistent with asymptotic theory (central limit theorem) that predicts that more data has diminishing returns.
That's not asymptotic theory or central limit theorem. It's a meek proposition that you learn to prove in first semester when studying math. A bounded and monotonous function will always converge.
> Higher velocity data does not improve percentage accuracy and makes accuracy levels worse!
What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". "Higher velocity" is marketing lingo related to their Kinesis Data Streams service. But I assume that he refers to the usual loss of performance observed with all online-learning algos. Feeding knowledge one by one will mostly just overwrite learned generalizations. Or maybe he is indicating that they are compromising algo quality for faster execution.
> It is important to pick a single metric to improve, even if it is not perfect, but to use it as the basis for measuring performance improvement.
I think that is wrong. Single metrics never capture complex behavior well and will lead to distortions if fed back to the system. I mean it's normal to use one metric for the error. But saying this is the real deal sounds ridiculous to me as this is being done most of the time anyway and probably is going to change in the future. Backprop only works with one metric at the moment - that's just the fact.
> Pat noted that improvement and learning is often very slow – sort of like a slow weight loss program, where you lose weight very slowly. Processes may only be improving by 20 basis points a quarter, or 80 basis points a year. That isn’t a lot, but over a decade, it really makes a difference.
Now he's contradicting himself as he gives a reason for why big data is beneficial.
> His final word of advice – students should be broad in their knowledge of a lot of things, but need to be very deep in one area.
And again he is contradicting himself. Because if you make an analogy from student to a learning algorithm he now gives TWO orthogonal metrics to optimize for.
- darawk 8y ago> What on earth is this supposed to mean? An ML algorithm doesn't care about how fast data "arrives". My guess is that he's referring to the resolution of a time series. E.g. Going from months to weeks is 'more' data, but makes your models worse.
- throwawaymath 8y agoGood point. If I retrofit the term "resolution" in place of "velocity", the author's statement does make a lot more sense. If that's what was intended they really should have used that terminology, because I was similarly confused at the term "velocity." Whenever I've worked with time series data I've always referred to the granularity of the time dimension as its "resolution" - this is also common in geospatial data. I don't think (but am happy to be corrected) that "velocity" is a term of art in timeseries analysis.
- darawk 8y agoYa I was confused by it at first too, it definitely could have been more clear. But it's also been my experience with time series that more resolution isn't always better.
- deleted 8y ago[deleted]
- erikb 8y agoIt seems like the dissonance you felt from reading comes from the article being written more in a business style of writing and thinking rather than a technical style which you are more used to. For instance the "velocity data" and "but over a decade it makes a difference" parts. From a technical perspective it's awesome if you know that thanks to the data science you can outcompete others over a decade. But in business terms thinking is quarter based (i.e. three month) for short term and yearly for long term. The current leader that has to spend his budget on data science or other stuff has to find an advantage in it in the next few weeks after implementation, because usually making it happen takes already a quarter or two. You might think they are idiots for thinking that short term, but their goals are also set in this way. So if they invest heavily in a topic and don't see any results for 3 quarters they might be replaced. So if you say algorithm X will cost him 80% of his budget to implement but only shows results ten years later, for him that's the same as "no results". It's just the game he has to play to stay in the game. I personally think this is part of why companies will not be able to make a drastic change in their DNA and instead will be replaced by the next generation of companies who comes out of start-ups, or be replaced by companies who already actively participate in the market in other areas and have the right DNA to take on the market. For instance Amazon is already more data driven than the traditional super market chains, therefore they now can attack that space with their modern technology. > And again he is contradicting himself. Because if you make an analogy from student to a learning algorithm he now gives TWO orthogonal metrics to optimize for. Loving that part. I bet few people have combined his advise for student learning with his advise on how to do ML earlier.