3 ms·

The annoying part is that ML is not sold to the world like this. I would say the truth (I believe) in this comment is the "dirty little secret" of our industry.

by typon 8y ago

The annoying part is that ML is not sold to the world like this. I would say the truth (I believe) in this comment is the "dirty little secret" of our industry. Everyone working on it knows this but the research and VC dollars are flowing in so no one wants to talk about it too much.

chrstphrhrt 8y ago
Can confirm, worked at an AI hype company for over a year and built all their systems outside the NN internals. There is going to be a correction.
- kuhhk 8y ago
  May I ask if this insight is why you left? What do you work on now?
  - chrstphrhrt 8y ago
    Actually left due to anti-engineering culture (no tech leaders), succumbed to the distortions of Conway's law due to imbalance between professional services and core/platform. That said, the nature of the lack of leadership was around the MBA mindset that data science is the be-all end-all of AI, which IME is so far from the truth. There needs to be many layers/spirals of naive methods (i.e. straight ahead engineering) as a vanguard on the front lines with the DS/NNs/NLP bringing up the flanks for aspects of the domains that become more well understood over time. So yeah, tail wagging the dog, cart before horse etc. If you just barrel ahead with a monolithic NN pretending that all classes are created equal (structureless blob), which is what our DS PhDs were doing, it will quickly ossify. In our case it was intent prediction so we got to pretty high accuracy, but the labels were indicating many different categories that actually had relations in nature that could not be expressed. Ironically it took some regular engineers to research HMC (hierarchical multi-label classification) and ensembles and implement a new model training framework to support them. Not sure what they teach in school but it doesn't seem to be very practical. EDIT: now working on messaging (SMS/MMS/RCS/ABC) with a focus on dialog management and logical rules with layering-in of progressively less naive stats methods strategically rather than blindly. It helps to have an existing revenue stream that can be leveraged to create even more value with AI features rather than the backwards hail Mary of not iterating from a foothold of existing traction.
RA_Fisher 8y ago
Statistics is quite valuable and grounded in theory. It's a trusty tool if used with respect (like a knife).
- candiodari 8y ago
  Yeah, too bad the theory requires a number of things to be true for data and models ... that are essentially never true. Once you think about 1) is the independance assumption really applicable here ? 2) does the underlying model satisfy the law of large numbers ? (NOT your model, the underlying model, ie. "reality") 3) am I really only interested in predictions within the data range ? The answer for all these questions is almost universally "no", and the theory says that statistics is not guaranteed to work under those circumstances. Correct statistical answers: 1) which average percentage of borrowers will default ? There is no answer to that question, because many non-random influences will make borrowers default in synchronized, and therefore very much non-independent ways (say, an economic crisis, an earthquake, an epidemic). Therefore you cannot correctly even calculate an average. 2) which average percentage of borrowers will default ? A model that describes whether an individual will default is going to be non-differentiable, and therefore certainly violate the law of large numbers (there will be sudden jumps in the default rate everywhere because of a million random reasons. An idiot tv anchor declares now the time to sell, and half the town finds out their loan is underwater, say). So underlying models involving humans essentially never satisfy the law of large numbers. 3) which average percentage of borrowers will default ? The number of factors changing that value do not match between the period your data is from (ie. the past) and the period you're predicting. Because there are so many real-world factors that affect your variable you can never avoid this situation. Therefore you cannot predict. All of these issues allow one to construct realistic scenarios where even trivial statistics will fail spectacularly.
  - RA_Fisher 8y ago
    It's true, statistics can't answer every question. There are some limits.

by typon 8y ago

chrstphrhrt 8y ago

Can confirm, worked at an AI hype company for over a year and built all their systems outside the NN internals. There is going to be a correction.

kuhhk 8y ago
May I ask if this insight is why you left? What do you work on now?
- chrstphrhrt 8y ago
  Actually left due to anti-engineering culture (no tech leaders), succumbed to the distortions of Conway's law due to imbalance between professional services and core/platform. That said, the nature of the lack of leadership was around the MBA mindset that data science is the be-all end-all of AI, which IME is so far from the truth. There needs to be many layers/spirals of naive methods (i.e. straight ahead engineering) as a vanguard on the front lines with the DS/NNs/NLP bringing up the flanks for aspects of the domains that become more well understood over time. So yeah, tail wagging the dog, cart before horse etc. If you just barrel ahead with a monolithic NN pretending that all classes are created equal (structureless blob), which is what our DS PhDs were doing, it will quickly ossify. In our case it was intent prediction so we got to pretty high accuracy, but the labels were indicating many different categories that actually had relations in nature that could not be expressed. Ironically it took some regular engineers to research HMC (hierarchical multi-label classification) and ensembles and implement a new model training framework to support them. Not sure what they teach in school but it doesn't seem to be very practical. EDIT: now working on messaging (SMS/MMS/RCS/ABC) with a focus on dialog management and logical rules with layering-in of progressively less naive stats methods strategically rather than blindly. It helps to have an existing revenue stream that can be leveraged to create even more value with AI features rather than the backwards hail Mary of not iterating from a foothold of existing traction.

RA_Fisher 8y ago

Statistics is quite valuable and grounded in theory. It's a trusty tool if used with respect (like a knife).

candiodari 8y ago
Yeah, too bad the theory requires a number of things to be true for data and models ... that are essentially never true. Once you think about 1) is the independance assumption really applicable here ? 2) does the underlying model satisfy the law of large numbers ? (NOT your model, the underlying model, ie. "reality") 3) am I really only interested in predictions within the data range ? The answer for all these questions is almost universally "no", and the theory says that statistics is not guaranteed to work under those circumstances. Correct statistical answers: 1) which average percentage of borrowers will default ? There is no answer to that question, because many non-random influences will make borrowers default in synchronized, and therefore very much non-independent ways (say, an economic crisis, an earthquake, an epidemic). Therefore you cannot correctly even calculate an average. 2) which average percentage of borrowers will default ? A model that describes whether an individual will default is going to be non-differentiable, and therefore certainly violate the law of large numbers (there will be sudden jumps in the default rate everywhere because of a million random reasons. An idiot tv anchor declares now the time to sell, and half the town finds out their loan is underwater, say). So underlying models involving humans essentially never satisfy the law of large numbers. 3) which average percentage of borrowers will default ? The number of factors changing that value do not match between the period your data is from (ie. the past) and the period you're predicting. Because there are so many real-world factors that affect your variable you can never avoid this situation. Therefore you cannot predict. All of these issues allow one to construct realistic scenarios where even trivial statistics will fail spectacularly.
- RA_Fisher 8y ago
  It's true, statistics can't answer every question. There are some limits.