10 ms·

Andrew Ng: Unbiggen AI

209 points by sbehere 4y ago

DeathArrow 4y ago
Pretty interesting. Mr. Ng claims that for some applications having a small set of quality data can be as good as using huge set of noisy data. I wonder if, assuming the data is of highest quality, with minimal noise, having more data will matter for training or not. And if it matters, on what degree?
- frozenport 4y ago
  This is at the heart of the ML training problem. In general you want to add more variants of data but not so much that the network doesn't get trained by them. Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) and prefer more of those. Now, why not simply add everything? Well in general it takes too long to train.
  - pbowyer 4y ago
    > Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) How do you identify these images? It sounds like I'd need to build small models to see the variance but I'm hoping that there's a more scientific way?
- kavalg 4y ago
  It is relatively easy to turn small and accurate data to bigger and less accurate data with various forms of augmentation. The opposite is harder.
TOMDM 4y ago
Yeah that'd be great. I also want cars that run on salt water. I'm not saying that small data ai is equally impossible, but simply saying "we should make this better thing" isn't enough.
- atulsnj 4y ago
  Atleast someone's working on it.
- Datenstrom 4y ago
  > simply saying "we should make this better thing" isn't enough. Besides the references to his company which has customers and a product that already works on these principles the literature currently shows that this is very much possible if you dig into the correct niches. Besides the SOTA in few-shot and meta-learning it is possible to smartly choose the correct few samples for the network that yield the same results. It has also been my primary focus for the past 5 years and the core of the company I founded.
  - riku_iki 4y ago
    > it is possible to smartly choose the correct few samples for the network that yield the same results. And then, someone is using pretrained 500B model, and fine-tuning your few examples, and getting new SOTA.
    - deleted 4y ago
      [deleted]
    - YeGoblynQueenne 4y ago
      They might get new SOTA because the metric is accuracy, but if the metric was accuracy weighted by sample efficiency, then SOTA would look a lot less impressive. Simplest way to weigh by sample efficiency: multiply accuracy by ratio of test set to training set sizes. Everyone's training/testing on 80/20 splits, so everybody's SOTA would go down by 3/4s.
- sanxiyn 4y ago
  It's more of "this direction seems higher ROI than that direction", in particular quality vs quantity of data. Already in 2018 SenseTime reported that for face recognition, clean dataset surpasses accuracy of 4x larger raw dataset. https://arxiv.org/abs/1807.11649 https://arxiv.org/abs/1807.11649
whatever1 4y ago
My understanding is that they are trying to automate the data preparation steps that seasoned ML practitioners are doing anyway today. The fact that he tries this in manufacturing makes the case stronger. In most manufacturing companies you do not have access to top ML talent. You have Greg who knows python and recently visualized some production metrics. If we could empower Greg with automated ML libraries that guide him in the data preparation steps in combination with precooked networks like autogluon, then manufacturing could become a huge beneficiary of the ML revolution.
- overkalix 4y ago
  Greg probably also knows SAS and AMPL, and has a good knowledge of ops research, which is within stone-tossing distance of whatever ML is pretending to be this week.
  - whatever1 4y ago
    OR and ML have their own space in manufacturing. OR is perfect when you can describe explicitly what the decision space is and what the restrictions are. ML is great fit when you want to identify and use patterns. Quality control with machine vision is a good application for ML. NLP for PDF documents is a huge field for manufacturing as well. Companies have so much data in email attachments that they do not currently take advantage of.
    - overkalix 4y ago
      > OR is perfect when you can describe explicitly what the decision space is and what the restrictions are. As opposed to having to figure it out later from the outputs of a black box? > Quality control with machine vision is a good application for ML. I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc. > NLP for PDF documents is a huge field for manufacturing as well. NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes? I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
itissid 4y ago
That is the problem with generalization and cop outs like these. It's no good to people in the field doing actual work where the devil is in the detail. Big data is fairly important to a lot of things, for example I was listening to Tesla's use of Deep net models where they mentioned that there were literally so many variations of Stop Signs that they needed to learn what was really in the "tail" of the distribution of Stop Sign types to construct reliable AI
- vasco 4y ago
  Interestingly, when you learn how to drive you need to see approximately one example and you're able to identify them all.
  - corndoge 4y ago
    Is there some underlying point to this statement? It comes off as a passive dismissal of something but I'm not sure what. It might be helpful to directly state what you're trying to say so that other people can engage with it.
    - vasco 4y ago
      It doesn't seem like the other replies are having trouble engaging with it. Since we're giving each other advice, you should use your down votes instead of pontificating that other people's comments are "passive dismissals" when you don't like or understand them.
  - teruakohatu 4y ago
    That is called transfer learning. You might only need to see one photo of a sign to identify it in real life (although arguably learner drivers take a while to notice signs) but that is only because you have been training on identifying generic objects since you left the womb. You brain already knows how to select the most important features of a sign. The shape, the size and the color. You have also learned how to understand the text on the sign. A new born baby does not have that ability. This is applied in ANN as well. Transfer learning is using a pre-trained neural network, which has already learned identifying objects, and then using it to train on identifying a new, usually smaller, set of objects using, usually, a lot less training data. That is what Andrew is talking about in the article.
xiphias2 4y ago
I can imagine that customizing AI solutions in an automated way is quite important, but writing that as the next wave is probably an overstatement. Of course few shot learning is important for models, but for example for Pathways it was already part of the evaluation.
deleted 4y ago
[deleted]
a-dub 4y ago
data quality is important. every ai project i've worked on has started with visualizing the data and thinking about it. it's easy to get complacent and focus on building big datasets. in practice, looking at the data often reveals issues sometimes in data quality and sometimes scope of what's in there (if you're missing key examples, it's simply not going to work). most ml is actually data engineering.
atbpaca 4y ago
Glad to see the term ML being used more often than AI in the comments as it looks like most "AI" models are trained for image classification. Having said that, the idea of "doing more with less" sounds interesting and I wonder what it means exactly. Does it mean taking a dataset of 50 images and to create 1000s of synthetic images from it?
- spupe 4y ago
  Yeah I was very interested about that point in particular. I think synthetic data is one of the ideas, but I got the sense that he also means helping to identify what makes a data set good, even if small. It looks like Andrew Ng is developing a platform for automatically detecting whether a dataset is suitable and, if not, what are the steps to improve it. A sort of automated ML consultant, allowing you to sell capabilities much cheaper than if you needed to consult an actual expert.
notsag-hn 4y ago
I was going to interview at LandingAI. I was asked before the interview to install a spyware browser extension to monitor my traffic to detect if I was cheating during the interview. I respectfully declined and didn't have that interview.
- kevsim 4y ago
  Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad.
  - tablespoon 4y ago
    > Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad. Not necessarily, at least on the first point. Someone could be getting coached. A few years ago, a coworker of mine hired a contractor onto his team and was convinced the person who actually showed up was not the person who he interviewed (over the phone). He also thought the guy who did show up was getting a lot of help day-to-day from somewhere. The guy was a contractor, so it wasn't a huge problem because we could drop him quickly, but I would have never expected someone would do anything like that. However, it kind of makes sense as a scam: be a decent developer, get a stable of unhirable incompetents, and rotate them through companies while taking a cut of their salary.
    - mFixman 4y ago
      You cannot really prevent those kinds of cheats. Even if you use the most insidious spyware a coach can advice the interviewee from a different device. The only way to prevent those kind of scams is to put all employees in probation for the first months of work and fire them if they don't perform, like it's common in the UK.
      - 4y ago
aj7 4y ago
“I once built a face recognition system using 350 million images.” Did this make any of you a little queasy?
- mdp2021 4y ago
  Well noted! Explicitly: where does such database come from?
  - mkl 4y ago
    Frames of video could make the number sky-high like that without involving enormous numbers of people.
    - mdp2021 4y ago
      Right. Such as, extras in movies. Because, "350 million public faces" does not seem "cognitively digestible", but it could just be «350 million images» including large variations of the same faces, consistently with the "frames of video" idea.
leobg 4y ago
What are some ML data annotation tools that guide you towards those data points where the model gets confused? I hear Prodigy does this. Any others?
- jstx1 4y ago
  What's the role of these tools? Can't a developer just write the code to get those data points? At a first glance it seems like the hassle of integrating such a product into an existing ML codebase/pipeline is larger than solving the problem by hand.
  - leobg 4y ago
    What I mean is an annotation tool that interacts with the model itself in such a way that it will present to the user exactly those training examples next that will have the greatest impact in helping the model learn. So an annotation tool that provides a user interface for annotating data quickly (with keyboard shortcuts etc.). And looped into inference through the model to be trained, so you always get presented with the very training example that, out of the ones available, the model currently would be most unsure about.
a_square_peg 4y ago
I’ve been wondering about the limits of data-centric approach – there seems to be this implicit notion that more data equals better performing ML or AI. I think it would be interesting to imagine a point of diminishing return on additional data if we consider that our ability to perceive is probably largely based on two parts - sensory input and knowledge. Note that I’m making an explicit distinction here on the difference between data and knowledge. For instance, an English speaker and a non-English speaker may listen to someone speaking English and while the auditory signals received by both are the same, the meaning of the speech will only be perceived by the English speaker. When we’re learning a new language, it’s this ‘knowledge’ aspect that we’re enhancing in our brain, however that is encoded. This knowledge part is what allows us to see what’s not there but should be (e.g. the curious incident of the dog in the night) and when the data is inconsistent (e.g. all the nuclear close calls). I’m really not sure how this ‘knowledge’ part will be approached by the AI community but feel like we’re already close to having squeezed out as much as we can from just the data side of things. Somewhat related, we have a saying in Korean – ‘you see as much as you know’.
- Longwelwind 4y ago
  Can't you consider that knowledge is a function of previous data? In your example, the 2 individuals actually didn't receive the same amount of data because the English speakers received data previously that allowed him to build some kind of "knowledge" that allows him to solve specific related tasks (understanding a spoken sentence). This would be the equivalent of transfer learning where "knowledge" is a model trained on previous, more general, data.
  - mjburgess 4y ago
    Nope, it's never a function of data -- because data is always ambiguous. It is never possible just to infer the conceptual model of the data from the data alone. Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions. Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.
tacosbane 4y ago
can we build an AI to detect that the AI goalposts keep getting moved?
- girvo 4y ago
  A simple “return true;” should suffice, but to be honest that’s what makes the field fascinating to me as an outsider
kappi 4y ago
For industrial application, there are already mature systems based on CV. For majority of those applications, there is no need for deep learning or multilayer CNN. Shocked to see Andrew Ng talking like a marketing guy.

209 points by sbehere 4y ago

Pretty interesting. Mr. Ng claims that for some applications having a small set of quality data can be as good as using huge set of noisy data. I wonder if, assuming the data is of highest quality, with minimal noise, having more data will matter for training or not. And if it matters, on what degree?

frozenport 4y ago
This is at the heart of the ML training problem. In general you want to add more variants of data but not so much that the network doesn't get trained by them. Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) and prefer more of those. Now, why not simply add everything? Well in general it takes too long to train.
- pbowyer 4y ago
  > Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) How do you identify these images? It sounds like I'd need to build small models to see the variance but I'm hoping that there's a more scientific way?
kavalg 4y ago
It is relatively easy to turn small and accurate data to bigger and less accurate data with various forms of augmentation. The opposite is harder.

TOMDM 4y ago

Yeah that'd be great. I also want cars that run on salt water. I'm not saying that small data ai is equally impossible, but simply saying "we should make this better thing" isn't enough.

atulsnj 4y ago
Atleast someone's working on it.
Datenstrom 4y ago
> simply saying "we should make this better thing" isn't enough. Besides the references to his company which has customers and a product that already works on these principles the literature currently shows that this is very much possible if you dig into the correct niches. Besides the SOTA in few-shot and meta-learning it is possible to smartly choose the correct few samples for the network that yield the same results. It has also been my primary focus for the past 5 years and the core of the company I founded.
- riku_iki 4y ago
  > it is possible to smartly choose the correct few samples for the network that yield the same results. And then, someone is using pretrained 500B model, and fine-tuning your few examples, and getting new SOTA.
  - deleted 4y ago
    [deleted]
  - YeGoblynQueenne 4y ago
    They might get new SOTA because the metric is accuracy, but if the metric was accuracy weighted by sample efficiency, then SOTA would look a lot less impressive. Simplest way to weigh by sample efficiency: multiply accuracy by ratio of test set to training set sizes. Everyone's training/testing on 80/20 splits, so everybody's SOTA would go down by 3/4s.
sanxiyn 4y ago
It's more of "this direction seems higher ROI than that direction", in particular quality vs quantity of data. Already in 2018 SenseTime reported that for face recognition, clean dataset surpasses accuracy of 4x larger raw dataset. https://arxiv.org/abs/1807.11649 https://arxiv.org/abs/1807.11649

whatever1 4y ago

My understanding is that they are trying to automate the data preparation steps that seasoned ML practitioners are doing anyway today. The fact that he tries this in manufacturing makes the case stronger. In most manufacturing companies you do not have access to top ML talent. You have Greg who knows python and recently visualized some production metrics. If we could empower Greg with automated ML libraries that guide him in the data preparation steps in combination with precooked networks like autogluon, then manufacturing could become a huge beneficiary of the ML revolution.

overkalix 4y ago
Greg probably also knows SAS and AMPL, and has a good knowledge of ops research, which is within stone-tossing distance of whatever ML is pretending to be this week.
- whatever1 4y ago
  OR and ML have their own space in manufacturing. OR is perfect when you can describe explicitly what the decision space is and what the restrictions are. ML is great fit when you want to identify and use patterns. Quality control with machine vision is a good application for ML. NLP for PDF documents is a huge field for manufacturing as well. Companies have so much data in email attachments that they do not currently take advantage of.
  - overkalix 4y ago
    > OR is perfect when you can describe explicitly what the decision space is and what the restrictions are. As opposed to having to figure it out later from the outputs of a black box? > Quality control with machine vision is a good application for ML. I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc. > NLP for PDF documents is a huge field for manufacturing as well. NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes? I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.

itissid 4y ago

That is the problem with generalization and cop outs like these. It's no good to people in the field doing actual work where the devil is in the detail. Big data is fairly important to a lot of things, for example I was listening to Tesla's use of Deep net models where they mentioned that there were literally so many variations of Stop Signs that they needed to learn what was really in the "tail" of the distribution of Stop Sign types to construct reliable AI

vasco 4y ago
Interestingly, when you learn how to drive you need to see approximately one example and you're able to identify them all.
- corndoge 4y ago
  Is there some underlying point to this statement? It comes off as a passive dismissal of something but I'm not sure what. It might be helpful to directly state what you're trying to say so that other people can engage with it.
  - vasco 4y ago
    It doesn't seem like the other replies are having trouble engaging with it. Since we're giving each other advice, you should use your down votes instead of pontificating that other people's comments are "passive dismissals" when you don't like or understand them.
- teruakohatu 4y ago
  That is called transfer learning. You might only need to see one photo of a sign to identify it in real life (although arguably learner drivers take a while to notice signs) but that is only because you have been training on identifying generic objects since you left the womb. You brain already knows how to select the most important features of a sign. The shape, the size and the color. You have also learned how to understand the text on the sign. A new born baby does not have that ability. This is applied in ANN as well. Transfer learning is using a pre-trained neural network, which has already learned identifying objects, and then using it to train on identifying a new, usually smaller, set of objects using, usually, a lot less training data. That is what Andrew is talking about in the article.

xiphias2 4y ago

I can imagine that customizing AI solutions in an automated way is quite important, but writing that as the next wave is probably an overstatement. Of course few shot learning is important for models, but for example for Pathways it was already part of the evaluation.

deleted 4y ago

[deleted]

a-dub 4y ago

data quality is important. every ai project i've worked on has started with visualizing the data and thinking about it. it's easy to get complacent and focus on building big datasets. in practice, looking at the data often reveals issues sometimes in data quality and sometimes scope of what's in there (if you're missing key examples, it's simply not going to work). most ml is actually data engineering.

atbpaca 4y ago

Glad to see the term ML being used more often than AI in the comments as it looks like most "AI" models are trained for image classification. Having said that, the idea of "doing more with less" sounds interesting and I wonder what it means exactly. Does it mean taking a dataset of 50 images and to create 1000s of synthetic images from it?

spupe 4y ago
Yeah I was very interested about that point in particular. I think synthetic data is one of the ideas, but I got the sense that he also means helping to identify what makes a data set good, even if small. It looks like Andrew Ng is developing a platform for automatically detecting whether a dataset is suitable and, if not, what are the steps to improve it. A sort of automated ML consultant, allowing you to sell capabilities much cheaper than if you needed to consult an actual expert.

notsag-hn 4y ago

I was going to interview at LandingAI. I was asked before the interview to install a spyware browser extension to monitor my traffic to detect if I was cheating during the interview. I respectfully declined and didn't have that interview.

kevsim 4y ago
Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad.
- tablespoon 4y ago
  > Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad. Not necessarily, at least on the first point. Someone could be getting coached. A few years ago, a coworker of mine hired a contractor onto his team and was convinced the person who actually showed up was not the person who he interviewed (over the phone). He also thought the guy who did show up was getting a lot of help day-to-day from somewhere. The guy was a contractor, so it wasn't a huge problem because we could drop him quickly, but I would have never expected someone would do anything like that. However, it kind of makes sense as a scam: be a decent developer, get a stable of unhirable incompetents, and rotate them through companies while taking a cut of their salary.
  - mFixman 4y ago
    You cannot really prevent those kinds of cheats. Even if you use the most insidious spyware a coach can advice the interviewee from a different device. The only way to prevent those kind of scams is to put all employees in probation for the first months of work and fire them if they don't perform, like it's common in the UK.
    - 4y ago

aj7 4y ago

“I once built a face recognition system using 350 million images.” Did this make any of you a little queasy?

mdp2021 4y ago
Well noted! Explicitly: where does such database come from?
- mkl 4y ago
  Frames of video could make the number sky-high like that without involving enormous numbers of people.
  - mdp2021 4y ago
    Right. Such as, extras in movies. Because, "350 million public faces" does not seem "cognitively digestible", but it could just be «350 million images» including large variations of the same faces, consistently with the "frames of video" idea.

leobg 4y ago

What are some ML data annotation tools that guide you towards those data points where the model gets confused? I hear Prodigy does this. Any others?

jstx1 4y ago
What's the role of these tools? Can't a developer just write the code to get those data points? At a first glance it seems like the hassle of integrating such a product into an existing ML codebase/pipeline is larger than solving the problem by hand.
- leobg 4y ago
  What I mean is an annotation tool that interacts with the model itself in such a way that it will present to the user exactly those training examples next that will have the greatest impact in helping the model learn. So an annotation tool that provides a user interface for annotating data quickly (with keyboard shortcuts etc.). And looped into inference through the model to be trained, so you always get presented with the very training example that, out of the ones available, the model currently would be most unsure about.

a_square_peg 4y ago

I’ve been wondering about the limits of data-centric approach – there seems to be this implicit notion that more data equals better performing ML or AI. I think it would be interesting to imagine a point of diminishing return on additional data if we consider that our ability to perceive is probably largely based on two parts - sensory input and knowledge. Note that I’m making an explicit distinction here on the difference between data and knowledge. For instance, an English speaker and a non-English speaker may listen to someone speaking English and while the auditory signals received by both are the same, the meaning of the speech will only be perceived by the English speaker. When we’re learning a new language, it’s this ‘knowledge’ aspect that we’re enhancing in our brain, however that is encoded. This knowledge part is what allows us to see what’s not there but should be (e.g. the curious incident of the dog in the night) and when the data is inconsistent (e.g. all the nuclear close calls). I’m really not sure how this ‘knowledge’ part will be approached by the AI community but feel like we’re already close to having squeezed out as much as we can from just the data side of things. Somewhat related, we have a saying in Korean – ‘you see as much as you know’.

Longwelwind 4y ago
Can't you consider that knowledge is a function of previous data? In your example, the 2 individuals actually didn't receive the same amount of data because the English speakers received data previously that allowed him to build some kind of "knowledge" that allows him to solve specific related tasks (understanding a spoken sentence). This would be the equivalent of transfer learning where "knowledge" is a model trained on previous, more general, data.
- mjburgess 4y ago
  Nope, it's never a function of data -- because data is always ambiguous. It is never possible just to infer the conceptual model of the data from the data alone. Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions. Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.

tacosbane 4y ago

can we build an AI to detect that the AI goalposts keep getting moved?

girvo 4y ago
A simple “return true;” should suffice, but to be honest that’s what makes the field fascinating to me as an outsider

kappi 4y ago

For industrial application, there are already mature systems based on CV. For majority of those applications, there is no need for deep learning or multilayer CNN. Shocked to see Andrew Ng talking like a marketing guy.