5 ms·

There are a couple points not generally made in discussions of data and the great AI race. Most data is crap. So the mountains of data that are supposedly an a

by vonnik 7y ago

There are a couple points not generally made in discussions of data and the great AI race. Most data is crap. So the mountains of data that are supposedly an advantage, in China or in the vaults of the large corporations, are not fit for purpose without a tremendous amount of data pre-processing, and even then... And that means the real chokepoint is data science talent, not data quantity. In other words, in many cases, the premises of this statement should be questioned. Secondly, a lot of research is focused on few-shot, one-shot or no-shot learning. That is, the AI industry will make this constraint increasingly obsolete. Thirdly, with synthetic data, it is only as good as the assumptions you made while creating it. But how did you reach those assumptions? By what argument should they be treated as a source of truth? If you are making those assumptions based on your wide experience with real-world data, well then, we run into the same scarcity, mediated by the mind of the person creating the synthetic data.

ggggtez 7y ago
Exactly this. Amazing, a picture of a white-mans arm holding a box of orange juice. But what if the person is a woman, or dark skinned, or has prosthetics, or is a child, or it's a bag/bottle instead of a box, or the lighting is different, or the camera is low resolution, or there is someone standing in the way... Anyone in this space is well aware that the benefit of big data isn't just the amount of data, but that's it's a real representative sample of the type of data you are actually going to need to work with. Big data solves the problem of people being bad at creating simulations. To suggest simulations as a solution to big data is kinda getting the relationship backwards.
- cbrun 7y ago
  Respectfully disagree. Again, I'm not suggesting SD will solve all problems. Big data is critical and will remain so. However, using a combination of SD and real data will make AI algorithms more robust than using big data alone. I do agree that the world is messy and it's hard to recreate the chaos and weirdness of the world. However, to think that at some point we won't be able to completely mimic the real world and all the variations out there is strange. Re: your example, its actually pretty easy to span millions of humans based on ethnicity, age, body mass, etc. It's just a matter of time until this problem gets solved.
  - notahacker 7y ago
    > However, to think that at some point we won't be able to completely mimic the real world and all the variations out there is strange. Why? I think it's strange to believe the opposite: that something as simple as a computer program designed by something as simple as the human mind definitely should be able to adequately simulate the complexity of the real world.
    - cbrun 7y ago
      Really? I think if we were to bring back our close ancestors (4-5 generations away) they'd look at our world like we see Harry Potter's: pure magic. I mean flying metal birds, fire that instantly turns on and off, machine that move around like ghosts, small boxes that can talk back, musicians on demand in a box? I think you get my point, you're selling humanity short. There's no limit to human ingenuity and there's a reason Elon Musk still doubts we're living in a simulation.
      - notahacker 7y ago
        There's a massive difference between 'develop technologies which are indistinguishable from magic to people who don't know how they work' and 'completely simulate the informational complexity of the world using the human minds and computers with comparatively limited information processing capabilities.' I'm not sure the second is even a logical possibility, never mind a practical one.
        hackernews65 7y ago
        Dude, if you're referencing information theory you clearly don't understand how to create tech. Obviously they're not trying to simulate the universe accurately lol. Simulating the real world for specific use-cases to solve computer vision for practical applications is super interesting. If they have customers, makes sense.
        cbrun 7y ago
        That's assuming human minds and computers won't exponentially increase their processing capabilities. Moore's law disproves the later and I hope Neuralink or some other crazy tech company will disprove the former. That's pretty much the hope of all transhumanists.
cbrun 7y ago
All valid points - thanks for the comment. 1. Totally agree with the point about data science talent being one of the bottlenecks. And in that area, the big tech companies already have the best & brightest workforce. Most data is crap is a bit of a stretch, but agree that a lot of pre-processing is required. But more and more platforms and/or services company are filling this gap. 2. 100%. This is a short term problem. 3. Agree to some extent. You can span latent spaces on top of an initial batch to overcome scarcity and use GANs to scale up. Data augmentation is not perfect but solves a lot of problems in production. Also specifically as it relates to computer vision, it's "easier" to know what the real world looks like and try to replicate it. It's hard in practice but the assumptions are mostly agreed upon.
gilbaz 7y ago
Good points! Most data is def crap lol. I don't see few shot, one-shot or no-shot getting anywhere close to standard supervised learning for anything practical. It really doesn't make sense in production settings at all. You have a function that you want to learn, let's say mapping between an RGB image to a segmentation map. For most applications you're never really in a situation where a production product is dealing with visual scenes/objects it has never seen before. In a factory, in smart stores, in cars, AR scenarios like I just don't see it happening. And then if this case is removed, I'm thinking, ok so when can I get good enough results from a tiny dataset? Machine Learning isn't magic, you're trying to learn a function with 100 million parameters using a dataset, I just don't see the math working out. More data provides better results, it inserts more information to create a more relevant mapping function from input to output. Third point is great! As long as the models are somehow based on real-world scans I think a lot of good can come from this. The funny thing is, there is so much bias in networks trained today precisely because the data captured is usually small and captured from a specific area/population/setting. If you had a great synthetic data generation engine you could at least generate equal representation of gender, age groups, ethnicities, ... etc. Overall great points!

by vonnik 7y ago

ggggtez 7y ago

Exactly this. Amazing, a picture of a white-mans arm holding a box of orange juice. But what if the person is a woman, or dark skinned, or has prosthetics, or is a child, or it's a bag/bottle instead of a box, or the lighting is different, or the camera is low resolution, or there is someone standing in the way... Anyone in this space is well aware that the benefit of big data isn't just the amount of data, but that's it's a real representative sample of the type of data you are actually going to need to work with. Big data solves the problem of people being bad at creating simulations. To suggest simulations as a solution to big data is kinda getting the relationship backwards.

cbrun 7y ago
Respectfully disagree. Again, I'm not suggesting SD will solve all problems. Big data is critical and will remain so. However, using a combination of SD and real data will make AI algorithms more robust than using big data alone. I do agree that the world is messy and it's hard to recreate the chaos and weirdness of the world. However, to think that at some point we won't be able to completely mimic the real world and all the variations out there is strange. Re: your example, its actually pretty easy to span millions of humans based on ethnicity, age, body mass, etc. It's just a matter of time until this problem gets solved.
- notahacker 7y ago
  > However, to think that at some point we won't be able to completely mimic the real world and all the variations out there is strange. Why? I think it's strange to believe the opposite: that something as simple as a computer program designed by something as simple as the human mind definitely should be able to adequately simulate the complexity of the real world.
  - cbrun 7y ago
    Really? I think if we were to bring back our close ancestors (4-5 generations away) they'd look at our world like we see Harry Potter's: pure magic. I mean flying metal birds, fire that instantly turns on and off, machine that move around like ghosts, small boxes that can talk back, musicians on demand in a box? I think you get my point, you're selling humanity short. There's no limit to human ingenuity and there's a reason Elon Musk still doubts we're living in a simulation.
    - notahacker 7y ago
      There's a massive difference between 'develop technologies which are indistinguishable from magic to people who don't know how they work' and 'completely simulate the informational complexity of the world using the human minds and computers with comparatively limited information processing capabilities.' I'm not sure the second is even a logical possibility, never mind a practical one.
      - hackernews65 7y ago
        Dude, if you're referencing information theory you clearly don't understand how to create tech. Obviously they're not trying to simulate the universe accurately lol. Simulating the real world for specific use-cases to solve computer vision for practical applications is super interesting. If they have customers, makes sense.
      - cbrun 7y ago
        That's assuming human minds and computers won't exponentially increase their processing capabilities. Moore's law disproves the later and I hope Neuralink or some other crazy tech company will disprove the former. That's pretty much the hope of all transhumanists.

cbrun 7y ago

All valid points - thanks for the comment. 1. Totally agree with the point about data science talent being one of the bottlenecks. And in that area, the big tech companies already have the best & brightest workforce. Most data is crap is a bit of a stretch, but agree that a lot of pre-processing is required. But more and more platforms and/or services company are filling this gap. 2. 100%. This is a short term problem. 3. Agree to some extent. You can span latent spaces on top of an initial batch to overcome scarcity and use GANs to scale up. Data augmentation is not perfect but solves a lot of problems in production. Also specifically as it relates to computer vision, it's "easier" to know what the real world looks like and try to replicate it. It's hard in practice but the assumptions are mostly agreed upon.

gilbaz 7y ago

Good points! Most data is def crap lol. I don't see few shot, one-shot or no-shot getting anywhere close to standard supervised learning for anything practical. It really doesn't make sense in production settings at all. You have a function that you want to learn, let's say mapping between an RGB image to a segmentation map. For most applications you're never really in a situation where a production product is dealing with visual scenes/objects it has never seen before. In a factory, in smart stores, in cars, AR scenarios like I just don't see it happening. And then if this case is removed, I'm thinking, ok so when can I get good enough results from a tiny dataset? Machine Learning isn't magic, you're trying to learn a function with 100 million parameters using a dataset, I just don't see the math working out. More data provides better results, it inserts more information to create a more relevant mapping function from input to output. Third point is great! As long as the models are somehow based on real-world scans I think a lot of good can come from this. The funny thing is, there is so much bias in networks trained today precisely because the data captured is usually small and captured from a specific area/population/setting. If you had a great synthetic data generation engine you could at least generate equal representation of gender, age groups, ethnicities, ... etc. Overall great points!