5 ms·

Strategy is, use expensive GPUs to train. When your model is ready to deploy at scale, take the matrices and deploy them in efficient and cheap ASICs. Nvidia

Strategy is, use expensive GPUs to train. When your model is ready to deploy at scale, take the matrices and deploy them in efficient and cheap ASICs. Nvidia wins if the product of the GPU is as easy to deploy at scale as possible for everyone, not just Google. A lot of people who don't have the scale are going to use GPU/CPU for the whole pipeline, and the people who have the scale weren't going to use GPUs in the long run, this just helps them get there faster and realize return on their GPUs.

joe_the_user 9y ago
Well, The broad paradigm of deep learning (and machine-learning generally) is a "train-test-deploy" cycle and this fits with it. The thing is that however successful this approach has been so far, it has rather clear limitations. Unlike a scientific discovery, what's being discovered isn't a universal rule but a heuristic between massive, real-world data and labels/qualities. As the world changes, the actual correlation changes but the deployed solution doesn't (and sure you can re-teach, re-test and so-forth but what if the required model changes, what if your experts leave, etc). So it seems like to make a deep-learning solution sustainable, you'd want a method of including learning in your deployed solution (how you'd do that may not yet be discovered but that doesn't mean it won't be discovered). But this approach seems to do the opposite. It bakes the basic learn-test-deploy process into silicone, making this approach more obligatory.
- amelius 9y ago
  > As the world changes, the actual correlation changes but the deployed solution doesn't Are you saying that self-driving cars might start crashing when pedestrians wear different fashion in the future, or when cars have different designs?
  - jacquesm 9y ago
    That's roughly the gist of it. If the validation set is a subset of the training set then you are making a whole pile of assumptions that need not necessarily be true. Since the network can't explain what those assumptions exactly are it may make conclusions that work empirically at training time and that pass validation but that have no solid theoretical underpinning as soon as those assumptions fail the systems will fail. See also: data leakage Continuous learning is hopefully going to address some aspects of this. https://deepmind.com/blog/enabling-continual-learning-in-neural-networks/ https://deepmind.com/blog/enabling-continual-learning-in-neu... And many more future tricks like it.
    - ethbro 9y ago
      Or essentially that deep learning is unable to discriminate between beliefs and facts. And we can't read the models well enough to tell which is which. So it would be like being handed a list of high level bridge building rules with no context, then being asked to build a bridge in a geological environment where one has never been built before. Will it work? Who knows! How would we say?
      - jacquesm 9y ago
        Well, it's early days. We already know it is possible to generate pathological inputs to elicit a false response from a network much like optical illusions can fool humans. For instance, in the context of adaptability of self driving cars: The retraining could theoretically be fixed using an OTA update that periodically pools new training data from in-service vehicles to further improve and generalize the models. You'd have to build that in from day one but presumably anybody busy with self driving cars would have this at the forefront of their thinking. Just so that when and if black clothing with white stripes becomes fashionable we don't end up with a large amount of roadkill because self driving cars interpret these as empty lanes.
  - ethbro 9y ago
    Whenever the sensed input changes. To use the car analogy: when additional or fewer sensors are used, when a different part number sensor is used, when driving laws are changed, when road signs or traffic lights are updated, when other cars begin behaving differently, etc.
  - RockyMcNuts 9y ago
    Other people adapt to self-driving cars. They walk in front of them expecting them to stop. They find and exploit artifacts in the way cars behave to make them do funny things. They take it beyond a tipping point, they create situations outside the training set. Maybe it crashes, or maybe it doesn't crash but gets stuck or doesn't go where it's supposed to.
- nightski 9y ago
  You act like the train-test-deploy cycle only happens once. But in reality wouldn't teams do this continuously? Training/improving models would happen regularly on the side. I'm not sure I'd call an "online" learning algorithm more reliable. They have plenty of tradeoffs.
  - joe_the_user 9y ago
    > "You act like the train-test-deploy cycle only happens once" My impression is currently the train-test-deploy cycle is very much not an automatic process but rather a tricky, difficult process which experts working and scratching their heads to get working. The phrase for getting a deep learning system working is often "graduate student descent". Sure, you could just blindly run the process again but there's a lot to know to get it right - why we don't just have "deep learning in a box" right now (though people are working it). So yeah, I am sure the process is re-done periodically but redoing may not be a slam dunk as is (though people are working on that too, people are working everything...). And I'm not advocating any particular approach, just observing current approaches.
- AndrewKemendo 9y ago
  You're ignoring completely Reinforcement Learning and the broader area of Unsupervised learning which is built exactly to counter the point of a "static" NN.
- RockyMcNuts 9y ago
  You can't do online optimization continuously but you can gather input and batch it and update the ASICs on a cycle. Like Google has guys with carts patching failed hardware... every week you reprogram a new set of ASICS and go around swapping them out.
  - RockyMcNuts 9y ago
    tbh I'm not sure you need to replace ASICs except when you change the architecture of the model...presumably an ASIC can load parameters at startup/runtime?
westbywest 9y ago
There is very little about design and manufacture of ASICs that is cheap, unless you're selling them by the millions. Two previous employers evaporated because they couldn't sell enough units to recoup VC investment in building semi-custom ASICs. The kicker is that each employer was already selling a previous generation of their product using more plug-and-play, less custom, ASIC libraries. The customized next-gen products were intended to be more performant and power efficient, which they were, but they still sunk both companies.

joe_the_user 9y ago

Well, The broad paradigm of deep learning (and machine-learning generally) is a "train-test-deploy" cycle and this fits with it. The thing is that however successful this approach has been so far, it has rather clear limitations. Unlike a scientific discovery, what's being discovered isn't a universal rule but a heuristic between massive, real-world data and labels/qualities. As the world changes, the actual correlation changes but the deployed solution doesn't (and sure you can re-teach, re-test and so-forth but what if the required model changes, what if your experts leave, etc). So it seems like to make a deep-learning solution sustainable, you'd want a method of including learning in your deployed solution (how you'd do that may not yet be discovered but that doesn't mean it won't be discovered). But this approach seems to do the opposite. It bakes the basic learn-test-deploy process into silicone, making this approach more obligatory.

amelius 9y ago
> As the world changes, the actual correlation changes but the deployed solution doesn't Are you saying that self-driving cars might start crashing when pedestrians wear different fashion in the future, or when cars have different designs?
- jacquesm 9y ago
  That's roughly the gist of it. If the validation set is a subset of the training set then you are making a whole pile of assumptions that need not necessarily be true. Since the network can't explain what those assumptions exactly are it may make conclusions that work empirically at training time and that pass validation but that have no solid theoretical underpinning as soon as those assumptions fail the systems will fail. See also: data leakage Continuous learning is hopefully going to address some aspects of this. https://deepmind.com/blog/enabling-continual-learning-in-neural-networks/ https://deepmind.com/blog/enabling-continual-learning-in-neu... And many more future tricks like it.
  - ethbro 9y ago
    Or essentially that deep learning is unable to discriminate between beliefs and facts. And we can't read the models well enough to tell which is which. So it would be like being handed a list of high level bridge building rules with no context, then being asked to build a bridge in a geological environment where one has never been built before. Will it work? Who knows! How would we say?
    - jacquesm 9y ago
      Well, it's early days. We already know it is possible to generate pathological inputs to elicit a false response from a network much like optical illusions can fool humans. For instance, in the context of adaptability of self driving cars: The retraining could theoretically be fixed using an OTA update that periodically pools new training data from in-service vehicles to further improve and generalize the models. You'd have to build that in from day one but presumably anybody busy with self driving cars would have this at the forefront of their thinking. Just so that when and if black clothing with white stripes becomes fashionable we don't end up with a large amount of roadkill because self driving cars interpret these as empty lanes.
- ethbro 9y ago
  Whenever the sensed input changes. To use the car analogy: when additional or fewer sensors are used, when a different part number sensor is used, when driving laws are changed, when road signs or traffic lights are updated, when other cars begin behaving differently, etc.
- RockyMcNuts 9y ago
  Other people adapt to self-driving cars. They walk in front of them expecting them to stop. They find and exploit artifacts in the way cars behave to make them do funny things. They take it beyond a tipping point, they create situations outside the training set. Maybe it crashes, or maybe it doesn't crash but gets stuck or doesn't go where it's supposed to.
nightski 9y ago
You act like the train-test-deploy cycle only happens once. But in reality wouldn't teams do this continuously? Training/improving models would happen regularly on the side. I'm not sure I'd call an "online" learning algorithm more reliable. They have plenty of tradeoffs.
- joe_the_user 9y ago
  > "You act like the train-test-deploy cycle only happens once" My impression is currently the train-test-deploy cycle is very much not an automatic process but rather a tricky, difficult process which experts working and scratching their heads to get working. The phrase for getting a deep learning system working is often "graduate student descent". Sure, you could just blindly run the process again but there's a lot to know to get it right - why we don't just have "deep learning in a box" right now (though people are working it). So yeah, I am sure the process is re-done periodically but redoing may not be a slam dunk as is (though people are working on that too, people are working everything...). And I'm not advocating any particular approach, just observing current approaches.
AndrewKemendo 9y ago
You're ignoring completely Reinforcement Learning and the broader area of Unsupervised learning which is built exactly to counter the point of a "static" NN.
RockyMcNuts 9y ago
You can't do online optimization continuously but you can gather input and batch it and update the ASICs on a cycle. Like Google has guys with carts patching failed hardware... every week you reprogram a new set of ASICS and go around swapping them out.
- RockyMcNuts 9y ago
  tbh I'm not sure you need to replace ASICs except when you change the architecture of the model...presumably an ASIC can load parameters at startup/runtime?

westbywest 9y ago

There is very little about design and manufacture of ASICs that is cheap, unless you're selling them by the millions. Two previous employers evaporated because they couldn't sell enough units to recoup VC investment in building semi-custom ASICs. The kicker is that each employer was already selling a previous generation of their product using more plug-and-play, less custom, ASIC libraries. The customized next-gen products were intended to be more performant and power efficient, which they were, but they still sunk both companies.