3 ms·
Given these chips are purely meant for machine learning the issue that "your wafer computer was always going to be sub-optimal at something" is less of an issue
by Smerity 7y ago
Given these chips are purely meant for machine learning the issue that "your wafer computer was always going to be sub-optimal at something" is less of an issue now than in traditional scientific programming setups, especially of the Trillogy / Transmeta days.
You have silicon in place to deal with physical defects at the hardware level.
You have backprop / machine learning in place to deal with physical deficiencies at the software level.
The programmer operates mostly at the objective / machine learning model level and can tweak the task setup as needed softly influenced by both the hardware architecture (explicit) and potential deficiencies (hopefully implicit).
The most extreme examples I've seen in papers or my own research: parts of a model left accidentally uninitialized (i.e. the weights were random) not impacting performance enough to be seen as an obvious bug (oops), tuning the size and specifics of a machine learning model to optimize hardware performance (i.e. the use of the adaptive softmax[1] and other "trade X for Y for better hardware efficiency and worry about task performance separately"), and even device placement optimization that outperforms human guided approaches[2].
Whilst the last was proven across equipment at a Google datacenter with various intermixed devices (CPUs, GPUs, devices separated by more or less latency, models requiring more or less data transferred across various bandwidth channels, ...) it's immediately obvious how this could extend to optimizing the performance of a given model on a sub-optimal wafer (or large variety of sub-optimal wafers) without human input.
Whilst reinforcement learning traditionally requires many samples that's perfectly fine when you're running it in an environment with a likely known distribution and can perform many millions of experiments per second testing various approaches.
For me working around wafer level hardware deficiencies with machine learning makes as much or more sense as MapReduce did for working around COTS hardware deficiences. Stop worrying about absolute quality, which is nearly unattainable anyway, and worry about the likely environment and task you're throwing against it.
[1]: https://arxiv.org/abs/1609.04309 https://arxiv.org/abs/1609.04309
[2]: https://arxiv.org/abs/1706.04972 https://arxiv.org/abs/1706.04972