3 ms·

>As for contrastive divergence with Restricted Boltzmann Machines overcoming the vanishing gradient problem. That's true, but has anyone even tried doing this w

>As for contrastive divergence with Restricted Boltzmann Machines overcoming the vanishing gradient problem. That's true, but has anyone even tried doing this with 7-bit floating point and demonstrated it even works? I wasn't referring to RBMs here, although it does look like I misremembered the details. See slides 74-76 of this presentation for a brief sketch [0]. In these feed-forward MLPs, the information content from each neuron is capped at one bit (which is quite a bit less than seven). I seem to recall him saying something similar (and with more detail) in a Google Tech Talk a few months later [1], although I'm not sure. [0]: http://www.ipam.ucla.edu/publications/gss2012/gss2012_10743.pdf http://www.ipam.ucla.edu/publications/gss2012/gss2012_10743.... [1]: http://www.youtube.com/watch?v=DleXA5ADG78 http://www.youtube.com/watch?v=DleXA5ADG78

varelse 13y ago
Indeed, but I quote: "The pre-training uses stochastic binary units. After pre-training we cheat and use backpropagation by pretending that they are deterministic units that send the REAL-VALUED OUTPUTS of logistics." So without trying to sound like a pedant, the manner in which the error is introduced is effectively a hyper parameter, no? The skills involved in doing this correctly will land you the big bucks these days: http://techcrunch.com/2014/01/26/google-deepmind/ http://techcrunch.com/2014/01/26/google-deepmind/ My problem is that I think the programming experience of an entire hardware stack based on 7-bit floating point would be abysmal and that companies that tried to port their software stack to it would die horribly (to be fair they'd deserve it though). OTOH If they'd just stick to the limited domains where this sort of thing is applicable, my perception of it would flip 180 degrees - it's probably a pretty cool image preprocessor. What it's not is the replacement for traditional CPUs. But the people behind this seem to have utter contempt for the engineers tasked with programming their magical doohickey, or putting this in TLDR terms: 1. Scads of 7-bit floating point processors 2. ??? 3. PROFIT!!!

varelse 13y ago

Indeed, but I quote: "The pre-training uses stochastic binary units. After pre-training we cheat and use backpropagation by pretending that they are deterministic units that send the REAL-VALUED OUTPUTS of logistics." So without trying to sound like a pedant, the manner in which the error is introduced is effectively a hyper parameter, no? The skills involved in doing this correctly will land you the big bucks these days: http://techcrunch.com/2014/01/26/google-deepmind/ http://techcrunch.com/2014/01/26/google-deepmind/ My problem is that I think the programming experience of an entire hardware stack based on 7-bit floating point would be abysmal and that companies that tried to port their software stack to it would die horribly (to be fair they'd deserve it though). OTOH If they'd just stick to the limited domains where this sort of thing is applicable, my perception of it would flip 180 degrees - it's probably a pretty cool image preprocessor. What it's not is the replacement for traditional CPUs. But the people behind this seem to have utter contempt for the engineers tasked with programming their magical doohickey, or putting this in TLDR terms: 1. Scads of 7-bit floating point processors 2. ??? 3. PROFIT!!!