5 ms·
Training of Physical Neural Networks
- deleted 2y ago[deleted]
- UncleOxidant 2y agoSo it sounds like these PNNs are essentially analog implementations of neural nets? Seems like an odd choice of naming to call them 'physical'.
- tomxor 2y agoANN is taken.
- TheLoafOfBread 2y agoI mean LoRA was taken too before LoRA became a thing
- tomxor 2y agoI don't mean globally, LoRA are at least in different domains. Artificial Neural Networks and Physical Neural Networks are both machine learning, discussion referring to both is highly probable, and the former far more established so it calling it an Analog Neural Network would never last long.
- TeMPOraL 2y agoYou can't call analog NN an AnalNN either, as then the "brain-gut axis" people will throw a fit.
- agarwaen163 2y agoWell, this is particularly frustrating because PNN is already taken as well, by the (imo) much more novel idea of Physics Informed Neural Networks (aka PINNs or PNNs). Why this isn't called Hardware Neural Nets is beyond me.
- pessimizer 2y agoMakes sense as opposed to "abstract." With the constant encoding and decoding that has to be done when things are going in an out of processors and storage (or sensors), digital processes are always in some sense simulations.
- ksd482 2y agoPNNs resemble neural networks, however at least part of the system is analog rather than digital, meaning that part or all the input/output data is encoded continuously in a physical parameter, and the weights can also be physical, with the ultimate goal of surpassing digital hardware in performance or efficiency. I am trying to understand what format does a node take in PNNs. Is it a transistor? Or is it more complex than that? Or, is it a combination of a few things such as analog signal and some other sensors which work together to form a single node that looks like the one we are all familiar with? Can anyone please help me understand what exactly is "physical" about PNNs?
- deleted 2y ago[deleted]
- sigmoid10 2y agoIt's just a general idea to implement the computation part of neurons directly in hardware instead of software. For example by calculating sums or products using voltages in circuits, i.e. analog computing. The actual implementation is up to the designer, who in turn will try to mimic a certain architecture.
- eightysixfour 2y agoHere you go: https://arstechnica.com/science/2018/07/neural-network-implemented-with-light-instead-of-electrons/ https://arstechnica.com/science/2018/07/neural-network-imple...
- deleted 2y ago[deleted]
- Shawnecy 2y agoMy knowledge in this area is incredibly limited, but I figured the paper would mention NanoWire Networks (NWNs) as an emerging physical neural network[0]. Last year, researchers from the University of Sydney and UCLA used NWNs to demonstrate online learning of handwritten digits with an accuracy of 93%. [0] = https://www.nature.com/articles/s41467-023-42470-5 https://www.nature.com/articles/s41467-023-42470-5
- programjames 2y agoThat doesn't implement a trainable network on hardware, it's just creating a "reservoir" of associations between the inputs.
- orbifold 2y agoClassifying MNIST digits with 93% accuracy can also be accomplished using a linear classifier. So it isn't clear to me what the advantage would be.
- tomxor 2y agoLast time I read about this the main practical difficulty was model transferability. The very thing that makes it so powerful and efficient is also the thing that make it uncopiable, because sensitivity to tiny physical differences in the devices inevitably gets encoded into the model during training. It seems intuitive this is an unavoidable, fundamental problem. Maybe that scares away big tech, but I quite like the idea of having invaluable, non-transferable, irreplaceable little devices. Not so easily deprecated by technological advances, flying in the face of consumerism, getting better with age, making people want to hold onto things.
- bongodongobob 2y agoReminds me of the evolutionary FPGA experiment that was dependent on magnetic flux or something. The same program wouldn't work on a different FPGA.
- cyberax 2y agoHere's the paper about it: https://www.researchgate.net/publication/2737441_An_Evolved_Circuit_Intrinsic_in_Silicon_Entwined_With_Physics https://www.researchgate.net/publication/2737441_An_Evolved_... And a more approachable article: https://www.damninteresting.com/on-the-origin-of-circuits/ https://www.damninteresting.com/on-the-origin-of-circuits/
- rusticpenn 2y agoWhat thy did was overfitting. We later found other ways of getting around the issue.
- actionfromafar 2y agoWould be interesting to hook up many FPGAs of the same model and train all of the at once. Programs with differing outputs on different individuals could be discarded. The program may still not transfer to another batch of FPGAs but at least you have a better chance of the working. Another idea is to just train a whole bunch of them individually, like putting your chips in school. :-D
- 2y ago
- craigmart 2y agoSchools?
- programjames 2y ago> These methods are typically slow because the number of gradient updates scales linearly with the number of learnable parameters in the network, posing a significant challenge for scaling up. This is a pretty big problem, though if you use information-bottleneck training you can train each layer simultaneously.
- deleted 2y ago[deleted]