3 ms·

Here's an idea: ReLU basically = a seesaw + ropes that attach at a movable distance from the fulcrum. Adjusting a 'weight' = sliding the attachment. Nonlinearit

Here's an idea: ReLU basically = a seesaw + ropes that attach at a movable distance from the fulcrum. Adjusting a 'weight' = sliding the attachment. Nonlinearity = rope can pull but not push. I'm afraid getting the details to work in practice would probably be horrible. Not steampunk but sort of biopunk: years ago someone programmed a DNA strand displacement mechanism to implement a particular fixed neural network, as a proof of concept.

xg15 4y ago
That's a good start, but I'd say it's about as far as the people in the article got - i.e. if you already know the weights, you can do inference fully in mechanics. That could be quite useful for pretrained stuff, no question. But I imagine, to go full steampunk, you'd also have to find a way to train the network fully mechanically. I.e. implement something like gradient descent without using a digital computer...
- abecedarius 4y ago
  Yeah, for backprop you need an error signal (probably not a rope now because it's signed), and to multiply it by the activation (which suggests another seesaw for the multiply), and to bump the connection horizontally proportional to the product. Then fan out the product into the previous layer's error. I have no experience at mechanical design -- I got to thinking about this out of irritation at a pop-science explanation of neural networks in the New York Times magazine which explained nothing. Could you help people to a real understanding (a basic one) by letting them play with a tiny network in tangible form, where you can see the activations and feel the forces? I guess making it work with all the details and practicalities would probably end up with the basic parts too complicated, and too many of them for this purpose of learning, but it'd be cool to try.

3 ms·

Here's an idea: ReLU basically = a seesaw + ropes that attach at a movable distance from the fulcrum. Adjusting a 'weight' = sliding the attachment. Nonlinearit

by abecedarius 4y ago

xg15 4y ago
That's a good start, but I'd say it's about as far as the people in the article got - i.e. if you already know the weights, you can do inference fully in mechanics. That could be quite useful for pretrained stuff, no question. But I imagine, to go full steampunk, you'd also have to find a way to train the network fully mechanically. I.e. implement something like gradient descent without using a digital computer...
- abecedarius 4y ago
  Yeah, for backprop you need an error signal (probably not a rope now because it's signed), and to multiply it by the activation (which suggests another seesaw for the multiply), and to bump the connection horizontally proportional to the product. Then fan out the product into the previous layer's error. I have no experience at mechanical design -- I got to thinking about this out of irritation at a pop-science explanation of neural networks in the New York Times magazine which explained nothing. Could you help people to a real understanding (a basic one) by letting them play with a tiny network in tangible form, where you can see the activations and feel the forces? I guess making it work with all the details and practicalities would probably end up with the basic parts too complicated, and too many of them for this purpose of learning, but it'd be cool to try.