4 ms·

Why can't you model each neuron as an individual object with a unique UID in code? I am fortunate enough to have a friend that's fairly accomplished in ML and

Why can't you model each neuron as an individual object with a unique UID in code? I am fortunate enough to have a friend that's fairly accomplished in ML and a really nice guy. He spent a video chat + code session with me where he helped me build an NN from scratch in Typescript, we replicated the network in the paper here: https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/ https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat... Here's the Codesandbox link, if you notice the console output it (approximately) matches the figures from the paper above: https://codesandbox.io/s/node-typescript-vqszi?file=/example/app.ts https://codesandbox.io/s/node-typescript-vqszi?file=/example... I don't really understand much about ML or neural networks, but from a code perspective, if you have these data structures: class Neuron { id: number | void = neuronUidGen.next().value; incoming: Connection[]; outgoing: Connection[]; rate: number; bias: number; output: number; error: number; } class Connection { id: number | void = connectionUidGen.next().value; from: Neuron; to: Neuron; weight: number; } Why couldn't you manipulate these however you see fit? There's a Layer implementation in there that's commented out because it doesn't function right that could roughly accomplish something like this theoretically, right?

cs702 6y ago
You could. But that's not the hard part. The hard part, for the foreseeable future, is that if you write your code like that, you will find it very difficult to get decent performance out of existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape. (Oversimplifying a bit.)
- goatlover 6y ago
  Could there be a transpiling step that converts the objects to arrays, targeting PyTorch or TensorFlow?
- gavinray 6y ago
  But the Neuron and Connections are just metadata objects like pointers that hold arrays, I don't understand. Surely in PyTorch/TF, the ND-Arrays of NN's must also be associated to an object reference? > existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape. const dot = (inputs: number[], weights: number[]) => { let total = 0; const length = Math.min(inputs.length, weights.length); for (let i = 0; i < length; i++) { total += inputs[i] * weights[i]; } return total; }; activate(input?: number): number { if (!this.incoming.length) { this.output = input; } else { const inputs = this.incoming.map(connection => connection.from.output) const weights = this.incoming.map(connection => connection.weight); this.output = squash(dot(inputs, weights) + this.bias); } return this.output; } This is on principle the same concept right?
  - cs702 6y ago
    Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=true https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=... (I oversimplified things in my comment above).

cs702 6y ago

You could. But that's not the hard part. The hard part, for the foreseeable future, is that if you write your code like that, you will find it very difficult to get decent performance out of existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape. (Oversimplifying a bit.)

goatlover 6y ago
Could there be a transpiling step that converts the objects to arrays, targeting PyTorch or TensorFlow?
gavinray 6y ago
But the Neuron and Connections are just metadata objects like pointers that hold arrays, I don't understand. Surely in PyTorch/TF, the ND-Arrays of NN's must also be associated to an object reference? > existing frameworks (PyTorch/TensorFlow) and accelerated hardware (GPUs/TPUs) which operate on neurons arranged as arrays of the same shape. const dot = (inputs: number[], weights: number[]) => { let total = 0; const length = Math.min(inputs.length, weights.length); for (let i = 0; i < length; i++) { total += inputs[i] * weights[i]; } return total; }; activate(input?: number): number { if (!this.incoming.length) { this.output = input; } else { const inputs = this.incoming.map(connection => connection.from.output) const weights = this.incoming.map(connection => connection.weight); this.output = squash(dot(inputs, weights) + this.bias); } return this.output; } This is on principle the same concept right?
- cs702 6y ago
  Unfortunately, the problem is more complex than that: https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=true https://dl.acm.org/doi/pdf/10.1145/3317550.3321441?download=... (I oversimplified things in my comment above).