3 ms·

Thats a bit far. Relu does check x>0 but thats just one non-linearity in the linear/non-linear sandwich that makes up universal function approximator theorem. I

• greenavocado 7mo ago
  Multiply-accumulate, then clamp negative values to zero. Every even-numbered variable is a weighted sum plus a bias (an affine transformation), and every odd-numbered variable is the ReLU gate (max(0, x)). Layer 2 feeds on the ReLU outputs of layer 1, and the final output is a plain linear combination of the last ReLU outputs // inputs: u, v // --- hidden layer 1 (3 neurons) --- let v0 = 0.616*u + 0.291*v - 0.135 let v1 = if 0 > v0 then 0 else v0 let v2 = -0.482*u + 0.735*v + 0.044 let v3 = if 0 > v2 then 0 else v2 let v4 = 0.261*u - 0.553*v + 0.310 let v5 = if 0 > v4 then 0 else v4 // --- hidden layer 2 (2 neurons) --- let v6 = 0.410*v1 - 0.378*v3 + 0.528*v5 + 0.091 let v7 = if 0 > v6 then 0 else v6 let v8 = -0.194*v1 + 0.617*v3 - 0.291*v5 - 0.058 let v9 = if 0 > v8 then 0 else v8 // --- output layer (binary classification) --- let v10 = 0.739*v7 - 0.415*v9 + 0.022 // sigmoid squashing v10 into the range (0, 1) let out = 1 / (1 + exp(-v10))
  • GeorgeTirebiter 7mo ago
    i let v0 = 0.616u + 0.291v - 0.135 let v1 = if 0 > v0 then 0 else v0 is there something 'less good' about: let v1 = if v0 < 0 then 0 else v0 Am I the only one who stutter-parses "0 > value" vs my counterexample? Is Yoda condition somehow better? Shouldn't we write: Let v1 = max 0 v0
• Vetch 7mo ago
  The relu/if-then-else is in fact centrally important as it enables computations with complex control flow (or more exactly, conditional signal flow or gating) schemes (particularly as you add more layers).