ReLU are indeed non-linear, despite the confusing name. The nonlinearity (plus at least two layers) are required to solve nonlinear problems (like the famous X

ReLU are indeed non-linear, despite the confusing name. The nonlinearity (plus at least two layers) are required to solve nonlinear problems (like the famous XOR example).