One could separate the training data into a validation and training set. Then you could try both the ReLU and softplus to see which performs better. I have no i

One could separate the training data into a validation and training set. Then you could try both the ReLU and softplus to see which performs better. I have no idea which type would---I just liked the idea of using a smooth activation function instead of a jagged one.