This idea exists (the broad field is called neural architecture search), although you have to parameterize it somehow to allow gradient descent to happen. Here

This idea exists (the broad field is called neural architecture search), although you have to parameterize it somehow to allow gradient descent to happen. Here are examples: https://arxiv.org/abs/2009.04759 https://arxiv.org/abs/2009.04759 https://arxiv.org/abs/1906.09529 https://arxiv.org/abs/1906.09529