We started with Serverless as well, but as soon as we started to do more complex stuff it showed its limitations and we ended up writing lots of CloudFormation

We started with Serverless as well, but as soon as we started to do more complex stuff it showed its limitations and we ended up writing lots of CloudFormation resources in serverless.yml. That was no benefit over pure CloudFormation anymore, especially as Serverless uses YAML, but relies on some notations from JSON-CloudFormation templates, I assume because it pre-dates the introduction of YAML for CloudFormation templates. In the end we switched over to CloudFormation with SAM (Serverless Application Model) and while that also still has some rough edges, we have been pretty happy since. So if you don't rely on the cross-provider features Serverless offers I suggest to take a look at CloudFormation + SAM instead.