End to end tests also tend to be more fragile and require a more complex test environment, this means you might end up spending too much time fixing broken stuf

End to end tests also tend to be more fragile and require a more complex test environment, this means you might end up spending too much time fixing broken stuff (tests, environment) and getting a lot of false failures. On the other hand if you start early you can take testability into the design and make at least some of the above problems go away, for example by having a modular architecture that allows easy mocking. I do recommend to minimize the use of UI tests (Selenium for web for example) as they tend to be even more fragile, focus on the most obvious sunny day scenarios and never ignore a broken test as it get rotten over time