I worked on a team that built an "ephemeral" system. Part of that was to ensure that no state was logged. They were so proud of it being built that way. Proble

I worked on a team that built an "ephemeral" system. Part of that was to ensure that no state was logged. They were so proud of it being built that way. Problem was, the system failed constantly and we had no way of knowing what the issue was without ssh'ing directly into a machine and retrying the steps by hand. I get what they were going for, but at the end of the day, their opinions on how ideal systems "should be" ended up destroying their project and ignored fundamentals. Example: one of the main deployment hosts was constantly crashing from disk space issues for over a year. Turns out when they provisioned the host with 4GB of storage and 64GB of RAM. They never troubleshot the host and just restarted its services whenever it crashed!