As a professional developer interested in extreme reliability, I have studied this and similar stories. I have studied high profile software flops, project fail

As a professional developer interested in extreme reliability, I have studied this and similar stories. I have studied high profile software flops, project failures as well as NTSB reports for aircraft and train accidents, reports of buildings failing etc. and find all those reports very illuminating in the sense of a kind of mindset that is necessary to ensure reliability. I am not working on safety-critical systems but still, the failure of the type of backend systems I work on can bring huge damage to the company. You can come out with a set of rules that can really improve the reliability of your application. * Ensure the system always moves between known, valid states. Errors can still be valid states if you plan for them correctly. I like to compare this to logical induction: if you start with correct state and on every operation ensure that if it executes on correct state it must result in correct state, then you guarantee that the system is always in correct state. * Ensure incorrect operations are impossible to execute. As an example, if you want to guard against large data loss, you can choose to not expose ability to remove data. In my current project we took away everybody's write access to the database, we removed all functions that can remove the data from the database and exchanged them with functions that either mask data (setting flags so that application ignores it) or move to archive. If you need to run an operation on database, you need to submit a pull request and go through Code Review. There is additional review before merge where the team goes through checklist to verify the change meets certain standards. * Ensure anything the client can throw at the application does not have a chance to escalate to other transactions. All inputs are limited and validated mercilessly, all queries are constructed so that it is possible to predict resources necessary to execute the query. Every type of query has a limit on how many queries of that type can be executed in parallel or in a unit of time and how much results it can return. All functionality is constructed in such a way that the performance per transaction does not degrade when load increases (for example by amortizing costs and batching work). Then every functionality is load tested and based on testing results it is certified with maximum load that it can process. Everything over that limit is preemptively declined. * If an algorithm or a solution cannot be guaranteed to work correctly it cannot be used. For example, if library cannot be guaranteed to use at most X amount of memory for the operation, then it is not suitable. * Every single failure of the system is investigated. No system is without faults. The only way to build reliable system is to not ignore errors and always try to figure out underlying failure in the process that let the fault happen. Most projects deal with errors only after the error is urgent enough or only to fix the immediate cause of the error, rather than the failure in the process. If you are interested in very reliable software, when you have it fail for whatever reason you need to look at all stages of your process and identify all places that were supposed to prevent the problem but did not.