I've been working on an application with such properties. Here are some patterns I have employed: Break it up into many completely independent components. In m

I've been working on an application with such properties. Here are some patterns I have employed: Break it up into many completely independent components. In my case, this is in the form of scripts which read from and write to queues. Each script reads from one queue and writes to another. A lot of redundancy. From one queue to the next, there are at least two pathways which the data can take. Lots of sanity checks. Wherever you are taking input, check that it has the expected format, shape, and content before processing it. More redundancy. Write two versions of the same script in two different languages and make the system run them side by side and compare the outputs. If the outputs differ, there is a problem, and you should switch to the script which produces the correct expected output (and alert the operator.) Avoid doing dangerous things. For example, querying the database using freeform strings is dangerous. So only query the database using sanity-checked identifiers which contain a predefined list of allowed characters which do not include quotes or anything else weird. Running scripts as a direct result of user request is dangerous, so serve only static HTML as much as possible. And so on.