Thanks for the video. Yup. So far the server side of my Web site has servers of 6 types -- Web server, session state store (server), log server, SQL Server, a

Thanks for the video. Yup. So far the server side of my Web site has servers of 6 types -- Web server, session state store (server), log server, SQL Server, and two core servers for the real work, doing some applied math to get the users the good stuff that will make them happy. Soon there will also be an ad server for a total of 7. As in the sense of the video, that sounds like a non-deterministic hell to debug. Ah, in this case, I don't think the debugging is so difficult. Mostly the reason is that so far all the interactions are conceptually and as in the architecture quite simple. E.g., the code seems to run fine, and I never encountered any significant problems getting the code, first version and first production version, running fine. But I do intend to add to the code lots more writes to the log server and be able to turn on/off those writes while the system is live. Then I could write some code to look at the log file and for each user session check all the traffic and work. Yes, things could fail. I do have a lot of reasonable value checks in the code and, thus, could get some diagnostics then. Eventually I'll want to implement some fault tolerance: E.g., if a log server becomes unresponsive, then switch to another one -- there maybe several anyway due to load and desire for redundancy. Similarly for the session state store -- a user might lose their session, but that would get logged, but the Web server would then start to use a different session state server. So far my site is simple enough in purpose, work, architecture, etc. that some reasonably careful thinking, documentation, coding, and testing should continue to be okay. But, sure, if the site gets to 1000 users a second, I'll have to rethink the architecture in big ways. Then I'll try to keep using the "keep it simple, stupid" KISS principle and be willing to waste a factor of 2 to 10 or so in checking and being sure and being able to do the right things for some failure scenarios. Uh, my Web site is not for banking, exchanging crypto tokens, etc.!!! Thanks, especially for the video. Right, for non-deterministic systems, can be tough to duplicate a problem in order to analyze it! So, tough, in practice, try never to have to confront directly the non-determinism -- i.e., have enough in tracing in logs that have in effect converted the work to something deterministic, the randomness removed. So, first cut lessons, try to avoid non-deterministic designs; if have one, then try to keep it dirt simple and likely relatively isolated from everything else; and if can't do all that, then during development and testing, at various levels, have lots of scaffolding and for production put in lots of checks, redundancy, thought, system management, monitoring, reporting, and analysis infrastructure, etc. E.g., I did and published some high end work in anomaly detection for real time monitoring to detect problems never seen before, and I intend to use that work when appropriate. The intuitive idea is that if the system is not humming like it usually does and should, then I'll discover that right away and, then, be able to zoom down a tree of partitions of the systems to localize the problem -- hopefully in a usefully large fraction of such cases. My work is good because it is multi-dimensional, distribution-free, has adjustable, known in advance, exact control over false alarm rate, and in a useful sense is the most powerful such monitoring possible from any means of processing the data. No way should ever have to see that the system occasionally gets sick and just look at the system, in racks in 100,000 square feet of floor space, and start to guess. Instead should have a LOT of darned good additional data from the various over-designed internal checks, logs, redundancies, etc. Really, the system should come close to just making it obvious about where the problem is. No joke: In 100,000 square feet of racks of humming servers, want some darned good health and wellness monitoring that, at least in the design, architecture, development, testing, and first months of production operation, appears to be able to localize problems. Suggesting that such work is important; not suggesting that it's easy or will get it good enough right away. Am suggesting that should take the problem seriously and try. For some important systems, may spend several times as much computing checking and monitoring as doing the actual production work. But no way do I ever want to have to know there is something wrong and have to be forced just to look at 100,000 square feet of servers and start with Ethernet diagnostic hardware, reading files, etc. I hope never have to consider a main memory dump. My first development computer got flaky -- I became able just to smell flaky. It really stinks!