3 ms·
> And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone
by marginalia_nu 5y ago
> And yes you can write logs that nobody will ever be able to read, but what's the point of that? When your manufacturing plant is a smoldering rubble, someone is going to want to launch an expedition for those hard drives and try and figure out what caused millions (or potentially billions) of dollars in damages, so if you just wrote everything to standard output and washed your hands of the rest, you might have to find another job anyway.
Reducing the number of points of failure is almost always a good idea. The network being down is one problem, not being able to ever know what your applications did when the network was down is another problem added to the first problem.
- marcosdumay 5y ago> Reducing the number of points of failure is almost always a good idea. Reducing the single points of failure is usually a good idea. You can easily reduce the number of points of failure by reducing redundancy and removing sanity checks from your process. Doing that does normally not improve your reliability. Sending the logs to your SAN for it to write in a couple of disks arrays is much more reliable than writing to stderr (stdout is not a reasonable place to log failures) and redirecting it to your local disk.
- marginalia_nu 5y agoReducing points of failure in general is a good idea too. Single points of failure are especially bad, but most major errors are the confluence of multiple things going wrong that while alone would be bad, but together make for an absolute disaster.
- marcosdumay 5y agoReducing points of failure in general is an iffy idea that can be good or bad depending on how you do it. A lot of complexity is there for good reason, a lot for bad reason, and some for no reason at all. It takes a very nuanced and specific work to determine what is what, and talking in generalities about it is a very bad idea on itself.
- geocar 5y ago> Reducing the number of points of failure is almost always a good idea. I think it is good to think about what happens when things fail: Things fail all the time and most of those failures are benign -- and even those that aren't can often be remedied by turning it off and on, but some of them aren't, so what happens then? If you run out of disk space, writing a log line to the disk complaining that you're out of disk space is no help, but what else can you do? You could send a network alert, or print something to hardcopy (if there's a printer nearby), or ring a bell if you have one attached -- all of these things are also potential points of failure, but they can also help prevent a meltdown. And if that sounds surprising, you might also be shocked the number of problems that can be solved with: <img src="cdn1/img.jpg" onerror="(function(t){t.onerror=function(){};t.src='cdn2/img.jpg'})(this)"> It's for this reason (I think) that people usually qualify your statement as reducing the number of "single" points of failure, instead of the way you put it, because having two single-points of failure isn't exactly like having two weak links in a chain, and in many cases, it's nothing like it at all. In any event, there's another advantage to thinking about the failure-process instead of trying to merely avoid failures: The biggest reason reducing points of failure is a bad idea is when it is more expensive to do so than to simply tolerate the failure: One thing that dominates the attention of startups looking to improve their availability beyond 99.999% uptime, is "removing the failure points" to get higher uptime, but if Amazon is down, what are Amazon's customers going to do? Are they going to buy from someplace else? Sometimes it is simply that more-established brands have the benefit of ignoring some problems, but you can be sure a large company like Amazon has someone who knows exactly how much more they can spend per second of uptime they may desire, and I think it is worth it for a small company to get good at that exercise as well, because sometimes the juice just isn't worth the squeeze. As a specific example: My ad servers (HTTP event collectors) are critical, and I want 100% uptime for them because missing a click means missing money, but the reporting dashboard (so the publishers know how much money they made, and the advertiser knows how much they owe me) doesn't need that, so as long as the user got redirected to the landing page correctly, and I can generate line charts and excel spreadsheets once a day, nobody cares that the reporting UI might be down for a few minutes a day-- they will simply press refresh, not knowing if it is their internet or mine, and combined with aggressive javascript-side caching and automatic retries, I am not quite sure any of my customers even notice my downtime. If instead, I focused on reducing the things that could go wrong (because as you say, it is almost always a good idea), I might spend time trying to improve the availability of my database simply because that's the thing that fails instead of whacking a setTimeout into the JavaScript since the impact is that the user would just retry on failure. That setTimeout saved me the cost of a whole server (and power and heat and electricity) and in the early days of my ad server, that would've been a lot of beer.