24 ms·
It sounds to me what some people would like is for a magical box they can throw their infrastructure into that will automatically shit test all the things that
by bob1029 6y ago
It sounds to me what some people would like is for a magical box they can throw their infrastructure into that will automatically shit test all the things that could potentially go wrong for them. This is poor engineering. Arbitrary, contrived error conditions do not constitute a rational test fixture. If you are not already aware of where failures might arise in your application and how to explicitly probe those areas, you are gambling at best. Not all errors are going to generate stack traces, and not all errors are going to be detectable by your users. What you would consider an error condition for one application may be a completely acceptable outcome for another.
This is the reliability engineering equivalent of building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it.
- cogman10 6y agoI disagree. Not handling failures correctly is a time honored tradition in programming. It is so easy to miss. For example, how often have you seen a malloc check for `ENOMEM`? Even though that is something that could be semi common. Even though that's definitely something you might be able to handle. Instead, most code will simply blow chunks when that sort of condition happens. Is the person that wrote it "wrong"? That's debatable. Some languages like Go make it even trickier to detect that someone forgot to handle an error condition. Nothing obvious in the code review (other than knowledge of the API in question) would get someone senior to catch those sorts of issues. So the question is, HOW do you catch those problems? The answer seems obvious to me, you simulate problems in integration tests. What happens when Service X simply disappears. What happens when a server restarts mid communication? Is everything handled or does this cause the apps to go into an non-recoverable mode? This are all great infrastructure tests that can catch a lot of edge case problems that may have been missed in code reviews. Even better, that sort of infrastructure testing can be generalized and apply to many applications. Making rare events common in an environment makes it a lot easier to catch hard to notice bugs that everyone writes. It's basically just Fuzz testing but for infrastructure. Fuzz testing has been shown to have a ton of value, infrastructure fuzzing seems like a natural valuable extension of that. Especially when high reliability and low maintenance is something everyone should want.
- im3w1l 6y agoAs a community we have basically decided against handling of out of memory; that it's better to crash, and design the program so that such a crash will not cause corrupt state. Consider over-commit. It means that the malloc call will succeed even if there isn't available memory. Instead the system just hopes you wont make use of the memory you asked for. And if you do make use of it, well then the OS might kill you. Just like that. No opportunity for error handling.
- eru 6y agoReliably crashing _is_ a way to handle out of memory errors. When you don't handle it, you get something closer to undefined behaviour instead.
- perl4ever 6y ago>For example, how often have you seen a malloc check for `ENOMEM`? I've never concerned myself with that, but it's apparently a lot more complicated than just doing that check. See: https://news.ycombinator.com/item?id=20143277 https://news.ycombinator.com/item?id=20143277 https://scvalex.net/posts/6/ https://scvalex.net/posts/6/ Some of the thread is hilarious, for instance: "malloc is not allowed to return a non-null pointer to a memory block that cannot be written to. Linux does it anyway, and in doing so blatantly violates the standard. ... That is my point, it doesn't fail. Either the kernel finds out a way to map the memory and it succeeds, or it kill the program and the instruction never runs. Code that doesn't run can't violate the standard."
- riskneutral 6y ago> building a data warehouse when you don't know what sorts of reports you want to run or how the data will generally be used after you collect it. Hi Bob, I can't tell you what reports I want or what we'll do with the data until you've first collected the data for analysis. Thanks!
- bob1029 6y agoWhy are you vacuuming up data if you don't even know what sort of outcomes you want to look for? How do you know you are even pulling from the right sources? When you do finally figure out why you are in business, you may find that all the data you gathered is worthless. Sometimes it's really innocuous stuff like storing datetime values without enough resolution or accuracy. Only when you know how you plan to use the data can you ensure you are gathering it properly. Teaching developers to collect all the data they can in hopes of being able to write arbitrary reports down the line is reckless behavior. Hyperscalers love pushing this ideology with inane bullshit like "Datalakes", so you can see where a conflict of interest may arise in the industry.
- 9nGQluzmnq3M 6y agoEver worked with an enterprise company? I've worked on a project (in a peripheral role, fortunately) where the entirety of the CIO's mandate was to "use AI", full stop. So the team was tasked with putting together a data warehouse that would collect random logs and app metrics, and then sprinkle AI fairy dust of their choice onto it to generate Insights(tm). This kind of thing happens all. the time. in large companies where everybody involved wants either money or bullet points for their resume, and the actual outcome is nearly irrelevant.
- majewsky 6y agoWhen I implement a new service, I have to make an executive decision about which metrics to collect (operation counts, durations, storage usage, etc.). Before the first incident, I cannot know with 100% certainty which metrics are useful. Only after the incident can I look at the metrics and see which ones did weird things before and during the incident, and that helps drive my decisions on what to alert on. I don't see how I could do it differently. I can only truly know which metrics are useful from experience with past incidents, but I can only gain this experience when the metrics were already collected during the past incident.
- irjustin 6y agoOkay I'll bite - I have problems with this line of thinking. You're right that you'll never be able to cover 100% of all cases, but using that logic, your specs will never test 100% of scenarios, so you shouldn't write specs. I think the problem is you assumed it was "a magical box that people can rely to cover 100% of test cases of". That's a poor leap. It's clearly not. It's a good system to test a set of network failures at varying degrees. Just like any engineered system, it needs to be documented what it can and cannot do. I also eagerly await my do it all, 100% magic box.