5 ms·
/* fixed nullreferenceexception based on black box crash report */
by mikewilliams 11y ago
/* fixed nullreferenceexception based on black box crash report */
- cordite 11y agovoid segfault_sigaction(int signal, siginfo_t *si, void *arg) { //Pretend it never happened return; }
- atemerev 11y agoOf course, the only right way is panic() and wait for supervisor to restart the process / VM. Idiomatic Erlang doesn't differentiate between "system" / "environment" errors and local bugs. If it has failed — restart it!
- geofft 11y agoI'll bite. What about Erlang makes it so that a restarted process doesn't run into the same bug when it gets to the same point, and panic again in an infinite loop? The only way I can imagine this working is if Erlang is so buggy and nondeterministic that it inserts crashes sometimes but not all of the time. But that's obviously absurd.
- cordite 11y agoWondered this too, it naively only makes sense for that case with neutrinos screwing your RAM over.
- liveoneggs 11y agosee section 3.4 here: http://erlang.org/documentation/doc-4.9.1/doc/design_principles/sup_princ.html http://erlang.org/documentation/doc-4.9.1/doc/design_princip... "3.4 The Restart Frequency Limit Mechanism"
- geofft 11y agoWell, okay, so your process crashes, you restart it, it crashes a few more times, then you kill it. What's the advantage there? How does this increase availability, beyond killing it the first time it crashes? It seems actively worse to allow users to retry requests that are doomed to failure than to put up a fail-whale or similar while the ops team is being paged.
- atemerev 11y agoBecause most production bugs are infrequent (otherwise they would be noticed by testing). They have to be logged and fixed, but not allowed to move the system into inconsistent state. Restart first, fix later.
- geofft 11y agoAre they? The bug discussed in this comment was extremely deterministic. There's a difference between infrequent in the sense that, across lots of users and lots of requests it happens rarely, and infrequent in the sense that, for one particular use, it only triggers sometimes. Also, the bug discussed in this article wasn't causing crashes. What would you propose be crashed and restarted in this case?
- toast0 11y agoIf it's some weird race condition crash, restarting (hopefully?) puts you in a known good state and you're unlikely to hit it again. If it quickly repeats, you've isolated the failure to happening within a narrow scope. This part isn't really Erlang magic, apache in pre-fork mode has a lot of the same properties. There may be some magic in supervision strategies, but I think the real magic is the amount of code you get to leave out by accepting the possibility of crashes and having concise ways to bail out on error cases. For example, to do an mnesia write and continue if successful and crash if not, you can write ok = mnesia:write(Record) Similarly, when you're writing a case statement (like a switch/case in C), if you expect only certain cases, you can leave out a default case, and just crash if you get weird input. I also find the catch Expression way of dealing with possible exceptions is often nicer than try/catch. It returns the exception so you can do something like case catch Expression of something_good -> ok; {'EXIT', badarg} -> not_so_great end and handle the errors you care about in the same place as where you handle the successes. Edited to add, re: failwhale, your HTTP entrypoints can usually be something like try real_work_and_output() catch E:R -> log_and_or_page(E,R) output_failwhale() end. As long as the failure in real_work_and_output is quick enough, you'll get your failwhale. Of course, if the problem is processing is too slow, you might want to set a global failwhale flag somewhere, but your ops team can hotload a patch if they need to fix the performance of the failwhale ;)
- simoncion 11y ago"It returns the exception so you can do something like case catch Expression of" Something to be aware of is the cost of a bare catch when an exception of type 'error' is thrown: "[W]hen the exception type is 'error', the catch will build a result containing the symbolic stack trace, and this will then in the first case [1] be immediately discarded, or in the second case matched on and then possibly discarded later. Whereas if you use try/catch, you can ensure that no stack trace is constructed at all to begin with." [0] Stack trace construction isn't free, so it makes sense to avoid it if you're not going to use it. I know that in either Erlang 17 or Erlang 18, parts of Mnesia were slightly refactored to move from bare catch to try/catch for this very reason. [0] http://erlang.org/pipermail/erlang-questions/2013-November/075928.html http://erlang.org/pipermail/erlang-questions/2013-November/0... [1] He's referring back to an example in the email
- chris_wot 11y agoYeah, that the way. To improve a latency issue, issue a panic. rolls eyes