4 ms·
Stacks printed by the receiver are generally incomplete/useless for systems that communicate. The error is often in the sender, but is reported by the receiver
by jrockway 2y ago
Stacks printed by the receiver are generally incomplete/useless for systems that communicate. The error is often in the sender, but is reported by the receiver. This isn't unique to Go and channels.
Consider a Node app that calls a C++ web service. The C++ web service returns "503 Service Unavailable" because your request causes the server to segfault. The useful stack trace for fixing this problem is on the C++ side in a core dump, not whatever Node prints when the HTTP request fails.)
What you want to do on the Go side is have your sender be able to relay an error to the receiver, and to wrap errors:
func doWork(w work) (result, error) {
return errors.New("too lazy to do work")
}
func doWorkQueue(q <-chan work, r chan<- result) {
for w := range q {
result, err := doWork(w)
if err != nil {
r <- errorResult(errors.Wrapf(err, "doWork(workID=%v)", w.ID))
continue
}
r <- successResult(w, result)
}
}
...
go doWorkQueue(...)
submitWork(work)
for r := range resultCh {
if err := r.Err(); err != nil {
log.Error("failed to do work: %v", errors.Wrap(err, "recv result"))
}
}
This now errors with something like:
main.go:20: failed to do work:
main.go:20: recv result:
main.go:9: doWork(workID=42):
main.go:2: too lazy to do work
Or if your `errors` library doesn't grab the caller func/line number, just "failed to do work: recv result: doWork(workID=42): too lazy to do work". This should be enough to debug the problem, regardless of what side the problem happens on.
(At work we use a hacked up copy of github.com/pkg/errors, which grabs the entire stack trace at each of the errors.Wrap/errors.New call sites. This results in an exceedingly verbose trace that takes up your entire screen, but is at the very least... thorough.)
The reason that error wrapping is essential is because stack traces don't capture critical information, like what iteration of the loop you're on, how many times your retry loop ran, which work id failed and returned "too lazy to do work", etc. The error wrapping is where you get to add this in. This is, again, the same as every language... the random exception you throw in your HTTP client when the server is down isn't aware of the work item id that caused this exception, so you have to catch and re-throw with that information or you have an undebuggable mess.
What's nice about Go is that it's really easy to add this contextual information, either with fmt.Errorf in the standard library, or with very small functions that capture some information automatically (errors.Wrap/errors.Wrapf). I will say that not doing this is my #1 complaint, and it's exceedingly common to just "return err". I have spent 4 years fixing the work codebase to wrap errors, and people still add new unwrapped errors (because our linter allows an escape hatch, errors.EnsureStack, which some people really like). It then results in some oncall engineer wasting a week debugging a simple problem. Sigh! But that's humans being humans; Go makes it very easy to do the right thing. You just have to tell your team to do the right thing and to make them want the right thing.
- cyberax 2y agoOf course stack traces are a silver bullet. However, it helps to be able to correlate stacks to requests from logs. It's especially helpful when you're debugging deadlocks.