6 ms·
I write HTTP/RPC and some TCP servers, implementing specs to parse files and data processing pipelines based on PubSub, NATs, etc. Whenever the server fails to
by bilinguliar 4y ago
I write HTTP/RPC and some TCP servers, implementing specs to parse files and data processing pipelines based on PubSub, NATs, etc.
Whenever the server fails to do what is needed - in most cases, it will retry with a backoff. In case it is not retryable - most often, it will exit, an alert will fire, and I will start handling an incident.
Background jobs run in a goroutine and send a non-retryable error to the channel.
go func() {
errChan <- s.ProcessData(ctx)
}()
Followed by a blocking select:
select {
case err := <- errChan:
log.Printf(“Server failed: %s”, err)
case <- ctx.Done():
log.Println(“Shutting down”)
}
Something along these lines.
- closeparen 4y ago>Whenever the server fails to do what is needed - in most cases, it will retry with a backoff. In case it is not retryable - most often, it will exit, an alert will fire, and I will start handling an incident. I don't follow. Requests have deadlines, so you can't keep trying forever. Often a persistent failure means the request itself is bad. Either invalid, or exposing an edge case that the system can't currently handle. These kinds of errors are always present at background levels in a high-scalability system; we have SLAs like 99.99% success rate to decide when there's really an incident. Crashing the entire server process because of one failed request sounds crazy.
- bilinguliar 4y agoMy bad. I misread your comment. Please see the response one level up.