3 ms·
> what are you expected to do when the service discovery system is unreachable, broken, or filled with bad data? Different things based on which one of these a
by q3k 7y ago
> what are you expected to do when the service discovery system is unreachable, broken, or filled with bad data?
Different things based on which one of these actually happened!
I would assume the code that caused this issue was along the lines of:
var addresses []string{}
for k, el := range etcdEntries {
var element data.Service
addr, err := json.Unmarshal(&element, el)
if err != nil {
return fmt.Errorf("could not unmarshal service entry %q: %w", k, err)
}
addresses = append(addresses, addr)
}
And the issue is the lack of graceful degradation, in the form of not failing the entire conversion if one entry is broken. Instead, this should be something like:
var addresses []string{}
for k, el := range etcdEntries {
var element data.Service
addr, err := json.Unmarshal(&element, el)
if err != nil {
metrics.ServiceEntryBroken("key", k).Up()
glog.Errorf("Could not unmarshal service entry %q: %v", k err)
continue
}
addresses = append(addresses, addr)
}
Of course, this takes much more effort. But such is the nature of writing high quality, reliable software.
Naturally, if your entire SD service was down, you would fail the entire thing much earlier. That is indeed the thing about Go errors being plain variables - you are intended to write error handling logic (or at least are forced to consider it), not just pack the buck down by default, via either optional types or exceptions.