5 ms·

> There are essentially three types of delivery semantics: at-most-once, at-least-once, and exactly-once. Oh, there's a fourth kind: "none-of-the-above", i.e.,

> There are essentially three types of delivery semantics: at-most-once, at-least-once, and exactly-once. Oh, there's a fourth kind: "none-of-the-above", i.e., neither at-most-once or at-least-once. The message gets delivered between [0, ∞] times. Maybe it gets delivered … maybe not. Your message is like UDP packet. A surprising number of systems exhibit this behavior, sadly.

ibotty 4y ago
I guess you mean "[0, ∞)". You cannot have infinitive delivery in computer systems.
- deathanatos 4y ago
  Oh IDK. So, if you look at it juuuust right, ELK can be considered as delivering messages … log messages. Normally our ELK system exhibited the behavior I described above: our logging component would make best efforts to ensure that log messages did get delivered. But there was no ID on the log message when it was submitted: double-submission would result in the message getting duplicated. The message was only removed from the queue of messages that needed to be submitted if the ACK was successfully received. The local queue was only so big: if the application continued to log but couldn't submit to ELK, well, it would just discard the messages², so messages might not get delivered period, even when the client was fine, (e.g., during network outages).¹ That was all fine and good (the consequences of a log message getting delivered twice or not delivered under normal circumstances is "whatever"). One day, the team managing ELK misconfigured it, causing all log submission to start failing. This … didn't get detected? (I mean, as a consumer of the service, what are you going to do, log the error?) Worse, some logs were first routed through the local syslog daemon. It decided to log that it couldn't send the message to a local log file, and then retry without backup, and never gave up. (And, of course, there's not log rotation on that local log file.) So we noticed the problem when the disk filled up … and then realized basically every VM in the fleet was doing this. So it felt like ∞] that day. But yes, mathematically, that bound should be ∞). ¹there is some decent discussion in the thread below my OP about whether this technically counts as "0 deliveries". ²the logic being that crashing due to inability to log is not worth it. It noted the failure on its stderr … but you had to know to look.
skyde 4y ago
if you don't explicitly re-transmit the UDP packet. It would be at-most-once delivery right? The IP routers in the path will not re-send the UDP packet on their own!
- tstrimple 4y ago
  Not with multi-casting.
- jerf 4y ago
  "The IP routers in the path will not re-send the UDP packet on their own!" First rule of networking: Every bad thing that can happen, will. If nothing else, by sheer bugginess you will certainly have something, at some point, retransmitting UDP packets for no good reason.
  - tshaddox 4y ago
    If you’re admitting into the discussion literally any conceivable buggy behavior from public routers you have no control over, then isn’t it pretty clear you can’t have any guarantees about anything whatsoever?
    - pixl97 4y ago
      Hello and welcome to the internet. Enjoy your stay! I think that is the point of these conversations. You shouldn't have application expectations that cannot be met in the real world. Even in your own data center you have far less control over your mirrored switches doing something dumb like sending a stream of packets twice out of their respective interfaces.
      - tshaddox 4y ago
        In some contexts, sure, you could say literally nothing is "guaranteed." But in this context, we're clearly talking about how distributed systems work in practice, and which behaviors you can reasonably achieve and what tradeoffs you must make. The article clearly states: > There are essentially three types of delivery semantics: at-most-once, at-least-once, and exactly-once. Of the three, the first two are feasible and widely used. But in the context you're talking about, even those first two are not "feasible." And in the context you're talking about, there's basically no point in talking about anything you can do to try to achieve certain behavior since there is technically no way to physically guarantee that. What you are saying is definitely not "the point of these conversations," because you're just saying that literally nothing can ever be absolutely guaranteed. Technically true, but not particularly helpful when you're designing real systems to solve real problems, and your systems will mostly operate on infrastructure that mostly does work as intended.
oefnak 4y ago
So that would be 'at least once'. Since if it is 0, it is not a delivery.
- deleted 4y ago
  [deleted]
- tkfu 4y ago
  That's silly. If we followed your logic, there is no such thing as "at most once" delivery. It's actually "exactly-once" delivery, since if it is 0 it is not a delivery.
_pastel 4y ago
Ah, the celery semantics.
dub 4y ago
> A surprising number of systems exhibit this behavior, sadly. I noticed [0, ∞] delivery semantics in a widely-used, internal/homegrown message delivery system at a big tech company once. The bug was easy to spot in the source code (which I was looking at for unrelated reasons), but the catch-22 is that engineers with the skills to notice these sorts of subtle but significant infra bugs are the same engineers who would've advised against building (or continuing to use) your own message delivery system in the first place when there are perfectly serviceable open source and SaaS options.
- dan-robertson 4y ago
  I think whether or not to build your own thing isn’t an obvious choice for a big company. You might have lots of other infrastructure to integrate with and adapting an existing solution might not work as well as making something from scratch that eg integrates with your storage layer or how you manage permissions. The choice may be between having a team dedicated to managing some third party thing (because in the real world these systems have bugs or cases that don’t perform well or need more hardware or rebalancing or whatever) and having a team dedicated to developing and managing an internal solution. The latter case can mean you get to take advantage of existing infrastructure, have less work integrating with this thing, and have in-house experts who can investigate and fix bugs. I don’t think it’s as simple as always preferring to bring in external things.
totetsu 4y ago
I'd like to get [0, ∞] printed on a t-shirt.

ibotty 4y ago

I guess you mean "[0, ∞)". You cannot have infinitive delivery in computer systems.

deathanatos 4y ago
Oh IDK. So, if you look at it juuuust right, ELK can be considered as delivering messages … log messages. Normally our ELK system exhibited the behavior I described above: our logging component would make best efforts to ensure that log messages did get delivered. But there was no ID on the log message when it was submitted: double-submission would result in the message getting duplicated. The message was only removed from the queue of messages that needed to be submitted if the ACK was successfully received. The local queue was only so big: if the application continued to log but couldn't submit to ELK, well, it would just discard the messages², so messages might not get delivered period, even when the client was fine, (e.g., during network outages).¹ That was all fine and good (the consequences of a log message getting delivered twice or not delivered under normal circumstances is "whatever"). One day, the team managing ELK misconfigured it, causing all log submission to start failing. This … didn't get detected? (I mean, as a consumer of the service, what are you going to do, log the error?) Worse, some logs were first routed through the local syslog daemon. It decided to log that it couldn't send the message to a local log file, and then retry without backup, and never gave up. (And, of course, there's not log rotation on that local log file.) So we noticed the problem when the disk filled up … and then realized basically every VM in the fleet was doing this. So it felt like ∞] that day. But yes, mathematically, that bound should be ∞). ¹there is some decent discussion in the thread below my OP about whether this technically counts as "0 deliveries". ²the logic being that crashing due to inability to log is not worth it. It noted the failure on its stderr … but you had to know to look.

skyde 4y ago

if you don't explicitly re-transmit the UDP packet. It would be at-most-once delivery right? The IP routers in the path will not re-send the UDP packet on their own!

tstrimple 4y ago
Not with multi-casting.
jerf 4y ago
"The IP routers in the path will not re-send the UDP packet on their own!" First rule of networking: Every bad thing that can happen, will. If nothing else, by sheer bugginess you will certainly have something, at some point, retransmitting UDP packets for no good reason.
- tshaddox 4y ago
  If you’re admitting into the discussion literally any conceivable buggy behavior from public routers you have no control over, then isn’t it pretty clear you can’t have any guarantees about anything whatsoever?
  - pixl97 4y ago
    Hello and welcome to the internet. Enjoy your stay! I think that is the point of these conversations. You shouldn't have application expectations that cannot be met in the real world. Even in your own data center you have far less control over your mirrored switches doing something dumb like sending a stream of packets twice out of their respective interfaces.
    - tshaddox 4y ago
      In some contexts, sure, you could say literally nothing is "guaranteed." But in this context, we're clearly talking about how distributed systems work in practice, and which behaviors you can reasonably achieve and what tradeoffs you must make. The article clearly states: > There are essentially three types of delivery semantics: at-most-once, at-least-once, and exactly-once. Of the three, the first two are feasible and widely used. But in the context you're talking about, even those first two are not "feasible." And in the context you're talking about, there's basically no point in talking about anything you can do to try to achieve certain behavior since there is technically no way to physically guarantee that. What you are saying is definitely not "the point of these conversations," because you're just saying that literally nothing can ever be absolutely guaranteed. Technically true, but not particularly helpful when you're designing real systems to solve real problems, and your systems will mostly operate on infrastructure that mostly does work as intended.

oefnak 4y ago

So that would be 'at least once'. Since if it is 0, it is not a delivery.

deleted 4y ago
[deleted]
tkfu 4y ago
That's silly. If we followed your logic, there is no such thing as "at most once" delivery. It's actually "exactly-once" delivery, since if it is 0 it is not a delivery.

_pastel 4y ago

Ah, the celery semantics.

dub 4y ago

> A surprising number of systems exhibit this behavior, sadly. I noticed [0, ∞] delivery semantics in a widely-used, internal/homegrown message delivery system at a big tech company once. The bug was easy to spot in the source code (which I was looking at for unrelated reasons), but the catch-22 is that engineers with the skills to notice these sorts of subtle but significant infra bugs are the same engineers who would've advised against building (or continuing to use) your own message delivery system in the first place when there are perfectly serviceable open source and SaaS options.

dan-robertson 4y ago
I think whether or not to build your own thing isn’t an obvious choice for a big company. You might have lots of other infrastructure to integrate with and adapting an existing solution might not work as well as making something from scratch that eg integrates with your storage layer or how you manage permissions. The choice may be between having a team dedicated to managing some third party thing (because in the real world these systems have bugs or cases that don’t perform well or need more hardware or rebalancing or whatever) and having a team dedicated to developing and managing an internal solution. The latter case can mean you get to take advantage of existing infrastructure, have less work integrating with this thing, and have in-house experts who can investigate and fix bugs. I don’t think it’s as simple as always preferring to bring in external things.

totetsu 4y ago

I'd like to get [0, ∞] printed on a t-shirt.