6 ms·

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

brightball 6y ago
I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes. The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.
- anonymousisme 6y ago
  Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.
  - indigodaddy 6y ago
    Was thinking the exact same thought as I finished up the article...
  - lalos 6y ago
    Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.
    - mcguire 6y ago
      "Professional responsibility", if nothing else.
    - calvinmorrison 6y ago
      until you hit something like "bug reports should be subitted to here, not there and filled out with appropriate information, matching triplicate documents and make sure to CC the grand puba, also CLOSED-WONTFIX" enough times and you just stop submitting requests.
    - saagarjha 6y ago
      Unfortunately, not every engineer has an employer that would let them do this…
ibejoeb 6y ago
I really want to know if the telemetrics (or whatever the thing was) pushed enough packets to actually warrant the config. Setting something to the max sounds like preopt to me. This must be a truly exceptional condition if it actually remained undiscovered since linux 3.
- sleepydog 6y ago
  I can't think of anything that would warrant a 2GB receive buffer. The buffer should be sized so that the receiving program has a reasonable amount of time to drain it before it becomes full. A large skylake VM in GCP can do 32 Gbps (lowercase b), so assuming worst conditions, a 2GB receive buffer would give the receiver 500ms to call recv(), which is a huge amount of time for something that should take microseconds, especially in the context of a client. Even if there was a specialized server that needed such a large receive buffer, it doesn't make sense to set the system-wide default so high.
  - deleted 6y ago
    [deleted]

Did you read the article? The customer had set `net.core.rmem_default = 2147483647` on purpose. Which exposed a Kernel bug. The whole herd would be having the same issue.

brightball 6y ago

I think what he's trying to suggest is that the customer may have been able to isolate the issue faster by walking through the provisioning settings for the machine to identify core changes. The bug report resulted in a core fix, which is a better result than if the customer had fixed it themselves of course.

anonymousisme 6y ago
Also, it was very nice of Google to follow up and submit the patch to LKML. IMHO this goes beyond the scope of their role. They could have taken a more selfish approach and accepted the bug as "normal" behavior, and advised their customer to not configure the buffer to such an enormous size.
- indigodaddy 6y ago
  Was thinking the exact same thought as I finished up the article...
- lalos 6y ago
  Any decent engineer would smile at the fact that they just found a bug in this type of open source stack and happily submit it. Feel this is more of a side effect of individual behavior rather than company policy.
  - mcguire 6y ago
    "Professional responsibility", if nothing else.
  - calvinmorrison 6y ago
    until you hit something like "bug reports should be subitted to here, not there and filled out with appropriate information, matching triplicate documents and make sure to CC the grand puba, also CLOSED-WONTFIX" enough times and you just stop submitting requests.
  - saagarjha 6y ago
    Unfortunately, not every engineer has an employer that would let them do this…

ibejoeb 6y ago

I really want to know if the telemetrics (or whatever the thing was) pushed enough packets to actually warrant the config. Setting something to the max sounds like preopt to me. This must be a truly exceptional condition if it actually remained undiscovered since linux 3.

sleepydog 6y ago
I can't think of anything that would warrant a 2GB receive buffer. The buffer should be sized so that the receiving program has a reasonable amount of time to drain it before it becomes full. A large skylake VM in GCP can do 32 Gbps (lowercase b), so assuming worst conditions, a 2GB receive buffer would give the receiver 500ms to call recv(), which is a huge amount of time for something that should take microseconds, especially in the context of a client. Even if there was a specialized server that needed such a large receive buffer, it doesn't make sense to set the system-wide default so high.
- deleted 6y ago
  [deleted]