5 ms·

Quick takes on the recent OpenAI public incident write-up

108 points by azhenley 2y ago

nijave 2y ago
Seems like automated node access could have also been helpful here. Kill the offending pods directly on the nodes to relieve API server pressure long enough to rollback
- dilyevsky 2y ago
  The only problem is you have to find out where those pods are and your primary source of this information is currently under dos attack by said pods
  - nijave 2y ago
    Not if you connect to all the nodes
geocrasher 2y ago
In short, the root cause was a new telemetry service configuration that unexpectedly generated massive Kubernetes API load across large clusters, overwhelming the control plane and breaking DNS-based service discovery. The DNS song seems appropriate. https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tune-of-let-it-be https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tun...
ec109685 2y ago
Surprised they don’t have a slower rollout across multiple regions / kubernetes clusters, given the K8s API’s are a spof as shown here where a change brought the control plane down. Also, stale-if-error is a far safer pattern for service discovery than ttl’d dns.
deleted 2y ago
[deleted]
dang 2y ago
Recent and related: ChatGPT Down - https://news.ycombinator.com/item?id=42394391 https://news.ycombinator.com/item?id=42394391 - Dec 2024 (30 comments)
JohnMakin 2y ago
I caused an API server outage once with a monitoring tool, however in my case it was a monstrosity of a 20,000 line script. We quickly realized what we had done and turned it off, and I have seen in very large clusters with 1000+ nodes that you need to be especially sensitive about monitoring API server resource usage depending on what precisely you are doing. Surprised they hadn't learned this lesson yet, given the likely scale of their workloads.
- dboreham 2y ago
  > 20,000 line script Dude.
  - JohnMakin 2y ago
    This was most definitely not my choice or preference, but you know how it goes.
  - fuzzy_biscuit 2y ago
    They meant manuscript I assume.
    - cbsmith 2y ago
      Yeah, "script" is just an abbreviation of "manuscript", right? ;-)
      - TeMPOraL 2y ago
        Right, and the (manu)script that size makes a long scroll. Humanity had to more or less invent pagination to deal with this. (I'll see myself out.)
        cbsmith 2y ago
        Thanks dad. ;-) Job well done.
jimmyl02 2y ago
splitting the control and data plane is a great way to improve resilience and prevent everything from being hard down. I wonder how it could be accomplished with service discovery / routing. maybe instead of relying on kubernetes DNS for discovery it can be closer to something like envoy.the control plane updates configs that are stored locally (and are eventually consistent) so even if the control plane dies the data plane has access to location information of other peer clusters.
- nijave 2y ago
  I imagine a peer to peer scheme could also work. All nodes are control plane and data plane and advertise/broadcast to other nodes (similar to how network routing tables are distributed) That'd be a pretty big architectural change, though
- cbsmith 2y ago
  > maybe instead of relying on kubernetes DNS for discovery it can be closer to something like envoy.the control plane updates configs that are stored locally (and are eventually consistent) so even if the control plane dies the data plane has access to location information of other peer clusters. That is generally how it is done... there's this never ending conflict between "push" architectures and "pull" architectures, and this scenario sure makes "push" seem better, and it is... until you're in one of those scenarios where "pull" is better. ;-)
dilyevsky 2y ago
Something doesn't add up - CoreDNS's kubernetes plugin should be serving Service RRs from its internal cache even if APIServer is down because it's using cache.Indexer. The records would be stale but unless their application pods all restarted, which they could not since APIServer was down, or all CoreDNS pods got restarted, which, again, they could not, just records expiring from the cache shouldn't have caused full discovery outage.
- dmauskop 2y ago
  I had the same thought! https://news.ycombinator.com/item?id=42446318 https://news.ycombinator.com/item?id=42446318 My guess is they were running CoreDNS on control plane nodes since that's the kubeadm default.
- jimmyl02 2y ago
  wouldn't it be coredns caches the information and records from API server for X amount of time (it seems like this might be 20 minutes?) then once the 20 minutes expired coredns would query api server, receive no response, then fail? I think the idea of just serving cached responses indefinitely when api server is unreachable is what you're describing but not sure if this is default. (and probably has other tradeoffs that I'm not sure about too)
  - dilyevsky 2y ago
    Based on my understanding of the plugin code it is the default. The way cache.Indexer works is it's continuously streaming resources from APIServer using Watch API and updates internal map. I think if Watch API is down it just sits there and doesn't purge anything but I haven't tested that. The 20 min expiry is probably referring to CodeDNS cache stanza which is a separate plugin[0]. [0] - https://coredns.io/plugins/cache https://coredns.io/plugins/cache
ilaksh 2y ago
Wow, sounds like a nightmare. Operations staff definitely have real jobs.
btown 2y ago
> In order to make that fix, we needed to access the Kubernetes control plane – which we could not do due to the increased load to the Kubernetes API servers. Something that I wish all databases and API servers would do, and that few actually do in practice, is to allocate a certain amount of headroom (memory and CPU) to "break glass in case of emergency" sessions. Have an interrupt fired periodically that listens exclusively on a port that will only be used for emergency instructions (but uses equal security measures to production, and is only visible internally). Ensure that it can allocate against a preallocated block of memory; allow it to schedule higher-priority threads. A small concession to make in the usual course of business, but when it's useful it's vital.
- dotancohen 2y ago
  I have a 1 GiB file to rm in case of emergency on every filesystem I manage - both personal and professional. I've only had to delete it maybe three or four times that I remember, but it's been a system-saver each time. I've long considered writing a process that just consumes some CPU, memory, and bandwidth for the same emergency buffer. I've even considered that it should even examine `last` every second and free up those resources when I SSH in.
  - tux3 2y ago
    It is also common for filesystems to reserve a small percentage to the root user. I think the ext4 default is still 5% (which can be quite a bit more than 1GB on modern drives!)
    - timschmidt 2y ago
      Yup. 'tune2fs -m0' has saved my bacon more than once.
    - dotancohen 2y ago
      I haven't used root directly for over a decade. Modern usage is to log in as an unprivileged user, and to use sudo for all root operations.
StarlaAtNight 2y ago
This quote cracked me up: “I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS”
- akshayshah 2y ago
  James Mickens is a comedic genius. The linked article always makes me laugh out loud. https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
  - rrr_oh_man 2y ago
    Excellent article, but the typesetting / justification in that pdf is horrendous. The text columns look like the side of that hallway rubber mat that my dog keeps chewing on. spend a lot of time trying edge. However, as someone lieve that true progress is mes, and for the chickens y zombies, and the polite to eat your brain to acquire be prepared; thus, in the e scientific breakthroughs, ast inevitably becomes he main thing that I ponder is post-apocalyptic survival ag-tag group of associates. cruit: a locksmith (to open ith has run out of ideas); row snakes at my enemies g is a reasonable way to ble in my ultimate success
    - remram 2y ago
      Converted to markdown: https://gist.github.com/remram44/767dd41676866cc2d30dae6e3a107ab9 https://gist.github.com/remram44/767dd41676866cc2d30dae6e3a1...
  - hyperdimension 2y ago
    That was an amazing read. Thanks for linking it.
feyman_r 2y ago
For me, this was stunning : “ 2:51pm to 3:20pm: The change was applied to all clusters” How can such a large change not be staged in some manner or the other? Feedback loops have a way of catching up later which is why it’s important to roll out gradually.
- antod 2y ago
  In the words of DevOps Borat... "To make error is human. To propagate error to all server in automatic way is #devops."
  - Havoc 2y ago
    So sad that the account isn’t posting anymore.
- TeMPOraL 2y ago
  DNS makes for an atypically slow feedback loop. If you're not aware of it, then for an otherwise safe-looking change, you may test and complete the gradual roll-out before the failure hits you.

108 points by azhenley 2y ago

Seems like automated node access could have also been helpful here. Kill the offending pods directly on the nodes to relieve API server pressure long enough to rollback

dilyevsky 2y ago
The only problem is you have to find out where those pods are and your primary source of this information is currently under dos attack by said pods
- nijave 2y ago
  Not if you connect to all the nodes

geocrasher 2y ago

In short, the root cause was a new telemetry service configuration that unexpectedly generated massive Kubernetes API load across large clusters, overwhelming the control plane and breaking DNS-based service discovery. The DNS song seems appropriate. https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tune-of-let-it-be https://soundcloud.com/ryan-flowers-916961339/dns-to-the-tun...

ec109685 2y ago

Surprised they don’t have a slower rollout across multiple regions / kubernetes clusters, given the K8s API’s are a spof as shown here where a change brought the control plane down. Also, stale-if-error is a far safer pattern for service discovery than ttl’d dns.

deleted 2y ago

[deleted]

dang 2y ago

Recent and related: ChatGPT Down - https://news.ycombinator.com/item?id=42394391 https://news.ycombinator.com/item?id=42394391 - Dec 2024 (30 comments)

JohnMakin 2y ago

I caused an API server outage once with a monitoring tool, however in my case it was a monstrosity of a 20,000 line script. We quickly realized what we had done and turned it off, and I have seen in very large clusters with 1000+ nodes that you need to be especially sensitive about monitoring API server resource usage depending on what precisely you are doing. Surprised they hadn't learned this lesson yet, given the likely scale of their workloads.

dboreham 2y ago
> 20,000 line script Dude.
- JohnMakin 2y ago
  This was most definitely not my choice or preference, but you know how it goes.
- fuzzy_biscuit 2y ago
  They meant manuscript I assume.
  - cbsmith 2y ago
    Yeah, "script" is just an abbreviation of "manuscript", right? ;-)
    - TeMPOraL 2y ago
      Right, and the (manu)script that size makes a long scroll. Humanity had to more or less invent pagination to deal with this. (I'll see myself out.)
      - cbsmith 2y ago
        Thanks dad. ;-) Job well done.

jimmyl02 2y ago

splitting the control and data plane is a great way to improve resilience and prevent everything from being hard down. I wonder how it could be accomplished with service discovery / routing. maybe instead of relying on kubernetes DNS for discovery it can be closer to something like envoy.the control plane updates configs that are stored locally (and are eventually consistent) so even if the control plane dies the data plane has access to location information of other peer clusters.

nijave 2y ago
I imagine a peer to peer scheme could also work. All nodes are control plane and data plane and advertise/broadcast to other nodes (similar to how network routing tables are distributed) That'd be a pretty big architectural change, though
cbsmith 2y ago
> maybe instead of relying on kubernetes DNS for discovery it can be closer to something like envoy.the control plane updates configs that are stored locally (and are eventually consistent) so even if the control plane dies the data plane has access to location information of other peer clusters. That is generally how it is done... there's this never ending conflict between "push" architectures and "pull" architectures, and this scenario sure makes "push" seem better, and it is... until you're in one of those scenarios where "pull" is better. ;-)

dilyevsky 2y ago

Something doesn't add up - CoreDNS's kubernetes plugin should be serving Service RRs from its internal cache even if APIServer is down because it's using cache.Indexer. The records would be stale but unless their application pods all restarted, which they could not since APIServer was down, or all CoreDNS pods got restarted, which, again, they could not, just records expiring from the cache shouldn't have caused full discovery outage.

dmauskop 2y ago
I had the same thought! https://news.ycombinator.com/item?id=42446318 https://news.ycombinator.com/item?id=42446318 My guess is they were running CoreDNS on control plane nodes since that's the kubeadm default.
jimmyl02 2y ago
wouldn't it be coredns caches the information and records from API server for X amount of time (it seems like this might be 20 minutes?) then once the 20 minutes expired coredns would query api server, receive no response, then fail? I think the idea of just serving cached responses indefinitely when api server is unreachable is what you're describing but not sure if this is default. (and probably has other tradeoffs that I'm not sure about too)
- dilyevsky 2y ago
  Based on my understanding of the plugin code it is the default. The way cache.Indexer works is it's continuously streaming resources from APIServer using Watch API and updates internal map. I think if Watch API is down it just sits there and doesn't purge anything but I haven't tested that. The 20 min expiry is probably referring to CodeDNS cache stanza which is a separate plugin[0]. [0] - https://coredns.io/plugins/cache https://coredns.io/plugins/cache

ilaksh 2y ago

Wow, sounds like a nightmare. Operations staff definitely have real jobs.

btown 2y ago

> In order to make that fix, we needed to access the Kubernetes control plane – which we could not do due to the increased load to the Kubernetes API servers. Something that I wish all databases and API servers would do, and that few actually do in practice, is to allocate a certain amount of headroom (memory and CPU) to "break glass in case of emergency" sessions. Have an interrupt fired periodically that listens exclusively on a port that will only be used for emergency instructions (but uses equal security measures to production, and is only visible internally). Ensure that it can allocate against a preallocated block of memory; allow it to schedule higher-priority threads. A small concession to make in the usual course of business, but when it's useful it's vital.

dotancohen 2y ago
I have a 1 GiB file to rm in case of emergency on every filesystem I manage - both personal and professional. I've only had to delete it maybe three or four times that I remember, but it's been a system-saver each time. I've long considered writing a process that just consumes some CPU, memory, and bandwidth for the same emergency buffer. I've even considered that it should even examine `last` every second and free up those resources when I SSH in.
- tux3 2y ago
  It is also common for filesystems to reserve a small percentage to the root user. I think the ext4 default is still 5% (which can be quite a bit more than 1GB on modern drives!)
  - timschmidt 2y ago
    Yup. 'tune2fs -m0' has saved my bacon more than once.
  - dotancohen 2y ago
    I haven't used root directly for over a decade. Modern usage is to log in as an unprivileged user, and to use sudo for all root operations.

StarlaAtNight 2y ago

This quote cracked me up: “I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS”

akshayshah 2y ago
James Mickens is a comedic genius. The linked article always makes me laugh out loud. https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf
- rrr_oh_man 2y ago
  Excellent article, but the typesetting / justification in that pdf is horrendous. The text columns look like the side of that hallway rubber mat that my dog keeps chewing on. spend a lot of time trying edge. However, as someone lieve that true progress is mes, and for the chickens y zombies, and the polite to eat your brain to acquire be prepared; thus, in the e scientific breakthroughs, ast inevitably becomes he main thing that I ponder is post-apocalyptic survival ag-tag group of associates. cruit: a locksmith (to open ith has run out of ideas); row snakes at my enemies g is a reasonable way to ble in my ultimate success
  - remram 2y ago
    Converted to markdown: https://gist.github.com/remram44/767dd41676866cc2d30dae6e3a107ab9 https://gist.github.com/remram44/767dd41676866cc2d30dae6e3a1...
- hyperdimension 2y ago
  That was an amazing read. Thanks for linking it.

feyman_r 2y ago

For me, this was stunning : “ 2:51pm to 3:20pm: The change was applied to all clusters” How can such a large change not be staged in some manner or the other? Feedback loops have a way of catching up later which is why it’s important to roll out gradually.

antod 2y ago
In the words of DevOps Borat... "To make error is human. To propagate error to all server in automatic way is #devops."
- Havoc 2y ago
  So sad that the account isn’t posting anymore.
TeMPOraL 2y ago
DNS makes for an atypically slow feedback loop. If you're not aware of it, then for an otherwise safe-looking change, you may test and complete the gradual roll-out before the failure hits you.