3 ms·

> where k8s is basically the only etcd customer left. Is that true. No one is really using it? I think one thing k8s would need is some obvious answer for sta

by tayo42 1y ago

> where k8s is basically the only etcd customer left. Is that true. No one is really using it? I think one thing k8s would need is some obvious answer for stateful systems(at scale, not mysql at a startup). I think there are some ways to do it? Where I work there is basically everything on k8s, then all the databases on their own crazy special systems to support they insist its impossible and costs to much. I work in the worst of all worlds now supporting this. re: comments about k8s should just schedule pods. mesos with aurora or marathon was basically that. If people wanted that those would have done better. The biggest users of mesos switched to k8s

haiku2077 1y ago
I had to go deep down the etcd rabbit hole several years ago. The problems I ran into: 1. etcd did an fsync on every write and required all nodes to complete a write to report a write as successful. This was not configurable and far higher a guarantee than most use cases actually need - most Kubernetes users are fine with snapshot + restore an older version of the data. But it really severely impacts performance. 2. At the time, etcd had a hard limit of 8GB. Not sure if this is still there. 3. Vanilla etcd was overly cautious about what to do if a majority of nodes went down. I ended up writing a wrapper program to automatically recover from this in most cases, which worked well in practice. In conclusion there was no situation where I saw etcd used that I wouldn't have preferred a highly available SQL DB. Indeed, k3s got it right using sqlite for small deployments.
- nh2 1y ago
  For (1), I definitely want my production HA databases to fsync every write. Of course configurability is good (e.g. for automated fasts tests you don't need it), but safe is a good default here, and if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable (e.g. 1000 fsyncs/second).
  - haiku2077 1y ago
    > I definitely want my production HA databases to fsync every write. I didn't! Our business DR plan only called for us to restore to an older version with short downtime, so fsync on every write on every node was a reduction in performance for no actual business purpose or benefit. IIRC we modified our database to run off ramdisk and snapshot every few minutes which ran way better and had no impact on our production recovery strategy. > if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable At the time one of the problems I ran into was that public cloud regions in southeast asia had significantly worse SSDs that couldn't keep up. This was on one of the big three cloud providers. 1000 fsyncs/second is a tiny fraction of the real world production load we required. An API that only accepts 1000 writes a second is very slow! Also, plenty of people run k8s clusters on commodity hardware. I ran one on an old gaming PC with a budget SSD for a while in my basement. Great use case for k3s.
    - nh2 1y ago
      > IIRC we modified our database to run off ramdisk If a ramdisk is sufficient for your use case, why would you use a Raft-based distributed networked consensus database like etcd in the first place? Its whole point is to protect you not only from power failure (like fsync does) but even from entire machine failure. And the network is usually higher latency than a local SSD anyway. > 1000 fsyncs/second is a tiny fraction of the real world production load we required Kubernetes uses etcd for storing the cluster state. Do you update the cluster state more than 1000 times per second? Curious what operation needs that.
      - mdaniel 1y ago
        > Do you update the cluster state more than 1000 times per second? Curious what operation needs that. Ooo, ooo, I know this one! It's for clusters with more than approximately 300 Nodes, as their status polling will actually knock over the primary etcd cluster. That's why kube-apiserver introduced this fun parameter: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#:~:text=--etcd-servers-overrides%20strings https://kubernetes.io/docs/reference/command-line-tools-refe... and the "but why" of https://kubernetes.io/docs/setup/best-practices/cluster-large/#etcd-storage https://kubernetes.io/docs/setup/best-practices/cluster-larg... I've only personally gotten clusters up to about 550 so I'm sure the next scaling wall is hiding between that and the 5000 limit they advertise
  - _bohm 1y ago
    Most database systems are designed to amortize fsyncs when they have high write throughput. You want every write to be fsync'd, but you don't want to actually call fsync for each individual write operation.
- dilyevsky 1y ago
  1 and 2 can be overridden via flag. 3 is practically the whole point of the software
  - haiku2077 1y ago
    With 3 I mean that in cases where there was an unambiguously correct way to recover from the situation, etcd did not automatically recover. My wrapper program would always recover from thise situations. (It's been a number of years and the exact details are hazy now, though.)
    - dilyevsky 1y ago
      If the majority of quorum is truly down, then you’re down. That is by design. There’s no good way to recover from this without potentially losing state so the system correctly does nothing at this point. Sure you can force it into working state with external intervention but that’s up to you
      - haiku2077 1y ago
        Like I said I'm hazy on the details, this was a small thing I did a long time ago. But I do remember our on-call having to deal with a lot of manual repair of etcd quorum, and I noticed the runbook to fix it had no steps that needed any human decision making, so I made that wrapper program to automate the recovery. It wasn't complex either, IIRC it was about one or two pages of code, mostly logging.
dilyevsky 1y ago
That is decisively not true. A number of very large companies use etcd directly for various needs

by tayo42 1y ago

haiku2077 1y ago

I had to go deep down the etcd rabbit hole several years ago. The problems I ran into: 1. etcd did an fsync on every write and required all nodes to complete a write to report a write as successful. This was not configurable and far higher a guarantee than most use cases actually need - most Kubernetes users are fine with snapshot + restore an older version of the data. But it really severely impacts performance. 2. At the time, etcd had a hard limit of 8GB. Not sure if this is still there. 3. Vanilla etcd was overly cautious about what to do if a majority of nodes went down. I ended up writing a wrapper program to automatically recover from this in most cases, which worked well in practice. In conclusion there was no situation where I saw etcd used that I wouldn't have preferred a highly available SQL DB. Indeed, k3s got it right using sqlite for small deployments.

nh2 1y ago
For (1), I definitely want my production HA databases to fsync every write. Of course configurability is good (e.g. for automated fasts tests you don't need it), but safe is a good default here, and if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable (e.g. 1000 fsyncs/second).
- haiku2077 1y ago
  > I definitely want my production HA databases to fsync every write. I didn't! Our business DR plan only called for us to restore to an older version with short downtime, so fsync on every write on every node was a reduction in performance for no actual business purpose or benefit. IIRC we modified our database to run off ramdisk and snapshot every few minutes which ran way better and had no impact on our production recovery strategy. > if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable At the time one of the problems I ran into was that public cloud regions in southeast asia had significantly worse SSDs that couldn't keep up. This was on one of the big three cloud providers. 1000 fsyncs/second is a tiny fraction of the real world production load we required. An API that only accepts 1000 writes a second is very slow! Also, plenty of people run k8s clusters on commodity hardware. I ran one on an old gaming PC with a budget SSD for a while in my basement. Great use case for k3s.
  - nh2 1y ago
    > IIRC we modified our database to run off ramdisk If a ramdisk is sufficient for your use case, why would you use a Raft-based distributed networked consensus database like etcd in the first place? Its whole point is to protect you not only from power failure (like fsync does) but even from entire machine failure. And the network is usually higher latency than a local SSD anyway. > 1000 fsyncs/second is a tiny fraction of the real world production load we required Kubernetes uses etcd for storing the cluster state. Do you update the cluster state more than 1000 times per second? Curious what operation needs that.
    - mdaniel 1y ago
      > Do you update the cluster state more than 1000 times per second? Curious what operation needs that. Ooo, ooo, I know this one! It's for clusters with more than approximately 300 Nodes, as their status polling will actually knock over the primary etcd cluster. That's why kube-apiserver introduced this fun parameter: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#:~:text=--etcd-servers-overrides%20strings https://kubernetes.io/docs/reference/command-line-tools-refe... and the "but why" of https://kubernetes.io/docs/setup/best-practices/cluster-large/#etcd-storage https://kubernetes.io/docs/setup/best-practices/cluster-larg... I've only personally gotten clusters up to about 550 so I'm sure the next scaling wall is hiding between that and the 5000 limit they advertise
- _bohm 1y ago
  Most database systems are designed to amortize fsyncs when they have high write throughput. You want every write to be fsync'd, but you don't want to actually call fsync for each individual write operation.
dilyevsky 1y ago
1 and 2 can be overridden via flag. 3 is practically the whole point of the software
- haiku2077 1y ago
  With 3 I mean that in cases where there was an unambiguously correct way to recover from the situation, etcd did not automatically recover. My wrapper program would always recover from thise situations. (It's been a number of years and the exact details are hazy now, though.)
  - dilyevsky 1y ago
    If the majority of quorum is truly down, then you’re down. That is by design. There’s no good way to recover from this without potentially losing state so the system correctly does nothing at this point. Sure you can force it into working state with external intervention but that’s up to you
    - haiku2077 1y ago
      Like I said I'm hazy on the details, this was a small thing I did a long time ago. But I do remember our on-call having to deal with a lot of manual repair of etcd quorum, and I noticed the runbook to fix it had no steps that needed any human decision making, so I made that wrapper program to automate the recovery. It wasn't complex either, IIRC it was about one or two pages of code, mostly logging.

dilyevsky 1y ago

That is decisively not true. A number of very large companies use etcd directly for various needs