4 ms·

For (1), I definitely want my production HA databases to fsync every write. Of course configurability is good (e.g. for automated fasts tests you don't need it

by nh2 1y ago

For (1), I definitely want my production HA databases to fsync every write. Of course configurability is good (e.g. for automated fasts tests you don't need it), but safe is a good default here, and if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable (e.g. 1000 fsyncs/second).

haiku2077 1y ago
> I definitely want my production HA databases to fsync every write. I didn't! Our business DR plan only called for us to restore to an older version with short downtime, so fsync on every write on every node was a reduction in performance for no actual business purpose or benefit. IIRC we modified our database to run off ramdisk and snapshot every few minutes which ran way better and had no impact on our production recovery strategy. > if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable At the time one of the problems I ran into was that public cloud regions in southeast asia had significantly worse SSDs that couldn't keep up. This was on one of the big three cloud providers. 1000 fsyncs/second is a tiny fraction of the real world production load we required. An API that only accepts 1000 writes a second is very slow! Also, plenty of people run k8s clusters on commodity hardware. I ran one on an old gaming PC with a budget SSD for a while in my basement. Great use case for k3s.
- nh2 1y ago
  > IIRC we modified our database to run off ramdisk If a ramdisk is sufficient for your use case, why would you use a Raft-based distributed networked consensus database like etcd in the first place? Its whole point is to protect you not only from power failure (like fsync does) but even from entire machine failure. And the network is usually higher latency than a local SSD anyway. > 1000 fsyncs/second is a tiny fraction of the real world production load we required Kubernetes uses etcd for storing the cluster state. Do you update the cluster state more than 1000 times per second? Curious what operation needs that.
  - mdaniel 1y ago
    > Do you update the cluster state more than 1000 times per second? Curious what operation needs that. Ooo, ooo, I know this one! It's for clusters with more than approximately 300 Nodes, as their status polling will actually knock over the primary etcd cluster. That's why kube-apiserver introduced this fun parameter: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#:~:text=--etcd-servers-overrides%20strings https://kubernetes.io/docs/reference/command-line-tools-refe... and the "but why" of https://kubernetes.io/docs/setup/best-practices/cluster-large/#etcd-storage https://kubernetes.io/docs/setup/best-practices/cluster-larg... I've only personally gotten clusters up to about 550 so I'm sure the next scaling wall is hiding between that and the 5000 limit they advertise
    - nh2 1y ago
      > their status polling will actually knock over the primary etcd cluster Polling the status sounds like a read-only operation, why would it trigger an fsync?
      - mdaniel 1y ago
        kubelet constantly checks-in to report "I am alive, and here is the state of affairs" so that kube-apiserver can make informed decisions about whether a Pod needs attention to align with its desired state So, I don't this second know what the formal reconciliation loop is called for that Node->apiserver handshake but it is a polling operation in that the connection isn't left open all the time, and it is a status-reporting operation. So that's how I ended up calling it "status polling." It is a write operation because whichever etcd is assigned to track the current state of the Events needs to be mutated to record the current state of the Events It actually could be that even a smaller sized cluster could get into this same situation if there were a bazillion tiny Pods (e.g. not directly correlated with the physical size of the cluster) but since the limits say one cannot have more than 110 Pods per Node, I'm guessing the Event magnification is easier to see with physically wider clusters
        nh2 1y ago
_bohm 1y ago
Most database systems are designed to amortize fsyncs when they have high write throughput. You want every write to be fsync'd, but you don't want to actually call fsync for each individual write operation.

by nh2 1y ago

haiku2077 1y ago

> I definitely want my production HA databases to fsync every write. I didn't! Our business DR plan only called for us to restore to an older version with short downtime, so fsync on every write on every node was a reduction in performance for no actual business purpose or benefit. IIRC we modified our database to run off ramdisk and snapshot every few minutes which ran way better and had no impact on our production recovery strategy. > if somebody sets up a Kubernetes cluster, they can and should afford enterprise SSDs where fsync of small data is fast and reliable At the time one of the problems I ran into was that public cloud regions in southeast asia had significantly worse SSDs that couldn't keep up. This was on one of the big three cloud providers. 1000 fsyncs/second is a tiny fraction of the real world production load we required. An API that only accepts 1000 writes a second is very slow! Also, plenty of people run k8s clusters on commodity hardware. I ran one on an old gaming PC with a budget SSD for a while in my basement. Great use case for k3s.

nh2 1y ago
> IIRC we modified our database to run off ramdisk If a ramdisk is sufficient for your use case, why would you use a Raft-based distributed networked consensus database like etcd in the first place? Its whole point is to protect you not only from power failure (like fsync does) but even from entire machine failure. And the network is usually higher latency than a local SSD anyway. > 1000 fsyncs/second is a tiny fraction of the real world production load we required Kubernetes uses etcd for storing the cluster state. Do you update the cluster state more than 1000 times per second? Curious what operation needs that.
- mdaniel 1y ago
  > Do you update the cluster state more than 1000 times per second? Curious what operation needs that. Ooo, ooo, I know this one! It's for clusters with more than approximately 300 Nodes, as their status polling will actually knock over the primary etcd cluster. That's why kube-apiserver introduced this fun parameter: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#:~:text=--etcd-servers-overrides%20strings https://kubernetes.io/docs/reference/command-line-tools-refe... and the "but why" of https://kubernetes.io/docs/setup/best-practices/cluster-large/#etcd-storage https://kubernetes.io/docs/setup/best-practices/cluster-larg... I've only personally gotten clusters up to about 550 so I'm sure the next scaling wall is hiding between that and the 5000 limit they advertise
  - nh2 1y ago
    > their status polling will actually knock over the primary etcd cluster Polling the status sounds like a read-only operation, why would it trigger an fsync?
    - mdaniel 1y ago
      kubelet constantly checks-in to report "I am alive, and here is the state of affairs" so that kube-apiserver can make informed decisions about whether a Pod needs attention to align with its desired state So, I don't this second know what the formal reconciliation loop is called for that Node->apiserver handshake but it is a polling operation in that the connection isn't left open all the time, and it is a status-reporting operation. So that's how I ended up calling it "status polling." It is a write operation because whichever etcd is assigned to track the current state of the Events needs to be mutated to record the current state of the Events It actually could be that even a smaller sized cluster could get into this same situation if there were a bazillion tiny Pods (e.g. not directly correlated with the physical size of the cluster) but since the limits say one cannot have more than 110 Pods per Node, I'm guessing the Event magnification is easier to see with physically wider clusters
      - nh2 1y ago

_bohm 1y ago

Most database systems are designed to amortize fsyncs when they have high write throughput. You want every write to be fsync'd, but you don't want to actually call fsync for each individual write operation.