5 ms·

Alrighty, a few questions: - what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch? - have y

by ypcx 7y ago

Alrighty, a few questions: - what k8s definitions do you use, e.g. do you use the official Helm Chart, or have you written your .yaml's from scratch? - have you practiced disaster recovery scenarios in the context of k8s? Can you describe them briefly? - how do you upgrade/redeploy the Pulsar k8s components, i.e. does this cause the Bookies to trigger a cluster rebalance, or does it trigger the Autorecovery - for the Bookies, do you use AWS EBS volumes with the EKS or just local instance storage (that is, if you use persistent topics) - do you use the Proxy pod's EKS k8s pod IPs as exposed on the AWS network, or do you use a NodePort type of service for the Proxy components (using the EKS node IPs) - have you been bitten by the recent EKS k8s network plugin bug (loss of pod connectivity), and/or how do you maintain your EKS cluster - do you run your EKS nodes in a multi-AZ setting?

addisonj 7y ago
for the k8s definitions, we started with the helm chart, rendered the template, and then moved it into kustomize, as that is our tool of choice ATM, IDK if I would recommend that approach for everyone (we expect we might move to helm v3 at some point) but it was a good choice for us. We have practiced some disaster recovery, but it isn't 100% exhaustive (is it ever?), however it is also aided by how Pulsar is designed. We have killed bookie nodes as well as lost all our state in zookeeper. The first is pretty easily handled by the replication factor of bookkeeper data and for zookeeper we do extra backup step and just dump the state to s3 and can restore it. What we haven't tested in practice but now how to do theoretically is to restore a k8s stateful set from EBS volume snapshots. However, we see that as a real edge case. In Pulsar, we offload our data to s3 after a few hours, so we only need to worry about potentially losing a few hours of data in BK, as the zookeeper state is very easy to just snapshot and restore from s3. In other words, we are still working on getting more and more confident with data and don't yet recommend teams use it for mission critical non-recoverable data, but there are a ton of uses cases for it now and we can continue to improve on the DR front We have done multiple upgrades and deploy all the time. Because bookkeeper nodes are in a stateful set and we have don't do automated rollouts, we manually have a process to replace the BK nodes. However, they don't trigger a re-balance as it closes gracefully and then re-attaches the EBS volume from the stateful set We use EBS volumes, we use a piops volumes for the journal and a larger slower volume for the ledger store. THis is one of the great parts of bookkeeper design is that the two disks pools are separate so we just need a small chunk of really fast storage and then the journaled data is copied over to the ledger volume by a background process. We figure for really high write throughput we could use instance storage for the journal volume and EBS for ledger, but that would have some complications on recovery but still easier than having to rebuild the whole ledger data. We use the pulsar proxy and expose it via a k8s service with the AWS specific NLB annotations. We haven't had any issues with the k8s plugin and haven't really had any issues with EKS version upgrades. We just add new nodes when we migrate the kubelets Yes, we have automation (via terraform) to allow us to add many different pools of compute and we use labels and taints to get specific apps mapped to specific pools of compute. For Pulsar, we run all the components multi-AZ
addisonj 7y ago
Oh forgot one aspect about DR: for critical data, we can easily turn on geo-replication (with a single API call) and have that data now in another region purely for DR purposes (or for cross region use cases)
happythought 7y ago
Do you have a link to the eks networking bug?
- ypcx 7y ago
  AWS was sending "ACTION REQUIRED" email to all EKS users about this. The Github issue is: https://github.com/aws/amazon-vpc-cni-k8s/issues/641 https://github.com/aws/amazon-vpc-cni-k8s/issues/641

by ypcx 7y ago

addisonj 7y ago

for the k8s definitions, we started with the helm chart, rendered the template, and then moved it into kustomize, as that is our tool of choice ATM, IDK if I would recommend that approach for everyone (we expect we might move to helm v3 at some point) but it was a good choice for us. We have practiced some disaster recovery, but it isn't 100% exhaustive (is it ever?), however it is also aided by how Pulsar is designed. We have killed bookie nodes as well as lost all our state in zookeeper. The first is pretty easily handled by the replication factor of bookkeeper data and for zookeeper we do extra backup step and just dump the state to s3 and can restore it. What we haven't tested in practice but now how to do theoretically is to restore a k8s stateful set from EBS volume snapshots. However, we see that as a real edge case. In Pulsar, we offload our data to s3 after a few hours, so we only need to worry about potentially losing a few hours of data in BK, as the zookeeper state is very easy to just snapshot and restore from s3. In other words, we are still working on getting more and more confident with data and don't yet recommend teams use it for mission critical non-recoverable data, but there are a ton of uses cases for it now and we can continue to improve on the DR front We have done multiple upgrades and deploy all the time. Because bookkeeper nodes are in a stateful set and we have don't do automated rollouts, we manually have a process to replace the BK nodes. However, they don't trigger a re-balance as it closes gracefully and then re-attaches the EBS volume from the stateful set We use EBS volumes, we use a piops volumes for the journal and a larger slower volume for the ledger store. THis is one of the great parts of bookkeeper design is that the two disks pools are separate so we just need a small chunk of really fast storage and then the journaled data is copied over to the ledger volume by a background process. We figure for really high write throughput we could use instance storage for the journal volume and EBS for ledger, but that would have some complications on recovery but still easier than having to rebuild the whole ledger data. We use the pulsar proxy and expose it via a k8s service with the AWS specific NLB annotations. We haven't had any issues with the k8s plugin and haven't really had any issues with EKS version upgrades. We just add new nodes when we migrate the kubelets Yes, we have automation (via terraform) to allow us to add many different pools of compute and we use labels and taints to get specific apps mapped to specific pools of compute. For Pulsar, we run all the components multi-AZ

addisonj 7y ago

Oh forgot one aspect about DR: for critical data, we can easily turn on geo-replication (with a single API call) and have that data now in another region purely for DR purposes (or for cross region use cases)

happythought 7y ago

Do you have a link to the eks networking bug?

ypcx 7y ago
AWS was sending "ACTION REQUIRED" email to all EKS users about this. The Github issue is: https://github.com/aws/amazon-vpc-cni-k8s/issues/641 https://github.com/aws/amazon-vpc-cni-k8s/issues/641