4 ms·

What's the general pattern for making sharded applications where each machine needs to be distinguishable on that count? For instance, say you're sharding acro

by arjie 10y ago

What's the general pattern for making sharded applications where each machine needs to be distinguishable on that count? For instance, say you're sharding across three machines. And now you want to up that to four. Each instance of the application is smart enough to do it provided it knows which shard it's in. We deploy fully baked AMIs containing an application. Do you make separate AMIs per shard? Or do you supply configuration afterwards? Or do you template and generate your plan file each time? I haven't found anything on this so I'm hoping someone can tell me.

jbkkd 10y ago
I'm guessing there is a finite number machine types, and they are well defined and do not change often? What you could do is generate a different packer configuration for each of those types (guessing you use packer since you mentioned AMIs), which will create a different image for each machine type. You can scale each type individually. Happy to help if you need specific advice.
- arjie 10y ago
  Ah, I wasn't terribly clear. Each node is actually the same machine type. It's something like Voldemort with partition-aware routing at the client[0]. Essentially, there's a small amount of configuration to determine which partitions a specific node should serve. Do people usually just build k AMIs if they have k partition groups that they want to serve? For a sort of simplistic version of the problem, imagine the data was splittable into 6 partitions and you hashmod the partitions across 3 groups. So instances in group 1 will serve partitions 0,4; instances in group 2 will server partitions 1,5; and so on. Does one build an AMI for group 1, one for group 2, and one for group 3? Does one provide configuration informing which group a specific node should serve through other means? Or does one stick that in a Terraform definition? 0: https://www.project-voldemort.com/voldemort/design.html https://www.project-voldemort.com/voldemort/design.html
  - jon-wood 10y ago
    Personally I'd build a single AMI with a startup script that extracts any configuration for things like which partition group the instance is in from user data (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-insta...)
    - arjie 10y ago
      Thank you. It seems to me like you'd need variation in your Terraform file to specify that. How do you do that? The DSL doesn't have control structures.
      - arjie 10y ago
        I should have looked at this more carefully. You can use `element` and `count.index` to walk across a list of things.
skMed 10y ago
I would prefer the image to look identical. Each host is going to have a unique name, and you're going to outsource your routing/discovery to the Client/another service anyway right? Generally though, if there is some custom config data that must be passed on BOOTSTRAPPING the node, you can use templated values for user-data on cloud-init based systems, or the remote_exec provisioner. This can even include the modified shard name (with count). If you're talking about updates AFTER the machine has been in service, I do not believe this is 'officially' supported by Terraform but it can be done [0]. You may want to look into a proper CM tool w/ idempotency for that sort of thing (Chef, Salt, etc.) 0: http://stackoverflow.com/questions/37865979/terraform-how-to-run-the-provisioner-on-existing-resources http://stackoverflow.com/questions/37865979/terraform-how-to...
- arjie 10y ago
  Oh, no, definitely the former approach. All right. That sounds perfectly all right to me. I was wondering if there was a different way people did things.

by arjie 10y ago

jbkkd 10y ago

I'm guessing there is a finite number machine types, and they are well defined and do not change often? What you could do is generate a different packer configuration for each of those types (guessing you use packer since you mentioned AMIs), which will create a different image for each machine type. You can scale each type individually. Happy to help if you need specific advice.

arjie 10y ago
Ah, I wasn't terribly clear. Each node is actually the same machine type. It's something like Voldemort with partition-aware routing at the client[0]. Essentially, there's a small amount of configuration to determine which partitions a specific node should serve. Do people usually just build k AMIs if they have k partition groups that they want to serve? For a sort of simplistic version of the problem, imagine the data was splittable into 6 partitions and you hashmod the partitions across 3 groups. So instances in group 1 will serve partitions 0,4; instances in group 2 will server partitions 1,5; and so on. Does one build an AMI for group 1, one for group 2, and one for group 3? Does one provide configuration informing which group a specific node should serve through other means? Or does one stick that in a Terraform definition? 0: https://www.project-voldemort.com/voldemort/design.html https://www.project-voldemort.com/voldemort/design.html
- jon-wood 10y ago
  Personally I'd build a single AMI with a startup script that extracts any configuration for things like which partition group the instance is in from user data (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-insta...)
  - arjie 10y ago
    Thank you. It seems to me like you'd need variation in your Terraform file to specify that. How do you do that? The DSL doesn't have control structures.
    - arjie 10y ago
      I should have looked at this more carefully. You can use `element` and `count.index` to walk across a list of things.

skMed 10y ago

I would prefer the image to look identical. Each host is going to have a unique name, and you're going to outsource your routing/discovery to the Client/another service anyway right? Generally though, if there is some custom config data that must be passed on BOOTSTRAPPING the node, you can use templated values for user-data on cloud-init based systems, or the remote_exec provisioner. This can even include the modified shard name (with count). If you're talking about updates AFTER the machine has been in service, I do not believe this is 'officially' supported by Terraform but it can be done [0]. You may want to look into a proper CM tool w/ idempotency for that sort of thing (Chef, Salt, etc.) 0: http://stackoverflow.com/questions/37865979/terraform-how-to-run-the-provisioner-on-existing-resources http://stackoverflow.com/questions/37865979/terraform-how-to...

arjie 10y ago
Oh, no, definitely the former approach. All right. That sounds perfectly all right to me. I was wondering if there was a different way people did things.