5 ms·

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's g

I work on an uber-like system, but with ~3 backend devs rather than 100s. We made the opposite decision, cloning our full stack for each new market. That's great for scalability, but is a nightmare for devops. If anything we want to find a way to move to one global system, And then slice down the bits that can be local: Eg. Create a local order matching service, but keep orders, payments, and user accounts global

brianwawok 10y ago
If you are doing devops, wouldn't deploying to 1 data center vs 20 be no more work? It's just a loop around a script.
- danpalmer 10y ago
  All it takes is for one server in one datacenter to be slightly different, or perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it, now you've got a deploy that will be a lot more complicated or error-prone than a loop around deployment to one location.
  - icebraining 10y ago
    All it takes is for one server in one datacenter to be slightly different But doesn't a global system still run on multiple DCs, at least for redundancy? perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it But if you have a single global system, you can't even make that decision. To be clear, I'm not arguing against it, I'm just trying to understand why, since my first instinct would be to divide geographically as well.
    - danpalmer 10y ago
      > But doesn't a global system still run on multiple DCs, at least for redundancy? If you've got 2 levels of separation - servers and 'groups' (whether they are datacenters, or whatever) - you've got 2 levels at which that special casing needs to happen. If you only have 1 level - servers - i.e. one deployment, even if that's across multiple datacenters, you only have 1 place to special case. I'd say that's easier. > But if you have a single global system, you can't even make that decision. Good point, but my point was that it will be simpler and less error prone in general. You might not be able to push the bugfix, or you might have to risk the deploy globally, but I think either would be better in the long run for a simpler deployment. It is a trade-off though.
  - brianwawok 10y ago
    > All it takes is for one server in one datacenter to be slightly different Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. > perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it Feature flags. Default off, but flip on a new path of code for a set of users. You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit). Not saying these are simple things to do, but if you are approaching servers with a devops mindsets, you literally should not care about number of servers or datacenters.
    - matwood 10y ago
      > Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. Yep, this is change control 101. I used to have a boss who would go in and edit sprocs on the production server and never get them into source control. I finally just encrypted everything on the server so the only way to push out new code was to commit to source control. A sledge hammer yes, but sometimes it is required. Use Terraform or some other tool to codify your entire infrastructure. >You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit). This is a great point. At first it is terrifying, but then you realize that a deployment is so easy that bugs are generally not a big deal. Bugs happen, so the goal should be to shorten the time between bug found and fix deployed.
    - krinchan 10y ago
      This is so weird. I hear this all the time. At my place of employment, we all have the ssh keys into our EC2 instances, but no one configures them. Ever. Period. Those ssh keys are purely for either validating changes in a test environment (like to .ebextensions) or diagnosing production issues (why did Puma fall over this time? why isn't syslog output making it to loggly?). Of course, we lean heavily on Elastic Beanstalk; autoscaling regularly kills old instances daily since we scale from 2 to 18 and back to 2 instances in a 24 hour period across about 9 microservices. So, if this ssh into boxes and change things is common, it means people aren't doing auto-scaling? THAT is scary.
- matthewrudy 10y ago
  There's the other aspect too, which is each country has their own cluster, so they can (and do) think of excuses to hold back updates just for them. So software gets out of sync, the mobile app releases need to take that into account too.
danpalmer 10y ago
As far as I'm aware, Hailo (>3 backend devs, not quite 100s) did exactly this as well, and the ex-Hailo devs I've spoken to considered it a pretty bad move. It took them ages to refactor into a global system if I remember rightly.
- deleted 10y ago
  [deleted]
- matthewrudy 10y ago
  Yep, I had a long chat with one of their engineers (Matt Heath) and seems like they had the exact same problems we have. And solved it with their restructure and move to microservices.
BenPlanetary 10y ago
Your user accounts aren't global? So does that mean if I travel to a new city I have to create a whole new account?
- steven2012 10y ago
  I've traveled to a dozen cities with Uber and didn't have to change my account, even in China. So the answer is no.
  - dbbk 10y ago
    They said they were an Uber-like system, not Uber itself.
- matthewrudy 10y ago
  Surprisingly although we have user accounts implemented, most of our customers don't use them. But yes, as it stands, if you create a user account it is distinct per country.
  - praneshp 10y ago
    Why would having federated logins be difficult to implement?
    - matthewrudy 10y ago
      It's just not important enough to do. Startup resources, business priorities, customer behaviour...
      - dbbk 10y ago
        Surely customers travelling to new cities/countries they don't know is significant behaviour?
tupshin 10y ago
Recipe: Create a global Cassandra cluster with regional datacenters. Use one keyspace per region Use per-keyspace replication to only replicate that region's data locally, and to one or more additional datacenters Have stateless app servers colocated with Cassandra in each DC handling all local traffic Run spark on top of Cassandra to do analytics, or to do the etl to a dedicated analytics system Optionally have a single "master" DC, with replicas of all data from all locations, that doesn't serve end user traffic, but is to allow efficient cross region analytics. Profit (optional step) And yes, the company I work for (Datastax) has a product and services to help make it simple.
- scaleout1 10y ago
  Thats an interesting approach. One question though, in our use case user often travel from city to city and country to country. How do you model that if you are only using local DC and local replications?
  - methehack 10y ago
    I think the regional keyspaces would be have to be caches -- denormalize it, basically. Pop/re-fresh people into the geo-based caches as they moved around. Truth sits behind it, centralized (perhaps partitioned in some way that makes sense globally but is sub-optimal from a regional cache perspective). Might not be worth it -- hard to know from here. :)
    - ajslater 10y ago
      afaik, this is how Facebook does it, but with regional sources of truth. If you signed up for FB in Paris and move to San Francisco, your master profile lives in Europe in perpetuity and you'll use your regional cache forever in the USA. The number of people moving far away from their home DC's should be a reasonably small fraction of the total for it not to matter.

brianwawok 10y ago

If you are doing devops, wouldn't deploying to 1 data center vs 20 be no more work? It's just a loop around a script.

danpalmer 10y ago
All it takes is for one server in one datacenter to be slightly different, or perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it, now you've got a deploy that will be a lot more complicated or error-prone than a loop around deployment to one location.
- icebraining 10y ago
  All it takes is for one server in one datacenter to be slightly different But doesn't a global system still run on multiple DCs, at least for redundancy? perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it But if you have a single global system, you can't even make that decision. To be clear, I'm not arguing against it, I'm just trying to understand why, since my first instinct would be to divide geographically as well.
  - danpalmer 10y ago
    > But doesn't a global system still run on multiple DCs, at least for redundancy? If you've got 2 levels of separation - servers and 'groups' (whether they are datacenters, or whatever) - you've got 2 levels at which that special casing needs to happen. If you only have 1 level - servers - i.e. one deployment, even if that's across multiple datacenters, you only have 1 place to special case. I'd say that's easier. > But if you have a single global system, you can't even make that decision. Good point, but my point was that it will be simpler and less error prone in general. You might not be able to push the bugfix, or you might have to risk the deploy globally, but I think either would be better in the long run for a simpler deployment. It is a trade-off though.
- brianwawok 10y ago
  > All it takes is for one server in one datacenter to be slightly different Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. > perhaps you had a bugfix that needed to go out for users in one area, but you couldn't take the risk of a flaky deploy for the areas that didn't need it Feature flags. Default off, but flip on a new path of code for a set of users. You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit). Not saying these are simple things to do, but if you are approaching servers with a devops mindsets, you literally should not care about number of servers or datacenters.
  - matwood 10y ago
    > Don't do that. No one is allowed to ssh to boxes. If you need to enforce it by blowing up and rebuilding all servers once per week, do that. Yep, this is change control 101. I used to have a boss who would go in and edit sprocs on the production server and never get them into source control. I finally just encrypted everything on the server so the only way to push out new code was to commit to source control. A sledge hammer yes, but sometimes it is required. Use Terraform or some other tool to codify your entire infrastructure. >You should deploy so often that it is routine. Deploy 50 times a day. You will find bugs at first, but eventually you should get to the point that you could deploy for every single commit and no one would notice. (Now they may not be a good idea depending on your risk tolerance and other things, but you should be ABLE to deploy every single commit). This is a great point. At first it is terrifying, but then you realize that a deployment is so easy that bugs are generally not a big deal. Bugs happen, so the goal should be to shorten the time between bug found and fix deployed.
  - krinchan 10y ago
    This is so weird. I hear this all the time. At my place of employment, we all have the ssh keys into our EC2 instances, but no one configures them. Ever. Period. Those ssh keys are purely for either validating changes in a test environment (like to .ebextensions) or diagnosing production issues (why did Puma fall over this time? why isn't syslog output making it to loggly?). Of course, we lean heavily on Elastic Beanstalk; autoscaling regularly kills old instances daily since we scale from 2 to 18 and back to 2 instances in a 24 hour period across about 9 microservices. So, if this ssh into boxes and change things is common, it means people aren't doing auto-scaling? THAT is scary.
matthewrudy 10y ago
There's the other aspect too, which is each country has their own cluster, so they can (and do) think of excuses to hold back updates just for them. So software gets out of sync, the mobile app releases need to take that into account too.

danpalmer 10y ago

As far as I'm aware, Hailo (>3 backend devs, not quite 100s) did exactly this as well, and the ex-Hailo devs I've spoken to considered it a pretty bad move. It took them ages to refactor into a global system if I remember rightly.

deleted 10y ago
[deleted]
matthewrudy 10y ago
Yep, I had a long chat with one of their engineers (Matt Heath) and seems like they had the exact same problems we have. And solved it with their restructure and move to microservices.

BenPlanetary 10y ago

Your user accounts aren't global? So does that mean if I travel to a new city I have to create a whole new account?

steven2012 10y ago
I've traveled to a dozen cities with Uber and didn't have to change my account, even in China. So the answer is no.
- dbbk 10y ago
  They said they were an Uber-like system, not Uber itself.
matthewrudy 10y ago
Surprisingly although we have user accounts implemented, most of our customers don't use them. But yes, as it stands, if you create a user account it is distinct per country.
- praneshp 10y ago
  Why would having federated logins be difficult to implement?
  - matthewrudy 10y ago
    It's just not important enough to do. Startup resources, business priorities, customer behaviour...
    - dbbk 10y ago
      Surely customers travelling to new cities/countries they don't know is significant behaviour?

tupshin 10y ago

Recipe: Create a global Cassandra cluster with regional datacenters. Use one keyspace per region Use per-keyspace replication to only replicate that region's data locally, and to one or more additional datacenters Have stateless app servers colocated with Cassandra in each DC handling all local traffic Run spark on top of Cassandra to do analytics, or to do the etl to a dedicated analytics system Optionally have a single "master" DC, with replicas of all data from all locations, that doesn't serve end user traffic, but is to allow efficient cross region analytics. Profit (optional step) And yes, the company I work for (Datastax) has a product and services to help make it simple.

scaleout1 10y ago
Thats an interesting approach. One question though, in our use case user often travel from city to city and country to country. How do you model that if you are only using local DC and local replications?
- methehack 10y ago
  I think the regional keyspaces would be have to be caches -- denormalize it, basically. Pop/re-fresh people into the geo-based caches as they moved around. Truth sits behind it, centralized (perhaps partitioned in some way that makes sense globally but is sub-optimal from a regional cache perspective). Might not be worth it -- hard to know from here. :)
  - ajslater 10y ago
    afaik, this is how Facebook does it, but with regional sources of truth. If you signed up for FB in Paris and move to San Francisco, your master profile lives in Europe in perpetuity and you'll use your regional cache forever in the USA. The number of people moving far away from their home DC's should be a reasonably small fraction of the total for it not to matter.