2 ms·
> they are operating a real time service that requires near perfect uptime 24/7 and work across multiple timezones in multiple languages, payment methods, and c
by thaumasiotes 7y ago
> they are operating a real time service that requires near perfect uptime 24/7 and work across multiple timezones in multiple languages, payment methods, and currencies
Uber is present around the world, but I don't see much reason Uber India needs to interact with Uber North America in real time. Operationally, they're completely independent -- no one is going to connect a ride in Canada to a ride in India -- so why do they need to be strongly coupled in terms of the software service? What would go wrong if Uber India only handled one timezone and didn't really support French?
- danpalmer 7y agoAn Uber competitor, Hailo, did this. Rather than adapting their services to support multiple languages, time zones, etc, they instead just ran a fully separate instance of the service for each region. Their costs, downtime, and tech debt spiralled out of control, mostly due to this architecture decision, and then spent a year or more rebuilding to try to get it under control. Uber probably have the better approach, at least I don’t hear bad things about Uber’s tech where I have heard multiple horror stories about Hailo’s.
- arusahni 7y agoDo you have a source for this? Not questioning the assertion - just interested in the issues and the design process for the solution.
- danpalmer 7y agoI'm afraid I don't have a single source – this is from discussions about tech teams, culture, engineering, etc, at meet-ups in London over the past ~7 years. They had a relatively large engineering team so there are plenty of ex-Hailo devs around here. There were some good things to come out of it, from what I understand. I believe Hailo were avid Cassandra users and many of the team went to Monzo who have deployed Cassandra very effectively.
- marcinzm 7y ago>What would go wrong if Uber India only handled one timezone and didn't really support French? People travel. That aside, you'd have 100+ different deployments that need to be independently updated, debugged, managed, etc. That takes people and effort and is still going to cause issues. For example, each country speaking French might have a different deployment so updating language files needs to take into account every single version running live.
- thaumasiotes 7y ago> People travel. So? People traveling to India are still going to be in India's timezone, and French covers virtually no one that English doesn't also cover.
- chrisseaton 7y agoBeing able to fly into almost any major international city, and then just open the Uber app with my existing account and get a car is a massive benefit of Uber over all competitors and it's the real power of Uber for me. (Lyft requires a US phone number, which I don't have, and local competitors I would have to research, install, sign-up for all individually.)
- geofft 7y agoSlack does (or did?) exactly this, I think. Each Slack workspace is its own deployment: its own web server, its own database, etc. This is loosely why accounts aren't shared across workspaces, you need to open a new tab per workspace, and so on. What this forces you to do is to get really good at configuration management and orchestrating deploys, so that when something goes wrong, you can fix it for all your customers at once. You can't treat any single instance as a thing that it's meaningful to log into (you might still want to log into things to debug problems after a customer report, but making changes is infeasible, and ideally you notice problems before the customer report so you have to be aggregating logs/events/metrics). So many of the same maturity lessons as required for building a single scalable instance continue to apply to you. The distinction is that you don't have a giant distributed system, and you don't have the corresponding operational burden of a giant distributed system, so you avoid outages like https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on-july-29th https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... . (To be clear - this isn't meant to shame Monzo, that was a genuinely difficult problem and a genuinely good incident response. But there are advantages in not having a single company-wide database that runs everything from your core business function to your customer support chat.) One of the engineering lessons I've been learning of late is you really want to avoid distributed systems wherever possible. They're cool, but the amount of engineering / operational effort required to get to a certain level of consistency or availability is way more than the effort required to get to that level of availability of a non-distributed system. It's fun as an engineer to solve hard problems, but it's better for the business to not have hard problems in the first place.