5 ms·
Uber's technology is reportedly 'hanging by a thread'
- nikropht 11y agoThey need EEs like Twitter has.
- agarcia-deniz 11y agoI wonder why the original article wasn't posted rather than the businessinsider article. edit: Paywall. I get it.
- arethuza 11y ago"setting up servers in a new data center on Halloween" That kind of problem seems remarkably common - the end of months and years can be peak times for businesses but it's also the kind of date that people pick for their project milestones ("we'll have the servers in by the end of the year").
- amatix 11y agoor "lets have new capacity in place a month before busy Halloween" ... progress slips... the project manager pushes and everyone forgets the reason for the original date.
- imaginenore 11y agoFrom my perspective, Uber backend can be sharded trivially. No driver in LA is going to get matched with a passenger in NYC.
- charliesome 11y agoI generally find it wise to avoid calling engineering problems faced by other organisations trivial. There are often a lot of complicating factors that aren't obvious from an outside perspective.
- latch 11y agoSure, but it's fun to talk about how you'd build it given some assumptions. I agree with your parent that it seems like a shardable dataset. To boot, I imagine that you could easily store all cars for a city/region in memory (type, x, y, driver id, status, ....). Next I'd grid the area (maybe 250m2?) into blocks. When a user does a search, figure out his or her block, and start looking for cars in the current block, expanding outwards. You read-lock the blocks as you examine them. You only write lock 2 blocks as a car moves from one block to another (the blocks themselves would have a list of cars, maybe an array, maybe an rtree, maybe another grid). It all falls apart when a single server can't handle all the load for a region. But then you could sub-divide the region, connect the servers with a queue (this car is now your responsibility), and let clients (not necessarily devices, but the api servers consuming these data servers) join the data.
- msandford 11y agoIf you can have one server per city then I agree. Especially if those cities are small and you never travel between them. But places like LA totally blow that theory as you're going to have to have multiple servers or shards to cover everywhere.
- latch 11y agoWhy LA, # of drivers, size of the city or # of users? Quick googling says LA is 500 miles squared, with Beijing being 6500. The only case where I see 1 server failing is # of requests. All their drivers in the world probably fit in a few GB of memory (if that). 99% of a car's movement requires no locking (they stay in the same block), so there's very little write locking...I dunno...give it a 24 core server (or more)... Maybe the problem is node and python. I don't know python runtime well enough, but this kind of setup is a nightmare for node. Sharing data across processes just isn't what it was meant to do (it rather fork and have a copy, but then your memory doubles per fork, and you have to keep it in sync). That's true for a lot of dynamic languages.
- 11y ago
- matthewrudy 11y agoLooking at Uber's APIs they're built to be sharded. But sharding comes at the cost of complexity. I work on an uber-like service and we use crude sharding between countries. The problem is: * sometimes your sharding is too coarse - eg. two cities which have no cross traffic * sometimes your sharding is too fine - eg. cross border traffic But the real complexity is; * managing configuration and logic differences between countries and cities * hosting so many different sharded clusters, routing between them, and keeping them all updated. And that's where all the work goes.
- deleted 11y ago[deleted]
- mwhuber 11y agoOne of Uber's architects talking about how they're handling that over via InfoQ http://goo.gl/bAoFHi http://goo.gl/bAoFHi
- sschueller 11y ago"reflected an amateurism with our overall engineering organization, its culture, its processes, and its operation.” That is not going to get the engineers on your side to solve the problem. Instead it will create even more friction between managers and engineers.
- late2part 11y agoOne nice thing about friction is that if it's done correctly, it can rub away the problems. Forcing people to deal with the reality of the situation often forces positive change.
- zipwitch 11y agoIndeed. "Uber Raided By Dutch Authorities, Seen As 'Criminal Organization'" http://yro.slashdot.org/story/15/09/29/2328232/uber-raided-by-dutch-authorities-seen-as-criminal-organization http://yro.slashdot.org/story/15/09/29/2328232/uber-raided-b...
- marrs 11y agoUnless the engineers agree. Even if they don't, if the assessment is true and they can't accept it then those engineers are probably a part of the problem.
- cjslep 11y ago> The engineers in charge of these systems have been "at odds," which has created friction, according to The Information. > Uber’s chief technology officer, Thuan Pham, later wrote to his staff that the mistake “reflects an amateurism with our overall engineering organization, its culture, its processes, and its operation.” This makes it sound as if the two engineers in charge of the Node.js and Python systems are bickering over which technology stack is better and refuse to compromise. I get there is worry about career progression if your backend option loses, the glory of being the engineer in charge of the entire backend of Uber, and the different specs of different technologies. But to hold the entire company hostage seems like it would be a career-ending move to me, regardless of sides. Then again, I am not in management.
- munin 11y ago> But to hold the entire company hostage seems like it would be a career-ending move to me, regardless of sides. It would be, if you lose. If you win, you are rich beyond the dreams of avarice. It would also be a career ending move to admit that your system is less than the other system and everyone will remember you as that loser whose system was done better by the current VP of engineering.
- ericfrederich 11y agoEasy... choose Node and put the Python engineer in charge or choose Python and put the Node guy in charge. Or redo the whole thing in Go.
- munin 11y agoThere is definitely some degree of executive decision their management could make but it would shred one of the two leads careers. This kind of hard call is why you're boss, so it says something about someone when they don't make this call (although we don't know the full story here, this situation happens enough in general). The post I was replying to was asking why an individual lead engineer in this scenario would do this. They have reasons to fight and no reason to concede. Their bosses have every reason to, at minimum, get both sides to "hug it out" so why haven't they?
- dantillberg 11y agoWhat benefits would they see/get from using their own co-located servers instead of using VMs on a cloud provider? For a fast-growing business, it would seem a huge win to not have to worry about physically scaling your infrastructure. And I can't imagine that their infrastructure size is so large (they're not, for example, indexing the internet) that they would get a huge cost savings from using their own hardware. But certainly, I must be missing something?
- latch 11y agoWhat benefit would they get from using cloud? A lot more money for much worse performance. It's like buying a monthly bus pass when you could buy a Honda Civic for less. Despite what Cloud providers want you to think, scaling (for most of us) is largely an architecture, design and software issue and it tends to be rather specific to your system (until you get to the point where you have to build your own). "Auto-scaling" doesn't help you make sure you aren't opening too many forking connections to your database, it doesn't eliminate coarse locks, un-optimized system calls, making sure you take advantage of cache locality, sharding, denormalization or really anything that requires effort. The game changes a bit with the SaaS stuff that PaaS vendors are selling (dynamodb, for example), but then you pay a lock-in price.
- inthewoods 11y agoSome serious link bait in that article title - sounds like they've got issues but they are working through it. The article ends by pointing out that New Years Eve went off without a hitch.
- probablyfiction 11y agoA more accurate title would have been "Uber's new CTO is doing pretty okay," but that doesn't entice people to click nearly as much
- OliverJones 11y agoYeah, this article is in "Business Insider" rather than "Tech Insider." Ya can tell by reading it. The real WTF is having that many developers without a solid ops team to match. At least they can charge extra for peak loads, which many of us cannot do.
- 9872 11y agoUber can't legally do it either. They just don't give a shit and do so anyway.
- icebraining 11y agoWhat laws ban surge pricing?
- kkhire 11y agoEvery industry will have dynamic pricing in the near future, transportation has only just begun to fully embrace it
- usefulcat 11y ago"The company's engineering staff has grown to 1,200 — a quarter of Uber's workforce — from just 400 people." 1200? I'd be very interested to know what they're all doing. Not saying that what the company is doing is easy on that scale, but it's hard to see how throwing a thousand people at the problem can be an effective solution. Unless many of them are working on new products? But Uber seems pretty young to be investing that much in R&D.
- PopeOfNope 11y agoDriverless Cars. They've basically hired the entire engineering department at Carnegie Mellon in Pittsburgh, where their R&D offices are located.