6 ms·

The architecture of Uber’s API gateway

148 points by zerop 5y ago

deleted 5y ago
[deleted]
alufers 5y ago
This GUI must have a pretty advanced versioning and rollback support, otherwise I can see one user borking the whole API with a bad change and nowwhere to check what happened.
- deleted 5y ago
  [deleted]
- throwawayuber29 5y ago
  Former Uber employee who worked on this system. It maps changes onto git changes (technically Phabricator diffs) and this system open the diff for you.
  - madan 5y ago
    Just to add, the changes are just config changes at this point and not code changes. One one user talking own all APIs, the system has ability to skip unmountable APIs, but we catch it during user interactions with tons of test and validations.
- menzoic 5y ago
  You only change one endpoint at a time. Fixes are rollforward since you would be reverting changes to other endpoints if you rolled back.
- junon 5y ago
  It did not. The deployment system at Uber was a f*ing nightmare. Things would fail, rollback, and then the logs would have their errors truncated or something. I wasted so many days deploying botched releases from coworkers. And the use of Phabricator at Uber was a nightmare. LLVM uses it correctly, IDK what Uber did but it was a PITA to do really anything. Pair that with the siloed off teams where "every team is its own startup" mentality and you have constant fighting, power grabs, being blocked all the time, etc.
  - Aeolun 5y ago
    Look at it positively. At least the rollback is working. Our version of rollback is to redeploy the old version and hope that the database migrations will work with the old one.
    - junon 5y ago
      That was absolutely a problem at Uber, too.
timhwang21 5y ago
Enjoying the article so far, but some things make it a bit hard to read. 1) code snippets are screen shots, and 2) some links point to an internal Google Docs page.
- abhishekparwal 5y ago
  oops.. thanks for sharing this. we will get that corrected.
madan 5y ago
One of the authors here, happy to answer questions.
- bilal4hmed 5y ago
  The issues that you had with Go 'Language naming conventions like ID, HTTP, and reserved keywords in Go (but not in Thrift) created failures that exposed the internal implementation details to the end users.' How did you go about solving those ? Did yall work with the Go team to resolve the other issues that were stumbled upon?
  - madan 5y ago
    Since the final artifact was generated, we were able to work around by annotating in thrift with alternate field names.
    - exhaze 5y ago
      Just make sure to not accidentally add an extra space in the Thrift annotation, or else it will globally bring down upfront pricing
- DandyDev 5y ago
  Did you consider using existing API Gateway solutions, either OSS or commercial? Why did you decide to build your own? Given that Uber is an engineering heavy and tech-centric organization, why did you choose to do configuration through the UI? Why not configuration and infrastructure as code?
tiwarivikash 5y ago
Which technologies have been used to Implement it? Framework? Language?
- nuclearnice1 5y ago
  > At the time of development of the gateway, our language choices were Go and Java. Our previous generation was in Node.js. While that was a very suitable language for building an IO-heavy gateway layer, we decided to align with the languages supported by the language platform teams at Uber. Go provided significant performance improvements. The lack of generics resulted in a significant amount of generated code during build time to a point where we were hitting the limits of the Go linker. We had to turn off the symbol table and debug information during the binary compilation. Language naming conventions like ID, HTTP, and reserved keywords in Go (but not in Thrift) created failures that exposed the internal implementation details to the end users.
  - abhishekparwal 5y ago
    One of the author this blog post. We had to do lot of other scaling optimizations like generating code for only IDL elements used rather than a big fat IDL, replacing ser/deser generated code with dynamic generation of code, etc. We will share details in further blog post on that.
taf2 5y ago
This is probably harsh but the description of their api gateway sounds like the description of corba from the late 90s… probably if you are just getting started and need something simple but powerful go with openresty… IMO it gives you all the benefits and it’s super fast/lightweight … you can get advanced with cookie sharing / signing or just do simple logging to statsd… it’s really good if you are starting out to get operational experience running nginx / statsd etc… IMO
- dtauzell 5y ago
  The people defining CORBA faced many of e same issues we face with micro services today.
  - TeMPOraL 5y ago
    After working with Distributed COM, I'd go as far as to say that people building DCOM software were doing microservices, way before that term was known or popular - microservices at much finer granularity (per object), and all the bells&whistles people make startups off today - like load balancing, service discovery, strong auth, etc. - were already built into the platform and properly integrated. Alas, it was ahead of its time; the legacy of C makes it hard to work with.
    - barrkel 5y ago
      Eh, but stateful and synchronous and chatty. I don't think DCOM has too many good ideas, beyond what it shares with CORBA - mostly, an IDL. It's good to have an IDL if you're going to be serious about heterogeneous implementations.
- plank_time 5y ago
  CORBA never scaled to the level that Uber needs to. Pre-pandemic, you’re talking hundreds of thousands to millions of requests per second globally. Also you never saw a CORBA project with thousands of engineers checking in code multiple times a day. All these are factors too. People on HN always think “all it’s doing is matching a rider with a driver, why is it so complicated? I can write it in a weekend!” Sure, but it won’t scale.
RKearney 5y ago
My last experience with Uber was horrendous. I had scheduled a trip for early the next morning. The app seemed to error out and didn't give me any conformation that my trip was scheduled or confirmed. I went to try again and received an error message that I already had a trip scheduled at that time. It's virtually impossible to contact Uber customer service. The best I found was some in-app chat which I had to lie to get to as their support options had no selection for what my issue was. I then gave up after the 5th or so canned response I received asking me to all but wipe my iPhone and return it to Apple. Never mind the fact that the issue also existed on uber.com and was reproducible. That was the last trip I ever used Uber for. It's one thing to have bugs in your product, but it's something else completely when the company does everything in their power to make no attempt whatsoever to understand or even entertain the thought that maybe, just maybe, it's not user error this time.
- tptacek 5y ago
  This has nothing at all to do with the article.
  - RKearney 5y ago
    I disagree. The article is about Uber’s API. My comment is about how Uber’s API clearly doesn’t have robust support for requests that did not fully execute. Based on my experience it appeared as though one system had processed my request to schedule the trip, but the rest of the system did not. This caused my account to essentially be soft locked by a trip that was not visible through the app (and by extension their API). I then added my antidotal experience with their lackluster support.
deleted 5y ago
[deleted]
0xbadcafebee 5y ago
It's a decent overview but you could easily do a whole blog series on each category of the gateway that they touch on. Is there a deeper dive for each part? Like, just the AuthNZ could get very complicated depending on requirements (as they have multiple implementations, that causes a mini crisis of what to rely on for what and how)
mvzvm 5y ago
I worked at Uber for a few years. This was a terrible layer. Everyone hated it when I was there. People ended up building logic directly into the API gateway because it was so difficult to use. I am so glad to never have to look at RTAPI again.
- deleted 5y ago
  [deleted]
- menzoic 5y ago
  This article is about the newer Edge Gateway and doesn't mention anything about RTAPI. When did you leave Uber?
  - mvzvm 5y ago
    RTAPI is the internal name. Why would it be mentioned here? I left too late. The engineering in that company was abysmal.
    - ojhughes 5y ago
      RTAPI is mentioned in the previous blog post [1], it sounds like the new edge gateway supercedes it [1] https://eng.uber.com/gatewayuberapi/ https://eng.uber.com/gatewayuberapi/
      - mvzvm 5y ago
        Nice find. I stand corrected.
xiwenc 5y ago
Was existing API management solutions like Mulesoft considered? And while on the reuse topic, will Uber open source this?
edwardmp 5y ago
If you guys could easily start over, would you have still picked Go given the problems you encountered? Or would Java have been a more likely candidate?
- papito 5y ago
  Why not just use OOP Scala? It's so much cleaner and more readable.
  - hamandcheese 5y ago
    I tried starting an “OOP Scala” project. The entire Scala ecosystem, at least the parts that seems good and active, all seem to be FP oriented. If you simply want a better Java, I’d try Kotlin. Or Java 15.
bsaul 5y ago
Side question : Anyone knows a gateway solution able to do blue/green deployment ?
- ojhughes 5y ago
  Envoy based gateways such as Istio ingress gateway or Ambassador use traffic shadowing to achieve this. Traefik also has shadowing
papito 5y ago
I think you will find that Uber engineering is just like any other place - a lot of silly mistakes while people learn on the company dime. This entry, for example, mentions how they avoided Go routines as a "performance concern" even without any data to prove it. It's bush league level to think you can do it better yourself. https://eng.uber.com/go-geofence-highest-query-per-second-service/ https://eng.uber.com/go-geofence-highest-query-per-second-se...
- thundergolfer 5y ago
  That post you linked is the one that got heavily criticised by a Bing Maps engineer for being under-engineered[1]. 1. https://medium.com/@buckhx/unwinding-uber-s-most-efficient-service-406413c5871d https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...
  - Aeolun 5y ago
    While the linked post maybbe right. I completely agree with the Uber engineers that it sounds too complicated. I can understand the double polygon search without thinking about it. I also understand the double polygon search with bounding box optimization, so I’m not sure why that wasn’t used.
    - Closi 5y ago
      If that’s the case, it’s weird that they would take the time to write a blog post about how great their algorithm is while effectively saying that they didn’t understand the other approaches and that they were too hard.

148 points by zerop 5y ago

deleted 5y ago

[deleted]

This GUI must have a pretty advanced versioning and rollback support, otherwise I can see one user borking the whole API with a bad change and nowwhere to check what happened.

deleted 5y ago
[deleted]
throwawayuber29 5y ago
Former Uber employee who worked on this system. It maps changes onto git changes (technically Phabricator diffs) and this system open the diff for you.
- madan 5y ago
  Just to add, the changes are just config changes at this point and not code changes. One one user talking own all APIs, the system has ability to skip unmountable APIs, but we catch it during user interactions with tons of test and validations.
menzoic 5y ago
You only change one endpoint at a time. Fixes are rollforward since you would be reverting changes to other endpoints if you rolled back.
junon 5y ago
It did not. The deployment system at Uber was a f*ing nightmare. Things would fail, rollback, and then the logs would have their errors truncated or something. I wasted so many days deploying botched releases from coworkers. And the use of Phabricator at Uber was a nightmare. LLVM uses it correctly, IDK what Uber did but it was a PITA to do really anything. Pair that with the siloed off teams where "every team is its own startup" mentality and you have constant fighting, power grabs, being blocked all the time, etc.
- Aeolun 5y ago
  Look at it positively. At least the rollback is working. Our version of rollback is to redeploy the old version and hope that the database migrations will work with the old one.
  - junon 5y ago
    That was absolutely a problem at Uber, too.

timhwang21 5y ago

Enjoying the article so far, but some things make it a bit hard to read. 1) code snippets are screen shots, and 2) some links point to an internal Google Docs page.

abhishekparwal 5y ago
oops.. thanks for sharing this. we will get that corrected.

madan 5y ago

One of the authors here, happy to answer questions.

bilal4hmed 5y ago
The issues that you had with Go 'Language naming conventions like ID, HTTP, and reserved keywords in Go (but not in Thrift) created failures that exposed the internal implementation details to the end users.' How did you go about solving those ? Did yall work with the Go team to resolve the other issues that were stumbled upon?
- madan 5y ago
  Since the final artifact was generated, we were able to work around by annotating in thrift with alternate field names.
  - exhaze 5y ago
    Just make sure to not accidentally add an extra space in the Thrift annotation, or else it will globally bring down upfront pricing
DandyDev 5y ago
Did you consider using existing API Gateway solutions, either OSS or commercial? Why did you decide to build your own? Given that Uber is an engineering heavy and tech-centric organization, why did you choose to do configuration through the UI? Why not configuration and infrastructure as code?

tiwarivikash 5y ago

Which technologies have been used to Implement it? Framework? Language?

nuclearnice1 5y ago
> At the time of development of the gateway, our language choices were Go and Java. Our previous generation was in Node.js. While that was a very suitable language for building an IO-heavy gateway layer, we decided to align with the languages supported by the language platform teams at Uber. Go provided significant performance improvements. The lack of generics resulted in a significant amount of generated code during build time to a point where we were hitting the limits of the Go linker. We had to turn off the symbol table and debug information during the binary compilation. Language naming conventions like ID, HTTP, and reserved keywords in Go (but not in Thrift) created failures that exposed the internal implementation details to the end users.
- abhishekparwal 5y ago
  One of the author this blog post. We had to do lot of other scaling optimizations like generating code for only IDL elements used rather than a big fat IDL, replacing ser/deser generated code with dynamic generation of code, etc. We will share details in further blog post on that.

taf2 5y ago

This is probably harsh but the description of their api gateway sounds like the description of corba from the late 90s… probably if you are just getting started and need something simple but powerful go with openresty… IMO it gives you all the benefits and it’s super fast/lightweight … you can get advanced with cookie sharing / signing or just do simple logging to statsd… it’s really good if you are starting out to get operational experience running nginx / statsd etc… IMO

dtauzell 5y ago
The people defining CORBA faced many of e same issues we face with micro services today.
- TeMPOraL 5y ago
  After working with Distributed COM, I'd go as far as to say that people building DCOM software were doing microservices, way before that term was known or popular - microservices at much finer granularity (per object), and all the bells&whistles people make startups off today - like load balancing, service discovery, strong auth, etc. - were already built into the platform and properly integrated. Alas, it was ahead of its time; the legacy of C makes it hard to work with.
  - barrkel 5y ago
    Eh, but stateful and synchronous and chatty. I don't think DCOM has too many good ideas, beyond what it shares with CORBA - mostly, an IDL. It's good to have an IDL if you're going to be serious about heterogeneous implementations.
plank_time 5y ago
CORBA never scaled to the level that Uber needs to. Pre-pandemic, you’re talking hundreds of thousands to millions of requests per second globally. Also you never saw a CORBA project with thousands of engineers checking in code multiple times a day. All these are factors too. People on HN always think “all it’s doing is matching a rider with a driver, why is it so complicated? I can write it in a weekend!” Sure, but it won’t scale.

RKearney 5y ago

My last experience with Uber was horrendous. I had scheduled a trip for early the next morning. The app seemed to error out and didn't give me any conformation that my trip was scheduled or confirmed. I went to try again and received an error message that I already had a trip scheduled at that time. It's virtually impossible to contact Uber customer service. The best I found was some in-app chat which I had to lie to get to as their support options had no selection for what my issue was. I then gave up after the 5th or so canned response I received asking me to all but wipe my iPhone and return it to Apple. Never mind the fact that the issue also existed on uber.com and was reproducible. That was the last trip I ever used Uber for. It's one thing to have bugs in your product, but it's something else completely when the company does everything in their power to make no attempt whatsoever to understand or even entertain the thought that maybe, just maybe, it's not user error this time.

tptacek 5y ago
This has nothing at all to do with the article.
- RKearney 5y ago
  I disagree. The article is about Uber’s API. My comment is about how Uber’s API clearly doesn’t have robust support for requests that did not fully execute. Based on my experience it appeared as though one system had processed my request to schedule the trip, but the rest of the system did not. This caused my account to essentially be soft locked by a trip that was not visible through the app (and by extension their API). I then added my antidotal experience with their lackluster support.

deleted 5y ago

[deleted]

0xbadcafebee 5y ago

It's a decent overview but you could easily do a whole blog series on each category of the gateway that they touch on. Is there a deeper dive for each part? Like, just the AuthNZ could get very complicated depending on requirements (as they have multiple implementations, that causes a mini crisis of what to rely on for what and how)

mvzvm 5y ago

I worked at Uber for a few years. This was a terrible layer. Everyone hated it when I was there. People ended up building logic directly into the API gateway because it was so difficult to use. I am so glad to never have to look at RTAPI again.

deleted 5y ago
[deleted]
menzoic 5y ago
This article is about the newer Edge Gateway and doesn't mention anything about RTAPI. When did you leave Uber?
- mvzvm 5y ago
  RTAPI is the internal name. Why would it be mentioned here? I left too late. The engineering in that company was abysmal.
  - ojhughes 5y ago
    RTAPI is mentioned in the previous blog post [1], it sounds like the new edge gateway supercedes it [1] https://eng.uber.com/gatewayuberapi/ https://eng.uber.com/gatewayuberapi/
    - mvzvm 5y ago
      Nice find. I stand corrected.

xiwenc 5y ago

Was existing API management solutions like Mulesoft considered? And while on the reuse topic, will Uber open source this?

edwardmp 5y ago

If you guys could easily start over, would you have still picked Go given the problems you encountered? Or would Java have been a more likely candidate?

papito 5y ago
Why not just use OOP Scala? It's so much cleaner and more readable.
- hamandcheese 5y ago
  I tried starting an “OOP Scala” project. The entire Scala ecosystem, at least the parts that seems good and active, all seem to be FP oriented. If you simply want a better Java, I’d try Kotlin. Or Java 15.

bsaul 5y ago

Side question : Anyone knows a gateway solution able to do blue/green deployment ?

ojhughes 5y ago
Envoy based gateways such as Istio ingress gateway or Ambassador use traffic shadowing to achieve this. Traefik also has shadowing

papito 5y ago

I think you will find that Uber engineering is just like any other place - a lot of silly mistakes while people learn on the company dime. This entry, for example, mentions how they avoided Go routines as a "performance concern" even without any data to prove it. It's bush league level to think you can do it better yourself. https://eng.uber.com/go-geofence-highest-query-per-second-service/ https://eng.uber.com/go-geofence-highest-query-per-second-se...

thundergolfer 5y ago
That post you linked is the one that got heavily criticised by a Bing Maps engineer for being under-engineered[1]. 1. https://medium.com/@buckhx/unwinding-uber-s-most-efficient-service-406413c5871d https://medium.com/@buckhx/unwinding-uber-s-most-efficient-s...
- Aeolun 5y ago
  While the linked post maybbe right. I completely agree with the Uber engineers that it sounds too complicated. I can understand the double polygon search without thinking about it. I also understand the double polygon search with bounding box optimization, so I’m not sure why that wasn’t used.
  - Closi 5y ago
    If that’s the case, it’s weird that they would take the time to write a blog post about how great their algorithm is while effectively saying that they didn’t understand the other approaches and that they were too hard.