9 ms·

Airbnb Is Moving Faster at Scale with GraphQL and Apollo

202 points by saranshk 8y ago

stockkid 8y ago
It is unclear from the article how exactly Airbnb is moving "10x faster" as the title claims. Unless backed by a benchmark, this kind of hyperbole shouldn't be an engineering article.
EduardoBautista 8y ago
As primarily an Ember.js user (with an interest in Elm), just looking at how many hoops people in the React ecosystem go through just to send a request just boggles the mind. Maybe I am in the minority, but writing boilerplate code all the time just isn't something that interests me.
- testnew2 8y ago
  That boilerplate has absolutely nothing to do with react.
- exogen 8y ago
  I write React every day and don't know what you're referring to – you can just call fetch() in a lifecycle hook and be done with it, just like in any other framework. But mostly I'm curious how your point relates to this blog post. Maybe you can clarify? The stuff they're doing with Apollo here that potentially comes across as boilerplate also gets query batching, caching, refetching, etc. for free (in addition to the fancy stuff they also talk about like automatic mocks). It's not like they're "just sending a request." Maybe you can show me how you do all that in another framework without some setup/wrappers/etc.?
  - EduardoBautista 8y ago
    Things Ember Data's `findRecord` does: * Checks if you already have the resource to prevent sending an unnecessary request (can be forced) * Retrieves the resource based on a convention for the URL * Parses the JSON based on a convention * Stores the resource in the "store" for use in the app Batching is also very simple thanks to a plugin from Netflix: https://github.com/Netflix/ember-batch-request https://github.com/Netflix/ember-batch-request The reason the Ember community doesn't talk about any of this stuff is because, from the developers perspective, nothing interesting happens, which is why developer productivity is high in Ember.
    - exogen 8y ago
      Why would a UI framework know anything about records? Keep in mind that it's explicitly not React's goal to know anything about requests, routing, records, data, or anything like that. That's why you hear React people talking about how they achieve this stuff. Ember is a framework with a much larger scope, while React is a view layer.
      - EduardoBautista 8y ago
        Ember Data can be used separately from Ember itself. In fact, the official documentation treats them as if they are. It's even a separate project on GitHub: https://github.com/emberjs/data https://github.com/emberjs/data
        exogen 8y ago
        If even in Ember-land it's an entirely separate package, then why make the comparison to React at all..? (Ember + Ember Data) is more like (React + Apollo) then, no? But you're just comparing it to React without anything else.
        EduardoBautista 8y ago
        My original comment said "React ecosystem", not just "React". Apollo is part of the ecosystem.
        exogen 8y ago
        Here's what just sending a request looks like with React + Apollo: https://codesandbox.io/s/q71px7njrw https://codesandbox.io/s/q71px7njrw Which boilerplate do you take issue with? There are a couple imports you might find ugly, I suppose. Most likely though the blog post we're talking about isn't a good impression of what "just making a request" looks like. :)
        lhorie 8y ago
- aboutruby 8y ago
  I also much prefer simple REST APIs, just much easier to understand among other things. (e.g. fetch("/api/users").then(...) and it's good to go)
  - exogen 8y ago
    It's certainly easier to start out that way. But there are also a lot of pitfalls, that systems like what's described in the article are designed to solve. For example: let's say when component A or component B get rendered, they need to fetch users in order to show something about them. You only want to make the request if either of these components is actually rendered on the screen. What if they both get rendered? Is something going to know to reuse the inflight request so you don't have two identical requests? Or is it just naively going to make unnecessary requests? What if some other thing already fetched the user list earlier, are either of them still going to make the request anyway? Or use the data that's already there? Another example: you've got your result from the users endpoint. Now you fetch some user information from the "blog author" endpoint. Information about the same user is in both responses. What's the authoritative client-side source of information about that user now? Are you now going to potentially be displaying stale/conflicting info about that user in 2+ different places? etc. You can always just keep things simple if you want, and people in React and GraphQL land are happy to do that, too. But a lot of people are also focused on building complex applications. This is just to give you some perspective on the "why" here.
    - aboutruby 8y ago
      In my cases, for 1) and 2) the parent component would be fetching the user and passing it down to the child components.
      - exogen 8y ago
        For (1), how does the parent know whether those components will actually be rendered? Keep in mind one of the requirements was "only fetch if they're actually rendered." Presumably some other components along the way could be deciding to conditionally render them, right? For (2), the complexity is still there, just in the parent: it needs an entity store and to merge/invalidate when the same entity appears in multiple responses. So still not just a simple case of making some REST API calls.
- Waterluvian 8y ago
  Are you talking about thunks and action creators and whatnot? All totally optional and ideally with a future plan in mind. Just call fetch.
  - xrd 8y ago
    I don't disagree with what you are suggesting. I personally prefer react over all other frameworks. And, there are many blog posts with outdated suggestions referring to the items you've referenced. Knowing which of those posts to skip over now that the proper solution has been named is what's really challenging about React (and, of course, JS in general). Probably the OP has seen two many of those posts and doesn't have the right frame of reference to know what to do, and Apollo looks like the same.
- yodon 8y ago
  > ... hoops people in the React ecosystem go through just to send a request boggles the mind React Hooks (currently in alpha for the next release of React) finally clean up that component lifecycle complexity I think you're referring to. Nader Dabit has a great post [0] on how to use React Hooks with GraphQL. [0]https://medium.com/open-graphql/react-hooks-for-graphql-3fa8ebdd6c62 https://medium.com/open-graphql/react-hooks-for-graphql-3fa8...
  - saranshk 8y ago
    Thanks for the blog post. I was going to look for a similar thing soon enough.
- bgentry 8y ago
  Shameless self-promotional plug: ember-apollo-client is pretty damn easy to get started with and fits in really nicely with the Ember way of doing things :) https://github.com/bgentry/ember-apollo-client https://github.com/bgentry/ember-apollo-client
- dpacmittal 8y ago
  Except Apollo is for sending graphql requests, not just any request.
- dorian-graph 8y ago
  You can end up writing a lot of boiler plate code to send requests in Elm, FYI.
Numberwang 8y ago
Airbnb is one of the slowest sites out there in my experience. Good to hear they are working on improving it.
bsbechtel 8y ago
I recently did a POC for my company on a mature Angular project to see what it would take to switch over to the Apollo stack. I ended up reducing LOC by 50%, adding caching, full offline support, and optimistic UI. I’m convinced if we had used the Apollo stack at the start of the project, our front end resource requirements might’ve been 30-40% less for the project. The things the Apollo team have done are really making a big impact on engineering teams that adopt their tools.
- hdra 8y ago
  I recently tried Apollo for a project as well, for the most part, its almost magic, doing most of the usually tedious things for me. But I often run into unexpected blocker on the most basic functionalities that I need. One of them that I haven't been able to really solve till now is around cache invalidation / removing deleted items from cache. How have you been finding Apollo around that area?
  - moltar 8y ago
    Take a look at fetchPolicy: https://www.apollographql.com/docs/react/api/react-apollo.html#query-props https://www.apollographql.com/docs/react/api/react-apollo.ht...
    - hdra 8y ago
      I'm not sure what you're pointing to, but are you suggesting to ignore the cache? Coz that is basically what I currently do. Throw away the cache for the lists, and fetch a fresh one everytime. While that "works", but it also throws away lots of the supposed benefits that Apollo brings to the table, and its also rather awkward when you need to "trigger" the fetch manually when the display doesn't involve something like page navigation
      - moltar 8y ago
        Imo “cache management” is outside of the Apollo scope. Everyone’s specific cache requirements are different and you should just implement your own.
fro0116 8y ago
I'd personally be interested in some details on how they're _generating_ the mock data for their server. As they're using that mock data for testing, the data returned has to be deterministic, so I'm guessing they're either making up mock data manually by hard-coding that data in resolvers on the mock server implementation, or using some kind of fake-data generating library to generate that data dynamically based on graphql type in the schema, with a fixed seed so that the data doesn't change between runs. I'm hoping to explore the latter approach a bit more as I feel it would be great for testing productivity to have mock data generated from the schema instead of having to manually write them for every new thing added.
- adamrneary 8y ago
  It's actually neither. :) We have a shared development environment with a persistent (and thus deterministic) dataset. So when one developer runs a query against that dataset, another developer could run the same query and get the same result. The best thing about this, of course, is that if your Storybook data looks at listing 112358, you can also open that same listing in development and see the same result in the product. Very powerful.
  - fro0116 8y ago
    Thank you for the quick reply! My question was more about how that persistent dataset in your shared development environment is created though. As that dataset has to first exist for people to query against it. Curious if that creation process is manual or automated somehow through inference on the types in the schema.
    - adamrneary 8y ago
      For sure. Unrelated to GraphQL in this case. Very manual (in a good way!). Sometimes it's achieved manually via the UI. Sometimes manually via scripts, etc. Imagine you're creating a new field for Business Travel, and you've added some new fields via the API. One way or another you're grabbing 1-2 listings in development, modifying those listings to reflect the desired change, and then in the dev setup, you're saying "grab listings x and y and update their data for use in Storybook and unit tests." In my experience, if the system is working properly, there's not a lot of room for type-driven inference. We often get a design with very explicit data present, and we want to bring that data in rather than calling on Casual or
tinyvm 8y ago
The state of the industry sometimes deeply sadden me. Creating a stack that complex just to render Static Content... Seriously ? I'm fairly confident that only the author of this project can do something with the codebase. It must be an absolute mess between Zeplin , Storybook , Apollo , GraphQL , Next, Yeoman etc... Just why ? Can't Airbnb invest in one good CMS and tooling solution ? Don't they have a CTO that define the company tech governance and tech stack ? Isn't building landing page for "Luxury Destination" one of their core businesses ? Do each Airbnb engineer create their own stack for a tiny part of the website ? It just buggers me to see something like this and remind me of their 'React Native Fiasco' where they decided to use React Native but their mobile engineers didn't like JS , so the engineers of each platform just wrote the app using binding to use Java or Objective-C. Sometimes I really tell myself that working for a FAANGS must be awesome, but then this type of content pops up and it just remind me I should either stay at my current job or create my own business to avoid all this.
- venantius 8y ago
  I used to work at Airbnb. Airbnb does not, in fact, have a CTO who dictates the company tech governance and/or stack. They prefer to run things in a federated manner, with individual teams making the decisions that they feel are best for them. While they're encouraged/required to draw up design docs and have them reviewed by an architecture review group, the group's recommendations are non-binding. This model has advantages and disadvantages. On the upside, it creates an environment where people can take risks and do things that haven't been done inside of the company before. On the other hand, it means people sometimes go out on a limb and push the company into supporting something that turns out not to be sustainable in the long term. As a matter of personal preference, I like to have a set toolchain that a company is built around. But it would be unwise to suggest that Airbnb's strategy hasn't worked out pretty well for them overall.
  - ForHackernews 8y ago
    I think AirBnb's strategy has worked out well for them for reasons completely unrelated to any of their technology choices. We're living in a time of cheap airfare, insanely inflated real estate in popular markets, and overburdened local governments unable to enforce regulations. That adds up to great success for Airbnb, regardless of what software runs their website.
gaius 8y ago
Is moving faster to evade taxes and regulations really something to encourage?
or113 8y ago
When looking at the network tab, I couldn't find anything related to graphql. Nor by searching overall the code. Does anyway know where exactly Airbnb use graphql?
- rayshan 8y ago
  Adam talked about server-side rendering in the video. You won't see the graphql queries in network activities.
brian005 8y ago
do you really need a professional hacker to get a faster and reliable job done for you? you need to contact brianscotthacker005@gmail.com for this to get you out of any issue or problem you want to get solved within 24 hours, i can guarantee you is job and a try ill convince you because he has helped me out and i have promise to promote his work.. Jamal referred
munchbunny 8y ago
With posts like this, I think it's important to remember that projects like this get created for one primary reason: it solves problems AirBnB has. If you're lucky, it solves problems you have too. I work on some tech giant cyber security stuff, and it's obvious if you look at it that what my team designs solves a tech giant scale problem in the context of past engineering decisions made to solve other tech giant scale problems this company had. I probably wouldn't recommend my team's approaches to anyone who has less than ~5000 engineers. From this experience I've adopted a heuristic: first consider the scale the system was built to address. If you are not in the ballpark of that scale, give it a long hard look before you adopted it.

202 points by saranshk 8y ago

It is unclear from the article how exactly Airbnb is moving "10x faster" as the title claims. Unless backed by a benchmark, this kind of hyperbole shouldn't be an engineering article.

EduardoBautista 8y ago

As primarily an Ember.js user (with an interest in Elm), just looking at how many hoops people in the React ecosystem go through just to send a request just boggles the mind. Maybe I am in the minority, but writing boilerplate code all the time just isn't something that interests me.

testnew2 8y ago
That boilerplate has absolutely nothing to do with react.
exogen 8y ago
I write React every day and don't know what you're referring to – you can just call fetch() in a lifecycle hook and be done with it, just like in any other framework. But mostly I'm curious how your point relates to this blog post. Maybe you can clarify? The stuff they're doing with Apollo here that potentially comes across as boilerplate also gets query batching, caching, refetching, etc. for free (in addition to the fancy stuff they also talk about like automatic mocks). It's not like they're "just sending a request." Maybe you can show me how you do all that in another framework without some setup/wrappers/etc.?
- EduardoBautista 8y ago
  Things Ember Data's `findRecord` does: * Checks if you already have the resource to prevent sending an unnecessary request (can be forced) * Retrieves the resource based on a convention for the URL * Parses the JSON based on a convention * Stores the resource in the "store" for use in the app Batching is also very simple thanks to a plugin from Netflix: https://github.com/Netflix/ember-batch-request https://github.com/Netflix/ember-batch-request The reason the Ember community doesn't talk about any of this stuff is because, from the developers perspective, nothing interesting happens, which is why developer productivity is high in Ember.
  - exogen 8y ago
    Why would a UI framework know anything about records? Keep in mind that it's explicitly not React's goal to know anything about requests, routing, records, data, or anything like that. That's why you hear React people talking about how they achieve this stuff. Ember is a framework with a much larger scope, while React is a view layer.
    - EduardoBautista 8y ago
      Ember Data can be used separately from Ember itself. In fact, the official documentation treats them as if they are. It's even a separate project on GitHub: https://github.com/emberjs/data https://github.com/emberjs/data
      - exogen 8y ago
        If even in Ember-land it's an entirely separate package, then why make the comparison to React at all..? (Ember + Ember Data) is more like (React + Apollo) then, no? But you're just comparing it to React without anything else.
        EduardoBautista 8y ago
        My original comment said "React ecosystem", not just "React". Apollo is part of the ecosystem.
        exogen 8y ago
        Here's what just sending a request looks like with React + Apollo: https://codesandbox.io/s/q71px7njrw https://codesandbox.io/s/q71px7njrw Which boilerplate do you take issue with? There are a couple imports you might find ugly, I suppose. Most likely though the blog post we're talking about isn't a good impression of what "just making a request" looks like. :)
        lhorie 8y ago
aboutruby 8y ago
I also much prefer simple REST APIs, just much easier to understand among other things. (e.g. fetch("/api/users").then(...) and it's good to go)
- exogen 8y ago
  It's certainly easier to start out that way. But there are also a lot of pitfalls, that systems like what's described in the article are designed to solve. For example: let's say when component A or component B get rendered, they need to fetch users in order to show something about them. You only want to make the request if either of these components is actually rendered on the screen. What if they both get rendered? Is something going to know to reuse the inflight request so you don't have two identical requests? Or is it just naively going to make unnecessary requests? What if some other thing already fetched the user list earlier, are either of them still going to make the request anyway? Or use the data that's already there? Another example: you've got your result from the users endpoint. Now you fetch some user information from the "blog author" endpoint. Information about the same user is in both responses. What's the authoritative client-side source of information about that user now? Are you now going to potentially be displaying stale/conflicting info about that user in 2+ different places? etc. You can always just keep things simple if you want, and people in React and GraphQL land are happy to do that, too. But a lot of people are also focused on building complex applications. This is just to give you some perspective on the "why" here.
  - aboutruby 8y ago
    In my cases, for 1) and 2) the parent component would be fetching the user and passing it down to the child components.
    - exogen 8y ago
      For (1), how does the parent know whether those components will actually be rendered? Keep in mind one of the requirements was "only fetch if they're actually rendered." Presumably some other components along the way could be deciding to conditionally render them, right? For (2), the complexity is still there, just in the parent: it needs an entity store and to merge/invalidate when the same entity appears in multiple responses. So still not just a simple case of making some REST API calls.
Waterluvian 8y ago
Are you talking about thunks and action creators and whatnot? All totally optional and ideally with a future plan in mind. Just call fetch.
- xrd 8y ago
  I don't disagree with what you are suggesting. I personally prefer react over all other frameworks. And, there are many blog posts with outdated suggestions referring to the items you've referenced. Knowing which of those posts to skip over now that the proper solution has been named is what's really challenging about React (and, of course, JS in general). Probably the OP has seen two many of those posts and doesn't have the right frame of reference to know what to do, and Apollo looks like the same.
yodon 8y ago
> ... hoops people in the React ecosystem go through just to send a request boggles the mind React Hooks (currently in alpha for the next release of React) finally clean up that component lifecycle complexity I think you're referring to. Nader Dabit has a great post [0] on how to use React Hooks with GraphQL. [0]https://medium.com/open-graphql/react-hooks-for-graphql-3fa8ebdd6c62 https://medium.com/open-graphql/react-hooks-for-graphql-3fa8...
- saranshk 8y ago
  Thanks for the blog post. I was going to look for a similar thing soon enough.
bgentry 8y ago
Shameless self-promotional plug: ember-apollo-client is pretty damn easy to get started with and fits in really nicely with the Ember way of doing things :) https://github.com/bgentry/ember-apollo-client https://github.com/bgentry/ember-apollo-client
dpacmittal 8y ago
Except Apollo is for sending graphql requests, not just any request.
dorian-graph 8y ago
You can end up writing a lot of boiler plate code to send requests in Elm, FYI.

Numberwang 8y ago

Airbnb is one of the slowest sites out there in my experience. Good to hear they are working on improving it.

bsbechtel 8y ago

I recently did a POC for my company on a mature Angular project to see what it would take to switch over to the Apollo stack. I ended up reducing LOC by 50%, adding caching, full offline support, and optimistic UI. I’m convinced if we had used the Apollo stack at the start of the project, our front end resource requirements might’ve been 30-40% less for the project. The things the Apollo team have done are really making a big impact on engineering teams that adopt their tools.

hdra 8y ago
I recently tried Apollo for a project as well, for the most part, its almost magic, doing most of the usually tedious things for me. But I often run into unexpected blocker on the most basic functionalities that I need. One of them that I haven't been able to really solve till now is around cache invalidation / removing deleted items from cache. How have you been finding Apollo around that area?
- moltar 8y ago
  Take a look at fetchPolicy: https://www.apollographql.com/docs/react/api/react-apollo.html#query-props https://www.apollographql.com/docs/react/api/react-apollo.ht...
  - hdra 8y ago
    I'm not sure what you're pointing to, but are you suggesting to ignore the cache? Coz that is basically what I currently do. Throw away the cache for the lists, and fetch a fresh one everytime. While that "works", but it also throws away lots of the supposed benefits that Apollo brings to the table, and its also rather awkward when you need to "trigger" the fetch manually when the display doesn't involve something like page navigation
    - moltar 8y ago
      Imo “cache management” is outside of the Apollo scope. Everyone’s specific cache requirements are different and you should just implement your own.

fro0116 8y ago

I'd personally be interested in some details on how they're _generating_ the mock data for their server. As they're using that mock data for testing, the data returned has to be deterministic, so I'm guessing they're either making up mock data manually by hard-coding that data in resolvers on the mock server implementation, or using some kind of fake-data generating library to generate that data dynamically based on graphql type in the schema, with a fixed seed so that the data doesn't change between runs. I'm hoping to explore the latter approach a bit more as I feel it would be great for testing productivity to have mock data generated from the schema instead of having to manually write them for every new thing added.

adamrneary 8y ago
It's actually neither. :) We have a shared development environment with a persistent (and thus deterministic) dataset. So when one developer runs a query against that dataset, another developer could run the same query and get the same result. The best thing about this, of course, is that if your Storybook data looks at listing 112358, you can also open that same listing in development and see the same result in the product. Very powerful.
- fro0116 8y ago
  Thank you for the quick reply! My question was more about how that persistent dataset in your shared development environment is created though. As that dataset has to first exist for people to query against it. Curious if that creation process is manual or automated somehow through inference on the types in the schema.
  - adamrneary 8y ago
    For sure. Unrelated to GraphQL in this case. Very manual (in a good way!). Sometimes it's achieved manually via the UI. Sometimes manually via scripts, etc. Imagine you're creating a new field for Business Travel, and you've added some new fields via the API. One way or another you're grabbing 1-2 listings in development, modifying those listings to reflect the desired change, and then in the dev setup, you're saying "grab listings x and y and update their data for use in Storybook and unit tests." In my experience, if the system is working properly, there's not a lot of room for type-driven inference. We often get a design with very explicit data present, and we want to bring that data in rather than calling on Casual or

tinyvm 8y ago

The state of the industry sometimes deeply sadden me. Creating a stack that complex just to render Static Content... Seriously ? I'm fairly confident that only the author of this project can do something with the codebase. It must be an absolute mess between Zeplin , Storybook , Apollo , GraphQL , Next, Yeoman etc... Just why ? Can't Airbnb invest in one good CMS and tooling solution ? Don't they have a CTO that define the company tech governance and tech stack ? Isn't building landing page for "Luxury Destination" one of their core businesses ? Do each Airbnb engineer create their own stack for a tiny part of the website ? It just buggers me to see something like this and remind me of their 'React Native Fiasco' where they decided to use React Native but their mobile engineers didn't like JS , so the engineers of each platform just wrote the app using binding to use Java or Objective-C. Sometimes I really tell myself that working for a FAANGS must be awesome, but then this type of content pops up and it just remind me I should either stay at my current job or create my own business to avoid all this.

venantius 8y ago
I used to work at Airbnb. Airbnb does not, in fact, have a CTO who dictates the company tech governance and/or stack. They prefer to run things in a federated manner, with individual teams making the decisions that they feel are best for them. While they're encouraged/required to draw up design docs and have them reviewed by an architecture review group, the group's recommendations are non-binding. This model has advantages and disadvantages. On the upside, it creates an environment where people can take risks and do things that haven't been done inside of the company before. On the other hand, it means people sometimes go out on a limb and push the company into supporting something that turns out not to be sustainable in the long term. As a matter of personal preference, I like to have a set toolchain that a company is built around. But it would be unwise to suggest that Airbnb's strategy hasn't worked out pretty well for them overall.
- ForHackernews 8y ago
  I think AirBnb's strategy has worked out well for them for reasons completely unrelated to any of their technology choices. We're living in a time of cheap airfare, insanely inflated real estate in popular markets, and overburdened local governments unable to enforce regulations. That adds up to great success for Airbnb, regardless of what software runs their website.

gaius 8y ago

Is moving faster to evade taxes and regulations really something to encourage?

or113 8y ago

When looking at the network tab, I couldn't find anything related to graphql. Nor by searching overall the code. Does anyway know where exactly Airbnb use graphql?

rayshan 8y ago
Adam talked about server-side rendering in the video. You won't see the graphql queries in network activities.

brian005 8y ago

do you really need a professional hacker to get a faster and reliable job done for you? you need to contact brianscotthacker005@gmail.com for this to get you out of any issue or problem you want to get solved within 24 hours, i can guarantee you is job and a try ill convince you because he has helped me out and i have promise to promote his work.. Jamal referred

munchbunny 8y ago

With posts like this, I think it's important to remember that projects like this get created for one primary reason: it solves problems AirBnB has. If you're lucky, it solves problems you have too. I work on some tech giant cyber security stuff, and it's obvious if you look at it that what my team designs solves a tech giant scale problem in the context of past engineering decisions made to solve other tech giant scale problems this company had. I probably wouldn't recommend my team's approaches to anyone who has less than ~5000 engineers. From this experience I've adopted a heuristic: first consider the scale the system was built to address. If you are not in the ballpark of that scale, give it a long hard look before you adopted it.