4 ms·

> ...you can just add a “fields” query param to those few key endpoints. Dylan Beattie actually talked about resource expansion (with HAL) about 6 years ago: h

> ...you can just add a “fields” query param to those few key endpoints. Dylan Beattie actually talked about resource expansion (with HAL) about 6 years ago: https://youtu.be/g8E1B7rTZBI?t=1654 https://youtu.be/g8E1B7rTZBI?t=1654 Essentially, it was a really nice and concise explanation of links to other data, problems with those and adding resource expansion to API endpoints. That said, it feels like HAL never really took off and the implementation of something like that is very framework specific. > ...or other approaches to solving this like GraphQL... GraphQL does seem to solve this, but it's like someone made an equivalent to SQL just for APIs, except that they forgot to include the bit that actually talks to your data store based on the queries and returns data, so you have to write that yourself (e.g. query resolvers). Unless you're using something like PostGraphile/Prisma/Hasura which exposes your database (provided you use one, which might be the situation in most cases) through GraphQL, but then you run into problems with intermediate data processing or additional access controls, that you might need to be API logic, before your data is returned. So, instead of the old approach: [client] <-RESTful request-> [your API] <-SQL query (with parameters)-> [SQL database] You might be tempted to go with something like this: [client] <-dynamic GraphQL query-> [GraphQL intermediate solution] <-SQL query (with parameters)-> [SQL database] But since that might prove to be untenable (unsafe), you might need something like: [client] <-dynamic GraphQL query-> [your API] <-dynamic GraphQL query-> [GraphQL intermediate solution] <-SQL query (with parameters)-> [SQL database] Which kind of feels worse from a complexity standpoint from where we were previously, especially if the queries are dynamic instead of largely static. Back to the topic, BFFs seem like a good idea, especially if they take over some of the gateway functionality as well. I've heard them be described as facade APIs in some cases.

acoard 4y ago
> Which kind of feels worse from a complexity standpoint from where we were previously, especially if the queries are dynamic instead of largely static. Agreed on increased complexity, but isn't the point largely the dynamic queries? You setup all this extra architecture, but in response you don't have to write as many "controllers" with custom API response. Just let the frontend request what it needs via GraphQL. So it's more upfront work, less down the line. Or is writing all the API level sanitizers and query resolvers more work?
- KronisLV 4y ago
  > Agreed on increased complexity, but isn't the point largely the dynamic queries? You setup all this extra architecture, but in response you don't have to write as many "controllers" with custom API response. Just let the frontend request what it needs via GraphQL. What about cases like: "The user should only be able to see the data of their own submitted requests, when the parent document is any of the given statuses: FOO and BAR and there is no related BAZ request that's been saved and is approved in the system by the manager. In cases, where the BAR request exists, ..." (a long list of other edge cases that might affect the visibility of any particular document or even individual fields in some form) My point is that in many systems the cases where you might want to just give someone the ability to retrieve the data they need is rather limited. So in those cases the dynamic nature of everything is more of a risk than anything else.
- bluesnowmonkey 4y ago
  At large scale, the queries you can handle are limited by the indexes you have in place, and you can’t index everything. So you already have to think about which queries you’ll really need. There’s also caching, which is another side of the same coin as indexing. You may want to cache certain common/expensive queries. You’ll probably need to think about the granularity of what gets cached, and how to invalidate it. Hard problems for a dynamic, “query anything” API. Also you probably need to control which data can be retrieved. KronisLV covered that.
randomdata 4y ago
> except that they forgot to include the bit that actually talks to your data store Which stands to reason given that GraphQL was designed for talking to services, not data stores. It was built for BFF. The backend is responsible for talking to the data store, as usual. [client] <-dynamic GraphQL query-> [GraphQL resolver] <-boring old RPC/REST/whatever-> {Set of backend services}
- KronisLV 4y ago
  But doesn't that essentially mean that you've simply shifted the problems of fetching data (overfetching, N+1 problems and resource expansion) a little bit to the right, without solving the actual issue? It would technically be faster because of lower latency between services in the same data center and sending the final results to the user in one go, but at the same time if your DB is hit 50 times to service that request, you still have a problem. Essentially you just give the client the ability to run dynamic queries (with the complexity of which, the resolvers and their logic, you now need to deal with), without actually improving anything else, at the cost of noticeably increased complexity and yet another abstraction layer. That tradeoff seems like a tough sell to me, much like an SQL engine that's not tightly coupled with a solution to actually store/retrieve the data would be like. Then again, there are solutions out there that attempt to do something like that already, so who knows.
  - randomdata 4y ago
    > It would technically be faster because of lower latency between services in the same data center and sending the final results to the user in one go Which is a pretty big deal when you have practically the whole planet as users, many of whom are found where low latency is a luxury. > but at the same time if your DB is hit 50 times to service that request, you still have a problem. Maybe. Or when you are Facebook maybe you don't care. Just throw another datacenter at the problem. > without actually improving anything else They found it improved discoverability for designers and non-technical folk. > That tradeoff seems like a tough sell to me If your organization doesn't look like Facebook, I imagine it wouldn't sell. It's very much designed for a purpose.
akrymski 4y ago
Indeed in most cases replacing simple code with GraphQL only adds complexity. The old approach has been more like this: [client] <- RESTful request -> [your API] <- Your data fetching logic -> [DB] <- SQL query (with parameters) -> [DB] Replacing your data fetching logic with GraphQL just means translating your JS/Python to GraphQL - something that isn't as flexible, turing complete, or predictable (in execution). If anything, I'd prefer to go the other way and get rid of SQL altogether - let me express queries in code: books.filter(book => book.author === "Alice") .group(book => book.year) .map(group => group.reduce(({ sum = 0 }, { year, sales }) => ({ year: year, sales: sum + sales }), {})) .sort((a, b) => b.year - a.year) which translates to: SELECT year, SUM(sales) FROM books WHERE author = 'Alice' GROUP BY year ORDER BY year DESC but is far more composable, debuggable and potentially more deterministic - I'd like more control over query execution planner, not less.

acoard 4y ago

> Which kind of feels worse from a complexity standpoint from where we were previously, especially if the queries are dynamic instead of largely static. Agreed on increased complexity, but isn't the point largely the dynamic queries? You setup all this extra architecture, but in response you don't have to write as many "controllers" with custom API response. Just let the frontend request what it needs via GraphQL. So it's more upfront work, less down the line. Or is writing all the API level sanitizers and query resolvers more work?

KronisLV 4y ago
> Agreed on increased complexity, but isn't the point largely the dynamic queries? You setup all this extra architecture, but in response you don't have to write as many "controllers" with custom API response. Just let the frontend request what it needs via GraphQL. What about cases like: "The user should only be able to see the data of their own submitted requests, when the parent document is any of the given statuses: FOO and BAR and there is no related BAZ request that's been saved and is approved in the system by the manager. In cases, where the BAR request exists, ..." (a long list of other edge cases that might affect the visibility of any particular document or even individual fields in some form) My point is that in many systems the cases where you might want to just give someone the ability to retrieve the data they need is rather limited. So in those cases the dynamic nature of everything is more of a risk than anything else.
bluesnowmonkey 4y ago
At large scale, the queries you can handle are limited by the indexes you have in place, and you can’t index everything. So you already have to think about which queries you’ll really need. There’s also caching, which is another side of the same coin as indexing. You may want to cache certain common/expensive queries. You’ll probably need to think about the granularity of what gets cached, and how to invalidate it. Hard problems for a dynamic, “query anything” API. Also you probably need to control which data can be retrieved. KronisLV covered that.

randomdata 4y ago

> except that they forgot to include the bit that actually talks to your data store Which stands to reason given that GraphQL was designed for talking to services, not data stores. It was built for BFF. The backend is responsible for talking to the data store, as usual. [client] <-dynamic GraphQL query-> [GraphQL resolver] <-boring old RPC/REST/whatever-> {Set of backend services}

KronisLV 4y ago
But doesn't that essentially mean that you've simply shifted the problems of fetching data (overfetching, N+1 problems and resource expansion) a little bit to the right, without solving the actual issue? It would technically be faster because of lower latency between services in the same data center and sending the final results to the user in one go, but at the same time if your DB is hit 50 times to service that request, you still have a problem. Essentially you just give the client the ability to run dynamic queries (with the complexity of which, the resolvers and their logic, you now need to deal with), without actually improving anything else, at the cost of noticeably increased complexity and yet another abstraction layer. That tradeoff seems like a tough sell to me, much like an SQL engine that's not tightly coupled with a solution to actually store/retrieve the data would be like. Then again, there are solutions out there that attempt to do something like that already, so who knows.
- randomdata 4y ago
  > It would technically be faster because of lower latency between services in the same data center and sending the final results to the user in one go Which is a pretty big deal when you have practically the whole planet as users, many of whom are found where low latency is a luxury. > but at the same time if your DB is hit 50 times to service that request, you still have a problem. Maybe. Or when you are Facebook maybe you don't care. Just throw another datacenter at the problem. > without actually improving anything else They found it improved discoverability for designers and non-technical folk. > That tradeoff seems like a tough sell to me If your organization doesn't look like Facebook, I imagine it wouldn't sell. It's very much designed for a purpose.

akrymski 4y ago

Indeed in most cases replacing simple code with GraphQL only adds complexity. The old approach has been more like this: [client] <- RESTful request -> [your API] <- Your data fetching logic -> [DB] <- SQL query (with parameters) -> [DB] Replacing your data fetching logic with GraphQL just means translating your JS/Python to GraphQL - something that isn't as flexible, turing complete, or predictable (in execution). If anything, I'd prefer to go the other way and get rid of SQL altogether - let me express queries in code: books.filter(book => book.author === "Alice") .group(book => book.year) .map(group => group.reduce(({ sum = 0 }, { year, sales }) => ({ year: year, sales: sum + sales }), {})) .sort((a, b) => b.year - a.year) which translates to: SELECT year, SUM(sales) FROM books WHERE author = 'Alice' GROUP BY year ORDER BY year DESC but is far more composable, debuggable and potentially more deterministic - I'd like more control over query execution planner, not less.