5 ms·

The recommendations of this document are definitely at odds with the design of the API my team is currently working on. We are trying to build a data access AP

The recommendations of this document are definitely at odds with the design of the API my team is currently working on. We are trying to build a data access API that can provide a functional and flexible view into data where we know some of the requirements and expectations of the initial users, but we don't know how it will be extended and transformed in the future. If we don't provide things such as customization of the returned fields, desired date ranges, and ordering, then I believe we would have to rely on waiting for feedback from users/devs that we did not satisfy and attempt to build new end-points to correct that deficiency. That seems to be almost guaranteed to get us off to the wrong start for a large portion of users and they might just decide it isn't worth asking us to make changes since we didn't appear to be catering to them in the first place. I understand that we can't meet all the unknown needs, but if we can build features into the API to meet general classes of needs, that seems better than ignoring them until the specifics become known and then supplying further endpoints that answer only those specifics. How would you extend this philosophy to a large data portal such as data.gov? Even in the FBI example, if there is no extensibility or transform-ability in the API, how are the unmet needs of users handled? Do you have a system that gathers complaints from users about the data they suspect is possible but not available and then build new endpoints for each distinct need? Do you just ignore the potential transformative uses that aren't directly applicable to the needs of the customers to which you are paying attention?

mfenniak 13y ago
Ooh, very interesting questions. :-) I'm not sure I have the answers for these questions, but I do have some ideas. Most APIs don't want to provide data analysis capabilities. They want to provide data. Data analysis is a really hard problem space. For the FBI example, the FBI doesn't want to build an analytics tool for you to use; that's not their core business, and it's not something that helps them achieve their goals. Data analysis versus data are two different problems with different classes of solutions; for example, a valid approach to delivering data might be an API to retrieve compressed, batch (nightly) downloads of events or deltas. That's not the only solution, of course, but I just mean to create an example that's a different point-of-view. If you intend to provide data analysis from your API, I would say that is the "intent" of your API and you should design it as such, with the appropriate flexibility and user-defined behaviour that is required to answer arbitrary queries from users. This is a hard API to design, build, and scale. When designing a data analysis API, you should address scenarios like a user making a valid API request that can't be delivered in real-time; is it your intention to make your API accessible in real-time, in which case you have to either crash horribly or tell this request to piss off? Or is it your intent to provide the analysis no matter what, in which case perhaps it would be valid to queue such queries, process them in batches with limited concurrency, and deliver the results when completed. Regarding the unmet needs of users, you should always provide a feedback mechanism for your API. This is something that varies a lot depending upon your customer; a lot of developers work in "enterprise" software, where the feedback channel is usually pretty apparent. Public APIs with an undefined audience will have a harder time getting that feedback, but usually you can bet that the importance of an issue is proportional to the number of times you hear about it.
dreamfactory 13y ago
A difficulty is that you can't remove options once they are published. A second problem is that you don't want your API to leak the underlying implementation or it will be brittle. But that doesn't mean that you shouldn't allow parameters - both examples given were a cop-out really. I've had good results with basic parameters - date ranges, index ranges for paging, ordering by a couple of important and relevant fields (could well include some kind of computed value like 'hotness'), and probably most importantly - a query parameter (search algorithm being blackboxed in API). If you need many more params, it may be a sign that the underlying resource is becoming overcomplex and can be decomposed - I've made applications where the API interfaces were completely uniform across resources, which worked out really well and encouraged takeup (the most important measure for an API). The other main question is whether to allow the caller to request which fields to return (cache-filling and complexity vs bandwidth).

mfenniak 13y ago

Ooh, very interesting questions. :-) I'm not sure I have the answers for these questions, but I do have some ideas. Most APIs don't want to provide data analysis capabilities. They want to provide data. Data analysis is a really hard problem space. For the FBI example, the FBI doesn't want to build an analytics tool for you to use; that's not their core business, and it's not something that helps them achieve their goals. Data analysis versus data are two different problems with different classes of solutions; for example, a valid approach to delivering data might be an API to retrieve compressed, batch (nightly) downloads of events or deltas. That's not the only solution, of course, but I just mean to create an example that's a different point-of-view. If you intend to provide data analysis from your API, I would say that is the "intent" of your API and you should design it as such, with the appropriate flexibility and user-defined behaviour that is required to answer arbitrary queries from users. This is a hard API to design, build, and scale. When designing a data analysis API, you should address scenarios like a user making a valid API request that can't be delivered in real-time; is it your intention to make your API accessible in real-time, in which case you have to either crash horribly or tell this request to piss off? Or is it your intent to provide the analysis no matter what, in which case perhaps it would be valid to queue such queries, process them in batches with limited concurrency, and deliver the results when completed. Regarding the unmet needs of users, you should always provide a feedback mechanism for your API. This is something that varies a lot depending upon your customer; a lot of developers work in "enterprise" software, where the feedback channel is usually pretty apparent. Public APIs with an undefined audience will have a harder time getting that feedback, but usually you can bet that the importance of an issue is proportional to the number of times you hear about it.

dreamfactory 13y ago

A difficulty is that you can't remove options once they are published. A second problem is that you don't want your API to leak the underlying implementation or it will be brittle. But that doesn't mean that you shouldn't allow parameters - both examples given were a cop-out really. I've had good results with basic parameters - date ranges, index ranges for paging, ordering by a couple of important and relevant fields (could well include some kind of computed value like 'hotness'), and probably most importantly - a query parameter (search algorithm being blackboxed in API). If you need many more params, it may be a sign that the underlying resource is becoming overcomplex and can be decomposed - I've made applications where the API interfaces were completely uniform across resources, which worked out really well and encouraged takeup (the most important measure for an API). The other main question is whether to allow the caller to request which fields to return (cache-filling and complexity vs bandwidth).