9 ms·

Everything Should Have An API

34 points by scottjackson 17y ago

sophacles 17y ago
This is ridiculous, we can't just let people have access to their data!... i mean OUR data. Just imagine, they may learn things we could have forced them to pay absurd fees to get! :P
DanielStraight 17y ago
As right as you are, making an API for your service so people can replace your crappy interface with their own good interface probably isn't the best business decision. Online banking sites can display ads. The bank can't inject ads into another client though. Also consider that the bank would get blamed in popular opinion if someone wrote an insecure client that was determined to be leaking data.
- scottjackson 17y ago
  I think it could be done -- I mean, Pine's an example of a secure client application that uses a kinda-API (SMTP) to deliver data. If we don't make APIs because someone might write an insecure client, would anyone make any APIs ever? I do agree with you though :) It's probably not going to happen in the real world any time soon. C'est la vie.
  - DanielStraight 17y ago
    I'm sure it could be done. But consider for example if Bank of America did this and someone created an awesome client app that basically became the main way anyone used their Bank of America account. If a bug is discovered in this application that means everyone's data is no longer secure, are users going to blame the app creator or Bank of America? Can Bank of America afford to take that chance?
    - scottjackson 17y ago
      You're right -- that's almost certainly the reason you don't see stuff like this happen. Though, didn't Bank of America make that iPhone app a little while ago that allowed you to photograph a cheque in order to deposit it? They're obviously OK with sending financial data around. I guess maybe it was OK in that case because they were the ones that wrote the app? In any case, you're right. Security is definitely the reason this kind of stuff hasn't been implemented. I still think that it's possible to write an open, secure interface to things. Email is the example I fall back on here. Like I mentioned above - sensitive information gets sent around in emails all the time, yet there are countless email clients and they're almost all secure (I say "almost all" to cover my ass if there's some that aren't). I think that if it can work for email, then maybe it can work in other domains.
      - grncdr 17y ago
        Unfortunately, email is a terrible example. While it's possible to have a modicum of security in regards to receiving your e-mail, spoofing emails is trivially easy (unless you use cryptographic signing, which isn't well supported by the majority of clients afaik)
    - dustingetz 17y ago
      yes, but what about when I mechanize my own API to my bank anyway? it's theoretically impossible to detect more effectively than cat-n-mouse.
      - DanielStraight 17y ago
        I don't see how that's related at all. I'm not saying Bank of America needs to work hard to prevent automated access to their online banking system. Right now, automated access counts as "undocumented behavior" at best. Bank of America is not likely to face public ridicule because someone hacked up an automated interface to their site which was not intended for that purpose and shot themselves in the foot. If, however, Bank of America says explicitally, "Hey, come use this API and applications built off it to access your account," and then one of those applications starts shooting people in the foot, people are going to wonder why Bank of America thought it was a good idea to let people access their accounts like that in the first place.
Raphael 17y ago
Couldn't agree more. This is what my next startup is based on.
mattmanser 17y ago
Personally it sounds like a security and privacy nightmare to me.
- scottjackson 17y ago
  Maybe. I think that if we get enough smart, awesome people working on it, it could be secure, safe and open all at the same time.
mrtron 17y ago
But at what cost?
hxa7241 17y ago
The scenario descriptions are distinctly reminiscent of Berners-Lee's scenario descriptions for the 'semantic web' several years ago. Of course, that is now rather out of fashion, but not entirely justifiably. The 'semantic' angle was somewhat fanciful, but the core idea was the data web: It is one thing to put APIs on everything, but another for all the data types and formats to work together. That was more the aspect that RDF was addressing. It is neither thoroughly convincing nor completely apposite, but worth factoring in to the consideration anyway. (The essential RDF concept was elegant, whatever anyone says.)
d4rt 17y ago
For my own personal use I think text based interfaces (with pipes) are superior, the higher level pluggable components in AppleScript/Automator seem more usable in general. That said, there was an excellent article recently with the experiences of a trainer in getting people to use *nix, who found the minimalistic "command mode" more usable due to it's text oriented simplicity.
tocomment 17y ago
Should creating email accounts have an API? How would the captcha be presented? How about adding reviews of products?
- scottjackson 17y ago
  I don't personally know because I'm just a peon with a qwerty keyboard, but I think there are some great developers and talented architects out there who could figure it out and make it happen.
- deleted 17y ago
  [deleted]
- silentbicycle 17y ago
  Don't captchas and open APIs fundamentally oppose each other? The point of an API is to have a clear interface so that programs can be built on a service, and captchas exist to try to block bots from using services.
  - tocomment 17y ago
    That's my point. Not everything <should> have an API since there are so many things we put captchas around.
- grncdr 17y ago
  That would be an easy API to implement, in psuedo-code: do: server->return any errors client->display errors client->ask user for account name client->server->request account name x while ( server->x is invalid account name ) << Repeat above loop for password >> do: server->return new captcha client->display captcha client->take user input client->server->send captcha text while(server->captcha does not match) server->return success message
  - stuartjmoore 17y ago
    Web browsers display CAPTCHAs every day with an API known as HTML.
grncdr 17y ago
As previously said, the financial incentive is simply not there. As much sense as it makes for you to have easy access to your information, there are too many businesses that rely on controlling that access. On the other hand, tools to do this with almost any site already exist. RSS is common enough, and where it doesn't exist, screen scraping isn't that hard (and getting easier all the time). If I'm not mistaken, this is the exact problem that Yahoo pipes are trying to address. Of course, sites that require a complicated login process (banking sites) are still out.
- stuartjmoore 17y ago
  I think Twitter is an example that owes its success to its API; it depends what your business plan is. As more companies become Google-esque data hogs, APIs will probably be encouraged.
stuartjmoore 17y ago
API is kind-of a misnomer in this case. API implies each service has its own set of rules, the data is open, but the access is different. Everything should talk to one another, but a service's name or url should never have to appear in any code. That's the user's job to input.
clawrencewenham 17y ago
Having a good API for everything would be almost as good as The Singularity (tm), but making a good and secure API is very hard and comes with a massive hidden cost. 1) An API can require even more planning to design than the application or service itself. 2) Mistakes made in the design or implementation of the API turn into liabilities you can't solve without breaking users of the API. 3) The cost of maintaining an API so that it and its documentation doesn't fall out of date with the app/service can be as high (and sometimes even higher) as the cost of maintaining the app itself. 4) It widens the attack surface. And those are just the ones off the top of my head.
- grncdr 17y ago
  I agree with all of your points, but I think many of these costs are minimized by the other point mentioned in the OP: following the UNIX philosophy. The smaller and simpler your service is, the better. If you have a service that can't be accessed with a simple API, you probably have more than one service, and would benefit from splitting it into multiple APIs. Another thing worth mentioning is the possibility of building your API first, and then building your service on top of it. If you are providing a web service, then it [usually] boils down to data access and management. Figure out what data you are managing for your users and what you need to do with it, then build your API around that. If you plan to have a public facing API from the start, this approach makes sense (in the same way that using javascript only for progressive enhancement of a working site makes sense). Of course, because you are an "internal user" of the API, you can bend the rules a little more than the average client, but the less you do this, the more useful your API is. Regarding points 2 and 4: if public APIs for data access become the norm, this will simply become the cost of doing business. Secure API's aren't that difficult, and updating an API while maintaining backwards compatibility can be done, especially if you transition these changes and notify all broken clients.
  - Periodic 17y ago
    For the UNIX philosophy I like the idea of small, useful tools loosely coupled. A big part of UNIX is that there are a lot of little tools that do one thing well. The smaller the tool is, less attack surface there will be, and each tool can interact with a common authentication/authorization interface to further simplify things.
dwiel 17y ago
The printing press enabled more people to read, though still few could write. The net has enabled more people to write, though still few can program. A net where everything has an API is an extremely helpful step in creating a system which enables more people to program.
rythie 17y ago
They shouldn't just have an API, they should use the same APIs where possible. Each site having it's own API is a major barrier.
ryah 17y ago
I want a build system with an external API.
SlyShy 17y ago
Yes. I was just doing my college applications and thinking this exact thing. The Common Application would look like a piddly idea if the college application process just had an API.
mrshoe 17y ago
I agree with 95% of what he says, but I'd make one change to the Unix philosophy: text files aren't the best primitive. Text files have a super low-level API (stat, seek, read). I think the system he describes would be much more usable if he picked slightly higher-level data abstractions. I would recommend Python's built-in types: list, tuple, dict, set. Most data can be easily represented by these types and they provide a higher level API than streams (files), which means the barriers to using them are much lower.
- scottjackson 17y ago
  That's probably fair. I honestly don't think we're at the point where we can abandon text files altogether, but you're right -- a lot of data could definitely be stored in slightly higher-level data structures. I still really like text files though :)
  - emmett 17y ago
    Text files are just a wire protocol. It would be nice to have a standard (similar to http) for what you send over the wire in general. Great post by the way, very thought provoking.
- silentbicycle 17y ago
  The Unix idea has never been that text files are the best primitive, it's that they're the best approach until you have a solid reason to do otherwise, and you should start simple. (Unix is more pragmatic than that - "worse is better", etc.) As long as you use plain text, you can use grep, diff, various version-control systems, etc., without having to rewrite any tools for your custom format. Still, sometimes you need more elaborate formats. SQLite is one good example of a portable, open, simple format that is complementary to flat files. JSON and Lua also seem very Unix-y to me. Unix typically also uses ELF, various compiled key-value databases, etc. Unix's choice of data types was also influenced by being written for computer that had 32k of RAM. That's, what, as much as a bar of soap nowadays?
dustingetz 17y ago
well, most data is exposed via a website, so if the api doesn't exist, just mechanize it. its not really feasible to rely on goodwill for these grand visions.
bkudria 17y ago
I want, I want, I want. How about you propose how this should be done? It's harder than it looks, and no, Unix is not the one-and-all answer to everything. Take one of his example scenarios: "When my alarm clock goes off in the morning, get the latest tech news podcast (downloaded overnight) from my computer via WiFi or a wired network connection. Play the podcast through the alarm clock’s speaker/s. If I hit “snooze” on the alarm clock, pause the podcast. Once I stop the alarm, remember where I was up to in the podcast and give that information back to my computer. Now when I go to listen to the podcast while I’m eating breakfast, it will continue playing where it left off." This has a number of hidden assumptions: * What is "the latest news podcast"? One user-selected source, or a blended filtered combination, or a crawled set selected based on previous behavior? * How does a (not the!) computer know how to talk to the alarm clock? I hope I don't have to create a home network myself, cause I know my Mom certainly can't. How is this connection authorized? What happens if the power goes out on the computer - what will the clock do? What if the clock batteries fail - will the computer play the podcast instead? * Is the saved position synced only the one source computer (I hope not). What if the network to the computer is down - can the alarm clock sync somewhere else? * Can I resume listening to the podcast on a different computer at breakfast? My partner/sibling/friend's computer? How do I log in? How is the position data saved along with my identity? How does the breakfast computer address the first computer, or the alarm clock? The problem here is not access to the external APIs (not that hard) but federation, identity management, authorization, privacy considerations, and addressing between these disparate systems. Proposal adopting existing standards for these requirements, or new proposals for standards are, of course, appreciated.
- scottjackson 17y ago
  Here's the thing -- I could come up with individual answers to those questions based on arbitrary decisions that I make ("the user selects the source", "it's implied that a WiFi network already operates in the house", "the position is only synced to the source computer", "you can listen to the podcast at a different computer"), but you're totally right -- arbitrarily-decided-on answers doth not a standard make. Like I said in another thread, I am just a peon with access to a keyboard. I trust that there are developers and system-setter-upper-ers much smarter than I who could put this into place.
milkshakes 17y ago
for anyone who hasn't seen it, http://www.ted.com/talks/tim_berners_lee_on_the_next_web.html http://www.ted.com/talks/tim_berners_lee_on_the_next_web.htm... provides a nice context

34 points by scottjackson 17y ago

This is ridiculous, we can't just let people have access to their data!... i mean OUR data. Just imagine, they may learn things we could have forced them to pay absurd fees to get! :P

DanielStraight 17y ago

As right as you are, making an API for your service so people can replace your crappy interface with their own good interface probably isn't the best business decision. Online banking sites can display ads. The bank can't inject ads into another client though. Also consider that the bank would get blamed in popular opinion if someone wrote an insecure client that was determined to be leaking data.

scottjackson 17y ago
I think it could be done -- I mean, Pine's an example of a secure client application that uses a kinda-API (SMTP) to deliver data. If we don't make APIs because someone might write an insecure client, would anyone make any APIs ever? I do agree with you though :) It's probably not going to happen in the real world any time soon. C'est la vie.
- DanielStraight 17y ago
  I'm sure it could be done. But consider for example if Bank of America did this and someone created an awesome client app that basically became the main way anyone used their Bank of America account. If a bug is discovered in this application that means everyone's data is no longer secure, are users going to blame the app creator or Bank of America? Can Bank of America afford to take that chance?
  - scottjackson 17y ago
    You're right -- that's almost certainly the reason you don't see stuff like this happen. Though, didn't Bank of America make that iPhone app a little while ago that allowed you to photograph a cheque in order to deposit it? They're obviously OK with sending financial data around. I guess maybe it was OK in that case because they were the ones that wrote the app? In any case, you're right. Security is definitely the reason this kind of stuff hasn't been implemented. I still think that it's possible to write an open, secure interface to things. Email is the example I fall back on here. Like I mentioned above - sensitive information gets sent around in emails all the time, yet there are countless email clients and they're almost all secure (I say "almost all" to cover my ass if there's some that aren't). I think that if it can work for email, then maybe it can work in other domains.
    - grncdr 17y ago
      Unfortunately, email is a terrible example. While it's possible to have a modicum of security in regards to receiving your e-mail, spoofing emails is trivially easy (unless you use cryptographic signing, which isn't well supported by the majority of clients afaik)
  - dustingetz 17y ago
    yes, but what about when I mechanize my own API to my bank anyway? it's theoretically impossible to detect more effectively than cat-n-mouse.
    - DanielStraight 17y ago
      I don't see how that's related at all. I'm not saying Bank of America needs to work hard to prevent automated access to their online banking system. Right now, automated access counts as "undocumented behavior" at best. Bank of America is not likely to face public ridicule because someone hacked up an automated interface to their site which was not intended for that purpose and shot themselves in the foot. If, however, Bank of America says explicitally, "Hey, come use this API and applications built off it to access your account," and then one of those applications starts shooting people in the foot, people are going to wonder why Bank of America thought it was a good idea to let people access their accounts like that in the first place.

Raphael 17y ago

Couldn't agree more. This is what my next startup is based on.

mattmanser 17y ago

Personally it sounds like a security and privacy nightmare to me.

scottjackson 17y ago
Maybe. I think that if we get enough smart, awesome people working on it, it could be secure, safe and open all at the same time.

mrtron 17y ago

But at what cost?

hxa7241 17y ago

The scenario descriptions are distinctly reminiscent of Berners-Lee's scenario descriptions for the 'semantic web' several years ago. Of course, that is now rather out of fashion, but not entirely justifiably. The 'semantic' angle was somewhat fanciful, but the core idea was the data web: It is one thing to put APIs on everything, but another for all the data types and formats to work together. That was more the aspect that RDF was addressing. It is neither thoroughly convincing nor completely apposite, but worth factoring in to the consideration anyway. (The essential RDF concept was elegant, whatever anyone says.)

d4rt 17y ago

For my own personal use I think text based interfaces (with pipes) are superior, the higher level pluggable components in AppleScript/Automator seem more usable in general. That said, there was an excellent article recently with the experiences of a trainer in getting people to use *nix, who found the minimalistic "command mode" more usable due to it's text oriented simplicity.

tocomment 17y ago

Should creating email accounts have an API? How would the captcha be presented? How about adding reviews of products?

scottjackson 17y ago
I don't personally know because I'm just a peon with a qwerty keyboard, but I think there are some great developers and talented architects out there who could figure it out and make it happen.
deleted 17y ago
[deleted]
silentbicycle 17y ago
Don't captchas and open APIs fundamentally oppose each other? The point of an API is to have a clear interface so that programs can be built on a service, and captchas exist to try to block bots from using services.
- tocomment 17y ago
  That's my point. Not everything <should> have an API since there are so many things we put captchas around.
grncdr 17y ago
That would be an easy API to implement, in psuedo-code: do: server->return any errors client->display errors client->ask user for account name client->server->request account name x while ( server->x is invalid account name ) << Repeat above loop for password >> do: server->return new captcha client->display captcha client->take user input client->server->send captcha text while(server->captcha does not match) server->return success message
- stuartjmoore 17y ago
  Web browsers display CAPTCHAs every day with an API known as HTML.

grncdr 17y ago

As previously said, the financial incentive is simply not there. As much sense as it makes for you to have easy access to your information, there are too many businesses that rely on controlling that access. On the other hand, tools to do this with almost any site already exist. RSS is common enough, and where it doesn't exist, screen scraping isn't that hard (and getting easier all the time). If I'm not mistaken, this is the exact problem that Yahoo pipes are trying to address. Of course, sites that require a complicated login process (banking sites) are still out.

stuartjmoore 17y ago
I think Twitter is an example that owes its success to its API; it depends what your business plan is. As more companies become Google-esque data hogs, APIs will probably be encouraged.

stuartjmoore 17y ago

API is kind-of a misnomer in this case. API implies each service has its own set of rules, the data is open, but the access is different. Everything should talk to one another, but a service's name or url should never have to appear in any code. That's the user's job to input.

clawrencewenham 17y ago

Having a good API for everything would be almost as good as The Singularity (tm), but making a good and secure API is very hard and comes with a massive hidden cost. 1) An API can require even more planning to design than the application or service itself. 2) Mistakes made in the design or implementation of the API turn into liabilities you can't solve without breaking users of the API. 3) The cost of maintaining an API so that it and its documentation doesn't fall out of date with the app/service can be as high (and sometimes even higher) as the cost of maintaining the app itself. 4) It widens the attack surface. And those are just the ones off the top of my head.

grncdr 17y ago
I agree with all of your points, but I think many of these costs are minimized by the other point mentioned in the OP: following the UNIX philosophy. The smaller and simpler your service is, the better. If you have a service that can't be accessed with a simple API, you probably have more than one service, and would benefit from splitting it into multiple APIs. Another thing worth mentioning is the possibility of building your API first, and then building your service on top of it. If you are providing a web service, then it [usually] boils down to data access and management. Figure out what data you are managing for your users and what you need to do with it, then build your API around that. If you plan to have a public facing API from the start, this approach makes sense (in the same way that using javascript only for progressive enhancement of a working site makes sense). Of course, because you are an "internal user" of the API, you can bend the rules a little more than the average client, but the less you do this, the more useful your API is. Regarding points 2 and 4: if public APIs for data access become the norm, this will simply become the cost of doing business. Secure API's aren't that difficult, and updating an API while maintaining backwards compatibility can be done, especially if you transition these changes and notify all broken clients.
- Periodic 17y ago
  For the UNIX philosophy I like the idea of small, useful tools loosely coupled. A big part of UNIX is that there are a lot of little tools that do one thing well. The smaller the tool is, less attack surface there will be, and each tool can interact with a common authentication/authorization interface to further simplify things.

dwiel 17y ago

The printing press enabled more people to read, though still few could write. The net has enabled more people to write, though still few can program. A net where everything has an API is an extremely helpful step in creating a system which enables more people to program.

rythie 17y ago

They shouldn't just have an API, they should use the same APIs where possible. Each site having it's own API is a major barrier.

ryah 17y ago

I want a build system with an external API.

SlyShy 17y ago

Yes. I was just doing my college applications and thinking this exact thing. The Common Application would look like a piddly idea if the college application process just had an API.

mrshoe 17y ago

I agree with 95% of what he says, but I'd make one change to the Unix philosophy: text files aren't the best primitive. Text files have a super low-level API (stat, seek, read). I think the system he describes would be much more usable if he picked slightly higher-level data abstractions. I would recommend Python's built-in types: list, tuple, dict, set. Most data can be easily represented by these types and they provide a higher level API than streams (files), which means the barriers to using them are much lower.

scottjackson 17y ago
That's probably fair. I honestly don't think we're at the point where we can abandon text files altogether, but you're right -- a lot of data could definitely be stored in slightly higher-level data structures. I still really like text files though :)
- emmett 17y ago
  Text files are just a wire protocol. It would be nice to have a standard (similar to http) for what you send over the wire in general. Great post by the way, very thought provoking.
silentbicycle 17y ago
The Unix idea has never been that text files are the best primitive, it's that they're the best approach until you have a solid reason to do otherwise, and you should start simple. (Unix is more pragmatic than that - "worse is better", etc.) As long as you use plain text, you can use grep, diff, various version-control systems, etc., without having to rewrite any tools for your custom format. Still, sometimes you need more elaborate formats. SQLite is one good example of a portable, open, simple format that is complementary to flat files. JSON and Lua also seem very Unix-y to me. Unix typically also uses ELF, various compiled key-value databases, etc. Unix's choice of data types was also influenced by being written for computer that had 32k of RAM. That's, what, as much as a bar of soap nowadays?

dustingetz 17y ago

well, most data is exposed via a website, so if the api doesn't exist, just mechanize it. its not really feasible to rely on goodwill for these grand visions.

bkudria 17y ago

I want, I want, I want. How about you propose how this should be done? It's harder than it looks, and no, Unix is not the one-and-all answer to everything. Take one of his example scenarios: "When my alarm clock goes off in the morning, get the latest tech news podcast (downloaded overnight) from my computer via WiFi or a wired network connection. Play the podcast through the alarm clock’s speaker/s. If I hit “snooze” on the alarm clock, pause the podcast. Once I stop the alarm, remember where I was up to in the podcast and give that information back to my computer. Now when I go to listen to the podcast while I’m eating breakfast, it will continue playing where it left off." This has a number of hidden assumptions: * What is "the latest news podcast"? One user-selected source, or a blended filtered combination, or a crawled set selected based on previous behavior? * How does a (not the!) computer know how to talk to the alarm clock? I hope I don't have to create a home network myself, cause I know my Mom certainly can't. How is this connection authorized? What happens if the power goes out on the computer - what will the clock do? What if the clock batteries fail - will the computer play the podcast instead? * Is the saved position synced only the one source computer (I hope not). What if the network to the computer is down - can the alarm clock sync somewhere else? * Can I resume listening to the podcast on a different computer at breakfast? My partner/sibling/friend's computer? How do I log in? How is the position data saved along with my identity? How does the breakfast computer address the first computer, or the alarm clock? The problem here is not access to the external APIs (not that hard) but federation, identity management, authorization, privacy considerations, and addressing between these disparate systems. Proposal adopting existing standards for these requirements, or new proposals for standards are, of course, appreciated.

scottjackson 17y ago
Here's the thing -- I could come up with individual answers to those questions based on arbitrary decisions that I make ("the user selects the source", "it's implied that a WiFi network already operates in the house", "the position is only synced to the source computer", "you can listen to the podcast at a different computer"), but you're totally right -- arbitrarily-decided-on answers doth not a standard make. Like I said in another thread, I am just a peon with access to a keyboard. I trust that there are developers and system-setter-upper-ers much smarter than I who could put this into place.

milkshakes 17y ago

for anyone who hasn't seen it, http://www.ted.com/talks/tim_berners_lee_on_the_next_web.html http://www.ted.com/talks/tim_berners_lee_on_the_next_web.htm... provides a nice context