10 ms·

The “API Mandate” memo at Amazon

381 points by laingc 5y ago

eyelidlessness 5y ago
This really shows in the products Amazon offers to developers. Everything that isn’t already an entrenched business success is some random internal tooling that got productized because everything does.
dannyw 5y ago
Is this actually still applied in 2021? Most memos from two decades ago aren't still followed.
- vmurthy 5y ago
  A perspective : Incentives, social proof and momentum are powerful things which might explain why this could still be applied in 2021 (I have no connection to Amazon or know anyone there so this is a theory). What do I mean? Imagine you are a newbie who just joined a project that has successfully done what Jeff said in the memo. The success of undertaking that would have meant that your colleagues will want to continue doing this (threat of firing or not :) ) and pretty soon you'll get sucked into it and hopefully see the rewards of such design. Soon, another team notices your team consistently getting things right and getting rewards so they have an incentive to follow (social proof). This becomes department wide next and so on. This is where momentum comes into picture. It is 2015 (let's say) and there are a dozen new departments. All of them reasonably want to get going quickly so they take up patterns that worked for other in the org. 6 more years pass by with more successes and there's no real incentive (at-least org-wide) to do something different to the one that works. My educated guess: The memo is still followed.
- __derek__ 5y ago
  > Is this actually still applied in 2021? Yes, absolutely. Arguably even more so now that Lambda and ECS/Fargate have reduced the cost of standing up a simple service.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
mmahemoff 5y ago
Always worth reading Yegge's insider take on this - https://gist.github.com/chitchcock/1281611 https://gist.github.com/chitchcock/1281611. The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate -- ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.
- Danieru 5y ago
  That Gist is the source of the "email": #7 is fake and was added by Yegge as a joke.
  - haolez 5y ago
    "But I'll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network." This is gold :D
  - throwaway894345 5y ago
    “Jeff Bezos doesn’t give a f*ck about your day”. I still laugh at that.
- lstamour 5y ago
  Background - Oct 12, 2011: > Google engineer Steve Yegge was trying to start a robust internal discussion, not post a viral hit, when he published a 4,570-word self-styled rant about what he sees as the company’s greatest flaw to Google+. Unfortunately for Yegge, he didn’t check the settings and shared his view on Google’s failure to grasp platforms over products — including Google+ — with everyone. > He later pulled it down on his own accord but he and Google aren’t asking that the copies already spread across the net be deleted. You can read the full post here and here, among other locations — and you should to get the real flavor about why Yegge thinks the company that does nearly everything right gets this fundamental so wrong. But a large chunk is also about his former employer Amazon, what it does wrong and how Jeff Bezos — Steve Jobs “without the fashion or design sense” — got it so right. Source: https://gigaom.com/2011/10/12/419-the-biggest-thing-amazon-got-right-the-platform/ https://gigaom.com/2011/10/12/419-the-biggest-thing-amazon-g... (Note, I think the post I'm citing might be incomplete as on my screen it stops at "Playstation Network" while the Gist continues...)
deleted 5y ago
[deleted]
lxe 5y ago
Things like these are taken as dogma or a religion -- and are applied to all things in an organization, for better or for worse. When there's a case for tighter coupling and less services, (and yes, there are cases for it), this memo gets brought up and microservices win the argument.
- drewcoo 5y ago
  What is an organizational case for tighter coupling and fewer services? Downsizing with an intent to not grow in the same direction again?
  - lumost 5y ago
    Each Service adds a marginal cost to maintenance, ops, infrastructure and debatably development speed (Easier to refactor in an IDE and semi-atomically deployed code than 50 independent services ). If you have a team of 5 people, launching 50 services is probably not as efficient as 10 services.
  - ChrisMarshallNY 5y ago
    As someone that tends to work this way, I can tell you that every API adds overhead. This is because they need to be documented, tested, and release-managed. Each of my APIs is a self-contained project, with its own lifecycle. That can, potentially, add a lot of overhead, and “concrete galosh”[0] to the project. And I am not a fan of dogma, in general. I like flexibility, and dogma is anathema to flexibility. [0] https://littlegreenviper.com/miscellany/concrete-galoshes/ https://littlegreenviper.com/miscellany/concrete-galoshes/
  - jimbokun 5y ago
    > Downsizing with an intent to not grow in the same direction again? More like we're not really big enough yet to justify multiple services, since everything still runs on a single beefy machine and we don't have enough experience running the system yet to really know where to put the service boundaries.
nn3 5y ago
So Amazon has no teams that write simple libraries for other teams? I imagine now that somewhere in amazon there is a "qsort" REST API that does all the sorting.
- __derek__ 5y ago
  Libraries exist, too. The glib example is pre-built clients for all those services, but there are also things like retry helpers.
- 8note 5y ago
  There's some frameworks, but libraries are generally under a "community support" model where if you're using the library, you fix the bugs you find. If you want a team supporting that qsort function,you want it behind a rest api
simonw 5y ago
Does anyone know who else was involved in constructing this memo? "There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever" Was Bezos deeply enough involved in Amazon's engineering to set those rules himself, or was the text of the memo influenced by a senior engineering group that he was working with?
- ayewo 5y ago
  Many people may not know this but Jeff Bezos clearly has a technical background as evidenced by this blurb from his Wikipedia entry: “graduated from Princeton University in 1986. He holds a degree in electrical engineering and computer science”. Of course he’s more known for his decision-making ability as an executive but he clearly has a solid understanding of computing fundamentals as his memo on Amazon S3 characterized it as “malloc for the Internet” [0][1]. 0: https://aws.amazon.com/blogs/aws/eight-years-and-counting-of-cloud-computing/ https://aws.amazon.com/blogs/aws/eight-years-and-counting-of... 1: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...
  - toyg 5y ago
    I mean, this pic of him from 1995 screams "nerd": https://www.seattletimes.com/business/amazon/the-seattle-times-first-story-about-jeff-bezos-when-you-had-to-hook-up-to-the-internet-22-years-ago/ https://www.seattletimes.com/business/amazon/the-seattle-tim... And his first office was literally for one desk. He clearly did some of the tech work himself.
- wging 5y ago
  The "text of the memo" is really just the article's author not understanding the context of Steve Yegge's years-later retelling, from which that text comes. "His Big Mandate went something along these lines" (Yegge's words right before the quoted text) doesn't even imply there was a singular 'memo' involved, and definitely (and obviously) wasn't meant to say that the text was actually what Bezos wrote. So the footnote "Whether or not it existed in this exact form" is unnecessarily ambiguous about whether these were Bezos' words. They're clearly not - besides Yegge not saying they are, they're a dead ringer for Yegge's style and not at all Bezos'.
flakiness 5y ago
Is this the same doc as "Distributed Computing Manifesto" mentioned in Werner Vogels' blog? [1] These legendary Amazon Memos haven't leaked, unlike Billg's various memos [2]. That is... journalistically unfortunate, I would say. I even wonder current Amazonian actually has the access to these docs. [1] https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html https://www.allthingsdistributed.com/2019/08/modern-applicat... [2] https://lettersofnote.com/2011/07/22/the-internet-tidal-wave/ https://lettersofnote.com/2011/07/22/the-internet-tidal-wave...
- ItsMonkk 5y ago
  I posted a comment on libraries vs frameworks yesterday[0], here's the relevant section. > Frameworks are easier to setup initially, but they do not scale. Why is it that Microsoft Windows has 13 different dialog generations? Because each is a framework on top of a framework on top of a framework. It's amazing that they can even get that done. > On the other side, OSS is generally built on libraries. When the 2 UNIX devs were in a basement building UNIX and were able to out-compete Multics[1], they did it because they were building libraries that could talk with each-other using pipes around the boundary. Applications that communicate based on input-output with no internal state behave just like pure functions do. Pure functions compose. When Linus built git in 10 days, he was able to do this because the core idea of git isn't actually that much work. The library is built out of composable blocks that neatly come together. Microsoft's TFS Source Control is a framework that acts on your behalf and therefore the bigger the project gets, you need n^2 people to work on it. Amazon's API mandate is the same exact unification as UNIX's pipe effect was to Operating Systems. Amazon's internal teams are building libraries whereas all other companies started at the same time were building frameworks. Jeff was brilliant to see this at the time, and I expect that this memo will be seen as just as pivotal as the Toyota Production System, and it might already have that prestige to some. Unfortunately for other companies, you can't retrofit into it and rewrites are always a terrible idea[2]. [0]: https://news.ycombinator.com/item?id=27570917 https://news.ycombinator.com/item?id=27570917 [1]: https://www.youtube.com/watch?v=3Ea3pkTCYx4 https://www.youtube.com/watch?v=3Ea3pkTCYx4, thanks to this HN comment(https://news.ycombinator.com/item?id=27494671 https://news.ycombinator.com/item?id=27494671) for this reference. [2]: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
barneygale 5y ago
> Anyone who doesn’t do this will be fired. > Thank you; have a nice day! What a colossal twat.
- xyzelement 5y ago
  It's really easy to be a nice guy by reacting the way you did, but if the quote above is real, it should be studied and celebrated. Let me break it down for you - Amazon bet the farm on this strategy and it worked out amazingly. Not following this strategy is equivalent to sabotaging the most important thing the company is doing. While I suspect this quote is tongue in cheek, it SHOULD be a fire able offense for someone to ignore company strategy because "they know better" or are too lazy or whatever.
  - yosamino 5y ago
    How did we end up in a place where we demand the system of government be democratic, with all the emotional language of freedom, self-determination and so on. And then we carved out an exception and made it so our places where we work are run as oppressive dictatorships... Dont agree with what the bossman said?: "You're fired!" And the people love it so much, they even democratically elected the poster child of that catchphrase.
    - oblio 5y ago
      You have it upside down. Everything else is authoritarian, except for (some) governments, because governments truly have the power of life or death over you. But everything else? Highly undemocratic.
    - jkepler 5y ago
      > How did we end up in a place where we demand the system of government be democratic, with all the emotional language of freedom, self-determination and so on. And then we carved out an exception and made it so our places where we work are run as oppressive dictatorships... Private enterprise rests on property rights. Thus Bezos, as owner, was free to write that memo, including spelling out the consequences of sabotaging the company's strategy through insubordination or incompetence. The only way this is dictatorial is if people are coerced to work there without a legal right to quit at any time. Their employment contracts given them that right (I assume) and they spell out the consequences of quitting without giving proper notice---which one could do. Thus, they're employees, not slaves in a dictatorship. That's not to say that Amazon or other large corporations don't have problems with mistreating workers. In fact, thinking of businesses as machine systems may encourage a mindset among management that risks dehumanizing the people who do the work. And this problem is far broader than amazon. However, dehumanizing workers isn't inherent in private property and owners' rights to run their firms as they see fit (within the constraint of law). Look at the Guinness company - privately owned, yet a pioneer in treating workers really well, and people have tasted the quality of their work round the world. Guinness believed people have inherent dignity (as a Christian he knew they were each made in God's image), so as a business owner he knew it was good for them and for his business to treat them well.[1] Dehumanizing employees is often a result of misaligned incentives in the legal system, of unjust laws that don't fit with reality, or more fundamentally a result of the deep levels of brokenness that exist in every human being. No one is perfect, and no human system is flawless. The distortion of private property and resulting authoritarianism in business that you ask about is a sad result of what the Bible calls sin. [1] https://www.amazon.com/Search-God-Guinness-Biography-Changed/dp/0718011333 https://www.amazon.com/Search-God-Guinness-Biography-Changed...
roughly 5y ago
As much as anyone who uses AWS and its various client libraries can attest to the areas this manifesto didn't solve, I think it's still very much the right way to go, and I think the success of AWS was at least significantly attributable to it. I've been part of several projects whose goal was to build Good interfaces on top of services after the fact, and I'm absolutely convinced that the shittiest interface in the world built in from the start trumps whatever you think you're going to be able to do later.
bogwog 5y ago
If every part of Amazon is really like that, then it'll make the task of breaking them up much easier!
- meowface 5y ago
  That's actually quite possibly true, if they were indeed broken up. But a hypothetical breakup might also forbid certain (or any) kinds of collaboration between any of them, which might include calling each other's APIs.
  - adolph 5y ago
    Otherwise: Later versions of the Hydra story add a regeneration feature to the monster: for every head chopped off, the Hydra would regrow two heads. https://en.wikipedia.org/wiki/Lernaean_Hydra https://en.wikipedia.org/wiki/Lernaean_Hydra
    - jjeaff 5y ago
      >Heracles required the assistance of his nephew Iolaus to cut off all of the monster's heads and burn the neck using a sword and fire.
  - not2b 5y ago
    It would mean that if two split off pieces of Amazon want to call each others' APIs, other companies could also do so under the same terms (if there is a charge, everyone pays the same rate). This could work.
macintux 5y ago
I wish GitHub under Microsoft followed this philosophy. So much of their repository management can be done through their APIs, but you hit some painful brick walls around things like enterprise security where you could really use centralized management. My business area uses around 200 repositories. APIs aren’t really optional at that scale.
- pushrax 5y ago
  There are multiple large Rails services (I'd guess GitHub included) that internally have a majority of contributors in favor of an API first approach, but it's not mandated absolutely. Shopify has a component boundary interface in Ruby that can be reflected into with GraphQL, and a lot of features are built for GraphQL first anyway. First party client side apps demand it. A lot of internal services use GraphQL to talk server-server as well. However, there's still a good chunk of monolithic logic left that hasn't been refactored yet. Refactoring efforts are mostly JIT when demanded.
justicezyx 5y ago
Have this mandate in Google You'll have every engineer complaining and stop working to fight their freedom and finally the change has to be reverted. And 5 years later, oh my god, Amazon is doing that, we need to move to that direction as well...
leishman 5y ago
> All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. I wonder how this is accomplished for event driven architectures built on shared event buses.
- CameronNemo 5y ago
  In that case, the bus, producers, and consumers would all be treated as separate services, no? The memo does mention pub sub, FWIW.
- fake-name 5y ago
  Presumably, you expose the ability to inject events onto the bus, and the ability to listen for events on the bus?
- bobnamob 5y ago
  SQS and SNS are services offered by AWS
markwkw 5y ago
It's interesting and often missed that this strategy implies a view of human collaboration and organization. Note that the memo says: 'All teams... Teams must... ...another team’s data store...'. It seems that someone figured out that a (5-10 person?) team is the right human collective size to design and build useful stuff, but that collective shall expose what they build to other teams via APIs. It's fine for the team to do things in a tightly coupled way - internally! In a way, tight collaboration between teammates can be reflected in tight coupling of internal components of whatever they build. But across teams the coupling shall be more formal and via APIs, reflecting looser and less powerful collaboration possible across teams.
- adolph 5y ago
  That someone would be Bezos and his pizza person. in time, two-pizza teams evolved into single-threaded leader (STL) teams, a term borrowed from computer science that means to only work on one thing at a time https://www.inc.com/jeff-haden/when-jeff-bezoss-two-pizza-teams-fell-short-he-turned-to-brilliant-model-amazon-uses-today.html https://www.inc.com/jeff-haden/when-jeff-bezoss-two-pizza-te...
- philwelch 5y ago
  It’s a form of using Conway’s Law to your own advantage instead of living in denial of it or treating it as an antipattern.
mcintyre1994 5y ago
I wonder how an example from the article like the API to post new listings to Amazon works in practice with the requirement to be designed to be open to outside developers. It seems like that’d force some sort of review process (and I’m not really sure who can review all new listings) between API call and public availability that might not be there if you eg. had a private API for approved employees.
- mxz3000 5y ago
  In my experience, most APIs my teamed designed/built were not meant to ever be publicly available. That is, we never considered public availability as a design factor. So I think this rule doesn't actually apply anymore. Then again, I don't know how public availability would change the API design really...
  - nivertech 5y ago
    API design concerns for public availability (just to name a few): - security - preventing abuse - API Anti-Corruption Layer - sanitizing inputs and outputs - i.e. not exposing DB IDs/PKs, or pagination cursors directly - versioning, backward- and forward- compatibility, deprecation strategy - usability, DX, Documentation - reducing bandwidth use: - caching - eliminate over-fetching - efficient wire format
ChrisMarshallNY 5y ago
This is the way I tend to work. I even write about it[0] (“Keeping Things Vague”). APIs (and opaque modules, in general) are key to the way I develop software. It works well. (0) https://littlegreenviper.com/miscellany/evolutionary-design-specification/#vague https://littlegreenviper.com/miscellany/evolutionary-design-...
travisgriggs 5y ago
How far does this principle scale? I've heard of this strategy and thought "that's kinda cool." Amazon is obviously successful in its domain, so it would be easy to assume some causality. And yet, there's another big tech company, fruit symbol I think, that has played the long game of continually fine tuning the interaction points to make the integration of their parts more than the whole. They get credit for attention to detail, leveraging their integration and their ability to move in ways they do because of their thorough integration across their whole hardware/software stack. They've recently been in the press because of just how much they were able to tune their integration to the problem with their first desktop computing chip. Again, we credit their success with some causality due to this strategy, which (to me) is very different than the Amazon strategy. Is it because Amazon is a cloud company and Apple is a hardware company that they have these different approaches to product development and each enjoy success? Or at the end of the day, are these "do it this way" less responsible than we'd like to believe?
- atomicity 5y ago
  It's sort of hard to compare these approaches, since it's not like Amazon and Apple only have 1 principle that they follow. I'll try to analyze them in a generic way. Disclaimer that I don't have experience in marketing or executive leadership. Apple's focus on design allows it to charge higher for its products and to build new high-margin products that people end up buying. It allows them to "scale" revenue by entering new markets with new products. Amazon has a similar focus on the customer which is centered around customer support. This approach also gives them the "brand reputation" to build new services (that businesses will pay for). You may note that Apple's approach works better for customers who pay without much planning/budgeting (like consumers/households), scales better with the number of customers (again like consumers/households), and scales more poorly with # of products (making it a worse fit for SaaS). Amazon's API mandate value is felt in how it allows them to make software development more efficient. Customers do not feel the impact directly. Instead, since data is exposed through well-defined APIs, new service (or product) development can be done with far less human communication, as mentioned in the article. However, while this makes inter-team efficiency better, it reduces intra-team efficiency by forcing developers to build things that they don't need. If services are too small, the APIs are not high-quality, or the service boundaries change too frequently, then it's possible that this approach doesn't make software engineering more efficient at Amazon.
jesstaa 5y ago
> All teams will henceforth expose their data and functionality through service interfaces. > Teams must communicate with each other through these interfaces. I read that as interfaces to the team, instead of interfaces to software the team is responsible for. Something like Mechanical Turk(https://www.mturk.com/ https://www.mturk.com/) but for internal team communication. I'm not sure if that was what was meant but that sounds interesting.
blntechie 5y ago
Deleted as I misread the year in the article.
tus89 5y ago
> It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. CORBA is quite close to direct linking, with a network in between. The developer does not see it as a service or protocol, but a library call, which is rather the point. And it's not very compatible with the next one: > All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. CORBA/COM never played well over the internet.
- pjmlp 5y ago
  Mostly because they got hit by J2EE rewrites fashion wave. Their are now back via the gRPC fashion wave, until something else "improves" it.
jayd16 5y ago
>It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. So how does Amazon manage this btw? I don't find AWS SDKs to be all that consistent or inconsistent. They are usually good _enough_. Is that all it takes? By contrast, Google seems to spend a lot of time on their single repo, build the world, approach. For the most part it seems beloved, or at least people try to recreate it outside google with things like Bazel. I feel like Google's approach is more popularized. Is that because its actually better, just advertised more, or simply consistent enough to explain? Can anyone shed light on Amazon's approach?
- ketzo 5y ago
  > I feel like Google's approach is more popularized I mean, the engineering architecture being described in this memo is, basically, microservices. That's certainly an extremely popular -- I would go so far as to say even vogue -- pattern for solving the problem of building software at scale.
  - nivertech 5y ago
    No, they're just services - a service per team. Microservices usually overdoing it, when you have multiple microservices per team, and sometimes even per developer. Also, microservices sometimes implemented incorrectly, where they're still communicate via shared databases, instead of encapsulating them and exposing them via service APIs only. This Memo was born because of the real business need, while many modern microservices deployments are the result of cargo-culting GAFAMs/FANGs.
    - pjmlp 5y ago
      Indeed, hence "the network is the computer" footnote that used to be written on Sun manuals. Many cargo cult web services of today can be written in Sun RPC APIs. But yeah it is C and its IDL isn't as cool as gRPC proto files.
- raverbashing 5y ago
  For real, if AWS internal APIs are as good as their outside-facing ones I fear the complexity. Their APIs are... correct. But have things like usability as a last concern. You also find some things not behaving exactly as you would expect and some things that are barely explained.
ridiculous_fish 5y ago
I've always wondered how far this extends: "All teams will henceforth expose their data and functionality through service interfaces." Does it include e.g. the team writing the on-device text renderer for a Kindle? What would such a service interface look like?
- minitoar 5y ago
  Presumably that team calls some sort of package management/build api to publish their new driver or whatever.
nivertech 5y ago
Does Amazon has some standards/conventions for inter-team APIs, i.e. something like Google's AIPs (API Improvement Proposals) [1,2]? [1] https://google.aip.dev/general https://google.aip.dev/general [2] https://google.aip.dev/ https://google.aip.dev/
- rejectedandsad 5y ago
  There’s an API bar raiser system for APIs called by many teams.
Newcomb5421 5y ago
Thanks for the information keep sharing https://www.walgreenslistens.biz/ https://www.walgreenslistens.biz/
kpmah 5y ago
I've worked at a place where this was cargo-culted and it worked horribly. All the worst aspects of 'services-first' design. I think this works much better when each team is working on what could be regarded as a complete end-to-end product e.g. a database. It works poorly when each team is working on part of a product.
- jimbokun 5y ago
  If you don't get the boundaries between the various services right, the effort is going to fail spectacularly.
augustl 5y ago
> no direct reads of another team’s data store This reminds me of how App Store used to ban game emulators interpreting ROMs downloaded from the internet, while at the same time allowing XML parsing. Does it really matter if the interface you're talking to over the network is code written by another team, or data written by another team? I suppose implementation details are often hidden away in data stores, but does it have to be that way?
- dragonwriter 5y ago
  > Does it really matter if the interface you’re talking to over the network is code written by another team, or data written by another team? Yes, because it means the depedencies between systems will consistently be API dependencies, not a mix of API and datastore structure dependencies, which means that the other team will only have to reduces the constraints on change.
- draw_down 5y ago
  Well, it isn’t really like that at all I would say. The reason for the Amazon rule is to prevent unseen coupling between teams. Explicit boundaries are critical for this to scale.
- mattikl 5y ago
  Direct reading effectively makes the data store an API, i.e. the API you provide is SQL SELECT over multiple tables instead of HTTP GET resource. This pretty much freezes the database schema, the team owning it can no longer change it, because they don't know how the users use it. It also limits the possibilities to provide optimizations on the data (like caching).
projectileboy 5y ago
Kind of lame that it doesn’t even mention Steve Yegge or his blog, which is almost certainly the source.
k__ 5y ago
"HTTP, Corba, Pubsub, custom protocols — doesn’t matter." That could have backfired pretty badly, haha.
- omginternets 5y ago
  I'm not sure which would be worse: CORBA or some custom protocol...
  - LanceH 5y ago
    Presumably the custom protocol would map well to at least one side of the API.
typedef_struct 5y ago
I spent 2020 at AWS and I can't think of a single part of their toolchain that exposed APIs as first-class citizens. Not Pipelines, not Apollo, not CRUX, not Brazil, not MCM. Its all command line applications and browser interfaces (the stuff first-year developers are most familiar with building, I suppose). I was familiar with this memo so it was quite a shock. Wish they'd taken their own advice.
- vermilingua 5y ago
  …and what do the cli apps and browser interfaces use to connect with the tools?
  - ncsurfus 5y ago
    Yeah, all of those tools have APIs…
Gayax 5y ago
It is exactly what author Cal Newport is recommending in his new book "A World Without Email" (March 2021): replacing the ping-pong of emails and meetings by standardized processes and requests via tools. It's simply incredible to see that Bezos saw this coming 20 years ago!
- tome 5y ago
  I came across an interesting blog discussing such ideas a few years ago. I don't think anything ever came of it though http://thingamy.typepad.com/ http://thingamy.typepad.com/
amazondrone12 5y ago
As a current Amazon employee I can confirm this is no longer the case or only applies to such a select few pieces of software that it doesn't actually mean what you think it does. The amount of day to day stuff that is relied upon that runs on greasemonkey scripts and web scraping is insane. API's for a ton of things don't exist or are have heavy gatekeeping. (eg. contact x to get onboarded to our API and they just ghost you). Of course you can get a high level leader to tell you to use a proper API in a mailing list email thread where you are trying to get help to solve a problem but they will not actually help you get working API access, all talk no action. A perfect example is there is this basic CRUD app that I rely upon and I see the current maintainer is working on all these feature requests, the problem for me is that the site has a ton of AJAX and takes like 20 mouse clicks to get retrieve basic information. So I open a ticket / feature request just to get a basic REST API for the reads and they politely tell me I am an idiot and close the request. So much pointless work is created at Amazon from lack of API access.
- geodel 5y ago
  This seems so true. A friend of mine got hired in Northern Virginia. He was Java developer for ten years. But at Amazon it is about same as you described.
- biztos 5y ago
  Sounds like the maintainer is optimizing for job security and/or work-life balance. And from all the comments on this post, it sounds like that’s a pretty rational course of action.
- phkahler 5y ago
  Sounds like they're allowing people to treat the systems as "infrastructure" and "applications" with only the infra having exposed APIs? That's super common in software of all sorts. There are a lot of people who can make good use of a decent API and appreciate it, but are not experienced (or smart, or have time, etc) enough to design a good API themselves. Most software is mortar, not bricks.
  - res0nat0r 5y ago
sgt101 5y ago
I worked at a company that read the Amazon memo virtually straight after it was published and decided to copy it. The CIO at the time predicated every system's teams' bonus on it. No API, no bonus. What followed was 18mths of people building enterprise messaging systems and "buses" of various types. Hundreds of pages of documentation on APIs was prepared and released. I think 2 API's with maybe 40 methods actually went live. There were no bonuses Many people left The people who stayed were unhappy It never happened, everyone quietly forgot about it after 18 mnths. The CIO stayed for another 3 or 4 years - he only left when a new CEO came in due to internal promotion. The new CEO used to run one of the P&L's/business units and hated the CIO with a passion. So... it's not the API mandate that made the difference at Amazon.
- Aperocky 5y ago
  That's the wrong way to do it though. You can't just have a free for all mentality when building APIs. The platform has to be there first. Everyone's API should just need simple descriptive json files and then a client can be generated to consume that API easily. Everyone can declare those json files and it would work everywhere across the business. This free for all with no standardization is bound to be a disaster.
- beastman82 5y ago
  That is quite the conclusion you've made from a single data point!
  - sgt101 5y ago
    Well - you can argue about the opposite as well! But I would claim that this is an existence proof - if the management will and focus is there for an API enabled business (it was) this case study proves that is not sufficient to create the kind of company that Amazon became. An API program isn't a magic bullet (as everyone knows by now I guess!)
    - bhupy 5y ago
      The following can be true at the same time: - An API program isn’t a magic bullet - An API program is a necessary, but not sufficient condition to replicate Amazon’s success You’re right that this case study proves that it alone is insufficient to create the kind of company that Amazon became, but does it also prove that it’s unnecessary to create the kind of company that Amazon became?
Hippocrates 5y ago
I love this. As others have mentioned I wonder how much input Jeff had, and from who, before making the mandate. I am experiencing the polar opposite of this mandate. The systems in my organization are always built to require human touch-points. What's worse, our CTO mistakes these menial interactions as "teamwork" and "collaboration" when they are really just toil to compensate for the lack of platform-level thinking. I love how a CEO can put this so bluntly, upend everyones work for a couple of years and build a juggernaut because of it. The idea runs parallel to one I have been championing throughout my career with SOAs which I call "self serve architecture". I want others in the organization to be able to pick up use my services to their benefit with zero input or help from me or my team. I tell my team to design the API as if it were GitHub's API. Practically, that means - There are up-to-date and easy docs that cover what you need to know. - People can gain access on their own (via some existing workplace/team based credential). At most we have to add them to a list somewhere. - The system will protect itself and inform users against problematic use (quotas, throttling, and visibility into this) - You have visibility into who your users are and what they are doing such that you can assess value, learn from usage, and communicate to consumers when necessary.
bruce343434 5y ago
Or: how to waste cycles and make your codebase run slow as shit on a networked computer cluster.
tomtato 5y ago
I had always read this as a metaphorical memo. Not necessarily that every team must have a software service running somewhere in infrastructure serving RESTful routes but rather as a way of thinking about agreements between groups. Your team's "API" could be documents describing: if you want us to do XYZ, send us a request via ABC, and you should expect a response in UVW format in x amount of time.
wly_cdgr 5y ago
Like all the best myth making propaganda this has some sense and truth to it
deeblering4 5y ago
> 6. Anyone who doesn’t do this will be fired. Sounds about right Azon turned throwaway compute instances and one-click 2 day delivery ecom into huge cash cows. They aren't some golden example of how to properly design infrastructure, it just worked out for them. luck/timing/reinvestment had a lot to do with it.
senderista 5y ago
I don’t think Yegge is a very reliable source in general, even though he was there at the time. I’m not sure why his version of these events is taken as gospel everywhere. I never saw Bezos concern himself with technical matters while I was at Amazon (though I was in AWS, which he seemed happy to leave to ajassy).
ChrisArchitect 5y ago
this is a post breaking down a pseudo-mythological memo from 2002 at a company where things have obviously changed and evolved, and there are multiple upvoted comments in here from amazon people saying this isn't the case or things have changed etc.

381 points by laingc 5y ago

This really shows in the products Amazon offers to developers. Everything that isn’t already an entrenched business success is some random internal tooling that got productized because everything does.

dannyw 5y ago

Is this actually still applied in 2021? Most memos from two decades ago aren't still followed.

vmurthy 5y ago
A perspective : Incentives, social proof and momentum are powerful things which might explain why this could still be applied in 2021 (I have no connection to Amazon or know anyone there so this is a theory). What do I mean? Imagine you are a newbie who just joined a project that has successfully done what Jeff said in the memo. The success of undertaking that would have meant that your colleagues will want to continue doing this (threat of firing or not :) ) and pretty soon you'll get sucked into it and hopefully see the rewards of such design. Soon, another team notices your team consistently getting things right and getting rewards so they have an incentive to follow (social proof). This becomes department wide next and so on. This is where momentum comes into picture. It is 2015 (let's say) and there are a dozen new departments. All of them reasonably want to get going quickly so they take up patterns that worked for other in the org. 6 more years pass by with more successes and there's no real incentive (at-least org-wide) to do something different to the one that works. My educated guess: The memo is still followed.
__derek__ 5y ago
> Is this actually still applied in 2021? Yes, absolutely. Arguably even more so now that Lambda and ECS/Fargate have reduced the cost of standing up a simple service.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

mmahemoff 5y ago

Always worth reading Yegge's insider take on this - https://gist.github.com/chitchcock/1281611 https://gist.github.com/chitchcock/1281611. The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate -- ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.

Danieru 5y ago
That Gist is the source of the "email": #7 is fake and was added by Yegge as a joke.
- haolez 5y ago
  "But I'll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network." This is gold :D
- throwaway894345 5y ago
  “Jeff Bezos doesn’t give a f*ck about your day”. I still laugh at that.
lstamour 5y ago
Background - Oct 12, 2011: > Google engineer Steve Yegge was trying to start a robust internal discussion, not post a viral hit, when he published a 4,570-word self-styled rant about what he sees as the company’s greatest flaw to Google+. Unfortunately for Yegge, he didn’t check the settings and shared his view on Google’s failure to grasp platforms over products — including Google+ — with everyone. > He later pulled it down on his own accord but he and Google aren’t asking that the copies already spread across the net be deleted. You can read the full post here and here, among other locations — and you should to get the real flavor about why Yegge thinks the company that does nearly everything right gets this fundamental so wrong. But a large chunk is also about his former employer Amazon, what it does wrong and how Jeff Bezos — Steve Jobs “without the fashion or design sense” — got it so right. Source: https://gigaom.com/2011/10/12/419-the-biggest-thing-amazon-got-right-the-platform/ https://gigaom.com/2011/10/12/419-the-biggest-thing-amazon-g... (Note, I think the post I'm citing might be incomplete as on my screen it stops at "Playstation Network" while the Gist continues...)

deleted 5y ago

[deleted]

lxe 5y ago

Things like these are taken as dogma or a religion -- and are applied to all things in an organization, for better or for worse. When there's a case for tighter coupling and less services, (and yes, there are cases for it), this memo gets brought up and microservices win the argument.

drewcoo 5y ago
What is an organizational case for tighter coupling and fewer services? Downsizing with an intent to not grow in the same direction again?
- lumost 5y ago
  Each Service adds a marginal cost to maintenance, ops, infrastructure and debatably development speed (Easier to refactor in an IDE and semi-atomically deployed code than 50 independent services ). If you have a team of 5 people, launching 50 services is probably not as efficient as 10 services.
- ChrisMarshallNY 5y ago
  As someone that tends to work this way, I can tell you that every API adds overhead. This is because they need to be documented, tested, and release-managed. Each of my APIs is a self-contained project, with its own lifecycle. That can, potentially, add a lot of overhead, and “concrete galosh”[0] to the project. And I am not a fan of dogma, in general. I like flexibility, and dogma is anathema to flexibility. [0] https://littlegreenviper.com/miscellany/concrete-galoshes/ https://littlegreenviper.com/miscellany/concrete-galoshes/
- jimbokun 5y ago
  > Downsizing with an intent to not grow in the same direction again? More like we're not really big enough yet to justify multiple services, since everything still runs on a single beefy machine and we don't have enough experience running the system yet to really know where to put the service boundaries.

nn3 5y ago

So Amazon has no teams that write simple libraries for other teams? I imagine now that somewhere in amazon there is a "qsort" REST API that does all the sorting.

__derek__ 5y ago
Libraries exist, too. The glib example is pre-built clients for all those services, but there are also things like retry helpers.
8note 5y ago
There's some frameworks, but libraries are generally under a "community support" model where if you're using the library, you fix the bugs you find. If you want a team supporting that qsort function,you want it behind a rest api

simonw 5y ago

Does anyone know who else was involved in constructing this memo? "There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever" Was Bezos deeply enough involved in Amazon's engineering to set those rules himself, or was the text of the memo influenced by a senior engineering group that he was working with?

ayewo 5y ago
Many people may not know this but Jeff Bezos clearly has a technical background as evidenced by this blurb from his Wikipedia entry: “graduated from Princeton University in 1986. He holds a degree in electrical engineering and computer science”. Of course he’s more known for his decision-making ability as an executive but he clearly has a solid understanding of computing fundamentals as his memo on Amazon S3 characterized it as “malloc for the Internet” [0][1]. 0: https://aws.amazon.com/blogs/aws/eight-years-and-counting-of-cloud-computing/ https://aws.amazon.com/blogs/aws/eight-years-and-counting-of... 1: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...
- toyg 5y ago
  I mean, this pic of him from 1995 screams "nerd": https://www.seattletimes.com/business/amazon/the-seattle-times-first-story-about-jeff-bezos-when-you-had-to-hook-up-to-the-internet-22-years-ago/ https://www.seattletimes.com/business/amazon/the-seattle-tim... And his first office was literally for one desk. He clearly did some of the tech work himself.
wging 5y ago
The "text of the memo" is really just the article's author not understanding the context of Steve Yegge's years-later retelling, from which that text comes. "His Big Mandate went something along these lines" (Yegge's words right before the quoted text) doesn't even imply there was a singular 'memo' involved, and definitely (and obviously) wasn't meant to say that the text was actually what Bezos wrote. So the footnote "Whether or not it existed in this exact form" is unnecessarily ambiguous about whether these were Bezos' words. They're clearly not - besides Yegge not saying they are, they're a dead ringer for Yegge's style and not at all Bezos'.

flakiness 5y ago

Is this the same doc as "Distributed Computing Manifesto" mentioned in Werner Vogels' blog? [1] These legendary Amazon Memos haven't leaked, unlike Billg's various memos [2]. That is... journalistically unfortunate, I would say. I even wonder current Amazonian actually has the access to these docs. [1] https://www.allthingsdistributed.com/2019/08/modern-applications-at-aws.html https://www.allthingsdistributed.com/2019/08/modern-applicat... [2] https://lettersofnote.com/2011/07/22/the-internet-tidal-wave/ https://lettersofnote.com/2011/07/22/the-internet-tidal-wave...

ItsMonkk 5y ago
I posted a comment on libraries vs frameworks yesterday[0], here's the relevant section. > Frameworks are easier to setup initially, but they do not scale. Why is it that Microsoft Windows has 13 different dialog generations? Because each is a framework on top of a framework on top of a framework. It's amazing that they can even get that done. > On the other side, OSS is generally built on libraries. When the 2 UNIX devs were in a basement building UNIX and were able to out-compete Multics[1], they did it because they were building libraries that could talk with each-other using pipes around the boundary. Applications that communicate based on input-output with no internal state behave just like pure functions do. Pure functions compose. When Linus built git in 10 days, he was able to do this because the core idea of git isn't actually that much work. The library is built out of composable blocks that neatly come together. Microsoft's TFS Source Control is a framework that acts on your behalf and therefore the bigger the project gets, you need n^2 people to work on it. Amazon's API mandate is the same exact unification as UNIX's pipe effect was to Operating Systems. Amazon's internal teams are building libraries whereas all other companies started at the same time were building frameworks. Jeff was brilliant to see this at the time, and I expect that this memo will be seen as just as pivotal as the Toyota Production System, and it might already have that prestige to some. Unfortunately for other companies, you can't retrofit into it and rewrites are always a terrible idea[2]. [0]: https://news.ycombinator.com/item?id=27570917 https://news.ycombinator.com/item?id=27570917 [1]: https://www.youtube.com/watch?v=3Ea3pkTCYx4 https://www.youtube.com/watch?v=3Ea3pkTCYx4, thanks to this HN comment(https://news.ycombinator.com/item?id=27494671 https://news.ycombinator.com/item?id=27494671) for this reference. [2]: https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...

barneygale 5y ago

> Anyone who doesn’t do this will be fired. > Thank you; have a nice day! What a colossal twat.

xyzelement 5y ago
It's really easy to be a nice guy by reacting the way you did, but if the quote above is real, it should be studied and celebrated. Let me break it down for you - Amazon bet the farm on this strategy and it worked out amazingly. Not following this strategy is equivalent to sabotaging the most important thing the company is doing. While I suspect this quote is tongue in cheek, it SHOULD be a fire able offense for someone to ignore company strategy because "they know better" or are too lazy or whatever.
- yosamino 5y ago
  How did we end up in a place where we demand the system of government be democratic, with all the emotional language of freedom, self-determination and so on. And then we carved out an exception and made it so our places where we work are run as oppressive dictatorships... Dont agree with what the bossman said?: "You're fired!" And the people love it so much, they even democratically elected the poster child of that catchphrase.
  - oblio 5y ago
    You have it upside down. Everything else is authoritarian, except for (some) governments, because governments truly have the power of life or death over you. But everything else? Highly undemocratic.
  - jkepler 5y ago
    > How did we end up in a place where we demand the system of government be democratic, with all the emotional language of freedom, self-determination and so on. And then we carved out an exception and made it so our places where we work are run as oppressive dictatorships... Private enterprise rests on property rights. Thus Bezos, as owner, was free to write that memo, including spelling out the consequences of sabotaging the company's strategy through insubordination or incompetence. The only way this is dictatorial is if people are coerced to work there without a legal right to quit at any time. Their employment contracts given them that right (I assume) and they spell out the consequences of quitting without giving proper notice---which one could do. Thus, they're employees, not slaves in a dictatorship. That's not to say that Amazon or other large corporations don't have problems with mistreating workers. In fact, thinking of businesses as machine systems may encourage a mindset among management that risks dehumanizing the people who do the work. And this problem is far broader than amazon. However, dehumanizing workers isn't inherent in private property and owners' rights to run their firms as they see fit (within the constraint of law). Look at the Guinness company - privately owned, yet a pioneer in treating workers really well, and people have tasted the quality of their work round the world. Guinness believed people have inherent dignity (as a Christian he knew they were each made in God's image), so as a business owner he knew it was good for them and for his business to treat them well.[1] Dehumanizing employees is often a result of misaligned incentives in the legal system, of unjust laws that don't fit with reality, or more fundamentally a result of the deep levels of brokenness that exist in every human being. No one is perfect, and no human system is flawless. The distortion of private property and resulting authoritarianism in business that you ask about is a sad result of what the Bible calls sin. [1] https://www.amazon.com/Search-God-Guinness-Biography-Changed/dp/0718011333 https://www.amazon.com/Search-God-Guinness-Biography-Changed...

roughly 5y ago

As much as anyone who uses AWS and its various client libraries can attest to the areas this manifesto didn't solve, I think it's still very much the right way to go, and I think the success of AWS was at least significantly attributable to it. I've been part of several projects whose goal was to build Good interfaces on top of services after the fact, and I'm absolutely convinced that the shittiest interface in the world built in from the start trumps whatever you think you're going to be able to do later.

bogwog 5y ago

If every part of Amazon is really like that, then it'll make the task of breaking them up much easier!

meowface 5y ago
That's actually quite possibly true, if they were indeed broken up. But a hypothetical breakup might also forbid certain (or any) kinds of collaboration between any of them, which might include calling each other's APIs.
- adolph 5y ago
  Otherwise: Later versions of the Hydra story add a regeneration feature to the monster: for every head chopped off, the Hydra would regrow two heads. https://en.wikipedia.org/wiki/Lernaean_Hydra https://en.wikipedia.org/wiki/Lernaean_Hydra
  - jjeaff 5y ago
    >Heracles required the assistance of his nephew Iolaus to cut off all of the monster's heads and burn the neck using a sword and fire.
- not2b 5y ago
  It would mean that if two split off pieces of Amazon want to call each others' APIs, other companies could also do so under the same terms (if there is a charge, everyone pays the same rate). This could work.

macintux 5y ago

I wish GitHub under Microsoft followed this philosophy. So much of their repository management can be done through their APIs, but you hit some painful brick walls around things like enterprise security where you could really use centralized management. My business area uses around 200 repositories. APIs aren’t really optional at that scale.

pushrax 5y ago
There are multiple large Rails services (I'd guess GitHub included) that internally have a majority of contributors in favor of an API first approach, but it's not mandated absolutely. Shopify has a component boundary interface in Ruby that can be reflected into with GraphQL, and a lot of features are built for GraphQL first anyway. First party client side apps demand it. A lot of internal services use GraphQL to talk server-server as well. However, there's still a good chunk of monolithic logic left that hasn't been refactored yet. Refactoring efforts are mostly JIT when demanded.

justicezyx 5y ago

Have this mandate in Google You'll have every engineer complaining and stop working to fight their freedom and finally the change has to be reverted. And 5 years later, oh my god, Amazon is doing that, we need to move to that direction as well...

leishman 5y ago

> All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. I wonder how this is accomplished for event driven architectures built on shared event buses.

CameronNemo 5y ago
In that case, the bus, producers, and consumers would all be treated as separate services, no? The memo does mention pub sub, FWIW.
fake-name 5y ago
Presumably, you expose the ability to inject events onto the bus, and the ability to listen for events on the bus?
bobnamob 5y ago
SQS and SNS are services offered by AWS

markwkw 5y ago

It's interesting and often missed that this strategy implies a view of human collaboration and organization. Note that the memo says: 'All teams... Teams must... ...another team’s data store...'. It seems that someone figured out that a (5-10 person?) team is the right human collective size to design and build useful stuff, but that collective shall expose what they build to other teams via APIs. It's fine for the team to do things in a tightly coupled way - internally! In a way, tight collaboration between teammates can be reflected in tight coupling of internal components of whatever they build. But across teams the coupling shall be more formal and via APIs, reflecting looser and less powerful collaboration possible across teams.

adolph 5y ago
That someone would be Bezos and his pizza person. in time, two-pizza teams evolved into single-threaded leader (STL) teams, a term borrowed from computer science that means to only work on one thing at a time https://www.inc.com/jeff-haden/when-jeff-bezoss-two-pizza-teams-fell-short-he-turned-to-brilliant-model-amazon-uses-today.html https://www.inc.com/jeff-haden/when-jeff-bezoss-two-pizza-te...
philwelch 5y ago
It’s a form of using Conway’s Law to your own advantage instead of living in denial of it or treating it as an antipattern.

mcintyre1994 5y ago

I wonder how an example from the article like the API to post new listings to Amazon works in practice with the requirement to be designed to be open to outside developers. It seems like that’d force some sort of review process (and I’m not really sure who can review all new listings) between API call and public availability that might not be there if you eg. had a private API for approved employees.

mxz3000 5y ago
In my experience, most APIs my teamed designed/built were not meant to ever be publicly available. That is, we never considered public availability as a design factor. So I think this rule doesn't actually apply anymore. Then again, I don't know how public availability would change the API design really...
- nivertech 5y ago
  API design concerns for public availability (just to name a few): - security - preventing abuse - API Anti-Corruption Layer - sanitizing inputs and outputs - i.e. not exposing DB IDs/PKs, or pagination cursors directly - versioning, backward- and forward- compatibility, deprecation strategy - usability, DX, Documentation - reducing bandwidth use: - caching - eliminate over-fetching - efficient wire format

ChrisMarshallNY 5y ago

This is the way I tend to work. I even write about it[0] (“Keeping Things Vague”). APIs (and opaque modules, in general) are key to the way I develop software. It works well. (0) https://littlegreenviper.com/miscellany/evolutionary-design-specification/#vague https://littlegreenviper.com/miscellany/evolutionary-design-...

travisgriggs 5y ago

How far does this principle scale? I've heard of this strategy and thought "that's kinda cool." Amazon is obviously successful in its domain, so it would be easy to assume some causality. And yet, there's another big tech company, fruit symbol I think, that has played the long game of continually fine tuning the interaction points to make the integration of their parts more than the whole. They get credit for attention to detail, leveraging their integration and their ability to move in ways they do because of their thorough integration across their whole hardware/software stack. They've recently been in the press because of just how much they were able to tune their integration to the problem with their first desktop computing chip. Again, we credit their success with some causality due to this strategy, which (to me) is very different than the Amazon strategy. Is it because Amazon is a cloud company and Apple is a hardware company that they have these different approaches to product development and each enjoy success? Or at the end of the day, are these "do it this way" less responsible than we'd like to believe?

atomicity 5y ago
It's sort of hard to compare these approaches, since it's not like Amazon and Apple only have 1 principle that they follow. I'll try to analyze them in a generic way. Disclaimer that I don't have experience in marketing or executive leadership. Apple's focus on design allows it to charge higher for its products and to build new high-margin products that people end up buying. It allows them to "scale" revenue by entering new markets with new products. Amazon has a similar focus on the customer which is centered around customer support. This approach also gives them the "brand reputation" to build new services (that businesses will pay for). You may note that Apple's approach works better for customers who pay without much planning/budgeting (like consumers/households), scales better with the number of customers (again like consumers/households), and scales more poorly with # of products (making it a worse fit for SaaS). Amazon's API mandate value is felt in how it allows them to make software development more efficient. Customers do not feel the impact directly. Instead, since data is exposed through well-defined APIs, new service (or product) development can be done with far less human communication, as mentioned in the article. However, while this makes inter-team efficiency better, it reduces intra-team efficiency by forcing developers to build things that they don't need. If services are too small, the APIs are not high-quality, or the service boundaries change too frequently, then it's possible that this approach doesn't make software engineering more efficient at Amazon.

jesstaa 5y ago

> All teams will henceforth expose their data and functionality through service interfaces. > Teams must communicate with each other through these interfaces. I read that as interfaces to the team, instead of interfaces to software the team is responsible for. Something like Mechanical Turk(https://www.mturk.com/ https://www.mturk.com/) but for internal team communication. I'm not sure if that was what was meant but that sounds interesting.

blntechie 5y ago

Deleted as I misread the year in the article.

tus89 5y ago

> It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. CORBA is quite close to direct linking, with a network in between. The developer does not see it as a service or protocol, but a library call, which is rather the point. And it's not very compatible with the next one: > All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. CORBA/COM never played well over the internet.

pjmlp 5y ago
Mostly because they got hit by J2EE rewrites fashion wave. Their are now back via the gRPC fashion wave, until something else "improves" it.

jayd16 5y ago

>It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. So how does Amazon manage this btw? I don't find AWS SDKs to be all that consistent or inconsistent. They are usually good _enough_. Is that all it takes? By contrast, Google seems to spend a lot of time on their single repo, build the world, approach. For the most part it seems beloved, or at least people try to recreate it outside google with things like Bazel. I feel like Google's approach is more popularized. Is that because its actually better, just advertised more, or simply consistent enough to explain? Can anyone shed light on Amazon's approach?

ketzo 5y ago
> I feel like Google's approach is more popularized I mean, the engineering architecture being described in this memo is, basically, microservices. That's certainly an extremely popular -- I would go so far as to say even vogue -- pattern for solving the problem of building software at scale.
- nivertech 5y ago
  No, they're just services - a service per team. Microservices usually overdoing it, when you have multiple microservices per team, and sometimes even per developer. Also, microservices sometimes implemented incorrectly, where they're still communicate via shared databases, instead of encapsulating them and exposing them via service APIs only. This Memo was born because of the real business need, while many modern microservices deployments are the result of cargo-culting GAFAMs/FANGs.
  - pjmlp 5y ago
    Indeed, hence "the network is the computer" footnote that used to be written on Sun manuals. Many cargo cult web services of today can be written in Sun RPC APIs. But yeah it is C and its IDL isn't as cool as gRPC proto files.
raverbashing 5y ago
For real, if AWS internal APIs are as good as their outside-facing ones I fear the complexity. Their APIs are... correct. But have things like usability as a last concern. You also find some things not behaving exactly as you would expect and some things that are barely explained.

ridiculous_fish 5y ago

I've always wondered how far this extends: "All teams will henceforth expose their data and functionality through service interfaces." Does it include e.g. the team writing the on-device text renderer for a Kindle? What would such a service interface look like?

minitoar 5y ago
Presumably that team calls some sort of package management/build api to publish their new driver or whatever.

nivertech 5y ago

Does Amazon has some standards/conventions for inter-team APIs, i.e. something like Google's AIPs (API Improvement Proposals) [1,2]? [1] https://google.aip.dev/general https://google.aip.dev/general [2] https://google.aip.dev/ https://google.aip.dev/

rejectedandsad 5y ago
There’s an API bar raiser system for APIs called by many teams.

Newcomb5421 5y ago

Thanks for the information keep sharing https://www.walgreenslistens.biz/ https://www.walgreenslistens.biz/

kpmah 5y ago

I've worked at a place where this was cargo-culted and it worked horribly. All the worst aspects of 'services-first' design. I think this works much better when each team is working on what could be regarded as a complete end-to-end product e.g. a database. It works poorly when each team is working on part of a product.

jimbokun 5y ago
If you don't get the boundaries between the various services right, the effort is going to fail spectacularly.

augustl 5y ago

> no direct reads of another team’s data store This reminds me of how App Store used to ban game emulators interpreting ROMs downloaded from the internet, while at the same time allowing XML parsing. Does it really matter if the interface you're talking to over the network is code written by another team, or data written by another team? I suppose implementation details are often hidden away in data stores, but does it have to be that way?

dragonwriter 5y ago
> Does it really matter if the interface you’re talking to over the network is code written by another team, or data written by another team? Yes, because it means the depedencies between systems will consistently be API dependencies, not a mix of API and datastore structure dependencies, which means that the other team will only have to reduces the constraints on change.
draw_down 5y ago
Well, it isn’t really like that at all I would say. The reason for the Amazon rule is to prevent unseen coupling between teams. Explicit boundaries are critical for this to scale.
mattikl 5y ago
Direct reading effectively makes the data store an API, i.e. the API you provide is SQL SELECT over multiple tables instead of HTTP GET resource. This pretty much freezes the database schema, the team owning it can no longer change it, because they don't know how the users use it. It also limits the possibilities to provide optimizations on the data (like caching).

projectileboy 5y ago

Kind of lame that it doesn’t even mention Steve Yegge or his blog, which is almost certainly the source.

k__ 5y ago

"HTTP, Corba, Pubsub, custom protocols — doesn’t matter." That could have backfired pretty badly, haha.

omginternets 5y ago
I'm not sure which would be worse: CORBA or some custom protocol...
- LanceH 5y ago
  Presumably the custom protocol would map well to at least one side of the API.

typedef_struct 5y ago

I spent 2020 at AWS and I can't think of a single part of their toolchain that exposed APIs as first-class citizens. Not Pipelines, not Apollo, not CRUX, not Brazil, not MCM. Its all command line applications and browser interfaces (the stuff first-year developers are most familiar with building, I suppose). I was familiar with this memo so it was quite a shock. Wish they'd taken their own advice.

vermilingua 5y ago
…and what do the cli apps and browser interfaces use to connect with the tools?
- ncsurfus 5y ago
  Yeah, all of those tools have APIs…

Gayax 5y ago

It is exactly what author Cal Newport is recommending in his new book "A World Without Email" (March 2021): replacing the ping-pong of emails and meetings by standardized processes and requests via tools. It's simply incredible to see that Bezos saw this coming 20 years ago!

tome 5y ago
I came across an interesting blog discussing such ideas a few years ago. I don't think anything ever came of it though http://thingamy.typepad.com/ http://thingamy.typepad.com/

amazondrone12 5y ago

As a current Amazon employee I can confirm this is no longer the case or only applies to such a select few pieces of software that it doesn't actually mean what you think it does. The amount of day to day stuff that is relied upon that runs on greasemonkey scripts and web scraping is insane. API's for a ton of things don't exist or are have heavy gatekeeping. (eg. contact x to get onboarded to our API and they just ghost you). Of course you can get a high level leader to tell you to use a proper API in a mailing list email thread where you are trying to get help to solve a problem but they will not actually help you get working API access, all talk no action. A perfect example is there is this basic CRUD app that I rely upon and I see the current maintainer is working on all these feature requests, the problem for me is that the site has a ton of AJAX and takes like 20 mouse clicks to get retrieve basic information. So I open a ticket / feature request just to get a basic REST API for the reads and they politely tell me I am an idiot and close the request. So much pointless work is created at Amazon from lack of API access.

geodel 5y ago
This seems so true. A friend of mine got hired in Northern Virginia. He was Java developer for ten years. But at Amazon it is about same as you described.
biztos 5y ago
Sounds like the maintainer is optimizing for job security and/or work-life balance. And from all the comments on this post, it sounds like that’s a pretty rational course of action.
phkahler 5y ago
Sounds like they're allowing people to treat the systems as "infrastructure" and "applications" with only the infra having exposed APIs? That's super common in software of all sorts. There are a lot of people who can make good use of a decent API and appreciate it, but are not experienced (or smart, or have time, etc) enough to design a good API themselves. Most software is mortar, not bricks.
- res0nat0r 5y ago

sgt101 5y ago

I worked at a company that read the Amazon memo virtually straight after it was published and decided to copy it. The CIO at the time predicated every system's teams' bonus on it. No API, no bonus. What followed was 18mths of people building enterprise messaging systems and "buses" of various types. Hundreds of pages of documentation on APIs was prepared and released. I think 2 API's with maybe 40 methods actually went live. There were no bonuses Many people left The people who stayed were unhappy It never happened, everyone quietly forgot about it after 18 mnths. The CIO stayed for another 3 or 4 years - he only left when a new CEO came in due to internal promotion. The new CEO used to run one of the P&L's/business units and hated the CIO with a passion. So... it's not the API mandate that made the difference at Amazon.

Aperocky 5y ago
That's the wrong way to do it though. You can't just have a free for all mentality when building APIs. The platform has to be there first. Everyone's API should just need simple descriptive json files and then a client can be generated to consume that API easily. Everyone can declare those json files and it would work everywhere across the business. This free for all with no standardization is bound to be a disaster.
beastman82 5y ago
That is quite the conclusion you've made from a single data point!
- sgt101 5y ago
  Well - you can argue about the opposite as well! But I would claim that this is an existence proof - if the management will and focus is there for an API enabled business (it was) this case study proves that is not sufficient to create the kind of company that Amazon became. An API program isn't a magic bullet (as everyone knows by now I guess!)
  - bhupy 5y ago
    The following can be true at the same time: - An API program isn’t a magic bullet - An API program is a necessary, but not sufficient condition to replicate Amazon’s success You’re right that this case study proves that it alone is insufficient to create the kind of company that Amazon became, but does it also prove that it’s unnecessary to create the kind of company that Amazon became?

Hippocrates 5y ago

I love this. As others have mentioned I wonder how much input Jeff had, and from who, before making the mandate. I am experiencing the polar opposite of this mandate. The systems in my organization are always built to require human touch-points. What's worse, our CTO mistakes these menial interactions as "teamwork" and "collaboration" when they are really just toil to compensate for the lack of platform-level thinking. I love how a CEO can put this so bluntly, upend everyones work for a couple of years and build a juggernaut because of it. The idea runs parallel to one I have been championing throughout my career with SOAs which I call "self serve architecture". I want others in the organization to be able to pick up use my services to their benefit with zero input or help from me or my team. I tell my team to design the API as if it were GitHub's API. Practically, that means - There are up-to-date and easy docs that cover what you need to know. - People can gain access on their own (via some existing workplace/team based credential). At most we have to add them to a list somewhere. - The system will protect itself and inform users against problematic use (quotas, throttling, and visibility into this) - You have visibility into who your users are and what they are doing such that you can assess value, learn from usage, and communicate to consumers when necessary.

bruce343434 5y ago

Or: how to waste cycles and make your codebase run slow as shit on a networked computer cluster.

tomtato 5y ago

I had always read this as a metaphorical memo. Not necessarily that every team must have a software service running somewhere in infrastructure serving RESTful routes but rather as a way of thinking about agreements between groups. Your team's "API" could be documents describing: if you want us to do XYZ, send us a request via ABC, and you should expect a response in UVW format in x amount of time.

wly_cdgr 5y ago

Like all the best myth making propaganda this has some sense and truth to it

deeblering4 5y ago

> 6. Anyone who doesn’t do this will be fired. Sounds about right Azon turned throwaway compute instances and one-click 2 day delivery ecom into huge cash cows. They aren't some golden example of how to properly design infrastructure, it just worked out for them. luck/timing/reinvestment had a lot to do with it.

senderista 5y ago

I don’t think Yegge is a very reliable source in general, even though he was there at the time. I’m not sure why his version of these events is taken as gospel everywhere. I never saw Bezos concern himself with technical matters while I was at Amazon (though I was in AWS, which he seemed happy to leave to ajassy).

ChrisArchitect 5y ago

this is a post breaking down a pseudo-mythological memo from 2002 at a company where things have obviously changed and evolved, and there are multiple upvoted comments in here from amazon people saying this isn't the case or things have changed etc.