36 ms·

The 2002 mandate for internal communication systems at Amazon

634 points by anacleto 7y ago

Twirrim 7y ago
Yegge's post was very interesting reading, and I took similar learnings away from it. I was at Amazon at the time, however, and there were things that certainly weren't true any more: >3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network. API first... except if you want to be a number of certain new services that somehow managed to get away with not presenting an API, even though an API would make every service team's life easier. > 5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. Except, similar to above, where teams apparently decided they didn't want to think that way at all and management just let them. > 6) Anyone who doesn’t do this will be fired. Unless your exception is perceived providing value to the company. Then you'll get lauded, and everyone is told they'll need to use your js laden, web only interface, and to hell with any automation. Mostly those exceptions just codified further in my mind about just how right the Bezos email Yegge paraphrased actually was.
- leoh 7y ago
  How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"
  - thaeli 7y ago
    Some methods I've commonly seen in Enterprise Duct Tape: Screen scrape the other service and do data exchange via a Selenium script. Directly interact with the other service's database. CSV files and nightly batch jobs.
    - adambyrtek 7y ago
      This is one of the niches where (S)FTP and batch processing is still alive and kicking.
      - mason55 7y ago
        Yeah, SFTP + CSV file is still the standard for enterprise software. The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction. Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement. But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
        deleted 7y ago
        [deleted]
        dfox 7y ago
        My experience is that usual reason for RPC-over-SFTP is that it is the only thing that corporate IT security does not control and thus cannot make inflexible. Adding another SOAP/JSON/whatever endpoint tends to be multiyear project, while creating new directory on shared SFTP server is a way to implement the functionality in few hours.
      - crtlaltdel 7y ago
        its also quite common in fixed data logging applications, such as exports from bms.
    - throwaway122kk 7y ago
      I raise you a service which gets it's configuration from a table on confluence page
  - bdamm 7y ago
    Call out to a shell? Pass structs between C-based programs using sockets? Use runtime marshalling? Make your "API" be just different arbitrary JSON objects mapping to maps? Hide the entire API behind a lazy cache with undocumented side effects when cache misses occur? I wouldn't consider any of these "API"s although they are interfaces.
    - morelisp 7y ago
      > Make your "API" be just different arbitrary JSON objects mapping to maps? But enough about the web...
  - morelisp 7y ago
    A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers) - that, but with some mmap'd region (because the next team of developers doesn't know how to select()) - that, but with a local file (because the next team of developers doesn't know how to mmap()) - that, but with some NFS file (for scaling!) - that, but with some hadoop fs file (for big data!) Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.
    - myself248 7y ago
      > rowhammering the data you want into the next job This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
    - btown 7y ago
      > Dump some file in a directory and hope something is treating that directory like a queue Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/ https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
      - shkkmo 7y ago
        ACH is not the lifeblood of the American economy. If you removed ACH there would be unimaginable distruption but I can't see that economic activity would completely cease. I think the lifeblood of the American economy is our population.
    - saalweachter 7y ago
      Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."
  - closeparen 7y ago
    My company increasingly understands "platform" to mean "codebase to build your feature into" rather than "API to consume from your own codebase."
  - rob-olmos 7y ago
    Faxes, like in US health care. With staff to handle it at the endpoints.
  - jcrites 7y ago
    Calling parts of your application "services" means that you're already thinking in terms of the "services/API" metaphor. If you're not using services, you might be, for example: - Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.) - Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.
  - jhallenworld 7y ago
    IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer work for IBM)
    - flukus 7y ago
      Pricing: Contact Us. So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
      - nameistaken 7y ago
        Cries in Webseal.
      - jugg1es 7y ago
        IBM's business model is totally antiquated and exhausting for modern processes. We have had a nightmare trying to get IBM MDM's solution to finally admit they were not actually cloud-ready after saying repeatedly that they were. No TLS support for DB2 out of the box for K8 support, documentation sucks. But contact us for pricing. IBM sucks.
    - closeparen 7y ago
      This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes. In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.
  - cpitman 7y ago
    Another favorite: You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".
- LordHumungous 7y ago
  Rules are meant to be broken as the saying goes.
- akhilcacharya 7y ago
  Was Amazon the first major company to invest fully in SOA for internal use like this?
  - morelisp 7y ago
    The critical decision here is not SOA for internal use, but SOA designed for external use, for internal use. This is still uncommon.
- mav3rick 7y ago
  I have to ask. Isn't this just microservices ? Also isn't this terrible for latency and debugging ?
  - eutropia 7y ago
    This email from 2002 describes system design decisions which led to amazon developing AWS. I wasn't a professional programmer at the time, but as I understand it, I think service-oriented design patterns were rare, let alone microservices as a concept.
  - lmm 7y ago
    > Isn't this just microservices ? Yeah it is (though not so "micro-": it's really one service per team, which is the scale that actually makes sense IMO). This is kind of a "seinfeld is unfunny" case: this memo was in 2002, and predated (arguably helped create) most of the modern microservice approach. > Also isn't this terrible for latency and debugging ? When two codebases are owned by different teams (and potentially in different languages), it's far better to have them separated by a network connection and a well-defined HTTP API than living in the same address space where they can corrupt each other. Since the interfaces are well defined, you can isolate any problem down to a request that's not getting the correct answer from the service it's calling, and then hand that over to the other team to investigate. Since the request and response are plain text, it's easy to see what's happening. If responding to a single request from the end customer requires chaining through several independent codebases maintained by different teams, you have bigger problems than request latency. Each team should own a piece of end-to-end functionality, not a horizontal layer.
    - huherto 7y ago
      Agree. The "Micro" prefix is very miss leading. It should be one per (pizza) team. Otherwise you end up with too many, and it becomes a mess.
    - I_AM_A_SMURF 7y ago
      Not only that, but you can also track performance/crash rate/etc of each service individually and over time, which is a really powerful tool to have. Infinitely better than a monolithic system at that scale.
- ozgune 7y ago
  (Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quite a strong and opinionated statement. I'd agree that Jeff made this change to improve communication infrastructure for Amazon.com. I wouldn't agree that he made it to enable AWS - for two reasons. First, Amazon started thinking about EC2 and S3 way way later than this. Second, Kindle and SimpleDB (now Dynamo) were similar independent bets. These projects were happening at the time of moving to SOA, but they weren't tied to it. The one thing common across all of these products is - making informed bets on market fundamentals and enabling teams to deliver on them. Now, I know of a few fairly senior Amazonians who read those forums. They have more context than I do. So, if they chime in and say that Jeff sent this email to enable AWS, I'll gladly take it. :) Until then, I wouldn't extrapolate Steve Yegge's post to mean something about Jeff B.'s intentions, and build an overarching argument over it.
  - lifeisstillgood 7y ago
    >> making informed bets on market fundamentals and enabling teams to deliver on them. Given the recent 20,000 startup ideas post, this is the perfect algorithm to cut through the noise
  - nexuist 7y ago
    > Second, Kindle and SimpleDB (now Dynamo) Minor nitpick, these are two separate services. SDB is still independently accessible and probably will be for a while. That being said they definitely don't encourage any new use of SDB and push Dynamo instead. Kind of a shame because a modern NoSQL store with SQL support would be great for rapid prototyping before a concrete data schema is established.
    - guidoism 7y ago
      I actually really liked Simple DB as a place for configuration.
FrojoS 7y ago
Something like Conway’s Law was also recently cited by Elon Musk (jump to 3:30) https://www.reddit.com/r/SpaceXLounge/comments/dbttaw/everyday_astronaut_a_conversation_with_elon_musk/ https://www.reddit.com/r/SpaceXLounge/comments/dbttaw/everyd...
- degenerate 7y ago
  Direct link to 3:30: https://youtu.be/cIQ36Kt7UVg?t=206 https://youtu.be/cIQ36Kt7UVg?t=206
Waterluvian 7y ago
> 6) Anyone who doesn’t do this will be fired. So I've never worked at a company over 150 people. Is this... a normal thing for an email? Maybe I'm just one of those softies but an email with that line would throw me off my day and cause a serious hit to my morale and confidence of working there.
- inimino 7y ago
  This is Yegge's over-the-top style of humor, none of these are actual quotes from the email.
  - wayoutthere 7y ago
    To be fair, "do this or you're fired" is just assumed of every request at Amazon, because they follow through on it often.
    - jrd259 7y ago
      In my five-year experience at Amazon that has never been true.
- deleted 7y ago
  [deleted]
- jeffbarr 7y ago
  I strongly believe that Steve was exaggerating for effect here. In my 17 years at Amazon I have never seen or heard of a threat of this nature. The overall intent of the email was to tell teams to decouple, decentralize, and to own their own destinies.
  - Waterluvian 7y ago
    Thanks for the reality check. Phew.
  - michaelcampbell 7y ago
    Yes, IIRC in Steve's original blog post he points out that that particular point was not in Bezos' email and notes he put it in there for dramatic effect.
cm2012 7y ago
I love some of Amazon's executive policies. From what I've read, everyone has to write a multi-page paper before executive meetings, and everyone has to read it, so the meeting goes smoothly with everyone understanding the issues. I hate how no one reads anything in most organizations.
- kylek 7y ago
  Not sure about execs, but this happens in engineering meetings (regarding new features being implemented or other semi-major changes). Whoever is initiating the meeting writes up a paper describing the terminology, the nature of the change and why it's needed, how it will be implemented etc. The entire dev team (+ maybe other dev teams within the group), management (the initiator's boss + 1 level above, maybe other dev team managers too) start the meeting with hard printouts of the paper, armed with red pens. The meeting "starts" with ~15 mins or so of silence for everyone to review the paper in the room from start to finish. Then the paper is reviewed end to end and torn up on the way. Often there are multiple of these meetings (i.e. first one went badly or if things change along the way of building/implementing it and questions come up)
  - QuercusMax 7y ago
    That sounds like a low-fi, synchronous, in-person version of reviewing a Google Doc (with comments, suggestions, etc).
    - kansface 7y ago
      Alternatively, the people who are scheduled for the meeting are guaranteed to spend 15 reviewing the subject and having their questions answered in a timely fashion. All of the stakeholders are likely in the room so a decision can actually be made.
    - BookmarkSaver 7y ago
      It forces real-time immediate interaction and discussion rather than dealing with people that have varying levels of understanding and who may or may not have bothered to actually read the whole thing. And it forces all of the comments and suggestions to be hashed out in 30-60 minutes, rather than over weeks as people only check the doc maybe once a day if you are lucky so every back and forth takes forever, especially when a lot of it is dealing with the aforementioned sources of understanding. A single hour-long in-person meeting can forgo a month of back-and-forth online.
sputknick 7y ago
I worked at an organization that had a similar declaration. Here's how it played out: 1. Everyone is super excited for other teams to share their data 2. Everyone wants an exception from sharing their own data because it's too hard or too sensitive to share. 3. Eventually everything gets shared, but it takes 3-4 times longer than it really should.
- deleted 7y ago
  [deleted]
- mtberatwork 7y ago
  Yup and getting anyone to write coherent documentation for their new interfaces is like pulling teeth.
  - dodobirdlord 7y ago
    Yes, but you'll at least have the API definition. And you work at the same company, so you can show up at the desks of the team responsible and demand answers. And if it breaks in production you get to page them and they have to wake up and help you! The threat of pages is a great way to coerce decent documentation. It's an important principle at Amazon that if a production service has a dependency on you, then you are also a production service. Another benefit of breaking things up into SOA is monitoring individual services. If your API is returning 500s then it's your fault and your problem (at least until you can root cause to one of your own dependencies that's returning 500s, then you can pass the buck).
- rb808 7y ago
  4. A couple of years later you want to stop an obsolete interface but you can't because a handful of systems use it and they dont have budget to change.
  - jrd259 7y ago
    True this happens, but you're still better off that if you had tight coupling. In the absolute worst case you can make a shim implementation to support the obsolete use case. You can not do this when callers are directly reading your database/memory structure.
darkstar999 7y ago
Author should ctrl-f for the many erroneous double spaces. </ocd>
- namdnay 7y ago
  Macbook keyboard maybe?
duxup 7y ago
Eat your own dogfood. You can't sell to customers effectively if your flagship product only works because it has access to resources the customers will never have... and it is designed around that flagship's needs and not your customer's needs.
- morelisp 7y ago
  AWS is largely a side-effect of this memo, not its instigator. At the time, Amazon's dog food would be books/clothes/literal dog food.
Invictus0 7y ago
A lot of interesting thoughts here but the author doesn't really wrap them into a conclusion. A whole lot of words to say "they all work and it depends".
williesleg 7y ago
Bezos has a time machine. Only way to explain it.
prepend 7y ago
I wish the actual body of the email was available and published. I’ve only read Yegge’s account of the note and didn’t see it in any of Bezos’ books. I suppose it’s nice that the email, or really any amazon emails, has not been leaked.
- boldslogan 7y ago
  I only could find his autobiography. What other books does he have / wrote?
  - prepend 7y ago
    Brad Stone wrote a Bezos/Amazon bio called “Everything Store.”
busterarm 7y ago
Reading Bezos' mandate email puts a smile across my face, every time.
arkitaip 7y ago
If you use an adblocker like uBlock Origin, you can add the following rule: news.ycombinator.com##.pagetop Unfortunately it removes ALL of the top navbar but I've found it really useful to get around HN's damaging and useless gamification metric.
- ianmobbs 7y ago
  What?
xyzzyz 7y ago
> While the third point makes all the difference in the world, what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. > Having teams acting like individual APIs and interacting with one another through interfaces over the network was the catalyst of a series of consequent actions that eventually made possible the realization of AWS in a way that couldn’t have been possible otherwise. Google has worked this way since time immemorial. That’s what protocol buffers are for: to create services and pass data between them using well defined interfaces.
- defen 7y ago
  > Google has worked this way since time immemorial. Do internal Google services exclusively use the Google Cloud Platform APIs? The implication is that internal Amazon services exclusively use AWS APIs. I've never worked at either company though, so I don't know if it's true. Perhaps someone could clarify.
  - hello_moto 7y ago
    > The implication is that internal Amazon services exclusively use AWS APIs. Untrue. Some even use older version of AWS "core" services (S3), not the latest and greatest version of AWS services.
    - dastbe 7y ago
      though in some cases I can think of, the "legacy" service is now just a facade over an AWS service.
  - xyzzyz 7y ago
    No, because internal APIs for the same backends are more powerful and easier to integrate with. It's not easy to make the external APIs as useful and powerful as internal ones: for one, you can trust your users more to do the right thing and not try to exploit you for profit, it's much less committment to offer certain functionalities, since it's easier to roll them back if only users are internal, etc. Google is slowly getting there in feature parity, but so much effort has been invested in Google internal ecosystem even before GCP has even existed that, as good as GCP is, nobody wants to bet on it when obviously superior alternative is available.
darksaints 7y ago
At least as of 3 years ago when I left, the software systems that drove the mandate towards SOA were still massive systems that communicated almost purely through a monolithic Oracle database. It was the software system(s) that was responsible for all automation and accounting at fulfillment centers. This is one of those rare times where I actually think a full rewrite from scratch would have been a better idea.
- dodobirdlord 7y ago
  They got there in the end. https://aws.amazon.com/solutions/case-studies/amazon-fulfillment-aurora/ https://aws.amazon.com/solutions/case-studies/amazon-fulfill...
totaldude87 7y ago
>> Anyone who doesn’t do this will be fired Right, motivating everyone.. check..
- deleted 7y ago
  [deleted]
dang 7y ago
Yegge's article never says it was an email. What should the title be? Edit: I've taken a rather lame crack at it and am open to improvements.
- x2f10 7y ago
  Memo?
- jcrites 7y ago
  The circumstances that Yegge described happened somewhat before my time, but I suppose you could call it an "internal goal" or "internal mandate". Amazon's not really big on "mandates" in general, but the term seems to fit Yegge's characterization of what happened. "Internal goal" would be another way to phrase it. E.g., "The single most important technical goal in the history of Amazon".
  - tayleeganj 7y ago
    oh hey ur the Nova nigga
- maxmcd 7y ago
  love the new title
jrochkind1 7y ago
> what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. I'm not following what he means. What is the thing he is describing as "an internal communication system" here? That made all the rest possible? What is/was this internal communications system?
- akhilcacharya 7y ago
  I'm assuming Yegge was referring to the RPC framework.
  - jrochkind1 7y ago
    "an internal communication system" does sound like something like an "RPC framework", but Yegge's paraphrase actually says "It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. Bezos doesn’t care." I read this as saying different teams/services don't have to use the same thing either. That doesn't sound like an "RPC framework" or "an internal communications system" at all. It seems to leave the door open to everyone doing things in a diverse mishmash. Which isn't what I'd call "an internal communications system" at all. But was/is there in fact an Amazon-specific "RPC framework", that all Amazon services use, some consistent framework used consistently accross services? I haven't heard much about this before so am curious to learn more. I haven't heard of an Amazon 'RPC framework' before, or what it's called, or what. And OP doesn't specify it either; does the rest of the audience know what's being talked about, and I'm just missing context? If that is the thing that the OP thinks is really what Amazon got right... then the interesting thing is figuring out how it went from the paraphrased email, which doesn't actually demand such a thing, to.... such a thing. Who designed or chose this "RPC framework"? When? How? How'd they get everyone to use the same one? If that's the thing Amazon got right, there are some steps missing between the Yegge-paraphrased email and there, since the email doesn't actually even call for such a thing. Or is that not what happened at all, and I'm still not sure what OP means by "an internal communication system" being the thing Amazon got right.
    - morelisp 7y ago
thefounder 7y ago
The issue if you are developing using such requirements is that the product will end-up quite expensive. A simple messaging or authentication feature becomes a fully fledged multi-tier service maybe with super admins, owners/admins and clients. Dev budget is not an issue for Amazon though...
jordache 7y ago
is this trying to be stratechry in format?
iagooar 7y ago
> 6) Anyone who doesn’t do this will be fired. I would have so much loved this approach in the last corporate job I had. It would have changed so many things in such a short time...
d--b 7y ago
The thing to point out is Bezos is a real techie, and while any business guy would have built amazon on top of msft or google cloud, the fact that he knows about infrastructure made it possible for Amazon to build AWS
eigen-vector 7y ago
This was not exactly a Jeff Bezos mandate but the result of an engineering brainstorm. The mandate came more out of a "how to scale Amazon for the next decade" discussion. In large companies, one where distributed/independent teams are as important as distributed systems this ended up being the only way to operate. Initially, during the good old days of Amazon, there was what you'd call a single datawarehouse. It made sense initially for every system that processed an order to access the data by querying that data warehouse—this meant that the processes would be distributed (different services), while the data would be centralized. It also meant that any change to the way the data is stored in the datawarehouse meant deploying code to a hundred places. The most important problem that this addressed was however different. A centralized datawarehouse meant that every customer request bubbled up into N queries to the datawarehouse (where N is the number of services that needed access to the data—billing, ordering, tracking...). The mandate summarized in one line would be this—"the data is the one that should go to the services, not the other way round." Voila, microservices.
- dang 7y ago
  Ok, we'll take Bezos out of the title above.
  - eigen-vector 7y ago
    Thank you, dang! This is certainly a more accurate title.
ineedasername 7y ago
The article mentions this as dog fooding, but does that really apply here? Did they do this with the idea in mind that they'd turn this stuff into a product? It struck me as Bezos wanting things built for the future, reducing technical debt, and the product-ification was an excellent byproduct, but perhaps not intentional.
- morelisp 7y ago
  At the time Amazon was building out their merchant portal as a white-label-ish service for other large retailers to sell products online. The 'customers' in the memo would be other merchants, and the early AWS offering (e.g. SQS) reflect this. "Elastic" clouds weren't really on the menu yet but obviously part of the point is that you can offer it to customers regardless of where the architectural fad goes.
  - function_seven 7y ago
    I remember the early days of target.com and (I think?) toysrus.com being thinly skinned versions of Amazon.
    - gowld 7y ago
      Yep. Also one of the large British retailers and a few others.
    - ineedasername 7y ago
      Yes, if I remember correctly even Borders Books got in on the action there.
dmh2000 7y ago
here's an article about how the idea of AWS came about. the main takeaway is that it evolved and the article has a lot of 'we' in it, not only 'jeff' https://techcrunch.com/2016/07/02/andy-jassys-brief-history-of-the-genesis-of-aws/ https://techcrunch.com/2016/07/02/andy-jassys-brief-history-...
lytfyre 7y ago
IIRC when Yegge accidentally posted that rant, the entirety of Amazon corp got IP banned from Hacker News from _everyone_ rushing to view and comment.
ga-vu 7y ago
Do other (Silicon Valley) companies do the same?
goatinaboat 7y ago
At a previous company, a senior manager took Yegge’s blog post and presented it internally as his own original work. Hilarity ensued.
tomduncalf 7y ago
Found the original post from Yegge a really interesting and thought provoking read. Didn’t realise from the context that he originally accidentally posted it as a public rather than private Google+ post! His follow up post explaining this, and with an interesting anecdote about presenting to Jeff Bezos, is archived here (seeing as G+ has, ironically (or not) given the context, shut down): https://gist.github.com/dexterous/1383377#file-the-post-retraction-message https://gist.github.com/dexterous/1383377#file-the-post-retr...
what_to_do_next 7y ago
Leonardo Federico needs to take a penmanship class.
brown9-2 7y ago
It’s such a loss that Yegge doesn’t blog anymore.
- rctay89 7y ago
  ...you were saying? :) https://medium.com/@steve.yegge/google-to-grab-one-year-later-3e1e4df321f3 https://medium.com/@steve.yegge/google-to-grab-one-year-late...
thrower123 7y ago
Why does the title of this keep getting flopped around? It's shifted three or four times today. I thought it was supposed to be the title, or the subtitle, and avoid paraphrasing.
iamleppert 7y ago
We have robotic baristas here in SF, but no one uses them. Why? People want to have their food prepared and served by a real human being, in most cases. The food tastes better when it's served to you by a real person.
- cthalupa 7y ago
  I believe you replied to the wrong story, unless robot baristas serving you coffee was meant to be a metaphor for using APIs for programmatic communication vs. SFTPing csv files, or something.
emmelaich 7y ago
> doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols So a jdbc interface and published schema would count?
throwawayy98121 7y ago
Hi! I’m a senior engineer at Amazon. Throwaway account but I’ll try to respond to questions if anyone cares to ask. Yeah we use services heavily, but there’s plenty of teams dumping data to S3 or using a data lake. There’s also the “we need to do this but management doesn’t see value so let’s dump it on the intern or SDE 1, who we won’t really mentor or guide and then blame, forcing them to switch teams as soon as they can.” If you work at another company and think we have our stuff figured out at Amazon, we really don’t. We have brilliant people, many of who are straight up assholes who will throw you under the bus. We have people who are kind and will help you gain all kinds of engineering skills. We also have people who are scum of the Earth shit people who work at Amazon because I don’t think any other sane company or workplace would tolerate them. We have extremes on the garbage people end of the spectrum, unfortunately. Sorry long rant - point being - it’s good to learn how we do things. The internal email on services is pretty unique. I learned about it when I was an SDE 1 back in the day. But - don’t take it as gospel. It doesn’t mean you need to build services. I can think of any number of examples where we follow anti patterns because no one gives a shit about the pattern, whether it’s a service, a bucket, a queue, or a file attached to the system used for scrum tasks, or shit passed over email... we care about value at the end of the day. If you don’t provide sufficient value at Amazons bar, they have no problem tossing you out the window.
notacoward 7y ago
A few things I'd add today: * Every service must provide latency and error-rate metrics. * Every service must be capable of generating and/or responding to backpressure when things become overloaded. * Every service must be prepared to support multitenancy.

634 points by anacleto 7y ago

Yegge's post was very interesting reading, and I took similar learnings away from it. I was at Amazon at the time, however, and there were things that certainly weren't true any more: >3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network. API first... except if you want to be a number of certain new services that somehow managed to get away with not presenting an API, even though an API would make every service team's life easier. > 5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. Except, similar to above, where teams apparently decided they didn't want to think that way at all and management just let them. > 6) Anyone who doesn’t do this will be fired. Unless your exception is perceived providing value to the company. Then you'll get lauded, and everyone is told they'll need to use your js laden, web only interface, and to hell with any automation. Mostly those exceptions just codified further in my mind about just how right the Bezos email Yegge paraphrased actually was.

leoh 7y ago
How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"
- thaeli 7y ago
  Some methods I've commonly seen in Enterprise Duct Tape: Screen scrape the other service and do data exchange via a Selenium script. Directly interact with the other service's database. CSV files and nightly batch jobs.
  - adambyrtek 7y ago
    This is one of the niches where (S)FTP and batch processing is still alive and kicking.
    - mason55 7y ago
      Yeah, SFTP + CSV file is still the standard for enterprise software. The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction. Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement. But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
      - deleted 7y ago
        [deleted]
      - dfox 7y ago
        My experience is that usual reason for RPC-over-SFTP is that it is the only thing that corporate IT security does not control and thus cannot make inflexible. Adding another SOAP/JSON/whatever endpoint tends to be multiyear project, while creating new directory on shared SFTP server is a way to implement the functionality in few hours.
    - crtlaltdel 7y ago
      its also quite common in fixed data logging applications, such as exports from bms.
  - throwaway122kk 7y ago
    I raise you a service which gets it's configuration from a table on confluence page
- bdamm 7y ago
  Call out to a shell? Pass structs between C-based programs using sockets? Use runtime marshalling? Make your "API" be just different arbitrary JSON objects mapping to maps? Hide the entire API behind a lazy cache with undocumented side effects when cache misses occur? I wouldn't consider any of these "API"s although they are interfaces.
  - morelisp 7y ago
    > Make your "API" be just different arbitrary JSON objects mapping to maps? But enough about the web...
- morelisp 7y ago
  A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers) - that, but with some mmap'd region (because the next team of developers doesn't know how to select()) - that, but with a local file (because the next team of developers doesn't know how to mmap()) - that, but with some NFS file (for scaling!) - that, but with some hadoop fs file (for big data!) Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.
  - myself248 7y ago
    > rowhammering the data you want into the next job This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
  - btown 7y ago
    > Dump some file in a directory and hope something is treating that directory like a queue Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/ https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
    - shkkmo 7y ago
      ACH is not the lifeblood of the American economy. If you removed ACH there would be unimaginable distruption but I can't see that economic activity would completely cease. I think the lifeblood of the American economy is our population.
  - saalweachter 7y ago
    Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."
- closeparen 7y ago
  My company increasingly understands "platform" to mean "codebase to build your feature into" rather than "API to consume from your own codebase."
- rob-olmos 7y ago
  Faxes, like in US health care. With staff to handle it at the endpoints.
- jcrites 7y ago
  Calling parts of your application "services" means that you're already thinking in terms of the "services/API" metaphor. If you're not using services, you might be, for example: - Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.) - Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.
- jhallenworld 7y ago
  IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer work for IBM)
  - flukus 7y ago
    Pricing: Contact Us. So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
    - nameistaken 7y ago
      Cries in Webseal.
    - jugg1es 7y ago
      IBM's business model is totally antiquated and exhausting for modern processes. We have had a nightmare trying to get IBM MDM's solution to finally admit they were not actually cloud-ready after saying repeatedly that they were. No TLS support for DB2 out of the box for K8 support, documentation sucks. But contact us for pricing. IBM sucks.
  - closeparen 7y ago
    This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes. In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.
- cpitman 7y ago
  Another favorite: You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".
LordHumungous 7y ago
Rules are meant to be broken as the saying goes.
akhilcacharya 7y ago
Was Amazon the first major company to invest fully in SOA for internal use like this?
- morelisp 7y ago
  The critical decision here is not SOA for internal use, but SOA designed for external use, for internal use. This is still uncommon.
mav3rick 7y ago
I have to ask. Isn't this just microservices ? Also isn't this terrible for latency and debugging ?
- eutropia 7y ago
  This email from 2002 describes system design decisions which led to amazon developing AWS. I wasn't a professional programmer at the time, but as I understand it, I think service-oriented design patterns were rare, let alone microservices as a concept.
- lmm 7y ago
  > Isn't this just microservices ? Yeah it is (though not so "micro-": it's really one service per team, which is the scale that actually makes sense IMO). This is kind of a "seinfeld is unfunny" case: this memo was in 2002, and predated (arguably helped create) most of the modern microservice approach. > Also isn't this terrible for latency and debugging ? When two codebases are owned by different teams (and potentially in different languages), it's far better to have them separated by a network connection and a well-defined HTTP API than living in the same address space where they can corrupt each other. Since the interfaces are well defined, you can isolate any problem down to a request that's not getting the correct answer from the service it's calling, and then hand that over to the other team to investigate. Since the request and response are plain text, it's easy to see what's happening. If responding to a single request from the end customer requires chaining through several independent codebases maintained by different teams, you have bigger problems than request latency. Each team should own a piece of end-to-end functionality, not a horizontal layer.
  - huherto 7y ago
    Agree. The "Micro" prefix is very miss leading. It should be one per (pizza) team. Otherwise you end up with too many, and it becomes a mess.
  - I_AM_A_SMURF 7y ago
    Not only that, but you can also track performance/crash rate/etc of each service individually and over time, which is a really powerful tool to have. Infinitely better than a monolithic system at that scale.
ozgune 7y ago
(Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quite a strong and opinionated statement. I'd agree that Jeff made this change to improve communication infrastructure for Amazon.com. I wouldn't agree that he made it to enable AWS - for two reasons. First, Amazon started thinking about EC2 and S3 way way later than this. Second, Kindle and SimpleDB (now Dynamo) were similar independent bets. These projects were happening at the time of moving to SOA, but they weren't tied to it. The one thing common across all of these products is - making informed bets on market fundamentals and enabling teams to deliver on them. Now, I know of a few fairly senior Amazonians who read those forums. They have more context than I do. So, if they chime in and say that Jeff sent this email to enable AWS, I'll gladly take it. :) Until then, I wouldn't extrapolate Steve Yegge's post to mean something about Jeff B.'s intentions, and build an overarching argument over it.
- lifeisstillgood 7y ago
  >> making informed bets on market fundamentals and enabling teams to deliver on them. Given the recent 20,000 startup ideas post, this is the perfect algorithm to cut through the noise
- nexuist 7y ago
  > Second, Kindle and SimpleDB (now Dynamo) Minor nitpick, these are two separate services. SDB is still independently accessible and probably will be for a while. That being said they definitely don't encourage any new use of SDB and push Dynamo instead. Kind of a shame because a modern NoSQL store with SQL support would be great for rapid prototyping before a concrete data schema is established.
  - guidoism 7y ago
    I actually really liked Simple DB as a place for configuration.

FrojoS 7y ago

Something like Conway’s Law was also recently cited by Elon Musk (jump to 3:30) https://www.reddit.com/r/SpaceXLounge/comments/dbttaw/everyday_astronaut_a_conversation_with_elon_musk/ https://www.reddit.com/r/SpaceXLounge/comments/dbttaw/everyd...

degenerate 7y ago
Direct link to 3:30: https://youtu.be/cIQ36Kt7UVg?t=206 https://youtu.be/cIQ36Kt7UVg?t=206

Waterluvian 7y ago

> 6) Anyone who doesn’t do this will be fired. So I've never worked at a company over 150 people. Is this... a normal thing for an email? Maybe I'm just one of those softies but an email with that line would throw me off my day and cause a serious hit to my morale and confidence of working there.

inimino 7y ago
This is Yegge's over-the-top style of humor, none of these are actual quotes from the email.
- wayoutthere 7y ago
  To be fair, "do this or you're fired" is just assumed of every request at Amazon, because they follow through on it often.
  - jrd259 7y ago
    In my five-year experience at Amazon that has never been true.
deleted 7y ago
[deleted]
jeffbarr 7y ago
I strongly believe that Steve was exaggerating for effect here. In my 17 years at Amazon I have never seen or heard of a threat of this nature. The overall intent of the email was to tell teams to decouple, decentralize, and to own their own destinies.
- Waterluvian 7y ago
  Thanks for the reality check. Phew.
- michaelcampbell 7y ago
  Yes, IIRC in Steve's original blog post he points out that that particular point was not in Bezos' email and notes he put it in there for dramatic effect.

cm2012 7y ago

I love some of Amazon's executive policies. From what I've read, everyone has to write a multi-page paper before executive meetings, and everyone has to read it, so the meeting goes smoothly with everyone understanding the issues. I hate how no one reads anything in most organizations.

kylek 7y ago
Not sure about execs, but this happens in engineering meetings (regarding new features being implemented or other semi-major changes). Whoever is initiating the meeting writes up a paper describing the terminology, the nature of the change and why it's needed, how it will be implemented etc. The entire dev team (+ maybe other dev teams within the group), management (the initiator's boss + 1 level above, maybe other dev team managers too) start the meeting with hard printouts of the paper, armed with red pens. The meeting "starts" with ~15 mins or so of silence for everyone to review the paper in the room from start to finish. Then the paper is reviewed end to end and torn up on the way. Often there are multiple of these meetings (i.e. first one went badly or if things change along the way of building/implementing it and questions come up)
- QuercusMax 7y ago
  That sounds like a low-fi, synchronous, in-person version of reviewing a Google Doc (with comments, suggestions, etc).
  - kansface 7y ago
    Alternatively, the people who are scheduled for the meeting are guaranteed to spend 15 reviewing the subject and having their questions answered in a timely fashion. All of the stakeholders are likely in the room so a decision can actually be made.
  - BookmarkSaver 7y ago
    It forces real-time immediate interaction and discussion rather than dealing with people that have varying levels of understanding and who may or may not have bothered to actually read the whole thing. And it forces all of the comments and suggestions to be hashed out in 30-60 minutes, rather than over weeks as people only check the doc maybe once a day if you are lucky so every back and forth takes forever, especially when a lot of it is dealing with the aforementioned sources of understanding. A single hour-long in-person meeting can forgo a month of back-and-forth online.

sputknick 7y ago

I worked at an organization that had a similar declaration. Here's how it played out: 1. Everyone is super excited for other teams to share their data 2. Everyone wants an exception from sharing their own data because it's too hard or too sensitive to share. 3. Eventually everything gets shared, but it takes 3-4 times longer than it really should.

deleted 7y ago
[deleted]
mtberatwork 7y ago
Yup and getting anyone to write coherent documentation for their new interfaces is like pulling teeth.
- dodobirdlord 7y ago
  Yes, but you'll at least have the API definition. And you work at the same company, so you can show up at the desks of the team responsible and demand answers. And if it breaks in production you get to page them and they have to wake up and help you! The threat of pages is a great way to coerce decent documentation. It's an important principle at Amazon that if a production service has a dependency on you, then you are also a production service. Another benefit of breaking things up into SOA is monitoring individual services. If your API is returning 500s then it's your fault and your problem (at least until you can root cause to one of your own dependencies that's returning 500s, then you can pass the buck).
rb808 7y ago
4. A couple of years later you want to stop an obsolete interface but you can't because a handful of systems use it and they dont have budget to change.
- jrd259 7y ago
  True this happens, but you're still better off that if you had tight coupling. In the absolute worst case you can make a shim implementation to support the obsolete use case. You can not do this when callers are directly reading your database/memory structure.

darkstar999 7y ago

Author should ctrl-f for the many erroneous double spaces. </ocd>

namdnay 7y ago
Macbook keyboard maybe?

duxup 7y ago

Eat your own dogfood. You can't sell to customers effectively if your flagship product only works because it has access to resources the customers will never have... and it is designed around that flagship's needs and not your customer's needs.

morelisp 7y ago
AWS is largely a side-effect of this memo, not its instigator. At the time, Amazon's dog food would be books/clothes/literal dog food.

Invictus0 7y ago

A lot of interesting thoughts here but the author doesn't really wrap them into a conclusion. A whole lot of words to say "they all work and it depends".

williesleg 7y ago

Bezos has a time machine. Only way to explain it.

prepend 7y ago

I wish the actual body of the email was available and published. I’ve only read Yegge’s account of the note and didn’t see it in any of Bezos’ books. I suppose it’s nice that the email, or really any amazon emails, has not been leaked.

boldslogan 7y ago
I only could find his autobiography. What other books does he have / wrote?
- prepend 7y ago
  Brad Stone wrote a Bezos/Amazon bio called “Everything Store.”

busterarm 7y ago

Reading Bezos' mandate email puts a smile across my face, every time.

arkitaip 7y ago

If you use an adblocker like uBlock Origin, you can add the following rule: news.ycombinator.com##.pagetop Unfortunately it removes ALL of the top navbar but I've found it really useful to get around HN's damaging and useless gamification metric.

ianmobbs 7y ago
What?

xyzzyz 7y ago

> While the third point makes all the difference in the world, what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. > Having teams acting like individual APIs and interacting with one another through interfaces over the network was the catalyst of a series of consequent actions that eventually made possible the realization of AWS in a way that couldn’t have been possible otherwise. Google has worked this way since time immemorial. That’s what protocol buffers are for: to create services and pass data between them using well defined interfaces.

defen 7y ago
> Google has worked this way since time immemorial. Do internal Google services exclusively use the Google Cloud Platform APIs? The implication is that internal Amazon services exclusively use AWS APIs. I've never worked at either company though, so I don't know if it's true. Perhaps someone could clarify.
- hello_moto 7y ago
  > The implication is that internal Amazon services exclusively use AWS APIs. Untrue. Some even use older version of AWS "core" services (S3), not the latest and greatest version of AWS services.
  - dastbe 7y ago
    though in some cases I can think of, the "legacy" service is now just a facade over an AWS service.
- xyzzyz 7y ago
  No, because internal APIs for the same backends are more powerful and easier to integrate with. It's not easy to make the external APIs as useful and powerful as internal ones: for one, you can trust your users more to do the right thing and not try to exploit you for profit, it's much less committment to offer certain functionalities, since it's easier to roll them back if only users are internal, etc. Google is slowly getting there in feature parity, but so much effort has been invested in Google internal ecosystem even before GCP has even existed that, as good as GCP is, nobody wants to bet on it when obviously superior alternative is available.

darksaints 7y ago

At least as of 3 years ago when I left, the software systems that drove the mandate towards SOA were still massive systems that communicated almost purely through a monolithic Oracle database. It was the software system(s) that was responsible for all automation and accounting at fulfillment centers. This is one of those rare times where I actually think a full rewrite from scratch would have been a better idea.

dodobirdlord 7y ago
They got there in the end. https://aws.amazon.com/solutions/case-studies/amazon-fulfillment-aurora/ https://aws.amazon.com/solutions/case-studies/amazon-fulfill...

totaldude87 7y ago

>> Anyone who doesn’t do this will be fired Right, motivating everyone.. check..

deleted 7y ago
[deleted]

dang 7y ago

Yegge's article never says it was an email. What should the title be? Edit: I've taken a rather lame crack at it and am open to improvements.

x2f10 7y ago
Memo?
jcrites 7y ago
The circumstances that Yegge described happened somewhat before my time, but I suppose you could call it an "internal goal" or "internal mandate". Amazon's not really big on "mandates" in general, but the term seems to fit Yegge's characterization of what happened. "Internal goal" would be another way to phrase it. E.g., "The single most important technical goal in the history of Amazon".
- tayleeganj 7y ago
  oh hey ur the Nova nigga
maxmcd 7y ago
love the new title

jrochkind1 7y ago

> what Amazon really did get right that Google didn’t was an internal communication system designed to make all the rest possible. I'm not following what he means. What is the thing he is describing as "an internal communication system" here? That made all the rest possible? What is/was this internal communications system?

akhilcacharya 7y ago
I'm assuming Yegge was referring to the RPC framework.
- jrochkind1 7y ago
  "an internal communication system" does sound like something like an "RPC framework", but Yegge's paraphrase actually says "It doesn’t matter what technology they use. HTTP, Corba, Pubsub, custom protocols — doesn’t matter. Bezos doesn’t care." I read this as saying different teams/services don't have to use the same thing either. That doesn't sound like an "RPC framework" or "an internal communications system" at all. It seems to leave the door open to everyone doing things in a diverse mishmash. Which isn't what I'd call "an internal communications system" at all. But was/is there in fact an Amazon-specific "RPC framework", that all Amazon services use, some consistent framework used consistently accross services? I haven't heard much about this before so am curious to learn more. I haven't heard of an Amazon 'RPC framework' before, or what it's called, or what. And OP doesn't specify it either; does the rest of the audience know what's being talked about, and I'm just missing context? If that is the thing that the OP thinks is really what Amazon got right... then the interesting thing is figuring out how it went from the paraphrased email, which doesn't actually demand such a thing, to.... such a thing. Who designed or chose this "RPC framework"? When? How? How'd they get everyone to use the same one? If that's the thing Amazon got right, there are some steps missing between the Yegge-paraphrased email and there, since the email doesn't actually even call for such a thing. Or is that not what happened at all, and I'm still not sure what OP means by "an internal communication system" being the thing Amazon got right.
  - morelisp 7y ago

thefounder 7y ago

The issue if you are developing using such requirements is that the product will end-up quite expensive. A simple messaging or authentication feature becomes a fully fledged multi-tier service maybe with super admins, owners/admins and clients. Dev budget is not an issue for Amazon though...

jordache 7y ago

is this trying to be stratechry in format?

iagooar 7y ago

> 6) Anyone who doesn’t do this will be fired. I would have so much loved this approach in the last corporate job I had. It would have changed so many things in such a short time...

d--b 7y ago

The thing to point out is Bezos is a real techie, and while any business guy would have built amazon on top of msft or google cloud, the fact that he knows about infrastructure made it possible for Amazon to build AWS

eigen-vector 7y ago

This was not exactly a Jeff Bezos mandate but the result of an engineering brainstorm. The mandate came more out of a "how to scale Amazon for the next decade" discussion. In large companies, one where distributed/independent teams are as important as distributed systems this ended up being the only way to operate. Initially, during the good old days of Amazon, there was what you'd call a single datawarehouse. It made sense initially for every system that processed an order to access the data by querying that data warehouse—this meant that the processes would be distributed (different services), while the data would be centralized. It also meant that any change to the way the data is stored in the datawarehouse meant deploying code to a hundred places. The most important problem that this addressed was however different. A centralized datawarehouse meant that every customer request bubbled up into N queries to the datawarehouse (where N is the number of services that needed access to the data—billing, ordering, tracking...). The mandate summarized in one line would be this—"the data is the one that should go to the services, not the other way round." Voila, microservices.

dang 7y ago
Ok, we'll take Bezos out of the title above.
- eigen-vector 7y ago
  Thank you, dang! This is certainly a more accurate title.

ineedasername 7y ago

The article mentions this as dog fooding, but does that really apply here? Did they do this with the idea in mind that they'd turn this stuff into a product? It struck me as Bezos wanting things built for the future, reducing technical debt, and the product-ification was an excellent byproduct, but perhaps not intentional.

morelisp 7y ago
At the time Amazon was building out their merchant portal as a white-label-ish service for other large retailers to sell products online. The 'customers' in the memo would be other merchants, and the early AWS offering (e.g. SQS) reflect this. "Elastic" clouds weren't really on the menu yet but obviously part of the point is that you can offer it to customers regardless of where the architectural fad goes.
- function_seven 7y ago
  I remember the early days of target.com and (I think?) toysrus.com being thinly skinned versions of Amazon.
  - gowld 7y ago
    Yep. Also one of the large British retailers and a few others.
  - ineedasername 7y ago
    Yes, if I remember correctly even Borders Books got in on the action there.

dmh2000 7y ago

here's an article about how the idea of AWS came about. the main takeaway is that it evolved and the article has a lot of 'we' in it, not only 'jeff' https://techcrunch.com/2016/07/02/andy-jassys-brief-history-of-the-genesis-of-aws/ https://techcrunch.com/2016/07/02/andy-jassys-brief-history-...

lytfyre 7y ago

IIRC when Yegge accidentally posted that rant, the entirety of Amazon corp got IP banned from Hacker News from _everyone_ rushing to view and comment.

ga-vu 7y ago

Do other (Silicon Valley) companies do the same?

goatinaboat 7y ago

At a previous company, a senior manager took Yegge’s blog post and presented it internally as his own original work. Hilarity ensued.

tomduncalf 7y ago

Found the original post from Yegge a really interesting and thought provoking read. Didn’t realise from the context that he originally accidentally posted it as a public rather than private Google+ post! His follow up post explaining this, and with an interesting anecdote about presenting to Jeff Bezos, is archived here (seeing as G+ has, ironically (or not) given the context, shut down): https://gist.github.com/dexterous/1383377#file-the-post-retraction-message https://gist.github.com/dexterous/1383377#file-the-post-retr...

what_to_do_next 7y ago

Leonardo Federico needs to take a penmanship class.

brown9-2 7y ago

It’s such a loss that Yegge doesn’t blog anymore.

rctay89 7y ago
...you were saying? :) https://medium.com/@steve.yegge/google-to-grab-one-year-later-3e1e4df321f3 https://medium.com/@steve.yegge/google-to-grab-one-year-late...

thrower123 7y ago

Why does the title of this keep getting flopped around? It's shifted three or four times today. I thought it was supposed to be the title, or the subtitle, and avoid paraphrasing.

iamleppert 7y ago

We have robotic baristas here in SF, but no one uses them. Why? People want to have their food prepared and served by a real human being, in most cases. The food tastes better when it's served to you by a real person.

cthalupa 7y ago
I believe you replied to the wrong story, unless robot baristas serving you coffee was meant to be a metaphor for using APIs for programmatic communication vs. SFTPing csv files, or something.

emmelaich 7y ago

> doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols So a jdbc interface and published schema would count?

throwawayy98121 7y ago

Hi! I’m a senior engineer at Amazon. Throwaway account but I’ll try to respond to questions if anyone cares to ask. Yeah we use services heavily, but there’s plenty of teams dumping data to S3 or using a data lake. There’s also the “we need to do this but management doesn’t see value so let’s dump it on the intern or SDE 1, who we won’t really mentor or guide and then blame, forcing them to switch teams as soon as they can.” If you work at another company and think we have our stuff figured out at Amazon, we really don’t. We have brilliant people, many of who are straight up assholes who will throw you under the bus. We have people who are kind and will help you gain all kinds of engineering skills. We also have people who are scum of the Earth shit people who work at Amazon because I don’t think any other sane company or workplace would tolerate them. We have extremes on the garbage people end of the spectrum, unfortunately. Sorry long rant - point being - it’s good to learn how we do things. The internal email on services is pretty unique. I learned about it when I was an SDE 1 back in the day. But - don’t take it as gospel. It doesn’t mean you need to build services. I can think of any number of examples where we follow anti patterns because no one gives a shit about the pattern, whether it’s a service, a bucket, a queue, or a file attached to the system used for scrum tasks, or shit passed over email... we care about value at the end of the day. If you don’t provide sufficient value at Amazons bar, they have no problem tossing you out the window.

notacoward 7y ago

A few things I'd add today: * Every service must provide latency and error-rate metrics. * Every service must be capable of generating and/or responding to backpressure when things become overloaded. * Every service must be prepared to support multitenancy.