9 ms·

Yegge's post was very interesting reading, and I took similar learnings away from it. I was at Amazon at the time, however, and there were things that certainl

Yegge's post was very interesting reading, and I took similar learnings away from it. I was at Amazon at the time, however, and there were things that certainly weren't true any more: >3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network. API first... except if you want to be a number of certain new services that somehow managed to get away with not presenting an API, even though an API would make every service team's life easier. > 5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions. Except, similar to above, where teams apparently decided they didn't want to think that way at all and management just let them. > 6) Anyone who doesn’t do this will be fired. Unless your exception is perceived providing value to the company. Then you'll get lauded, and everyone is told they'll need to use your js laden, web only interface, and to hell with any automation. Mostly those exceptions just codified further in my mind about just how right the Bezos email Yegge paraphrased actually was.

leoh 7y ago
How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"
- thaeli 7y ago
  Some methods I've commonly seen in Enterprise Duct Tape: Screen scrape the other service and do data exchange via a Selenium script. Directly interact with the other service's database. CSV files and nightly batch jobs.
  - adambyrtek 7y ago
    This is one of the niches where (S)FTP and batch processing is still alive and kicking.
    - mason55 7y ago
      Yeah, SFTP + CSV file is still the standard for enterprise software. The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction. Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement. But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
      - deleted 7y ago
        [deleted]
      - dfox 7y ago
        My experience is that usual reason for RPC-over-SFTP is that it is the only thing that corporate IT security does not control and thus cannot make inflexible. Adding another SOAP/JSON/whatever endpoint tends to be multiyear project, while creating new directory on shared SFTP server is a way to implement the functionality in few hours.
    - crtlaltdel 7y ago
      its also quite common in fixed data logging applications, such as exports from bms.
  - throwaway122kk 7y ago
    I raise you a service which gets it's configuration from a table on confluence page
    - morelisp 7y ago
      Confluence is at least canonically XHTML so this is better than a lot of data lake bullshit I've seen.
    - mason55 7y ago
      It's funny to think about but the reality is that it's better than a lot of other options. - The page has automatic history & merge conflict resolution - There's built-in role-based security to control both visibility and actions (read-only vs. read-write) - You can work on a draft and then "deploy" by publishing your changes - You can respond to hooks/notifications when the page is updated Considering that the alternative is either editing a text file on disk or teaching business users to use git, a Confluence page is not so bad.
    - IggleSniggle 7y ago
      As crazy as that may seem on the face of it, it’s actually kind of genius for merging the roles of those who need to configure / don’t get git, and those who need to develop against the configuration.
    - throwaway40324 7y ago
      and I thought the screen scraping / Selenium solution was wack! Wow!
    - throwaway40324 7y ago
      DAAC - Documentation As A Configuration
  - hnick 7y ago
    Oh damn you just reminded me. They tried to bring in "Robotics" (e.g. Blue Prism) here. For in-house apps. Apparently it was so hard to deal with the developers that instead of exposing an API they would automate clicking around on Internet Explorer browser windows. Thankfully I haven't heard much about it lately.
  - badpun 7y ago
    Directly interacting with other service's DB is an "Enterprise Integration Pattern" AFAIR? It can make sense in lots of cases.
    - arethuza 7y ago
      Reading data from another application's database is pretty common (although even this can cause chaos if done without some care) but writing to application databases is often a very bad idea and often explicitly forbidden by CRM/ERP vendors.
    - dragonwriter 7y ago
      Shared database is, in fact,a classic enterprise integration pattern, and much of classic relational database engineering assumes that multiple application will share the same database; in the classic ideal, each would use it through a set of (read/write, ad needed) application-specific views so as to avoid exposing implementation details of the base tables and to permit each application, and the base database, to change with minimum disruption to any of the others.
- bdamm 7y ago
  Call out to a shell? Pass structs between C-based programs using sockets? Use runtime marshalling? Make your "API" be just different arbitrary JSON objects mapping to maps? Hide the entire API behind a lazy cache with undocumented side effects when cache misses occur? I wouldn't consider any of these "API"s although they are interfaces.
  - morelisp 7y ago
    > Make your "API" be just different arbitrary JSON objects mapping to maps? But enough about the web...
- morelisp 7y ago
  A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers) - that, but with some mmap'd region (because the next team of developers doesn't know how to select()) - that, but with a local file (because the next team of developers doesn't know how to mmap()) - that, but with some NFS file (for scaling!) - that, but with some hadoop fs file (for big data!) Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.
  - myself248 7y ago
    > rowhammering the data you want into the next job This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
  - btown 7y ago
    > Dump some file in a directory and hope something is treating that directory like a queue Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/ https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
    - shkkmo 7y ago
      ACH is not the lifeblood of the American economy. If you removed ACH there would be unimaginable distruption but I can't see that economic activity would completely cease. I think the lifeblood of the American economy is our population.
  - saalweachter 7y ago
    Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."
- closeparen 7y ago
  My company increasingly understands "platform" to mean "codebase to build your feature into" rather than "API to consume from your own codebase."
- rob-olmos 7y ago
  Faxes, like in US health care. With staff to handle it at the endpoints.
- jcrites 7y ago
  Calling parts of your application "services" means that you're already thinking in terms of the "services/API" metaphor. If you're not using services, you might be, for example: - Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.) - Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.
- jhallenworld 7y ago
  IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer work for IBM)
  - flukus 7y ago
    Pricing: Contact Us. So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
    - nameistaken 7y ago
      Cries in Webseal.
    - jugg1es 7y ago
      IBM's business model is totally antiquated and exhausting for modern processes. We have had a nightmare trying to get IBM MDM's solution to finally admit they were not actually cloud-ready after saying repeatedly that they were. No TLS support for DB2 out of the box for K8 support, documentation sucks. But contact us for pricing. IBM sucks.
  - closeparen 7y ago
    This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes. In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.
- cpitman 7y ago
  Another favorite: You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".
LordHumungous 7y ago
Rules are meant to be broken as the saying goes.
akhilcacharya 7y ago
Was Amazon the first major company to invest fully in SOA for internal use like this?
- morelisp 7y ago
  The critical decision here is not SOA for internal use, but SOA designed for external use, for internal use. This is still uncommon.
mav3rick 7y ago
I have to ask. Isn't this just microservices ? Also isn't this terrible for latency and debugging ?
- eutropia 7y ago
  This email from 2002 describes system design decisions which led to amazon developing AWS. I wasn't a professional programmer at the time, but as I understand it, I think service-oriented design patterns were rare, let alone microservices as a concept.
- lmm 7y ago
  > Isn't this just microservices ? Yeah it is (though not so "micro-": it's really one service per team, which is the scale that actually makes sense IMO). This is kind of a "seinfeld is unfunny" case: this memo was in 2002, and predated (arguably helped create) most of the modern microservice approach. > Also isn't this terrible for latency and debugging ? When two codebases are owned by different teams (and potentially in different languages), it's far better to have them separated by a network connection and a well-defined HTTP API than living in the same address space where they can corrupt each other. Since the interfaces are well defined, you can isolate any problem down to a request that's not getting the correct answer from the service it's calling, and then hand that over to the other team to investigate. Since the request and response are plain text, it's easy to see what's happening. If responding to a single request from the end customer requires chaining through several independent codebases maintained by different teams, you have bigger problems than request latency. Each team should own a piece of end-to-end functionality, not a horizontal layer.
  - huherto 7y ago
    Agree. The "Micro" prefix is very miss leading. It should be one per (pizza) team. Otherwise you end up with too many, and it becomes a mess.
  - I_AM_A_SMURF 7y ago
    Not only that, but you can also track performance/crash rate/etc of each service individually and over time, which is a really powerful tool to have. Infinitely better than a monolithic system at that scale.
ozgune 7y ago
(Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quite a strong and opinionated statement. I'd agree that Jeff made this change to improve communication infrastructure for Amazon.com. I wouldn't agree that he made it to enable AWS - for two reasons. First, Amazon started thinking about EC2 and S3 way way later than this. Second, Kindle and SimpleDB (now Dynamo) were similar independent bets. These projects were happening at the time of moving to SOA, but they weren't tied to it. The one thing common across all of these products is - making informed bets on market fundamentals and enabling teams to deliver on them. Now, I know of a few fairly senior Amazonians who read those forums. They have more context than I do. So, if they chime in and say that Jeff sent this email to enable AWS, I'll gladly take it. :) Until then, I wouldn't extrapolate Steve Yegge's post to mean something about Jeff B.'s intentions, and build an overarching argument over it.
- lifeisstillgood 7y ago
  >> making informed bets on market fundamentals and enabling teams to deliver on them. Given the recent 20,000 startup ideas post, this is the perfect algorithm to cut through the noise
- nexuist 7y ago
  > Second, Kindle and SimpleDB (now Dynamo) Minor nitpick, these are two separate services. SDB is still independently accessible and probably will be for a while. That being said they definitely don't encourage any new use of SDB and push Dynamo instead. Kind of a shame because a modern NoSQL store with SQL support would be great for rapid prototyping before a concrete data schema is established.
  - guidoism 7y ago
    I actually really liked Simple DB as a place for configuration.

leoh 7y ago

How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"

thaeli 7y ago
Some methods I've commonly seen in Enterprise Duct Tape: Screen scrape the other service and do data exchange via a Selenium script. Directly interact with the other service's database. CSV files and nightly batch jobs.
- adambyrtek 7y ago
  This is one of the niches where (S)FTP and batch processing is still alive and kicking.
  - mason55 7y ago
    Yeah, SFTP + CSV file is still the standard for enterprise software. The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction. Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement. But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
    - deleted 7y ago
      [deleted]
    - dfox 7y ago
      My experience is that usual reason for RPC-over-SFTP is that it is the only thing that corporate IT security does not control and thus cannot make inflexible. Adding another SOAP/JSON/whatever endpoint tends to be multiyear project, while creating new directory on shared SFTP server is a way to implement the functionality in few hours.
  - crtlaltdel 7y ago
    its also quite common in fixed data logging applications, such as exports from bms.
- throwaway122kk 7y ago
  I raise you a service which gets it's configuration from a table on confluence page
  - morelisp 7y ago
    Confluence is at least canonically XHTML so this is better than a lot of data lake bullshit I've seen.
  - mason55 7y ago
    It's funny to think about but the reality is that it's better than a lot of other options. - The page has automatic history & merge conflict resolution - There's built-in role-based security to control both visibility and actions (read-only vs. read-write) - You can work on a draft and then "deploy" by publishing your changes - You can respond to hooks/notifications when the page is updated Considering that the alternative is either editing a text file on disk or teaching business users to use git, a Confluence page is not so bad.
  - IggleSniggle 7y ago
    As crazy as that may seem on the face of it, it’s actually kind of genius for merging the roles of those who need to configure / don’t get git, and those who need to develop against the configuration.
  - throwaway40324 7y ago
    and I thought the screen scraping / Selenium solution was wack! Wow!
  - throwaway40324 7y ago
    DAAC - Documentation As A Configuration
- hnick 7y ago
  Oh damn you just reminded me. They tried to bring in "Robotics" (e.g. Blue Prism) here. For in-house apps. Apparently it was so hard to deal with the developers that instead of exposing an API they would automate clicking around on Internet Explorer browser windows. Thankfully I haven't heard much about it lately.
- badpun 7y ago
  Directly interacting with other service's DB is an "Enterprise Integration Pattern" AFAIR? It can make sense in lots of cases.
  - arethuza 7y ago
    Reading data from another application's database is pretty common (although even this can cause chaos if done without some care) but writing to application databases is often a very bad idea and often explicitly forbidden by CRM/ERP vendors.
  - dragonwriter 7y ago
    Shared database is, in fact,a classic enterprise integration pattern, and much of classic relational database engineering assumes that multiple application will share the same database; in the classic ideal, each would use it through a set of (read/write, ad needed) application-specific views so as to avoid exposing implementation details of the base tables and to permit each application, and the base database, to change with minimum disruption to any of the others.
bdamm 7y ago
Call out to a shell? Pass structs between C-based programs using sockets? Use runtime marshalling? Make your "API" be just different arbitrary JSON objects mapping to maps? Hide the entire API behind a lazy cache with undocumented side effects when cache misses occur? I wouldn't consider any of these "API"s although they are interfaces.
- morelisp 7y ago
  > Make your "API" be just different arbitrary JSON objects mapping to maps? But enough about the web...
morelisp 7y ago
A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers) - that, but with some mmap'd region (because the next team of developers doesn't know how to select()) - that, but with a local file (because the next team of developers doesn't know how to mmap()) - that, but with some NFS file (for scaling!) - that, but with some hadoop fs file (for big data!) Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.
- myself248 7y ago
  > rowhammering the data you want into the next job This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
- btown 7y ago
  > Dump some file in a directory and hope something is treating that directory like a queue Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/ https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
  - shkkmo 7y ago
    ACH is not the lifeblood of the American economy. If you removed ACH there would be unimaginable distruption but I can't see that economic activity would completely cease. I think the lifeblood of the American economy is our population.
- saalweachter 7y ago
  Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."
closeparen 7y ago
My company increasingly understands "platform" to mean "codebase to build your feature into" rather than "API to consume from your own codebase."
rob-olmos 7y ago
Faxes, like in US health care. With staff to handle it at the endpoints.
jcrites 7y ago
Calling parts of your application "services" means that you're already thinking in terms of the "services/API" metaphor. If you're not using services, you might be, for example: - Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.) - Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.
jhallenworld 7y ago
IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer work for IBM)
- flukus 7y ago
  Pricing: Contact Us. So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
  - nameistaken 7y ago
    Cries in Webseal.
  - jugg1es 7y ago
    IBM's business model is totally antiquated and exhausting for modern processes. We have had a nightmare trying to get IBM MDM's solution to finally admit they were not actually cloud-ready after saying repeatedly that they were. No TLS support for DB2 out of the box for K8 support, documentation sucks. But contact us for pricing. IBM sucks.
- closeparen 7y ago
  This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes. In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.
cpitman 7y ago
Another favorite: You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".

LordHumungous 7y ago

Rules are meant to be broken as the saying goes.

akhilcacharya 7y ago

Was Amazon the first major company to invest fully in SOA for internal use like this?

morelisp 7y ago
The critical decision here is not SOA for internal use, but SOA designed for external use, for internal use. This is still uncommon.

mav3rick 7y ago

I have to ask. Isn't this just microservices ? Also isn't this terrible for latency and debugging ?

eutropia 7y ago
This email from 2002 describes system design decisions which led to amazon developing AWS. I wasn't a professional programmer at the time, but as I understand it, I think service-oriented design patterns were rare, let alone microservices as a concept.
lmm 7y ago
> Isn't this just microservices ? Yeah it is (though not so "micro-": it's really one service per team, which is the scale that actually makes sense IMO). This is kind of a "seinfeld is unfunny" case: this memo was in 2002, and predated (arguably helped create) most of the modern microservice approach. > Also isn't this terrible for latency and debugging ? When two codebases are owned by different teams (and potentially in different languages), it's far better to have them separated by a network connection and a well-defined HTTP API than living in the same address space where they can corrupt each other. Since the interfaces are well defined, you can isolate any problem down to a request that's not getting the correct answer from the service it's calling, and then hand that over to the other team to investigate. Since the request and response are plain text, it's easy to see what's happening. If responding to a single request from the end customer requires chaining through several independent codebases maintained by different teams, you have bigger problems than request latency. Each team should own a piece of end-to-end functionality, not a horizontal layer.
- huherto 7y ago
  Agree. The "Micro" prefix is very miss leading. It should be one per (pizza) team. Otherwise you end up with too many, and it becomes a mess.
- I_AM_A_SMURF 7y ago
  Not only that, but you can also track performance/crash rate/etc of each service individually and over time, which is a really powerful tool to have. Infinitely better than a monolithic system at that scale.

ozgune 7y ago

(Former Amazonian, part of the team that drove the change to SOA at the time) > Now I think that this internal email is what has actually stuck with me the most. Bezos realized that he had to change the internal communication infrastructure [..]. > He understood that a radical organizational change was required to arrange the internal dynamics in a way that would allow the creation of something like AWS. This is quite a strong and opinionated statement. I'd agree that Jeff made this change to improve communication infrastructure for Amazon.com. I wouldn't agree that he made it to enable AWS - for two reasons. First, Amazon started thinking about EC2 and S3 way way later than this. Second, Kindle and SimpleDB (now Dynamo) were similar independent bets. These projects were happening at the time of moving to SOA, but they weren't tied to it. The one thing common across all of these products is - making informed bets on market fundamentals and enabling teams to deliver on them. Now, I know of a few fairly senior Amazonians who read those forums. They have more context than I do. So, if they chime in and say that Jeff sent this email to enable AWS, I'll gladly take it. :) Until then, I wouldn't extrapolate Steve Yegge's post to mean something about Jeff B.'s intentions, and build an overarching argument over it.

lifeisstillgood 7y ago
>> making informed bets on market fundamentals and enabling teams to deliver on them. Given the recent 20,000 startup ideas post, this is the perfect algorithm to cut through the noise
nexuist 7y ago
> Second, Kindle and SimpleDB (now Dynamo) Minor nitpick, these are two separate services. SDB is still independently accessible and probably will be for a while. That being said they definitely don't encourage any new use of SDB and push Dynamo instead. Kind of a shame because a modern NoSQL store with SQL support would be great for rapid prototyping before a concrete data schema is established.
- guidoism 7y ago
  I actually really liked Simple DB as a place for configuration.