6 ms·

How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"

by leoh 7y ago

thaeli 7y ago
Some methods I've commonly seen in Enterprise Duct Tape: Screen scrape the other service and do data exchange via a Selenium script. Directly interact with the other service's database. CSV files and nightly batch jobs.
- adambyrtek 7y ago
  This is one of the niches where (S)FTP and batch processing is still alive and kicking.
  - mason55 7y ago
    Yeah, SFTP + CSV file is still the standard for enterprise software. The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction. Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement. But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
    - deleted 7y ago
      [deleted]
    - dfox 7y ago
      My experience is that usual reason for RPC-over-SFTP is that it is the only thing that corporate IT security does not control and thus cannot make inflexible. Adding another SOAP/JSON/whatever endpoint tends to be multiyear project, while creating new directory on shared SFTP server is a way to implement the functionality in few hours.
  - crtlaltdel 7y ago
    its also quite common in fixed data logging applications, such as exports from bms.
- throwaway122kk 7y ago
  I raise you a service which gets it's configuration from a table on confluence page
  - morelisp 7y ago
    Confluence is at least canonically XHTML so this is better than a lot of data lake bullshit I've seen.
  - mason55 7y ago
    It's funny to think about but the reality is that it's better than a lot of other options. - The page has automatic history & merge conflict resolution - There's built-in role-based security to control both visibility and actions (read-only vs. read-write) - You can work on a draft and then "deploy" by publishing your changes - You can respond to hooks/notifications when the page is updated Considering that the alternative is either editing a text file on disk or teaching business users to use git, a Confluence page is not so bad.
  - IggleSniggle 7y ago
    As crazy as that may seem on the face of it, it’s actually kind of genius for merging the roles of those who need to configure / don’t get git, and those who need to develop against the configuration.
  - throwaway40324 7y ago
    and I thought the screen scraping / Selenium solution was wack! Wow!
  - throwaway40324 7y ago
    DAAC - Documentation As A Configuration
- hnick 7y ago
  Oh damn you just reminded me. They tried to bring in "Robotics" (e.g. Blue Prism) here. For in-house apps. Apparently it was so hard to deal with the developers that instead of exposing an API they would automate clicking around on Internet Explorer browser windows. Thankfully I haven't heard much about it lately.
- badpun 7y ago
  Directly interacting with other service's DB is an "Enterprise Integration Pattern" AFAIR? It can make sense in lots of cases.
  - arethuza 7y ago
    Reading data from another application's database is pretty common (although even this can cause chaos if done without some care) but writing to application databases is often a very bad idea and often explicitly forbidden by CRM/ERP vendors.
  - dragonwriter 7y ago
    Shared database is, in fact,a classic enterprise integration pattern, and much of classic relational database engineering assumes that multiple application will share the same database; in the classic ideal, each would use it through a set of (read/write, ad needed) application-specific views so as to avoid exposing implementation details of the base tables and to permit each application, and the base database, to change with minimum disruption to any of the others.
bdamm 7y ago
Call out to a shell? Pass structs between C-based programs using sockets? Use runtime marshalling? Make your "API" be just different arbitrary JSON objects mapping to maps? Hide the entire API behind a lazy cache with undocumented side effects when cache misses occur? I wouldn't consider any of these "API"s although they are interfaces.
- morelisp 7y ago
  > Make your "API" be just different arbitrary JSON objects mapping to maps? But enough about the web...
morelisp 7y ago
A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers) - that, but with some mmap'd region (because the next team of developers doesn't know how to select()) - that, but with a local file (because the next team of developers doesn't know how to mmap()) - that, but with some NFS file (for scaling!) - that, but with some hadoop fs file (for big data!) Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.
- myself248 7y ago
  > rowhammering the data you want into the next job This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
- btown 7y ago
  > Dump some file in a directory and hope something is treating that directory like a queue Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/ https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
  - shkkmo 7y ago
    ACH is not the lifeblood of the American economy. If you removed ACH there would be unimaginable distruption but I can't see that economic activity would completely cease. I think the lifeblood of the American economy is our population.
- saalweachter 7y ago
  Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."
closeparen 7y ago
My company increasingly understands "platform" to mean "codebase to build your feature into" rather than "API to consume from your own codebase."
rob-olmos 7y ago
Faxes, like in US health care. With staff to handle it at the endpoints.
jcrites 7y ago
Calling parts of your application "services" means that you're already thinking in terms of the "services/API" metaphor. If you're not using services, you might be, for example: - Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.) - Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.
jhallenworld 7y ago
IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer work for IBM)
- flukus 7y ago
  Pricing: Contact Us. So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
  - nameistaken 7y ago
    Cries in Webseal.
  - jugg1es 7y ago
    IBM's business model is totally antiquated and exhausting for modern processes. We have had a nightmare trying to get IBM MDM's solution to finally admit they were not actually cloud-ready after saying repeatedly that they were. No TLS support for DB2 out of the box for K8 support, documentation sucks. But contact us for pricing. IBM sucks.
- closeparen 7y ago
  This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes. In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.
cpitman 7y ago
Another favorite: You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".

How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"

by leoh 7y ago

How do services talk to each other without an API? Is it something like "put a non-well-documented object into a queue?"

thaeli 7y ago

Some methods I've commonly seen in Enterprise Duct Tape: Screen scrape the other service and do data exchange via a Selenium script. Directly interact with the other service's database. CSV files and nightly batch jobs.

adambyrtek 7y ago
This is one of the niches where (S)FTP and batch processing is still alive and kicking.
- mason55 7y ago
  Yeah, SFTP + CSV file is still the standard for enterprise software. The problem is that these kinds of things have to be built to the lowest common denominator, which is usually the customer anyway. The customer in enterprise software is usually not a tech company they typically have outdated IT policies and less skilled developers than a pure tech company would have. Even if the developers are capable of doing something like interacting with a queue they also need to be supported by a technology organization which can deal with that type of interaction. Some times you get lucky and someone in the past has pushed for that kind of modernization. Or your project really won't work without a more advanced interaction model and you have someone in the organization willing to go to bat for tech enhancement. But otherwise the default is "Control-M job to consume/produce a CSV file from/onto an SFTP"
  - deleted 7y ago
    [deleted]
  - dfox 7y ago
    My experience is that usual reason for RPC-over-SFTP is that it is the only thing that corporate IT security does not control and thus cannot make inflexible. Adding another SOAP/JSON/whatever endpoint tends to be multiyear project, while creating new directory on shared SFTP server is a way to implement the functionality in few hours.
- crtlaltdel 7y ago
  its also quite common in fixed data logging applications, such as exports from bms.
throwaway122kk 7y ago
I raise you a service which gets it's configuration from a table on confluence page
- morelisp 7y ago
  Confluence is at least canonically XHTML so this is better than a lot of data lake bullshit I've seen.
- mason55 7y ago
  It's funny to think about but the reality is that it's better than a lot of other options. - The page has automatic history & merge conflict resolution - There's built-in role-based security to control both visibility and actions (read-only vs. read-write) - You can work on a draft and then "deploy" by publishing your changes - You can respond to hooks/notifications when the page is updated Considering that the alternative is either editing a text file on disk or teaching business users to use git, a Confluence page is not so bad.
- IggleSniggle 7y ago
  As crazy as that may seem on the face of it, it’s actually kind of genius for merging the roles of those who need to configure / don’t get git, and those who need to develop against the configuration.
- throwaway40324 7y ago
  and I thought the screen scraping / Selenium solution was wack! Wow!
- throwaway40324 7y ago
  DAAC - Documentation As A Configuration
hnick 7y ago
Oh damn you just reminded me. They tried to bring in "Robotics" (e.g. Blue Prism) here. For in-house apps. Apparently it was so hard to deal with the developers that instead of exposing an API they would automate clicking around on Internet Explorer browser windows. Thankfully I haven't heard much about it lately.
badpun 7y ago
Directly interacting with other service's DB is an "Enterprise Integration Pattern" AFAIR? It can make sense in lots of cases.
- arethuza 7y ago
  Reading data from another application's database is pretty common (although even this can cause chaos if done without some care) but writing to application databases is often a very bad idea and often explicitly forbidden by CRM/ERP vendors.
- dragonwriter 7y ago
  Shared database is, in fact,a classic enterprise integration pattern, and much of classic relational database engineering assumes that multiple application will share the same database; in the classic ideal, each would use it through a set of (read/write, ad needed) application-specific views so as to avoid exposing implementation details of the base tables and to permit each application, and the base database, to change with minimum disruption to any of the others.

bdamm 7y ago

Call out to a shell? Pass structs between C-based programs using sockets? Use runtime marshalling? Make your "API" be just different arbitrary JSON objects mapping to maps? Hide the entire API behind a lazy cache with undocumented side effects when cache misses occur? I wouldn't consider any of these "API"s although they are interfaces.

morelisp 7y ago
> Make your "API" be just different arbitrary JSON objects mapping to maps? But enough about the web...

morelisp 7y ago

A queue if you're lucky! There's also: - Hire an intern / "Customer Service Representative" / "Technical Account Specialist" to manually copy data from one service into another - Dump some file in a directory and hope something is treating that directory like a queue - Read/write from the same database (/ same table) Or the classic Unix trajectory of increasingly bad service communication: - Read/write from the same local socket and (hopefully) same raw memory layouts (i.e. C structs) (because you've just taken your existing serialized process and begun fork()ing workers) - that, but with some mmap'd region (because the next team of developers doesn't know how to select()) - that, but with a local file (because the next team of developers doesn't know how to mmap()) - that, but with some NFS file (for scaling!) - that, but with some hadoop fs file (for big data!) Obviously all of these are at some level an 'application programming interface'. But then, technically so is rowhammering the data you want into the next job.

myself248 7y ago
> rowhammering the data you want into the next job This wouldn't quite fit the "obfuscated C" contest, but I feel like there should be a prize for a system that does useful work this way.
btown 7y ago
> Dump some file in a directory and hope something is treating that directory like a queue Or it's unencrypted files on an FTP server containing literally the lifeblood of the American economy: https://engineering.gusto.com/how-ach-works-a-developer-perspective-part-1/ https://engineering.gusto.com/how-ach-works-a-developer-pers... -_-
- shkkmo 7y ago
  ACH is not the lifeblood of the American economy. If you removed ACH there would be unimaginable distruption but I can't see that economic activity would completely cease. I think the lifeblood of the American economy is our population.
saalweachter 7y ago
Don't forget the most important step. "Think of the acronym CSV. Don't look up the definition of the format, just meditate on the idea of the format for a bit. Then write your data in the format you have just imagined is CSV, making whatever choices you feel personally best or most elegant regarding character escapes. Pass this file on to your downstream readers, assuring them it is a CSV file, without elaborating on how you have redefined that."

closeparen 7y ago

My company increasingly understands "platform" to mean "codebase to build your feature into" rather than "API to consume from your own codebase."

rob-olmos 7y ago

Faxes, like in US health care. With staff to handle it at the endpoints.

jcrites 7y ago

Calling parts of your application "services" means that you're already thinking in terms of the "services/API" metaphor. If you're not using services, you might be, for example: - Building one large application (monolith). Parts of the application communicate with each other via function calls. Everything runs in one large process. You can go quite a long way with this approach, especially for parts of the application that are stateless. (You can also build components of the application using a service/client metaphor within the process as well.) - Multiple separate applications might communicate with the same database, file system, or some other data store. Before we had distinct distributed systems components taking on the role of queues, event buses, and things like that, it was common to represent queues using folders or database tables. These approaches are still seen today, though they're uncommon in new applications.

jhallenworld 7y ago

IBM has an appliance for you: https://www.ibm.com/products/datapower-gateway https://www.ibm.com/products/datapower-gateway It's a middleware appliance that allows any API to talk to any other API and handles just about any data format. You can script it with javascript or XSLT. It can handle ad-hoc things like ftp polling. It has the added benefit that you can add security for outside facing clients. Disclaimer: I helped develop this appliance (but I no longer work for IBM)

flukus 7y ago
Pricing: Contact Us. So basically it will be expensive, require an army of IBM consultants and become yet another integration point instead of actually solving anything.
- nameistaken 7y ago
  Cries in Webseal.
- jugg1es 7y ago
  IBM's business model is totally antiquated and exhausting for modern processes. We have had a nightmare trying to get IBM MDM's solution to finally admit they were not actually cloud-ready after saying repeatedly that they were. No TLS support for DB2 out of the box for K8 support, documentation sucks. But contact us for pricing. IBM sucks.
closeparen 7y ago
This is the Enterprise Service Bus concept, right? I remember a pretty good conference talk about how we realized in the mid 2000s that these things are problematic and you probably want services communicating directly over dumb pipes. In showed an evolution from a monolithic spaghetti codebase, to an SOA, to realizing there are now spaghetti connections between services in an SOA, to a very clean looking ESB architecture, to showing how all the chaos is still there, just inside the ESB where it’s even harder to reason about or change.

cpitman 7y ago

Another favorite: You integrate a "service" by creating and linking a library that implements the entire service, including data access. Now try tracking down everything accessing the "service" database, or rolling out an upgrade to the "service".