6 ms·

Apply Unix Principles to Write Testable Microservices

135 points by ashish5887 8y ago

redact207 8y ago
The workflow/orchestrator pattern is key to keeping services decoupled and simplifying your code. I'm working on a Typescript library to encapsulate all the technical detail in https://github.com/node-ts/bus/tree/master/packages/bus-workflow https://github.com/node-ts/bus/tree/master/packages/bus-work... so that you only worry about message inputs and outputs keeping it easy to see the logical flows of your system. It's Inspired from the EIP pattern (https://www.enterpriseintegrationpatterns.com/patterns/messaging/ProcessManager.html https://www.enterpriseintegrationpatterns.com/patterns/messa...) and frameworks from other languages that have had this type of thing for years. The author also hints about an alternative, commonly known as "routing slips" (https://www.enterpriseintegrationpatterns.com/patterns/messaging/RoutingTable.html https://www.enterpriseintegrationpatterns.com/patterns/messa...), where the "next step" is loaded into the message header and gets passed down to the next handler as part of a broader process. This removes the need for a centralised orchestrator, but has a huge downside of being tough to handle failure paths, compensating logic or switching logic.
dsirola 8y ago
Sounds like a nightmare to me. Something that could've been a module in a regular application is split across multiple services introducing multiple network calls that can fail (not to mention the response time once the workflow grows). If people who don't know how to write modular code start using microservices as a solution, the only thing that changes is that you end up with distributed big ball of mud instead of the regular big ball of mud.
- redact207 8y ago
  It's worth clarifying that workflows are generally used to coordinate long running processes that can last up to days - think coordinating receiving an order, picking, dispatch etc. It's not intended to fit within the space of an http request/response. As for should everything be in a module? Idk, but it's a much nicer place to start than microservices
- romanovcode 8y ago
  Congratulations, you've just described 90% of microservices.
- jbverschoor 8y ago
  https://mathservice.my.org/api/v1/multiply https://mathservice.my.org/api/v1/multiply
deleted 8y ago
[deleted]
AzzieElbab 8y ago
Funny, I just read man pages for the ip command and I am no longer sure about "do one thing..." tningy
- module0000 7y ago
  `ip` and `openssl` are both mega-guilty of this! At least the man pages for the `ip` sub-commands are sanely named though.. example: `man openssl` to find how to use `rand` sub-command, instructs you to reference `man rand`. `man ip` to find how to use the `address` sub-command, instructs you to reference `man ip-address`.
itamarst 8y ago
Microservices are an organizational pattern: "we have 500 programmers working on one application, how the hell do we do it." Solution: break up application into independent projects, each team of developers works on one of them, have extra team that builds tooling to make everyone happy. Using microservices with a small teams is a terrible idea, usually, because you're switching to a distributed system when you don't need to. And distributed systems are hard to get right. And you're not even following the organizational pattern, because you have one team with N microservices.
- BerislavLopac 8y ago
  Essentially, microservices are Conway's Law in practice.
- pjmlp 8y ago
  And even then, those 500 programmers can be distributed across application modules, delivering libraries, no need for microservices.
  - maxxxxx 8y ago
    Agreed. If you don’t have the discipline to maintain libraries and their interfaces then you won’t have the discipline to maintain micro services either. I am sure there are good use cases for micro services but using them to compensate for bad coordination won’t help.
  - barrkel 7y ago
    Kind of. If you need to scale compute across multiple nodes, starting out as some kind of service abstraction over a method call will help. But most people don't need that scale.
    - pjmlp 7y ago
      We already learned back in the days of SUN RPC how scaling across method calls works in practice across the network.
gambler 8y ago
In 1997 at OOPSLA Alan Kay said "every object on the internet should have a URL". The response at that time (and many years afterward) could be generally described as smugnorant mockery. A bunch of smart-asses decided that "it doesn't work", mostly by sharing gossip about CORBA failures, rather than looking into relevant research and experiments. 20+ years later we get the same idea presented as some groundbreaking revelation, except it's made orders of magnitude more complicated by the virtue of being an ad-hoc accretion of "hip" technologies. All created with zero overarching vision by people who clearly aren't even aware of past research or ideas. The crowds applaud - themselves. This is depressing.
- gambler 8y ago
  Let me just point out some ideas that seem to have been forgotten: - The idea that machines can exchange live objects, which can bring their "friends" along. (Before anyone says something about this not working - this is very similar to how we load JavaScript in the browser on nearly every page on the web. Except way more formalized and universal.) - The idea that concurrency can be achieved by versioning object changes and reconciling different version of reality when you need to. Pseudo-time, etc. - The idea that objects that respond the same to the same messages are equivalent. Which extends to the notion that a node on the network can be modeled as an object and conversely a local object can be made a node on the network. ... Joe Armstrong had a lot of good intuitions when designing Erlang. I's a pity that after a brief surge in popularity Erlang/Elixir have been downgraded to "not hip enough" status. ... Another thing. If you like Unix pipes, you have to love late-bound dynamically typed objects that communicate through message passing, because Unix pipes can be generalized as such. Yet I routinely see people who claim to like the former and hate the latter. It's especially depressing how many people don't see the point of "message passing" as the fundamental building block of programming. Just a couple examples why it matters: - This progression. Object interface -> inter-object protocol (extends to sequences and state) -> object communication language (allows objects/system to derive protocols). - If objects communicate through messages then object interactions with its environment are also messages. Which means you can create virtualized environments for an object by restricting or manipulating those messages.
  - 7y ago
newaccoutnas 8y ago
There's some irony in the example. > ls -a | grep “gear” Using 2 services to do the job of 1 (find)
- mbrumlow 8y ago
  I feel like you missed the point of the article. Programs like find where taking the functions of two (and more) smaller more finite subsystems and rolling them into one, more complicated system. That added complication assumes all the functionality you would ever need is baked in -- in this case it might be, but this was just a example to start out with. $ man find | wc -l 1020 $ man ls | wc -l 207 $ man grep | wc -l 400

135 points by ashish5887 8y ago

The workflow/orchestrator pattern is key to keeping services decoupled and simplifying your code. I'm working on a Typescript library to encapsulate all the technical detail in https://github.com/node-ts/bus/tree/master/packages/bus-workflow https://github.com/node-ts/bus/tree/master/packages/bus-work... so that you only worry about message inputs and outputs keeping it easy to see the logical flows of your system. It's Inspired from the EIP pattern (https://www.enterpriseintegrationpatterns.com/patterns/messaging/ProcessManager.html https://www.enterpriseintegrationpatterns.com/patterns/messa...) and frameworks from other languages that have had this type of thing for years. The author also hints about an alternative, commonly known as "routing slips" (https://www.enterpriseintegrationpatterns.com/patterns/messaging/RoutingTable.html https://www.enterpriseintegrationpatterns.com/patterns/messa...), where the "next step" is loaded into the message header and gets passed down to the next handler as part of a broader process. This removes the need for a centralised orchestrator, but has a huge downside of being tough to handle failure paths, compensating logic or switching logic.

dsirola 8y ago

Sounds like a nightmare to me. Something that could've been a module in a regular application is split across multiple services introducing multiple network calls that can fail (not to mention the response time once the workflow grows). If people who don't know how to write modular code start using microservices as a solution, the only thing that changes is that you end up with distributed big ball of mud instead of the regular big ball of mud.

redact207 8y ago
It's worth clarifying that workflows are generally used to coordinate long running processes that can last up to days - think coordinating receiving an order, picking, dispatch etc. It's not intended to fit within the space of an http request/response. As for should everything be in a module? Idk, but it's a much nicer place to start than microservices
romanovcode 8y ago
Congratulations, you've just described 90% of microservices.
jbverschoor 8y ago
https://mathservice.my.org/api/v1/multiply https://mathservice.my.org/api/v1/multiply

deleted 8y ago

[deleted]

AzzieElbab 8y ago

Funny, I just read man pages for the ip command and I am no longer sure about "do one thing..." tningy

module0000 7y ago
`ip` and `openssl` are both mega-guilty of this! At least the man pages for the `ip` sub-commands are sanely named though.. example: `man openssl` to find how to use `rand` sub-command, instructs you to reference `man rand`. `man ip` to find how to use the `address` sub-command, instructs you to reference `man ip-address`.

itamarst 8y ago

Microservices are an organizational pattern: "we have 500 programmers working on one application, how the hell do we do it." Solution: break up application into independent projects, each team of developers works on one of them, have extra team that builds tooling to make everyone happy. Using microservices with a small teams is a terrible idea, usually, because you're switching to a distributed system when you don't need to. And distributed systems are hard to get right. And you're not even following the organizational pattern, because you have one team with N microservices.

BerislavLopac 8y ago
Essentially, microservices are Conway's Law in practice.
pjmlp 8y ago
And even then, those 500 programmers can be distributed across application modules, delivering libraries, no need for microservices.
- maxxxxx 8y ago
  Agreed. If you don’t have the discipline to maintain libraries and their interfaces then you won’t have the discipline to maintain micro services either. I am sure there are good use cases for micro services but using them to compensate for bad coordination won’t help.
- barrkel 7y ago
  Kind of. If you need to scale compute across multiple nodes, starting out as some kind of service abstraction over a method call will help. But most people don't need that scale.
  - pjmlp 7y ago
    We already learned back in the days of SUN RPC how scaling across method calls works in practice across the network.

gambler 8y ago

In 1997 at OOPSLA Alan Kay said "every object on the internet should have a URL". The response at that time (and many years afterward) could be generally described as smugnorant mockery. A bunch of smart-asses decided that "it doesn't work", mostly by sharing gossip about CORBA failures, rather than looking into relevant research and experiments. 20+ years later we get the same idea presented as some groundbreaking revelation, except it's made orders of magnitude more complicated by the virtue of being an ad-hoc accretion of "hip" technologies. All created with zero overarching vision by people who clearly aren't even aware of past research or ideas. The crowds applaud - themselves. This is depressing.

gambler 8y ago
Let me just point out some ideas that seem to have been forgotten: - The idea that machines can exchange live objects, which can bring their "friends" along. (Before anyone says something about this not working - this is very similar to how we load JavaScript in the browser on nearly every page on the web. Except way more formalized and universal.) - The idea that concurrency can be achieved by versioning object changes and reconciling different version of reality when you need to. Pseudo-time, etc. - The idea that objects that respond the same to the same messages are equivalent. Which extends to the notion that a node on the network can be modeled as an object and conversely a local object can be made a node on the network. ... Joe Armstrong had a lot of good intuitions when designing Erlang. I's a pity that after a brief surge in popularity Erlang/Elixir have been downgraded to "not hip enough" status. ... Another thing. If you like Unix pipes, you have to love late-bound dynamically typed objects that communicate through message passing, because Unix pipes can be generalized as such. Yet I routinely see people who claim to like the former and hate the latter. It's especially depressing how many people don't see the point of "message passing" as the fundamental building block of programming. Just a couple examples why it matters: - This progression. Object interface -> inter-object protocol (extends to sequences and state) -> object communication language (allows objects/system to derive protocols). - If objects communicate through messages then object interactions with its environment are also messages. Which means you can create virtualized environments for an object by restricting or manipulating those messages.
- 7y ago

newaccoutnas 8y ago

There's some irony in the example. > ls -a | grep “gear” Using 2 services to do the job of 1 (find)

mbrumlow 8y ago
I feel like you missed the point of the article. Programs like find where taking the functions of two (and more) smaller more finite subsystems and rolling them into one, more complicated system. That added complication assumes all the functionality you would ever need is baked in -- in this case it might be, but this was just a example to start out with. $ man find | wc -l 1020 $ man ls | wc -l 207 $ man grep | wc -l 400