3 ms·

Not sure how to approach that comment really, I don't think it matters if the pattern used is microservices or not, the basic premise that we're still way far o

by asim 4y ago

Not sure how to approach that comment really, I don't think it matters if the pattern used is microservices or not, the basic premise that we're still way far off reusability of software that's basically identical across thousands of orgs stands. Most devs are rebuilding identical software stacks to serve a very small different business layer product. There's probably 80% of software that we could open source as reusable building blocks. Do they need to be microservices? No, but if the logic was in libraries and we had to reimplement per language it would be really a waste of time. Could we package this in some other way? Someone please show me that solution because I'll stop talking about microservices and go use that. Ultimately from what I've seen at companies like Amazon, Google, etc is reusability is about services development exposed as APIs with common libraries, frameworks and platforms for that development/deployment, itself, that's it. Its not buzzwordy, but microservices encapsulates that model. Amazon/Google, etc are just hundreds of small teams all using a shared model of development, so the argument that you don't need microservices at small scale is not the issue here, it's that we outside of orgs don't have this level of reuse that they do, so we waste most of our time spinning our wheels building the same software, where as if we had a set of reusable building blocks or APIs on top of cloud infrastructure, we could actually then go solve even more interesting ideas rather than it being something only possible at massive orgs.

throw827474737 4y ago
But then I look e.g. at your weather microservice, and see again a (configurable, but still expecting some already standardized format) wrapper around an existing API that once more mingles the data all around? Sorry, but really not getting what this is achieving/solving?
- asim 4y ago
  Building a standard set of interfaces consumed as APIs regardless of what the backend implementation is. Something that exists on a network rather than as a library that can be executed and changed ad-hoc without having to upgrade thousands of binaries. It also provides opportunities to do far more interesting things with the data when you have enough users at scale e.g predicting demand, caching data, etc.
  - throw827474737 4y ago
    What is the standard you built, how is it specced? In code that I just looked at? Man I feel old and stupid, but really still not got it any further :( You made it even worse now.. "lives on the network.. without updating thousand binaries" - huh what, how? You sound a bit like Urbit btw, not sure if appropriate ref though.
    - asim 4y ago
      lol, really don't want to be in the same sentence as Urbit so please no. Everything is defined as protobuf interfaces, which is a standard used by Google and everyone else now that gRPC is so dominant. So the idea is, define the API in protobuf, code generate and implement the handlers for it. The service can be called by other services on the platform using that code generation and then an API Gateway, which Micro provides can be used to call services externally using the same format but using HTTP/JSON. To take that even further, M3O (m3o.com) codifies protobuf to openapi specs and then generates client libraries on top. You can see some of that in https://github.com/m3o/m3o https://github.com/m3o/m3o.
      - davidjfelix 4y ago
        Total tangent, but gRPC is not dominant and I think that belief is cargo-cultism.
        simonw 4y ago
        Yeah I've managed to avoid gRPC for my whole career to date, and I've done a LOT of work on web services and microservices.
      - throw827474737 4y ago
        Thanks, that made now more sense. I'd put this condensed together with https://micro.dev/blog/2022/09/27/real-world-micro-services.html https://micro.dev/blog/2022/09/27/real-world-micro-services.... more prominently to the readme of https://github.com/micro/services https://github.com/micro/services ! Looking at that github alone makes it hard to commect the context.
cowl 4y ago
And the premise that there can be reusability of software in that scale is the flow. > Most devs are rebuilding identical software stacks to serve a very small different business layer product. The problem is plainly because we don't recognize that the "Identical Business layer" does not exist. On the surface they seem identical but then each has its own features/prerequisites/peculiarities etc. And you end up with having to fight that "reusable framework" to handle all this because to be reusable it has to stay generic enough and the more generic something is the more useless it is for the concreate problem and more time consuming it is to reuse it. In the golden era of Web Services the moto was to make reusable software same as electricity is reusable through standards (just plug in your device) forgetting that Electricity is reusable only because it doesn't really do anything. All implementation of the useful work is in the device you plug. And what's more these devices now are forced to work on non efficient standards, Hey I need just 2W of power to work but to interface with the Reusable thing I must waste 5w power to conversion. Fortunately we still don't have that kind of reusability in Software and hope we will never have to.
- masukomi 4y ago
  at the large scale sure. at the small scale no. For example, look at all the different implementations of JSONPath in all the various language that do exactly the same thing: take in some JSON, extract some data from a specific point in it based on a string input. do we _really_ need to rewrite that every time we start a project in a different language that doesn't already have an implementation? Or, could it be loaded/used via a service or other resource that's made accessible in some language agnostic way?
- HeyLaughingBoy 4y ago
  I work for a company that build embedded systems, commonly on a core family of processors. The way we get around this is to have an ever-expanding core set of reusable software that can be customized for each project. At the beginning of a project, you fork that core and add it to your project. If you identify a bug in the core, then you discuss it with the maintainers, do a PR and the fix gets merged back for everyone to use. Similarly for extensions that are likely to be usable across the company. Stuff that's only relevant to your project stays on your fork. It's not perfect, but it saves an amazing amount of duplicated work. The problem is getting to be that the framework is getting big enough that people new to the company don't always know what's in it, and accidentally end up implementing something that already existed. We're still working on a solution to this.
deepGem 4y ago
Spring cloud ecosystem is headed in this direction. Most of the common needs for micro service development such as Service mesh, service discovery, circuit breakers are getting incorporated into the ecosystem so as to reduce repetitive development. Not sure how many devs are actually using this. They are building upon Netflix offerings. https://spring.io/projects/spring-cloud-netflix https://spring.io/projects/spring-cloud-netflix

by asim 4y ago

throw827474737 4y ago

But then I look e.g. at your weather microservice, and see again a (configurable, but still expecting some already standardized format) wrapper around an existing API that once more mingles the data all around? Sorry, but really not getting what this is achieving/solving?

asim 4y ago
Building a standard set of interfaces consumed as APIs regardless of what the backend implementation is. Something that exists on a network rather than as a library that can be executed and changed ad-hoc without having to upgrade thousands of binaries. It also provides opportunities to do far more interesting things with the data when you have enough users at scale e.g predicting demand, caching data, etc.
- throw827474737 4y ago
  What is the standard you built, how is it specced? In code that I just looked at? Man I feel old and stupid, but really still not got it any further :( You made it even worse now.. "lives on the network.. without updating thousand binaries" - huh what, how? You sound a bit like Urbit btw, not sure if appropriate ref though.
  - asim 4y ago
    lol, really don't want to be in the same sentence as Urbit so please no. Everything is defined as protobuf interfaces, which is a standard used by Google and everyone else now that gRPC is so dominant. So the idea is, define the API in protobuf, code generate and implement the handlers for it. The service can be called by other services on the platform using that code generation and then an API Gateway, which Micro provides can be used to call services externally using the same format but using HTTP/JSON. To take that even further, M3O (m3o.com) codifies protobuf to openapi specs and then generates client libraries on top. You can see some of that in https://github.com/m3o/m3o https://github.com/m3o/m3o.
    - davidjfelix 4y ago
      Total tangent, but gRPC is not dominant and I think that belief is cargo-cultism.
      - simonw 4y ago
        Yeah I've managed to avoid gRPC for my whole career to date, and I've done a LOT of work on web services and microservices.
    - throw827474737 4y ago
      Thanks, that made now more sense. I'd put this condensed together with https://micro.dev/blog/2022/09/27/real-world-micro-services.html https://micro.dev/blog/2022/09/27/real-world-micro-services.... more prominently to the readme of https://github.com/micro/services https://github.com/micro/services ! Looking at that github alone makes it hard to commect the context.

cowl 4y ago

And the premise that there can be reusability of software in that scale is the flow. > Most devs are rebuilding identical software stacks to serve a very small different business layer product. The problem is plainly because we don't recognize that the "Identical Business layer" does not exist. On the surface they seem identical but then each has its own features/prerequisites/peculiarities etc. And you end up with having to fight that "reusable framework" to handle all this because to be reusable it has to stay generic enough and the more generic something is the more useless it is for the concreate problem and more time consuming it is to reuse it. In the golden era of Web Services the moto was to make reusable software same as electricity is reusable through standards (just plug in your device) forgetting that Electricity is reusable only because it doesn't really do anything. All implementation of the useful work is in the device you plug. And what's more these devices now are forced to work on non efficient standards, Hey I need just 2W of power to work but to interface with the Reusable thing I must waste 5w power to conversion. Fortunately we still don't have that kind of reusability in Software and hope we will never have to.

masukomi 4y ago
at the large scale sure. at the small scale no. For example, look at all the different implementations of JSONPath in all the various language that do exactly the same thing: take in some JSON, extract some data from a specific point in it based on a string input. do we _really_ need to rewrite that every time we start a project in a different language that doesn't already have an implementation? Or, could it be loaded/used via a service or other resource that's made accessible in some language agnostic way?
HeyLaughingBoy 4y ago
I work for a company that build embedded systems, commonly on a core family of processors. The way we get around this is to have an ever-expanding core set of reusable software that can be customized for each project. At the beginning of a project, you fork that core and add it to your project. If you identify a bug in the core, then you discuss it with the maintainers, do a PR and the fix gets merged back for everyone to use. Similarly for extensions that are likely to be usable across the company. Stuff that's only relevant to your project stays on your fork. It's not perfect, but it saves an amazing amount of duplicated work. The problem is getting to be that the framework is getting big enough that people new to the company don't always know what's in it, and accidentally end up implementing something that already existed. We're still working on a solution to this.

deepGem 4y ago

Spring cloud ecosystem is headed in this direction. Most of the common needs for micro service development such as Service mesh, service discovery, circuit breakers are getting incorporated into the ecosystem so as to reduce repetitive development. Not sure how many devs are actually using this. They are building upon Netflix offerings. https://spring.io/projects/spring-cloud-netflix https://spring.io/projects/spring-cloud-netflix