5 ms·

One of the biggest issues we face in the industry is the lack of reusability in software Disagree with this core premise. The biggest problems we face are tha

by weego 4y ago

One of the biggest issues we face in the industry is the lack of reusability in software Disagree with this core premise. The biggest problems we face are that every developer is coached into believing everything they make needs to be unicorn scale and quality, that everything they build should conform to impractical levels of scrutiny and long-termism and that it is unavoidable for even basic development needs to have incredibly high levels of complexity in code / build / release tooling. Microservices are just the latest weapon in furthering this agenda for the good of a tiny minority of people who value indulging their time and business costs in esoteric software philosophy goals.

asim 4y ago
Not sure how to approach that comment really, I don't think it matters if the pattern used is microservices or not, the basic premise that we're still way far off reusability of software that's basically identical across thousands of orgs stands. Most devs are rebuilding identical software stacks to serve a very small different business layer product. There's probably 80% of software that we could open source as reusable building blocks. Do they need to be microservices? No, but if the logic was in libraries and we had to reimplement per language it would be really a waste of time. Could we package this in some other way? Someone please show me that solution because I'll stop talking about microservices and go use that. Ultimately from what I've seen at companies like Amazon, Google, etc is reusability is about services development exposed as APIs with common libraries, frameworks and platforms for that development/deployment, itself, that's it. Its not buzzwordy, but microservices encapsulates that model. Amazon/Google, etc are just hundreds of small teams all using a shared model of development, so the argument that you don't need microservices at small scale is not the issue here, it's that we outside of orgs don't have this level of reuse that they do, so we waste most of our time spinning our wheels building the same software, where as if we had a set of reusable building blocks or APIs on top of cloud infrastructure, we could actually then go solve even more interesting ideas rather than it being something only possible at massive orgs.
- throw827474737 4y ago
  But then I look e.g. at your weather microservice, and see again a (configurable, but still expecting some already standardized format) wrapper around an existing API that once more mingles the data all around? Sorry, but really not getting what this is achieving/solving?
  - asim 4y ago
    Building a standard set of interfaces consumed as APIs regardless of what the backend implementation is. Something that exists on a network rather than as a library that can be executed and changed ad-hoc without having to upgrade thousands of binaries. It also provides opportunities to do far more interesting things with the data when you have enough users at scale e.g predicting demand, caching data, etc.
    - throw827474737 4y ago
      What is the standard you built, how is it specced? In code that I just looked at? Man I feel old and stupid, but really still not got it any further :( You made it even worse now.. "lives on the network.. without updating thousand binaries" - huh what, how? You sound a bit like Urbit btw, not sure if appropriate ref though.
      - asim 4y ago
        lol, really don't want to be in the same sentence as Urbit so please no. Everything is defined as protobuf interfaces, which is a standard used by Google and everyone else now that gRPC is so dominant. So the idea is, define the API in protobuf, code generate and implement the handlers for it. The service can be called by other services on the platform using that code generation and then an API Gateway, which Micro provides can be used to call services externally using the same format but using HTTP/JSON. To take that even further, M3O (m3o.com) codifies protobuf to openapi specs and then generates client libraries on top. You can see some of that in https://github.com/m3o/m3o https://github.com/m3o/m3o.
sneak 4y ago
Microservices with an api proxy were nice for me when I ran a small team because different subteams could use their preferred language, and set their own release cadence, and autoscale separately. It wasn’t so much about scale or quality as it was letting different teams independently build/deploy/maintain/scale different routes on the same public api server hostname. We also got to centralize the read cache memory pool in the frontend, so each backend service is not potentially caching duplicate stuff, saving on memory across the org.
pfarrell 4y ago
Perlisism #74: “Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?” http://cs.yale.edu/homes/perlis-alan/quotes.html http://cs.yale.edu/homes/perlis-alan/quotes.html
magic_hamster 4y ago
This is fairly true, except unicorn level. In a unicorn start-up where things happen extremely fast and your valuation skyrockets, it's more likely the code isn't exactly beautiful and tidy. I understand where the author comes from. You want to use tools developed in company instead of being cut off when you leave.
- asim 4y ago
  Yea like you're part of this club that has exclusive access to something, you contribute to it, deliver value, see it grow and then it's gone when you leave. It exists within a silo and for the better part of a decade that's really irked me but I haven't quite figured out how to solve for that problem beyond doing it in a shared open source repo and a shared platform. I think the issue is it's bigger than any one person and you have to find a way to sell thousands of people on the idea. My starting point was code and now I wonder could I have approached this differently? Is there another path in which this would actually succeed? I'm still trying to figure it out and it's driving me crazy. Next I'll be writing a protocol no joke https://github.com/micro/network https://github.com/micro/network
- papito 4y ago
  Just basic due diligence about performance and knowing your storage engine can easily get you to and even at unicorn level. Stack Overflow regularly bragged about how their eight servers and a few MSSQL replicas were basically always in a coma, even at peak hours. That's just good engineering. Microservices themselves solve nothing out of the box.
efsavage 4y ago
>> One of the biggest issues we face in the industry is the lack of reusability in software > The biggest problems we face are that every developer is coached into believing everything they make needs to be unicorn scale and quality I kind of disagree with both of these statements. Reusability/DRY/etc. is just a useful technique or pattern. If you can do it, cool. If it's not worth it, factoring in complexity and cost, carry on. Building something stable, reliable, and dependable, which is what I take "unicorn scale" to mean, is not necessary, it is simply good engineering. If you're thinking this is necessary for every project, you're not necessarily a bad developer, just a specialist, and the only real problem is if you don't realize that. Software is a blend of engineering, art, and craft. Knowing what mix the current problem calls for is really the core pillar of being an effective developer. Too much engineering in your prototype and you've undermined it's flexibility. Too much craft in your production service and you've created a maintenance nightmare. Not enough art in your API and nobody wants to use it.
- deleted 4y ago
  [deleted]
- deleted 4y ago
  [deleted]
- zerkten 4y ago
  I don't take "unicorn scale" to mean "stable, reliable, and dependable" in the context of the original post. You seem to be a reasonable person that values building "stable, reliable, and dependable" systems. There is another segment of the tech world that have used the wrong tools and techniques for their domain based on experiences shared by the largest companies. This is often resume-driven development. It results in a lot of toil and rework for those maintaining those systems. The challenge is that assessing outcomes is very contextual. There are teams that will be successful with a completely overengineered system built with what seemed like an unrealistic toolkit, and there will be teams that fail with an overengineered system and seemingly better toolset and team. For example, I'm aware of pharma teams using K8s and Svelte for small apps when a Django or .NET app would be sufficient.
debaserab2 4y ago
Microservices are usually a solution to an org. level scaling problem before a technical scaling problem*. It's easier to coordinate development resources around small codebases than it is larger ones. It's a human problem, not a technical one. * Not always true of course
- acjohnson55 4y ago
  I disagree that they actually solve that problem. Instead, they bake organizational fragmentation into the tech infrastructure. This is one way to accomplish encapsulation, but it comes with a bunch of costs. The reason I think it doesn't address scaling is that delivering functionality that crosses org/service boundaries becomes more complex, potentially reducing scalability.
ravenstine 4y ago
So much time is wasted with DRY and abstracting everything. There's a time and a place for those, but they're usually treated as theology, the result often being much worse software than something that developers can understand without having spent years on a project.

by weego 4y ago

asim 4y ago

Not sure how to approach that comment really, I don't think it matters if the pattern used is microservices or not, the basic premise that we're still way far off reusability of software that's basically identical across thousands of orgs stands. Most devs are rebuilding identical software stacks to serve a very small different business layer product. There's probably 80% of software that we could open source as reusable building blocks. Do they need to be microservices? No, but if the logic was in libraries and we had to reimplement per language it would be really a waste of time. Could we package this in some other way? Someone please show me that solution because I'll stop talking about microservices and go use that. Ultimately from what I've seen at companies like Amazon, Google, etc is reusability is about services development exposed as APIs with common libraries, frameworks and platforms for that development/deployment, itself, that's it. Its not buzzwordy, but microservices encapsulates that model. Amazon/Google, etc are just hundreds of small teams all using a shared model of development, so the argument that you don't need microservices at small scale is not the issue here, it's that we outside of orgs don't have this level of reuse that they do, so we waste most of our time spinning our wheels building the same software, where as if we had a set of reusable building blocks or APIs on top of cloud infrastructure, we could actually then go solve even more interesting ideas rather than it being something only possible at massive orgs.

throw827474737 4y ago
But then I look e.g. at your weather microservice, and see again a (configurable, but still expecting some already standardized format) wrapper around an existing API that once more mingles the data all around? Sorry, but really not getting what this is achieving/solving?
- asim 4y ago
  Building a standard set of interfaces consumed as APIs regardless of what the backend implementation is. Something that exists on a network rather than as a library that can be executed and changed ad-hoc without having to upgrade thousands of binaries. It also provides opportunities to do far more interesting things with the data when you have enough users at scale e.g predicting demand, caching data, etc.
  - throw827474737 4y ago
    What is the standard you built, how is it specced? In code that I just looked at? Man I feel old and stupid, but really still not got it any further :( You made it even worse now.. "lives on the network.. without updating thousand binaries" - huh what, how? You sound a bit like Urbit btw, not sure if appropriate ref though.
    - asim 4y ago
      lol, really don't want to be in the same sentence as Urbit so please no. Everything is defined as protobuf interfaces, which is a standard used by Google and everyone else now that gRPC is so dominant. So the idea is, define the API in protobuf, code generate and implement the handlers for it. The service can be called by other services on the platform using that code generation and then an API Gateway, which Micro provides can be used to call services externally using the same format but using HTTP/JSON. To take that even further, M3O (m3o.com) codifies protobuf to openapi specs and then generates client libraries on top. You can see some of that in https://github.com/m3o/m3o https://github.com/m3o/m3o.

sneak 4y ago

Microservices with an api proxy were nice for me when I ran a small team because different subteams could use their preferred language, and set their own release cadence, and autoscale separately. It wasn’t so much about scale or quality as it was letting different teams independently build/deploy/maintain/scale different routes on the same public api server hostname. We also got to centralize the read cache memory pool in the frontend, so each backend service is not potentially caching duplicate stuff, saving on memory across the org.

pfarrell 4y ago

Perlisism #74: “Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?” http://cs.yale.edu/homes/perlis-alan/quotes.html http://cs.yale.edu/homes/perlis-alan/quotes.html

magic_hamster 4y ago

This is fairly true, except unicorn level. In a unicorn start-up where things happen extremely fast and your valuation skyrockets, it's more likely the code isn't exactly beautiful and tidy. I understand where the author comes from. You want to use tools developed in company instead of being cut off when you leave.

asim 4y ago
Yea like you're part of this club that has exclusive access to something, you contribute to it, deliver value, see it grow and then it's gone when you leave. It exists within a silo and for the better part of a decade that's really irked me but I haven't quite figured out how to solve for that problem beyond doing it in a shared open source repo and a shared platform. I think the issue is it's bigger than any one person and you have to find a way to sell thousands of people on the idea. My starting point was code and now I wonder could I have approached this differently? Is there another path in which this would actually succeed? I'm still trying to figure it out and it's driving me crazy. Next I'll be writing a protocol no joke https://github.com/micro/network https://github.com/micro/network
papito 4y ago
Just basic due diligence about performance and knowing your storage engine can easily get you to and even at unicorn level. Stack Overflow regularly bragged about how their eight servers and a few MSSQL replicas were basically always in a coma, even at peak hours. That's just good engineering. Microservices themselves solve nothing out of the box.

efsavage 4y ago

>> One of the biggest issues we face in the industry is the lack of reusability in software > The biggest problems we face are that every developer is coached into believing everything they make needs to be unicorn scale and quality I kind of disagree with both of these statements. Reusability/DRY/etc. is just a useful technique or pattern. If you can do it, cool. If it's not worth it, factoring in complexity and cost, carry on. Building something stable, reliable, and dependable, which is what I take "unicorn scale" to mean, is not necessary, it is simply good engineering. If you're thinking this is necessary for every project, you're not necessarily a bad developer, just a specialist, and the only real problem is if you don't realize that. Software is a blend of engineering, art, and craft. Knowing what mix the current problem calls for is really the core pillar of being an effective developer. Too much engineering in your prototype and you've undermined it's flexibility. Too much craft in your production service and you've created a maintenance nightmare. Not enough art in your API and nobody wants to use it.

deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
zerkten 4y ago
I don't take "unicorn scale" to mean "stable, reliable, and dependable" in the context of the original post. You seem to be a reasonable person that values building "stable, reliable, and dependable" systems. There is another segment of the tech world that have used the wrong tools and techniques for their domain based on experiences shared by the largest companies. This is often resume-driven development. It results in a lot of toil and rework for those maintaining those systems. The challenge is that assessing outcomes is very contextual. There are teams that will be successful with a completely overengineered system built with what seemed like an unrealistic toolkit, and there will be teams that fail with an overengineered system and seemingly better toolset and team. For example, I'm aware of pharma teams using K8s and Svelte for small apps when a Django or .NET app would be sufficient.

debaserab2 4y ago

Microservices are usually a solution to an org. level scaling problem before a technical scaling problem*. It's easier to coordinate development resources around small codebases than it is larger ones. It's a human problem, not a technical one. * Not always true of course

acjohnson55 4y ago
I disagree that they actually solve that problem. Instead, they bake organizational fragmentation into the tech infrastructure. This is one way to accomplish encapsulation, but it comes with a bunch of costs. The reason I think it doesn't address scaling is that delivering functionality that crosses org/service boundaries becomes more complex, potentially reducing scalability.

ravenstine 4y ago

So much time is wasted with DRY and abstracting everything. There's a time and a place for those, but they're usually treated as theology, the result often being much worse software than something that developers can understand without having spent years on a project.