5 ms·

My overall reaction is that this is a great piece for products/teams that have reached significant scale, once the job to be done is too big and complex for one

My overall reaction is that this is a great piece for products/teams that have reached significant scale, once the job to be done is too big and complex for one team to own, end-to-end, or there are truly reusable concerns that can be separated from the core product (e.g. auth, observability). Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead. And if we're talking about network APIs, there's a ton of distributed systems complexity to account for. > The problem with the DevOps movement is that it ended up taking “shifting left” to the extreme. In this sense, development teams weren’t so much empowered to deliver software faster; rather, they were over-encumbered with infrastructure tasks that were outside of their expertise. This. In truth, I think this is a major misinterpretation of DevOps, which is meant to empower devs without loading them down with incidental complexity. But I experienced exactly this misinterpretation at the first place I worked that had embraced DevOps culture.

braza 3y ago
> Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead. I agree in principle, but there is a lot of “unseen coordination/communication” costs that it’s easy taken for granted. When I was working on telecom doing interfacing with carriers (e.g T-Mobile, Verizon, etc.) on thing that I noticed was how simple was to work with those folks: Ok, this is the standard XML, those are the endpoints, that’s the list of error codes, the rate limit is X requests per second, a bunch of files will be on this FTP at 5AM daily basis, and if you face more than 100ms latency from our side just call this number. Working with “product” companies without silos most of the time it’s design by committee, folks that won’t keep the service running wanting to have a say in our payload wanting us to change our overly reliable RabbitMQ to use their Kafka.
- acjohnson55 3y ago
  > When I was working on telecom doing interfacing with carriers (e.g T-Mobile, Verizon, etc.) on thing that I noticed was how simple was to work with those folks: Ok, this is the standard XML, those are the endpoints, that’s the list of error codes, the rate limit is X requests per second, a bunch of files will be on this FTP at 5AM daily basis, and if you face more than 100ms latency from our side just call this number. To me, that probably reflects the maturity of the services the carriers provide. And presumably that there's an explicit customer-producer relationship? These things justify the complexity of maintaining a well curated and operated API. > Working with “product” companies without silos most of the time it’s design by committee, folks that won’t keep the service running wanting to have a say in our payload wanting us to change our overly reliable RabbitMQ to use their Kafka. If I understand what you're saying, you've experienced platform people telling you what tech to use, without having real skin in the game for operating your services? If so, that sounds very irritating. To me, a truly silo-less approach would not have that. To the extent that there are platform teams with a say in architecture, I think they should develop requirements around the external characteristics of the deliverable (performance, cost, observability, contract with other teams, etc) and largely leave the implementation concerns to the people developing and running the service.
- The_Colonel 3y ago
  This is an interesting example, becase telco has an actual API standards committee (TM Forum). Telcos have decades of experience and extremely well defined (and to a large degree shared / interchangeable) domain model. It's an ideal scenario for APIs. Meanwhile your product companies each develop different product, there's little standardization. API designers have only vague idea what the API will be used for and how. Fast evolution is important.
  - adamzochowski 3y ago
    TM Forum provides a bloated format. It has everything and a kitchen sink. This means that something simple like customer name can be written in multiple places, and different architects will recommend different location. Two different silos within same company won't be able to talk to each other because although they are using same format, they interpret the usage different. Additionally, this is XML/XSD, different teams will end up using different version of the standard. They won't be able to interface with each other without additional layers of translations. It is not uncommon for one team to need to load multiple versions of the XSDs because different end points use different versions.
- sgarland 3y ago
  > folks that won’t keep the service running wanting to have a say in our payload wanting us to change This is my single biggest gripe with DevOps. If you’re not going to be fixing it, why on earth do you get a say as to how I build it? It’s nearly _always_ a one-way street, too – when’s the last time Ops successfully ordered Dev to change some specific part of their code (modulo things like, “hey, you really need to add a rate-limiter”)?
  - acjohnson55 3y ago
    IMO, this phenomenon is not specific to DevOps. I've seen this happen when there are official architects or architecture committees. One of the problems about debating DevOps is that there's no single agreed upon meaning. That probably also impedes its success. To me, the essence is that there are engineers whose job it is to give product and service devs the ability to do their own operations through software that simplifies basic shared concerns. That does not mean mandating solutions.
- rqtwteye 3y ago
  “Working with “product” companies without silos most of the time it’s design by committee,” With siloes you may end up with design by ego trip.
  - danielovichdk 3y ago
    Egos only exists if the team let them exist. A strong leader will make sure this is not happening.
    - jpc0 3y ago
      A strong leader at the same time can cause it to happen...
acjohnson55 3y ago
As a coda to this, I think that we are entering a new world where services truly can have minimal operational overhead. At my last company, we were all in on serverless, with AWS tools like SQS coupling things together. It worked extremely well for keeping the architecture simple to operate and approachable for new people. But even better, I think we want logical services that interface with each other as though they were in a single process. We want the ability to write code as though it is monolithic (e.g. lexical scope and native-language APIs), while availing ourselves of the advantages of independently deployed services. I believe projects like Temporal point the way. I haven't had the opportunity to use it, but philosophically, I think it's the right direction.
- throw04323 3y ago
  I don't have experience with it myself, but Polylith architecture looks interesting. There you compose services based on shared components. You can start developing it as a monolith, and then extract components to separate services by just changing the interface between components. https://polylith.gitbook.io/polylith/ https://polylith.gitbook.io/polylith/
  - acjohnson55 3y ago
    If I'm understanding Polylith, it seems like a way to do monoglot monorepo in a way that avoids explicitly using libraries. That looks pretty interesting and similar to what Artsy did with Ezel for front-end development. https://github.com/artsy/ezel https://github.com/artsy/ezel
- jakjak123 3y ago
  I have been working mostly with Kafka for five years, and I found SQS to be a little weird to work with. How do you keep an overview of all services that consume a queue? What if you want multiple services to read the same data in the same order? Granted I am very new to AWS.
  - 8note 3y ago
    I think you're really looking to use kinesis rather than SQS. SNS+SQS make a pub/sub setup, but every queue subscribed to the event is fully separate, and you wouldn't expect to try to couple the different listening queues together. You could make all the queues into SQS fifo queues, and put them all on the same sorting group, but I think network time could still break the ordering on different subscribers? The simple way to use SQS is to have a queue for each message type you plan to consume, and treat it as a bucket of work to do, where you can control how fast you pull the work out
toss1 3y ago
Yes APIs are expensive and require significant overhead. But clean, maintained interfaces between separate domains are far LESS expensive than multi-domain/team mashups. Maintaining separate teams AND well- maintained interfaces surfaces and makes explicit the myriad inter-silo interactions. Instead of a quick backchannel conversation and a quick hack, it is an explicit conversation and an API adjustment. Yes, it's more expensive up front, but far cheaper and more powerful in the long run. So for high-speed prototyping, not so suitable- breaj the silos, do the backchannel hack & move on (this version will likely be binned or massively refactored anyway). But when the domain is more stable, modular with solid interfaces is the way to go.
jakjak123 3y ago
I'm not saying its a silver bullet, because sharing data via database makes change management way worse.
6gvONxR4sf7o 3y ago
> My overall reaction is that this is a great piece for products/teams that have reached significant scale, once the job to be done is too big and complex for one team to own, end-to-end, or there are truly reusable concerns that can be separated from the core product (e.g. auth, observability). I'd say it's critical in the small scale too. When you get down to a single person size, this is just what well factored code is: you write down little reasonably independent pieces that can be learned about and thought about without having to think about all the other things. After all, we've all had that experience when you go back and revisit your code after vacation or something, and it might as well have been written by someone else. I was on a tiny team not long ago where my teammates kept writing tightly coupled systems, then rewriting everything from scratch every time. It was hell. Our product moved slowly, broke constantly, and we couldn't build off of our prior work, and could barely even build off of each others' work, so velocity stayed constant (read: slow). (as a tangent, the communication patterns of remote work seem to make this more important) Siloing teams, siloing concerns, and writing modular code are all kind of the same thing, just at different scales.
steveBK123 3y ago
> Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead. An under appreciated point. Something that is affordable for giant monopoly profit driven FAANGs, but harder in smaller orgs. Even within my team, the "level of difficulty" of writing an API depends on its user base. If I am the primary user, it's easy. There's one really sharp kind on my team, if he is going to use it, then 2x the difficulty. If I need it to handle the median dev on my team, then 2x again. By the time you get to the below median dev, add another 2x. So even within my team, the intended user base can change the difficult from 1x->8x. What are some of the pain points - input validation that drives useful informative errors, flexibility on inputs like having sensible defaults to reduce the number of inputs users must pass, performance/scaling, and edge cases. >> In this sense, development teams weren’t so much empowered to deliver software faster; rather, they were over-encumbered with infrastructure tasks that were outside of their expertise. And agreed on this point. Amusingly I have had HN thread arguments just this week with DevOps advocates telling me that akshully their job isn't to empower devs, but some sort of "tail that wags that dog" interpretation around devops organizational / standardization / cost / etc.

braza 3y ago

> Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead. I agree in principle, but there is a lot of “unseen coordination/communication” costs that it’s easy taken for granted. When I was working on telecom doing interfacing with carriers (e.g T-Mobile, Verizon, etc.) on thing that I noticed was how simple was to work with those folks: Ok, this is the standard XML, those are the endpoints, that’s the list of error codes, the rate limit is X requests per second, a bunch of files will be on this FTP at 5AM daily basis, and if you face more than 100ms latency from our side just call this number. Working with “product” companies without silos most of the time it’s design by committee, folks that won’t keep the service running wanting to have a say in our payload wanting us to change our overly reliable RabbitMQ to use their Kafka.

acjohnson55 3y ago
> When I was working on telecom doing interfacing with carriers (e.g T-Mobile, Verizon, etc.) on thing that I noticed was how simple was to work with those folks: Ok, this is the standard XML, those are the endpoints, that’s the list of error codes, the rate limit is X requests per second, a bunch of files will be on this FTP at 5AM daily basis, and if you face more than 100ms latency from our side just call this number. To me, that probably reflects the maturity of the services the carriers provide. And presumably that there's an explicit customer-producer relationship? These things justify the complexity of maintaining a well curated and operated API. > Working with “product” companies without silos most of the time it’s design by committee, folks that won’t keep the service running wanting to have a say in our payload wanting us to change our overly reliable RabbitMQ to use their Kafka. If I understand what you're saying, you've experienced platform people telling you what tech to use, without having real skin in the game for operating your services? If so, that sounds very irritating. To me, a truly silo-less approach would not have that. To the extent that there are platform teams with a say in architecture, I think they should develop requirements around the external characteristics of the deliverable (performance, cost, observability, contract with other teams, etc) and largely leave the implementation concerns to the people developing and running the service.
The_Colonel 3y ago
This is an interesting example, becase telco has an actual API standards committee (TM Forum). Telcos have decades of experience and extremely well defined (and to a large degree shared / interchangeable) domain model. It's an ideal scenario for APIs. Meanwhile your product companies each develop different product, there's little standardization. API designers have only vague idea what the API will be used for and how. Fast evolution is important.
- adamzochowski 3y ago
  TM Forum provides a bloated format. It has everything and a kitchen sink. This means that something simple like customer name can be written in multiple places, and different architects will recommend different location. Two different silos within same company won't be able to talk to each other because although they are using same format, they interpret the usage different. Additionally, this is XML/XSD, different teams will end up using different version of the standard. They won't be able to interface with each other without additional layers of translations. It is not uncommon for one team to need to load multiple versions of the XSDs because different end points use different versions.
sgarland 3y ago
> folks that won’t keep the service running wanting to have a say in our payload wanting us to change This is my single biggest gripe with DevOps. If you’re not going to be fixing it, why on earth do you get a say as to how I build it? It’s nearly _always_ a one-way street, too – when’s the last time Ops successfully ordered Dev to change some specific part of their code (modulo things like, “hey, you really need to add a rate-limiter”)?
- acjohnson55 3y ago
  IMO, this phenomenon is not specific to DevOps. I've seen this happen when there are official architects or architecture committees. One of the problems about debating DevOps is that there's no single agreed upon meaning. That probably also impedes its success. To me, the essence is that there are engineers whose job it is to give product and service devs the ability to do their own operations through software that simplifies basic shared concerns. That does not mean mandating solutions.
rqtwteye 3y ago
“Working with “product” companies without silos most of the time it’s design by committee,” With siloes you may end up with design by ego trip.
- danielovichdk 3y ago
  Egos only exists if the team let them exist. A strong leader will make sure this is not happening.
  - jpc0 3y ago
    A strong leader at the same time can cause it to happen...

acjohnson55 3y ago

As a coda to this, I think that we are entering a new world where services truly can have minimal operational overhead. At my last company, we were all in on serverless, with AWS tools like SQS coupling things together. It worked extremely well for keeping the architecture simple to operate and approachable for new people. But even better, I think we want logical services that interface with each other as though they were in a single process. We want the ability to write code as though it is monolithic (e.g. lexical scope and native-language APIs), while availing ourselves of the advantages of independently deployed services. I believe projects like Temporal point the way. I haven't had the opportunity to use it, but philosophically, I think it's the right direction.

throw04323 3y ago
I don't have experience with it myself, but Polylith architecture looks interesting. There you compose services based on shared components. You can start developing it as a monolith, and then extract components to separate services by just changing the interface between components. https://polylith.gitbook.io/polylith/ https://polylith.gitbook.io/polylith/
- acjohnson55 3y ago
  If I'm understanding Polylith, it seems like a way to do monoglot monorepo in a way that avoids explicitly using libraries. That looks pretty interesting and similar to what Artsy did with Ezel for front-end development. https://github.com/artsy/ezel https://github.com/artsy/ezel
jakjak123 3y ago
I have been working mostly with Kafka for five years, and I found SQS to be a little weird to work with. How do you keep an overview of all services that consume a queue? What if you want multiple services to read the same data in the same order? Granted I am very new to AWS.
- 8note 3y ago
  I think you're really looking to use kinesis rather than SQS. SNS+SQS make a pub/sub setup, but every queue subscribed to the event is fully separate, and you wouldn't expect to try to couple the different listening queues together. You could make all the queues into SQS fifo queues, and put them all on the same sorting group, but I think network time could still break the ordering on different subscribers? The simple way to use SQS is to have a queue for each message type you plan to consume, and treat it as a bucket of work to do, where you can control how fast you pull the work out

toss1 3y ago

Yes APIs are expensive and require significant overhead. But clean, maintained interfaces between separate domains are far LESS expensive than multi-domain/team mashups. Maintaining separate teams AND well- maintained interfaces surfaces and makes explicit the myriad inter-silo interactions. Instead of a quick backchannel conversation and a quick hack, it is an explicit conversation and an API adjustment. Yes, it's more expensive up front, but far cheaper and more powerful in the long run. So for high-speed prototyping, not so suitable- breaj the silos, do the backchannel hack & move on (this version will likely be binned or massively refactored anyway). But when the domain is more stable, modular with solid interfaces is the way to go.

jakjak123 3y ago

I'm not saying its a silver bullet, because sharing data via database makes change management way worse.

6gvONxR4sf7o 3y ago

> My overall reaction is that this is a great piece for products/teams that have reached significant scale, once the job to be done is too big and complex for one team to own, end-to-end, or there are truly reusable concerns that can be separated from the core product (e.g. auth, observability). I'd say it's critical in the small scale too. When you get down to a single person size, this is just what well factored code is: you write down little reasonably independent pieces that can be learned about and thought about without having to think about all the other things. After all, we've all had that experience when you go back and revisit your code after vacation or something, and it might as well have been written by someone else. I was on a tiny team not long ago where my teammates kept writing tightly coupled systems, then rewriting everything from scratch every time. It was hell. Our product moved slowly, broke constantly, and we couldn't build off of our prior work, and could barely even build off of each others' work, so velocity stayed constant (read: slow). (as a tangent, the communication patterns of remote work seem to make this more important) Siloing teams, siloing concerns, and writing modular code are all kind of the same thing, just at different scales.

steveBK123 3y ago

> Because interfacing via API is expensive. Writing APIs for others to use productively isn't easy and change management also adds a lot of overhead. An under appreciated point. Something that is affordable for giant monopoly profit driven FAANGs, but harder in smaller orgs. Even within my team, the "level of difficulty" of writing an API depends on its user base. If I am the primary user, it's easy. There's one really sharp kind on my team, if he is going to use it, then 2x the difficulty. If I need it to handle the median dev on my team, then 2x again. By the time you get to the below median dev, add another 2x. So even within my team, the intended user base can change the difficult from 1x->8x. What are some of the pain points - input validation that drives useful informative errors, flexibility on inputs like having sensible defaults to reduce the number of inputs users must pass, performance/scaling, and edge cases. >> In this sense, development teams weren’t so much empowered to deliver software faster; rather, they were over-encumbered with infrastructure tasks that were outside of their expertise. And agreed on this point. Amusingly I have had HN thread arguments just this week with DevOps advocates telling me that akshully their job isn't to empower devs, but some sort of "tail that wags that dog" interpretation around devops organizational / standardization / cost / etc.