6 ms·

I'm convinced that some people don't know any other way to break down a system into smaller parts. To these people, if it's not exposed as a API call it's just

I'm convinced that some people don't know any other way to break down a system into smaller parts. To these people, if it's not exposed as a API call it's just some opaque blob of code that cannot be understood or reused.

dkarl 1y ago
That's what I've observed empirically over my last half-dozen jobs. Many developers treat decomposition and contract design between services seriously, and work until they get it right. I've seen very few developers who put the same effort into decomposing the modules of a monolith and designing the interfaces between them, and never enough in the same team to stop a monolith from turning into a highly coupled amorphous blob. My grug brain conclusion: Grug see good microservice in many valley. Grug see grug tribe carry good microservice home and roast on spit. Grug taste good microservice, many time. Shaman tell of good monolith in vision. Grug also dream of good monolith. Maybe grug taste good monolith after die. Grug go hunt good microservice now.
- stavros 1y ago
  We've solved this problem by making the modules in the monolith only able to call each other from well-defined APIs, otherwise CI fails.
  - PaulHoule 1y ago
    In the Java world both Spring and Guice are meant to do this, and if you have an ISomething you've got the possibility of making an ILocalSomething and a IDistributedSomething and swap one for the other.
    - pbh101 1y ago
      This is generally a bad idea imo. You fundamentally will have a hard time if your api is opaquely network-dependent or not. I suppose, you’ll be ok if you assume there is a network call, but that means your client will need to pay that cost every time, even if using the ILocal.
      - PaulHoule 1y ago
        It depends on what the API is. For instance you might use something like JDBC or SQLAlchemy to access either a sqlite database or a postgres database. But you are right that the remote procedure call is a fraught concept for more reasons than one. On one hand there is the fundamental difference between a local procedure call that takes a few ns and a remote call which might take 1,000,000 longer. There's also the fact that most RPC mechanisms that call themselves RPC mechanisms are terribly complicated, like DCOM or the old Sun RPC. In some sense RPC became mainstream once people started pretending it was REST. People say it is not RPC but often you have a function in your front end Javascript like fetch_data(75) and that becomes GET /data/75 and your back end JAXB looks like @GET @Path("/{id}") public List<Data> fetchData(@PathParam("id") int id) { ... }
  - williamdclt 1y ago
    I honestly think it's the only way outside of one-person projects (and even then...), you need _some_ design pressure.
- pbh101 1y ago
  Maybe the friction imposed to mess up the well-factored microservice arch is sufficiently marginally higher than in the monolith that the perception of value in the factoring is higher, whereas the implicit expectation of factoring the monolith is that you’ll look away for five seconds and someone will ruin it.
- vharish 1y ago
  I think monoliths are not such a good idea anymore. Particularly with the direction development is going w.r.t the usage of LLMs, I think it's best to break things down. Ofcourse, it shouldn't be overdone.
  - discreteevent 1y ago
    > grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too > I think it's best to break things down Factoring system = break things down.
- manmal 1y ago
  Put the modules in different git repos and interfaces will get super clean eventually.
demosthanos 1y ago
> To these people, if it's not exposed as a API call it's just some opaque blob of code that cannot be understood or reused. I think this is correct as an explanation for the phenomenon, but it's not just a false perception on their part: for a lot of organizations it is actually true that the only way to preserve boundaries between systems over the course of years is to stick the network in between. Without a network layer enforcing module boundaries code does, in fact, tend to morph into a big ball of mud. I blame a few things for this: 1. Developers almost universally lack discipline. 2. Most programming languages are not designed to sufficiently account for #1. It's not a coincidence that microservices became popular shortly after Node.js and Python became the dominant web backend languages. A strong static type system is generally necessary (but not sufficient) to create clear boundaries between modules, and both Python and JavaScript have historically been even worse than usual for dynamic languages when it comes to having a strong modularity story. And while Python and JS have it worse than most, even most of our popular static languages are pretty lousy at giving developers the tools needed to clearly delineate module boundaries. Rust has a pretty decent starting point but it too could stand to be improved.
- giantrobot 1y ago
  3. Company structure poorly supports cross-team or department code ownership Many companies don't seem to do a good job coordinating between teams. Different teams have different incentives and priorities. If group A needs fixes/work from group B and B has been given some other priority, group A is stuck. By putting a network between modules different groups can limit blast damage from other teams' modules and more clearly show ownership when things go wrong. If group A's project fails because of B's module it still looks like A's code has the problem. Upper management rarely cares about nuance. They want to assign blame, especially if it's in another team or department. So teams under them always want clear boundaries of responsibility so they don't get thrown under the bus. The root cause of a lot of software problems is the organization that produces it more than any individual or even team working on it.
  - patrickmay 1y ago
    [O]rganizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations. — Melvin E. Conway, How Do Committees Invent?
cjfd 1y ago
Well, if people are really that stupid maybe they should just not be developers.
isoprophlex 1y ago
I swear I'm not making this up; a guy at my current client needed to join two CSV files. A one off thing for some business request. He wrote a REST api in Java, where you get the merged csv after POSTing your inputs. I must scream but I'm in a vacuum. Everyone is fine with this. (Also it takes a few seconds to process a 500 line test file and runs for ten minutes on the real 20k line input.)
- pbohun 1y ago
  Was it joining on some columns or just concatenating the files? I'm going to laugh pretty hard if it could just be done with: cat file1.csv file2.csv > combined.csv
  - Xenoamorphous 1y ago
    You need to account for the headers, which many (most?) csv files I've encountered have. So I guess something like this to skip the headers in the second file (this also assumes that headers don't have line breaks): cp file1.csv combined.csv && tail -n+2 file2.csv >> combined.csv
  - xnx 1y ago
    There are also a lot of command line options for joining by column like csvkit
- withinboredom 1y ago
  I mean, it would be faster to just import them into an in-memory sqlite database, run a `union all` query and then dump it to a csv... That's still probably the wrong way to do it, but 10 minutes for a 20k line file? That seems like poor engineering in the most basic sense.
  - strken 1y ago
    I'd probably think of xsv, go to its github repo, remember it's unmaintained and got replaced by qsv, and then use qsv.
  - 1y ago
9rx 1y ago
To be fair, microservices is about breaking people down into smaller parts, with the idea of mirroring services found in the macro economy, but within the microcosm of a single business. In other words, a business is broken down into different teams that operate in isolation from each other, just as individual businesses do at the macro scale. Any technical outcomes from that are merely a result of Conway's Law.

dkarl 1y ago

That's what I've observed empirically over my last half-dozen jobs. Many developers treat decomposition and contract design between services seriously, and work until they get it right. I've seen very few developers who put the same effort into decomposing the modules of a monolith and designing the interfaces between them, and never enough in the same team to stop a monolith from turning into a highly coupled amorphous blob. My grug brain conclusion: Grug see good microservice in many valley. Grug see grug tribe carry good microservice home and roast on spit. Grug taste good microservice, many time. Shaman tell of good monolith in vision. Grug also dream of good monolith. Maybe grug taste good monolith after die. Grug go hunt good microservice now.

stavros 1y ago
We've solved this problem by making the modules in the monolith only able to call each other from well-defined APIs, otherwise CI fails.
- PaulHoule 1y ago
  In the Java world both Spring and Guice are meant to do this, and if you have an ISomething you've got the possibility of making an ILocalSomething and a IDistributedSomething and swap one for the other.
  - pbh101 1y ago
    This is generally a bad idea imo. You fundamentally will have a hard time if your api is opaquely network-dependent or not. I suppose, you’ll be ok if you assume there is a network call, but that means your client will need to pay that cost every time, even if using the ILocal.
    - PaulHoule 1y ago
      It depends on what the API is. For instance you might use something like JDBC or SQLAlchemy to access either a sqlite database or a postgres database. But you are right that the remote procedure call is a fraught concept for more reasons than one. On one hand there is the fundamental difference between a local procedure call that takes a few ns and a remote call which might take 1,000,000 longer. There's also the fact that most RPC mechanisms that call themselves RPC mechanisms are terribly complicated, like DCOM or the old Sun RPC. In some sense RPC became mainstream once people started pretending it was REST. People say it is not RPC but often you have a function in your front end Javascript like fetch_data(75) and that becomes GET /data/75 and your back end JAXB looks like @GET @Path("/{id}") public List<Data> fetchData(@PathParam("id") int id) { ... }
- williamdclt 1y ago
  I honestly think it's the only way outside of one-person projects (and even then...), you need _some_ design pressure.
pbh101 1y ago
Maybe the friction imposed to mess up the well-factored microservice arch is sufficiently marginally higher than in the monolith that the perception of value in the factoring is higher, whereas the implicit expectation of factoring the monolith is that you’ll look away for five seconds and someone will ruin it.
vharish 1y ago
I think monoliths are not such a good idea anymore. Particularly with the direction development is going w.r.t the usage of LLMs, I think it's best to break things down. Ofcourse, it shouldn't be overdone.
- discreteevent 1y ago
  > grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too > I think it's best to break things down Factoring system = break things down.
manmal 1y ago
Put the modules in different git repos and interfaces will get super clean eventually.

demosthanos 1y ago

> To these people, if it's not exposed as a API call it's just some opaque blob of code that cannot be understood or reused. I think this is correct as an explanation for the phenomenon, but it's not just a false perception on their part: for a lot of organizations it is actually true that the only way to preserve boundaries between systems over the course of years is to stick the network in between. Without a network layer enforcing module boundaries code does, in fact, tend to morph into a big ball of mud. I blame a few things for this: 1. Developers almost universally lack discipline. 2. Most programming languages are not designed to sufficiently account for #1. It's not a coincidence that microservices became popular shortly after Node.js and Python became the dominant web backend languages. A strong static type system is generally necessary (but not sufficient) to create clear boundaries between modules, and both Python and JavaScript have historically been even worse than usual for dynamic languages when it comes to having a strong modularity story. And while Python and JS have it worse than most, even most of our popular static languages are pretty lousy at giving developers the tools needed to clearly delineate module boundaries. Rust has a pretty decent starting point but it too could stand to be improved.

giantrobot 1y ago
3. Company structure poorly supports cross-team or department code ownership Many companies don't seem to do a good job coordinating between teams. Different teams have different incentives and priorities. If group A needs fixes/work from group B and B has been given some other priority, group A is stuck. By putting a network between modules different groups can limit blast damage from other teams' modules and more clearly show ownership when things go wrong. If group A's project fails because of B's module it still looks like A's code has the problem. Upper management rarely cares about nuance. They want to assign blame, especially if it's in another team or department. So teams under them always want clear boundaries of responsibility so they don't get thrown under the bus. The root cause of a lot of software problems is the organization that produces it more than any individual or even team working on it.
- patrickmay 1y ago
  [O]rganizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations. — Melvin E. Conway, How Do Committees Invent?

cjfd 1y ago

Well, if people are really that stupid maybe they should just not be developers.

isoprophlex 1y ago

I swear I'm not making this up; a guy at my current client needed to join two CSV files. A one off thing for some business request. He wrote a REST api in Java, where you get the merged csv after POSTing your inputs. I must scream but I'm in a vacuum. Everyone is fine with this. (Also it takes a few seconds to process a 500 line test file and runs for ten minutes on the real 20k line input.)

pbohun 1y ago
Was it joining on some columns or just concatenating the files? I'm going to laugh pretty hard if it could just be done with: cat file1.csv file2.csv > combined.csv
- Xenoamorphous 1y ago
  You need to account for the headers, which many (most?) csv files I've encountered have. So I guess something like this to skip the headers in the second file (this also assumes that headers don't have line breaks): cp file1.csv combined.csv && tail -n+2 file2.csv >> combined.csv
- xnx 1y ago
  There are also a lot of command line options for joining by column like csvkit
withinboredom 1y ago
I mean, it would be faster to just import them into an in-memory sqlite database, run a `union all` query and then dump it to a csv... That's still probably the wrong way to do it, but 10 minutes for a 20k line file? That seems like poor engineering in the most basic sense.
- strken 1y ago
  I'd probably think of xsv, go to its github repo, remember it's unmaintained and got replaced by qsv, and then use qsv.
- 1y ago

9rx 1y ago

To be fair, microservices is about breaking people down into smaller parts, with the idea of mirroring services found in the macro economy, but within the microcosm of a single business. In other words, a business is broken down into different teams that operate in isolation from each other, just as individual businesses do at the macro scale. Any technical outcomes from that are merely a result of Conway's Law.