8 ms·

This screams “toy projects” to me. Separation of concerns is a hard-won insight. We call code without this “spaghetti” because it eventually becomes nearly impo

by dpark 5y ago

This screams “toy projects” to me. Separation of concerns is a hard-won insight. We call code without this “spaghetti” because it eventually becomes nearly impossible trace or to reason about. Even if you have all your engineers working all over the whole codebase, the codebase itself still needs separation of concerns to simplify reasoning about behavior. This is considered a good engineering practice because the alternative is that you need to hold the whole system in your head to understand any behaviors. I’ve seen codebases like this. It’s not pretty. Use of abstractions to simplify cognitive overhead is not a weakness. The idea that projects should take source copies instead of library dependencies is just kind of nuts, at least for large libraries. Yes, the proliferation of tiny libraries is obnoxious and adds a ton of pointless overhead. But for large libraries, taking a source copy is absolutely crazy. Imagine taking a source clone of Chromium into your project because you need a rendering engine. Yes, you can probably cut a bunch of code out as a result, but you will also now need to deal with security concerns and incorporation of new features until the end of time. And this is now way more expensive and you can’t just do a straightforward merge because you changed the code. 5 line package? Sure, just copy the code in (assuming licensing is in the clear). 500 dev-year codebase? Cloning that code is by far not a win.

akkartik 5y ago
"Separation of concerns is a hard-won insight." Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people. It's certainly reasonable to consider my projects toy. I consider them research: * https://github.com/akkartik/mu https://github.com/akkartik/mu * https://github.com/akkartik/teliva https://github.com/akkartik/teliva "The idea that projects should take source copies instead of library dependencies is just kind of nuts..." The idea that projects should take copies seems about symmetric to me with taking pointers. Call by value vs call by reference. We just haven't had 50 years of tooling to support copies. Where would we be by now if we had devoted equal resources to both branches? "...at least for large libraries." How are these large libraries going for ya? Log4j wasn't exactly a shining example of the human race at its best. We're trying to run before we can walk.
- hinkley 5y ago
  I'm arguing for separating just concerns, without entangling them with considerations of people. Conway’s Law says you’re going to be disappointed a lot. These impedance mismatches lead to throughput problems and worse.
  - akkartik 5y ago
    Can you elaborate on this comment? Based on your comments elsewhere I think we're likely to be largely in agreement. I tend to have a love/hate relationship with Conway's Law, spending half my time thinking about how to design stuff assuming it, and half my time trying to design stuff (unsuccessfully) to make it go away.
    - hinkley 5y ago
      A lot of pain comes from taking a “concern” that doesn’t align with the org chart, making simple tasks take weeks to accomplish because I need to get my boss to tell your boss to do it anyway or we don’t make payroll. People tend to get precious about things they own. See also the legal department or the financial department thinking they are the company, as if they had any revenue without all of those “cost centers” or legal liabilities.
      - akkartik 5y ago
        Yeah, I agree with this. There's a larger context here of capitalism evolving firms because they're more efficient, even though they dilute the price-communication value of markets to some extent. As a consequence your boss and my boss are talking about something that might affect millions of people. That results in an effect akin to regulatory capture. So one way to view my case against division of labor is as a case for pushing intelligence to the edges, as close as possible to the owner of the computer running the software. On this larger level my suggestions are not actually that radical. Capitalism has had a long trend to push intelligence down to the edges. Kings used to use hierarchy to push decisions down. Blitzkrieg was largely about creating an organization that pushes decision-making down. And we've been getting better at it. I think we (i.e. society) can be even better at it by pushing decision-making out of the org and towards the people using it.
hinkley 5y ago
The thing I tell people when I’m critiquing their tests is that nobody gives a shit about your tests until they break, at which point they’re interacting with your code when they’re already in a bad mood. They will associate that feeling with you. So do you want to make their bad day worse by writing abstruse code? Or should they be simple? I know I’m “done” with a test when I know why it’s red from what’s in the console. I shouldn’t even have to look in the source code. Until then, my tests are a work in progress. We could apply this philosophy a lot more widely.
- akkartik 5y ago
  Totally agreed! Here's a prototype from a few years ago where I tried to make this easier: https://github.com/akkartik/mu1#readme https://github.com/akkartik/mu1#readme (read just the first few paragraphs) I still think the full answer lies in this direction.
knowledge-clay 5y ago
> But for large libraries, taking a source copy is absolutely crazy. The solution is, when possible, to use (and write) smaller libraries. Many libraries are far larger and more complex than they need to be.
- dpark 5y ago
  Depends on what it is you need. If you really have a small need, then taking a giant dependency is very possibly a bad idea. But I have seen the opposite pretty often. “I don’t need that library. I just need this one thing. Okay, this other thing, and this. And this and this and this.” And by the time they’re done they’ve reimplemented 75% of the library in a far worse way. The most egregious example I ever saw was reimplementation of xml parsing with C++ substrings. They “didn’t need” a real xml parser. In the end, they found endless bugs with their shitty parsing logic and it turns it it was responsible for something like 50% of the (very large scale) system resource use.
datavirtue 5y ago
Taking source copies is thought to be especially FOR large libraries. You take what you need, leaving the "large" behind...if you can. You probably can. Though, libraries are developed with the assumption that they are to be consumed as one massive binary. Making it more difficult to run off with just the part(s) you need.
ramesh31 5y ago
>The idea that projects should take source copies instead of library dependencies is just kind of nuts, at least for large libraries. There's a medium to be found between static linking and dynamic dependencies which I think is one of the most crucial tradeoffs to understand in software engineering. NPM in my opinion goes a bit too far on the modularization of things. Especially in large projects where you can have multiple private repo URLs and different auth tokens needed for each, it can become a nightmare to orchestrate for CI/CD. Something that takes an NPM approach, but with a more fleshed out stdlib to eliminate things like left-pad is probably the way to go.

by dpark 5y ago

akkartik 5y ago

"Separation of concerns is a hard-won insight." Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people. It's certainly reasonable to consider my projects toy. I consider them research: * https://github.com/akkartik/mu https://github.com/akkartik/mu * https://github.com/akkartik/teliva https://github.com/akkartik/teliva "The idea that projects should take source copies instead of library dependencies is just kind of nuts..." The idea that projects should take copies seems about symmetric to me with taking pointers. Call by value vs call by reference. We just haven't had 50 years of tooling to support copies. Where would we be by now if we had devoted equal resources to both branches? "...at least for large libraries." How are these large libraries going for ya? Log4j wasn't exactly a shining example of the human race at its best. We're trying to run before we can walk.

hinkley 5y ago
I'm arguing for separating just concerns, without entangling them with considerations of people. Conway’s Law says you’re going to be disappointed a lot. These impedance mismatches lead to throughput problems and worse.
- akkartik 5y ago
  Can you elaborate on this comment? Based on your comments elsewhere I think we're likely to be largely in agreement. I tend to have a love/hate relationship with Conway's Law, spending half my time thinking about how to design stuff assuming it, and half my time trying to design stuff (unsuccessfully) to make it go away.
  - hinkley 5y ago
    A lot of pain comes from taking a “concern” that doesn’t align with the org chart, making simple tasks take weeks to accomplish because I need to get my boss to tell your boss to do it anyway or we don’t make payroll. People tend to get precious about things they own. See also the legal department or the financial department thinking they are the company, as if they had any revenue without all of those “cost centers” or legal liabilities.
    - akkartik 5y ago
      Yeah, I agree with this. There's a larger context here of capitalism evolving firms because they're more efficient, even though they dilute the price-communication value of markets to some extent. As a consequence your boss and my boss are talking about something that might affect millions of people. That results in an effect akin to regulatory capture. So one way to view my case against division of labor is as a case for pushing intelligence to the edges, as close as possible to the owner of the computer running the software. On this larger level my suggestions are not actually that radical. Capitalism has had a long trend to push intelligence down to the edges. Kings used to use hierarchy to push decisions down. Blitzkrieg was largely about creating an organization that pushes decision-making down. And we've been getting better at it. I think we (i.e. society) can be even better at it by pushing decision-making out of the org and towards the people using it.

hinkley 5y ago

The thing I tell people when I’m critiquing their tests is that nobody gives a shit about your tests until they break, at which point they’re interacting with your code when they’re already in a bad mood. They will associate that feeling with you. So do you want to make their bad day worse by writing abstruse code? Or should they be simple? I know I’m “done” with a test when I know why it’s red from what’s in the console. I shouldn’t even have to look in the source code. Until then, my tests are a work in progress. We could apply this philosophy a lot more widely.

akkartik 5y ago
Totally agreed! Here's a prototype from a few years ago where I tried to make this easier: https://github.com/akkartik/mu1#readme https://github.com/akkartik/mu1#readme (read just the first few paragraphs) I still think the full answer lies in this direction.

knowledge-clay 5y ago

> But for large libraries, taking a source copy is absolutely crazy. The solution is, when possible, to use (and write) smaller libraries. Many libraries are far larger and more complex than they need to be.

dpark 5y ago
Depends on what it is you need. If you really have a small need, then taking a giant dependency is very possibly a bad idea. But I have seen the opposite pretty often. “I don’t need that library. I just need this one thing. Okay, this other thing, and this. And this and this and this.” And by the time they’re done they’ve reimplemented 75% of the library in a far worse way. The most egregious example I ever saw was reimplementation of xml parsing with C++ substrings. They “didn’t need” a real xml parser. In the end, they found endless bugs with their shitty parsing logic and it turns it it was responsible for something like 50% of the (very large scale) system resource use.

datavirtue 5y ago

Taking source copies is thought to be especially FOR large libraries. You take what you need, leaving the "large" behind...if you can. You probably can. Though, libraries are developed with the assumption that they are to be consumed as one massive binary. Making it more difficult to run off with just the part(s) you need.

ramesh31 5y ago

>The idea that projects should take source copies instead of library dependencies is just kind of nuts, at least for large libraries. There's a medium to be found between static linking and dynamic dependencies which I think is one of the most crucial tradeoffs to understand in software engineering. NPM in my opinion goes a bit too far on the modularization of things. Especially in large projects where you can have multiple private repo URLs and different auth tokens needed for each, it can become a nightmare to orchestrate for CI/CD. Something that takes an NPM approach, but with a more fleshed out stdlib to eliminate things like left-pad is probably the way to go.