6 ms·

I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Clearly I'm still failing to articulate this. Perhaps https://n

I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Clearly I'm still failing to articulate this. Perhaps https://news.ycombinator.com/item?id=30019146#30040616 https://news.ycombinator.com/item?id=30019146#30040616 will help triangulate on what I mean. > Can you clarify what you see as the alternative? Implementing everything from scratch seems absurd and so costly that there’s no point in considering this an actual option. Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. I have a day job in tech and have to deal with some cognitive dissonance every day between my day job and my open source research. The one thing I have found valuable to take to my scale-obsessed tech job is to constantly be suspicious of dependencies and constantly ask if the operational burdens justify some new feature. Just switching mindset that way from software as asset to software as liability has, I'd like to believe, helped my org's decision-making. > We have probably invested dev-millennia into managing copies. This is exactly what source control does. This is not a new area of investment. Merging is a giant pain in the ass and very possibly always will be. Accepting merge pain better come with some huge benefits. Not all copying is the same. We've learned to copy the letter 'e' so well in our writings that we don't even think about it. In this context, even if I made a tool to make copying easier and merges more reliable, that would just cause people to take on more dependencies which defeats the whole point of understanding dependencies. So tooling would be counter-productive in that direction. The direction I want to focus on is: how can we help people understand the software they've copied into their applications? _That_ is the place where I want tooling to focus. Copying is just an implementation detail, a first, imperfect, heuristic coping mechanism for going from the world we have today to the world I want to move to that has 1000x more forks and 1000x more eyeballs looking at source code. You can see some (very toy) efforts in this direction at https://github.com/akkartik/teliva https://github.com/akkartik/teliva > It’s untenable to have, e.g., everyone who works on Windows be an expert in every part of the code. It's frustrating to say one thing in response to counter-argument A and have someone then bring up counter-argument B because I didn't talk about it right there in the response to counter-argument A. I think this is what Plato was talking about when he ranted about the problems with the newfangled technology of writing: https://newlearningonline.com/literacies/chapter-1/socrates-on-the-forgetfulness-that-comes-with-writing https://newlearningonline.com/literacies/chapter-1/socrates-.... I'm not saying everyone needs to be an expert in everything. I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. My Linux laptop doesn't need to be optimized to the hilt the way Google's server farms do. Using the same scheduling algo or whatever in my laptop imposes real costs on my ability to understand my computer, without giving me the benefits Google gets from the algo.

dpark 5y ago
> I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity. > Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it. Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost. Reimplementing and forking are generally not value adds. Building software is today about creating/adding value. Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years. > I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.
- akkartik 5y ago
  > If I write some code, I’m realistically the expert. Ha, disagree. This is overly general and reductionist. See, I know the C strcpy() function is a bad idea. In this I know more than Ken Thompson or whatever Unix God did at the time he wrote it. I also know more than everybody who wrote (at the time they wrote) every other library that's on my computer that uses strcpy(). Why the fuck is this still in my system? Division of labor accumulates bad ideas along with good ones.

dpark 5y ago

> I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity. > Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it. Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost. Reimplementing and forking are generally not value adds. Building software is today about creating/adding value. Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years. > I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.

akkartik 5y ago
> If I write some code, I’m realistically the expert. Ha, disagree. This is overly general and reductionist. See, I know the C strcpy() function is a bad idea. In this I know more than Ken Thompson or whatever Unix God did at the time he wrote it. I also know more than everybody who wrote (at the time they wrote) every other library that's on my computer that uses strcpy(). Why the fuck is this still in my system? Division of labor accumulates bad ideas along with good ones.