5 ms·

> Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people. The separation of people is largely an artifact

by dpark 5y ago

> Absolutely. I'm arguing for separating just concerns, without entangling them with considerations of people. The separation of people is largely an artifact of scale. At some point it doesn’t make sense to have everyone touching the whole code base. It’s untenable to have, e.g., everyone who works on Windows be an expert in every part of the code. It’s also generally unnecessary because in a large code base features are often nearly completely contained to one area of code. Some teams are really dysfunctional and won’t let people ever do work outside “their” area. This is absolutely an antipattern. > The idea that projects should take copies seems about symmetric to me with taking pointers. Call by value vs call by reference. We just haven't had 50 years of tooling to support copies. Where would we be by now if we had devoted equal resources to both branches? We have probably invested dev-millennia into managing copies. This is exactly what source control does. This is not a new area of investment. Merging is a giant pain in the ass and very possibly always will be. Accepting merge pain better come with some huge benefits. > How are these large libraries going for ya? Log4j wasn't exactly a shining example of the human race at its best. We're trying to run before we can walk. Can you clarify what you see as the alternative? Implementing everything from scratch seems absurd and so costly that there’s no point in considering this an actual option. Plus there’s no particular reason to believe that you wouldn’t care your own security bugs. Taking a source drop wouldn’t protect you if you happened to take one with the bug. And you’re forever bound to that version unless you want to pay a very high cost to reintegrate a new one. What viable option do you see that I don’t? (Picking on log4j right now is easy but are you seriously implementing your own SSH, for example?) For the record, these large libraries are going great. I’ve built numerous products that simply would not have ever been created without, say, SQL. The cost would simply have been too high. I’ve built others with libraries that the project outgrew, but it was still a big win to start with the library.

akkartik 5y ago
I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Clearly I'm still failing to articulate this. Perhaps https://news.ycombinator.com/item?id=30019146#30040616 https://news.ycombinator.com/item?id=30019146#30040616 will help triangulate on what I mean. > Can you clarify what you see as the alternative? Implementing everything from scratch seems absurd and so costly that there’s no point in considering this an actual option. Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. I have a day job in tech and have to deal with some cognitive dissonance every day between my day job and my open source research. The one thing I have found valuable to take to my scale-obsessed tech job is to constantly be suspicious of dependencies and constantly ask if the operational burdens justify some new feature. Just switching mindset that way from software as asset to software as liability has, I'd like to believe, helped my org's decision-making. > We have probably invested dev-millennia into managing copies. This is exactly what source control does. This is not a new area of investment. Merging is a giant pain in the ass and very possibly always will be. Accepting merge pain better come with some huge benefits. Not all copying is the same. We've learned to copy the letter 'e' so well in our writings that we don't even think about it. In this context, even if I made a tool to make copying easier and merges more reliable, that would just cause people to take on more dependencies which defeats the whole point of understanding dependencies. So tooling would be counter-productive in that direction. The direction I want to focus on is: how can we help people understand the software they've copied into their applications? _That_ is the place where I want tooling to focus. Copying is just an implementation detail, a first, imperfect, heuristic coping mechanism for going from the world we have today to the world I want to move to that has 1000x more forks and 1000x more eyeballs looking at source code. You can see some (very toy) efforts in this direction at https://github.com/akkartik/teliva https://github.com/akkartik/teliva > It’s untenable to have, e.g., everyone who works on Windows be an expert in every part of the code. It's frustrating to say one thing in response to counter-argument A and have someone then bring up counter-argument B because I didn't talk about it right there in the response to counter-argument A. I think this is what Plato was talking about when he ranted about the problems with the newfangled technology of writing: https://newlearningonline.com/literacies/chapter-1/socrates-on-the-forgetfulness-that-comes-with-writing https://newlearningonline.com/literacies/chapter-1/socrates-.... I'm not saying everyone needs to be an expert in everything. I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. My Linux laptop doesn't need to be optimized to the hilt the way Google's server farms do. Using the same scheduling algo or whatever in my laptop imposes real costs on my ability to understand my computer, without giving me the benefits Google gets from the algo.
- dpark 5y ago
  > I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity. > Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it. Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost. Reimplementing and forking are generally not value adds. Building software is today about creating/adding value. Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years. > I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.
  - akkartik 5y ago
    > If I write some code, I’m realistically the expert. Ha, disagree. This is overly general and reductionist. See, I know the C strcpy() function is a bad idea. In this I know more than Ken Thompson or whatever Unix God did at the time he wrote it. I also know more than everybody who wrote (at the time they wrote) every other library that's on my computer that uses strcpy(). Why the fuck is this still in my system? Division of labor accumulates bad ideas along with good ones.
tomnipotent 5y ago
> It’s untenable to have Absolutely this. There's a wide ranging of programming activities that are domain-agnostic, but most of what programmers do requires deeper knowledge in a specific domain to be useful.
akkartik 5y ago
> For the record, these large libraries are going great. I’ve built numerous products that simply would not have ever been created without, say, SQL. The cost would simply have been too high. I’ve built others with libraries that the project outgrew, but it was still a big win to start with the library. We definitely have great libraries out there. But they're the exception rather than the rule. I consider Redis and Sqlite in particular to be quite exemplary. SQL I'm more ambivalent about. Or rather, SQL the abstract ideal is great, but I'm more ambivalent about implementations like MySQL or PostgreSQL. As I've hopefully made clear already, I consider just raw speedup for an initial implementation of an app as a poor and incomplete metric for the quality of a library. At best it's a good quality for something you can use in prototypes, not the final version you build after you throw the prototype away. (Because we all agree you should throw the prototype away, right? :) For non-prototypes you also have to take into account the total cost of ownership. I think the ever expanding wavefront of "specialization" in roles like DBAs is a sign that MySQL and PostgreSQL are a bad trade for society. People being deployed to fill in for all the numerous gotchas arising from a crappy interface. MySQL and PostgreSQL make things more efficient in the short term, but they make us less resilient in the long term.
- dpark 5y ago
  > I think the ever expanding wavefront of "specialization" in roles like DBAs is a sign that MySQL and PostgreSQL are a bad trade for society. That’s a sign of a maturing field. Does the proliferation of medical specialties indicate a problem? I would argue that it simply arises from the amount of specialized knowledge that one can possess and utilize efficiently. The existence of SQL implementations allows for faster and more maintainable implementations. The set of engineers who could do what SQL allows without it is quite small. If SQL disappeared tomorrow, a thousand equivalent implementations would appear tomorrow because it delivers real value. Hand rolling queries in a general purpose programming language is a very poorly trade off. I’ve seen this sort of stuff before. It’s always a bug farm and every other engineer asks “why the hell didn’t you use SQL”? Give it enough time and you’ll see “Redis expert” show up as a specialty.
  - akkartik 5y ago

by dpark 5y ago

akkartik 5y ago

I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Clearly I'm still failing to articulate this. Perhaps https://news.ycombinator.com/item?id=30019146#30040616 https://news.ycombinator.com/item?id=30019146#30040616 will help triangulate on what I mean. > Can you clarify what you see as the alternative? Implementing everything from scratch seems absurd and so costly that there’s no point in considering this an actual option. Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. I have a day job in tech and have to deal with some cognitive dissonance every day between my day job and my open source research. The one thing I have found valuable to take to my scale-obsessed tech job is to constantly be suspicious of dependencies and constantly ask if the operational burdens justify some new feature. Just switching mindset that way from software as asset to software as liability has, I'd like to believe, helped my org's decision-making. > We have probably invested dev-millennia into managing copies. This is exactly what source control does. This is not a new area of investment. Merging is a giant pain in the ass and very possibly always will be. Accepting merge pain better come with some huge benefits. Not all copying is the same. We've learned to copy the letter 'e' so well in our writings that we don't even think about it. In this context, even if I made a tool to make copying easier and merges more reliable, that would just cause people to take on more dependencies which defeats the whole point of understanding dependencies. So tooling would be counter-productive in that direction. The direction I want to focus on is: how can we help people understand the software they've copied into their applications? _That_ is the place where I want tooling to focus. Copying is just an implementation detail, a first, imperfect, heuristic coping mechanism for going from the world we have today to the world I want to move to that has 1000x more forks and 1000x more eyeballs looking at source code. You can see some (very toy) efforts in this direction at https://github.com/akkartik/teliva https://github.com/akkartik/teliva > It’s untenable to have, e.g., everyone who works on Windows be an expert in every part of the code. It's frustrating to say one thing in response to counter-argument A and have someone then bring up counter-argument B because I didn't talk about it right there in the response to counter-argument A. I think this is what Plato was talking about when he ranted about the problems with the newfangled technology of writing: https://newlearningonline.com/literacies/chapter-1/socrates-on-the-forgetfulness-that-comes-with-writing https://newlearningonline.com/literacies/chapter-1/socrates-.... I'm not saying everyone needs to be an expert in everything. I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. My Linux laptop doesn't need to be optimized to the hilt the way Google's server farms do. Using the same scheduling algo or whatever in my laptop imposes real costs on my ability to understand my computer, without giving me the benefits Google gets from the algo.

dpark 5y ago
> I question the need for scale in 90% of the places where the tech industry has cargo-culted it. Certainly our industry does a lot of premature scaling. Sometimes this is the best bet, because it can be so difficult to scale later. But certainly sometimes it’s a huge waste and creates unnecessary complexity. > Not using, reimplementing and copying are the closest thing to solutions I have right now. You're right that they're not applicable to most people in their current context. Yeah. This again makes sense if you don’t really need the library. If you’re only using some tiny piece of it. Otherwise this looks a lot like Not Invented Here. Reimplementing huge pieces of functionality is rarely a win. Copying can be better but is again rarely a win. It works for cases like Chromium where a massive team was created to maintain the fork, but in general firing code comes with huge cost. Reimplementing and forking are generally not value adds. Building software is today about creating/adding value. Certainly it pays to be suspicious of new dependencies and ask if they are necessary. I’ve seen unnecessary dependencies cause significant pain. I have also seen variants of not invented here waste years. > I'm saying software should reduce the pressure on people to be experts so that we can late-bind experts to domains. Not every software sub-system should need expertise at the scale at which it is used in every possible context. I don’t understand this criticism. If I write some code, I’m realistically the expert. The person who code reviewed for me is likely the next most knowledgeable. It’s not about wanting to silo expertise. It’s about the reality that certain people will understand certain subsystems better because they’ve worked there. If I need to get something done, I’m going to first try to get the experts to do it, not because someone who typically works in another area can’t, but because it will probably take the “outsider” twice as long. It’s not a statement of what’s right or wrong, just what’s efficient. Certainly I’d knowledge becomes too silo’d, it’s a massive risk. But some siloing can be significantly more efficient, at least on large or complex systems.
- akkartik 5y ago
  > If I write some code, I’m realistically the expert. Ha, disagree. This is overly general and reductionist. See, I know the C strcpy() function is a bad idea. In this I know more than Ken Thompson or whatever Unix God did at the time he wrote it. I also know more than everybody who wrote (at the time they wrote) every other library that's on my computer that uses strcpy(). Why the fuck is this still in my system? Division of labor accumulates bad ideas along with good ones.

tomnipotent 5y ago

> It’s untenable to have Absolutely this. There's a wide ranging of programming activities that are domain-agnostic, but most of what programmers do requires deeper knowledge in a specific domain to be useful.

akkartik 5y ago

> For the record, these large libraries are going great. I’ve built numerous products that simply would not have ever been created without, say, SQL. The cost would simply have been too high. I’ve built others with libraries that the project outgrew, but it was still a big win to start with the library. We definitely have great libraries out there. But they're the exception rather than the rule. I consider Redis and Sqlite in particular to be quite exemplary. SQL I'm more ambivalent about. Or rather, SQL the abstract ideal is great, but I'm more ambivalent about implementations like MySQL or PostgreSQL. As I've hopefully made clear already, I consider just raw speedup for an initial implementation of an app as a poor and incomplete metric for the quality of a library. At best it's a good quality for something you can use in prototypes, not the final version you build after you throw the prototype away. (Because we all agree you should throw the prototype away, right? :) For non-prototypes you also have to take into account the total cost of ownership. I think the ever expanding wavefront of "specialization" in roles like DBAs is a sign that MySQL and PostgreSQL are a bad trade for society. People being deployed to fill in for all the numerous gotchas arising from a crappy interface. MySQL and PostgreSQL make things more efficient in the short term, but they make us less resilient in the long term.

dpark 5y ago
> I think the ever expanding wavefront of "specialization" in roles like DBAs is a sign that MySQL and PostgreSQL are a bad trade for society. That’s a sign of a maturing field. Does the proliferation of medical specialties indicate a problem? I would argue that it simply arises from the amount of specialized knowledge that one can possess and utilize efficiently. The existence of SQL implementations allows for faster and more maintainable implementations. The set of engineers who could do what SQL allows without it is quite small. If SQL disappeared tomorrow, a thousand equivalent implementations would appear tomorrow because it delivers real value. Hand rolling queries in a general purpose programming language is a very poorly trade off. I’ve seen this sort of stuff before. It’s always a bug farm and every other engineer asks “why the hell didn’t you use SQL”? Give it enough time and you’ll see “Redis expert” show up as a specialty.
- akkartik 5y ago