11 ms·

Hi Alan, In "The Power of the Context" (2004) you wrote: ...In programming there is a wide-spread 1st order theory that one shouldn’t build one’s own tool

by di 10y ago

Hi Alan, In "The Power of the Context" (2004) you wrote: ...In programming there is a wide-spread 1st order theory that one shouldn’t build one’s own tools, languages, and especially operating systems. This is true—an incredible amount of time and energy has gone down these ratholes. On the 2nd hand, if you can build your own tools, languages and operating systems, then you absolutely should because the leverage that can be obtained (and often the time not wasted in trying to fix other people’s not quite right tools) can be incredible. I love this quote because it justifies a DIY attitude of experimentation and reverse engineering, etc., that generally I think we could use more of. However, more often than not, I find the sentiment paralyzing. There's so much that one could probably learn to build themselves, but as things become more and more complex, one has to be able to make a rational tradeoff between spending the time and energy in the rathole, or not. I can't spend all day rebuilding everything I can simply because I can. My question is: how does one decide when to DIY, and when to use what's already been built?

deleted 10y ago
[deleted]
jjnoakes 10y ago
I tend to do both in parallel and the first one done wins. That is, if I have a problem that requires a library or program, and I don't know of one, I semi-simultaneously try to find a library/program that exists out there (scanning forums, googling around, reading stack overflow, searching github, going to language repositories for the languages I care about, etc) and also in parallel try to formulate in my mind what the ideal solution would look like for my particular problem. As time goes by, I get closer to finding a good enough library/program and closer to being able to picture what a solution would look like if I wrote it. At some point I either find what I need (it's good enough or it's perfect) or I get to the point where I understand enough about the solution I'm envisioning that I write it up myself.
- igorgue 10y ago
  If you don't have the time or energy for such projects then you CAN'T do them. The answer is there.
- quantumhobbit 10y ago
  Yes. If it takes me longer to figure out how to use your library or framework than to just implement the functionality myself, there is no point in using the library. Some people claim you should still use the 3rd party solution because of the cost of supporting the extra code you have written. But bugs can exist in both my code and the 3rd party code and I understand how to fix bugs in my code much more easily.
  - MaulingMonkey 10y ago
    Other points of consideration: My coworkers might not already know some library, but they definitely won't know my library. My coworker's code is just about as "3rd party" as any library - as is code I wrote as little as 6 months ago. Also my job owns that code, so rolling my own means I need to write another clone every time I switch employers - assuming there's no patents or overly litigious lawyers to worry about. But you're of course correct that there is, eventually, a point where it no longer makes sense to use the library. > Some people claim you should still use the 3rd party solution because of the cost of supporting the extra code you have written. But bugs can exist in both my code and the 3rd party code and I understand how to fix bugs in my code much more easily. The problem is I got so tired of fixing bugs in coworker / former coworker code that I eventually replaced their stuff with off the shelf libraries, just so the bugs would go away. And in practice, they did go away. And it caught several usage bugs because the library had better sanity checks. And to this day, those former coworkers would use the same justifications, in total earnestness. I've never said "gee, I wish we used some custom bespoke implementation for this". I'll wish a good implementation had been made commonly available as a reusable library, perhaps. But bespoke just means fewer eyes and fewer bugfixes.
    - jjnoakes 10y ago
      It's all trade-offs. If there happens to be a well-tested third party library that does what you want, doesn't increase your attack surface more than necessary, is supported by the community, is easy to get up and running with, and has a compatible license with what you are using it in, then by all means go for it. For me and my work, I tend to find that something from the above list is lacking enough that it makes more sense to write it in-house. Not always, and not as a rule, but it works out that way quite a bit. I would also argue that if coworkers couldn't write a library without a prohibitive number of bugs, then they won't be able to write application or glue code either. So maybe your issue wasn't in-house vs third party libraries, but the quality control and/or developer aptitude around you.
stcredzero 10y ago
Isn't the answer contained in the quote? Do a cost/benefit analysis of the "amount of time and energy" that would go "down these ratholes" versus the "the time not wasted in trying to fix other people’s not quite right tools."
- austinjp 10y ago
  But how can you assess this until you have gone down those rat holes?
  - stcredzero 10y ago
    For many particular examples, there have already been enough rathole spelunkers to provide useful data. Maybe start looking in the places where there isn't already useful data?
    - DasIch 10y ago
      Any area in which enough such spelunkers are found is unlikely to be significantly improved by adding your own effort.
      - TheAdventMaster 10y ago
        Agreed. It's often much, much harder to articulate why an idea is bad or a rat hole. You just move on. I've come up with explanation by analogy. You can demonstrate quite easily in mathematics how you can create a system of notation or a function that quickly becomes impossible to compute. A number that is too large, or an algorithm that would take infinity amount of time and resources to solve... It seems to be in nature that bad ideas are easy. Good ideas are harder, because they tend to be refinements of what already exists and what is already good. So pursue good ideas. Pursue the thing that you have thought about and decided has the best balance between values and highest chance to succeed. Sometimes it's just a strong gut feeling. Go for it, but set limits, because you don't want to fall prey to a gut feeling originating from strong intuition but an equally strong lack of fundamental understanding.
alankay1 10y ago
This is a tough question. (And always has been in a sense, because every era has had projects where the tool building has sunk the project into a black hole.) It really helped at Parc to work with real geniuses like Chuck Thacker and Dan Ingalls (and quite a few more). There is a very thin boundary between making the 2nd order work vs getting wiped out by the effort. Another perspective on this is to think about "not getting caught by dependencies" -- what if there were really good independent module systems -- perhaps aided by hardware -- that allowed both worlds to work together (so one doesn't get buried under "useful patches", etc.) One of my favorite things to watch at Parc was how well Dan Ingalls was able to bootstrap a new system out of an old one by really using what objects are good for, and especially where the new system was even much better at facilitating the next bootstrap. I'm not a big Unix fan -- it was too late on the scene for the level of ideas that it had -- but if you take the cultural history it came from, there were several things they tried to do that were admirable -- including really having a tiny kernel and using Unix processes for all systems building (this was a very useful version of "OOP" -- you just couldn't have small objects because of the way processes were implemented). It was quite sad to see how this pretty nice mix and match approach gradually decayed into huge loads and dependencies. Part of this was that the rather good idea of parsing non-command messages in each process -- we used this in the first Smalltalk at Parc -- became much too ad hoc because there was not a strong attempt to intertwine a real language around the message structures (this very same thing happened with http -- just think of what this could have been if anyone had been noticing ...)
- pault 10y ago
  > I'm not a big Unix fan What is your preferred technology stack?
  - astrodust 10y ago
    What's a good non-UNIX open-source operating system that's useful for day-to-day work, or at least academically significant enough that it's worth diving in to?
    - 10y ago
shostack 10y ago
Specifically with regards to languages and OS's, I wonder how much that cost/benefit equation shifts as things have become so much more complex, and as we continue to pile on abstraction layer after abstraction layer.
- astrobe_ 10y ago
  I think the problem is not complexity but size. Most of the source for the Linux kernel is in the drivers, for instance. As for languages, most of the weight is in the libraries.
50CNT 10y ago
I think you have to weigh your qualms against the difficulty of implementation. They're both spectra, one from 'completely unusable' to 'perfect in its sublime beauty', the other from 'there's a complete solution for this' to 'i need to learn VHDL for this'. There's some factors that help shift these spectra. Configurability helps. If I can change a config to get the behavior I want, that is incredible, thank you. Open source helps. Getting to see how they did it reduces reverse engineering work immensely if I ever have to dig in. Modularity helps. If I can just plop in my module instead of doing brain surgery on other modules, that makes it a lot easier. Good components help. Say I need a webscraper and know python. Imagine there was only selenium and not even urllib, but some low level TCP/IP library. I get a choice between heavy but easy or slim but high maintenance. But there's the sexy requests library, and there is the beautiful beautifulsoup4. I tell requests what to get, tell bs4 what I want from it, and I'm done. Another great example for this is emacs. python-mode + elpy (almost complete solution), hide-show mode, electric-pair mode, and if anything still bugs me, it is fixable. If it were OOP, I'd inherit a lot of powerful functions, but I can always override anything that is wrong. Expertise helps. If I have written a kernel module, that's another avenue to solving problems I have. Expertise is a special case here worth more attention. It's the main thing that changes for any single programmer, and can skew this equation immensely. Expertise grows when you struggle with new things. Preferably just outside what you know and are comfortable with. Considering that, DIY whenever you can afford to DIY (eg. pay the upfront cost of acquiring expertise), DIY whenever it is just outside what you can do, or DIY when it makes a lot of sense (eg. squarely in your domain of expertise, and there's a benefit to be had). In concrete examples, that means don't DIY when you're on a tight deadline, don't attempt to write your own kernel after learning about variables, don't write your own parser generator when say, YACC, solves your problem just fine.

by di 10y ago

deleted 10y ago

[deleted]

jjnoakes 10y ago

I tend to do both in parallel and the first one done wins. That is, if I have a problem that requires a library or program, and I don't know of one, I semi-simultaneously try to find a library/program that exists out there (scanning forums, googling around, reading stack overflow, searching github, going to language repositories for the languages I care about, etc) and also in parallel try to formulate in my mind what the ideal solution would look like for my particular problem. As time goes by, I get closer to finding a good enough library/program and closer to being able to picture what a solution would look like if I wrote it. At some point I either find what I need (it's good enough or it's perfect) or I get to the point where I understand enough about the solution I'm envisioning that I write it up myself.

igorgue 10y ago
If you don't have the time or energy for such projects then you CAN'T do them. The answer is there.
quantumhobbit 10y ago
Yes. If it takes me longer to figure out how to use your library or framework than to just implement the functionality myself, there is no point in using the library. Some people claim you should still use the 3rd party solution because of the cost of supporting the extra code you have written. But bugs can exist in both my code and the 3rd party code and I understand how to fix bugs in my code much more easily.
- MaulingMonkey 10y ago
  Other points of consideration: My coworkers might not already know some library, but they definitely won't know my library. My coworker's code is just about as "3rd party" as any library - as is code I wrote as little as 6 months ago. Also my job owns that code, so rolling my own means I need to write another clone every time I switch employers - assuming there's no patents or overly litigious lawyers to worry about. But you're of course correct that there is, eventually, a point where it no longer makes sense to use the library. > Some people claim you should still use the 3rd party solution because of the cost of supporting the extra code you have written. But bugs can exist in both my code and the 3rd party code and I understand how to fix bugs in my code much more easily. The problem is I got so tired of fixing bugs in coworker / former coworker code that I eventually replaced their stuff with off the shelf libraries, just so the bugs would go away. And in practice, they did go away. And it caught several usage bugs because the library had better sanity checks. And to this day, those former coworkers would use the same justifications, in total earnestness. I've never said "gee, I wish we used some custom bespoke implementation for this". I'll wish a good implementation had been made commonly available as a reusable library, perhaps. But bespoke just means fewer eyes and fewer bugfixes.
  - jjnoakes 10y ago
    It's all trade-offs. If there happens to be a well-tested third party library that does what you want, doesn't increase your attack surface more than necessary, is supported by the community, is easy to get up and running with, and has a compatible license with what you are using it in, then by all means go for it. For me and my work, I tend to find that something from the above list is lacking enough that it makes more sense to write it in-house. Not always, and not as a rule, but it works out that way quite a bit. I would also argue that if coworkers couldn't write a library without a prohibitive number of bugs, then they won't be able to write application or glue code either. So maybe your issue wasn't in-house vs third party libraries, but the quality control and/or developer aptitude around you.

stcredzero 10y ago

Isn't the answer contained in the quote? Do a cost/benefit analysis of the "amount of time and energy" that would go "down these ratholes" versus the "the time not wasted in trying to fix other people’s not quite right tools."

austinjp 10y ago
But how can you assess this until you have gone down those rat holes?
- stcredzero 10y ago
  For many particular examples, there have already been enough rathole spelunkers to provide useful data. Maybe start looking in the places where there isn't already useful data?
  - DasIch 10y ago
    Any area in which enough such spelunkers are found is unlikely to be significantly improved by adding your own effort.
    - TheAdventMaster 10y ago
      Agreed. It's often much, much harder to articulate why an idea is bad or a rat hole. You just move on. I've come up with explanation by analogy. You can demonstrate quite easily in mathematics how you can create a system of notation or a function that quickly becomes impossible to compute. A number that is too large, or an algorithm that would take infinity amount of time and resources to solve... It seems to be in nature that bad ideas are easy. Good ideas are harder, because they tend to be refinements of what already exists and what is already good. So pursue good ideas. Pursue the thing that you have thought about and decided has the best balance between values and highest chance to succeed. Sometimes it's just a strong gut feeling. Go for it, but set limits, because you don't want to fall prey to a gut feeling originating from strong intuition but an equally strong lack of fundamental understanding.

alankay1 10y ago

This is a tough question. (And always has been in a sense, because every era has had projects where the tool building has sunk the project into a black hole.) It really helped at Parc to work with real geniuses like Chuck Thacker and Dan Ingalls (and quite a few more). There is a very thin boundary between making the 2nd order work vs getting wiped out by the effort. Another perspective on this is to think about "not getting caught by dependencies" -- what if there were really good independent module systems -- perhaps aided by hardware -- that allowed both worlds to work together (so one doesn't get buried under "useful patches", etc.) One of my favorite things to watch at Parc was how well Dan Ingalls was able to bootstrap a new system out of an old one by really using what objects are good for, and especially where the new system was even much better at facilitating the next bootstrap. I'm not a big Unix fan -- it was too late on the scene for the level of ideas that it had -- but if you take the cultural history it came from, there were several things they tried to do that were admirable -- including really having a tiny kernel and using Unix processes for all systems building (this was a very useful version of "OOP" -- you just couldn't have small objects because of the way processes were implemented). It was quite sad to see how this pretty nice mix and match approach gradually decayed into huge loads and dependencies. Part of this was that the rather good idea of parsing non-command messages in each process -- we used this in the first Smalltalk at Parc -- became much too ad hoc because there was not a strong attempt to intertwine a real language around the message structures (this very same thing happened with http -- just think of what this could have been if anyone had been noticing ...)

pault 10y ago
> I'm not a big Unix fan What is your preferred technology stack?
- astrodust 10y ago
  What's a good non-UNIX open-source operating system that's useful for day-to-day work, or at least academically significant enough that it's worth diving in to?
  - 10y ago

shostack 10y ago

Specifically with regards to languages and OS's, I wonder how much that cost/benefit equation shifts as things have become so much more complex, and as we continue to pile on abstraction layer after abstraction layer.

astrobe_ 10y ago
I think the problem is not complexity but size. Most of the source for the Linux kernel is in the drivers, for instance. As for languages, most of the weight is in the libraries.

50CNT 10y ago

I think you have to weigh your qualms against the difficulty of implementation. They're both spectra, one from 'completely unusable' to 'perfect in its sublime beauty', the other from 'there's a complete solution for this' to 'i need to learn VHDL for this'. There's some factors that help shift these spectra. Configurability helps. If I can change a config to get the behavior I want, that is incredible, thank you. Open source helps. Getting to see how they did it reduces reverse engineering work immensely if I ever have to dig in. Modularity helps. If I can just plop in my module instead of doing brain surgery on other modules, that makes it a lot easier. Good components help. Say I need a webscraper and know python. Imagine there was only selenium and not even urllib, but some low level TCP/IP library. I get a choice between heavy but easy or slim but high maintenance. But there's the sexy requests library, and there is the beautiful beautifulsoup4. I tell requests what to get, tell bs4 what I want from it, and I'm done. Another great example for this is emacs. python-mode + elpy (almost complete solution), hide-show mode, electric-pair mode, and if anything still bugs me, it is fixable. If it were OOP, I'd inherit a lot of powerful functions, but I can always override anything that is wrong. Expertise helps. If I have written a kernel module, that's another avenue to solving problems I have. Expertise is a special case here worth more attention. It's the main thing that changes for any single programmer, and can skew this equation immensely. Expertise grows when you struggle with new things. Preferably just outside what you know and are comfortable with. Considering that, DIY whenever you can afford to DIY (eg. pay the upfront cost of acquiring expertise), DIY whenever it is just outside what you can do, or DIY when it makes a lot of sense (eg. squarely in your domain of expertise, and there's a benefit to be had). In concrete examples, that means don't DIY when you're on a tight deadline, don't attempt to write your own kernel after learning about variables, don't write your own parser generator when say, YACC, solves your problem just fine.