12 ms·

Why I often implement things from scratch (2006)

144 points by clarry 7y ago

amelius 7y ago
Sending data between two machines should not be difficult even in 2006, and this post shows that it isn't. However the author seems to want to imply that this would hold for any task, which is of course totally false.
pvtmert 7y ago
if any given task is trivial (eg. can be done within minutes manually) author's method is the way I go too. Instead of downloading node.js application with numerous dependencies, I also prefer implementing it in few lines of code. Some can claim running a 'proper' application (like nodejs/express with JWT authentication) is better. But if your business not solely depend on your solution, trivial stuff wins on the long run IMHO. Of course security/access-control is debatable...
- codazoda 7y ago
  I am working on a pet project to build a slot machine. I'm looking at it from a regulatory angle as well. As such, I want to understand the code as much as possible. I did some initial tests in Node. When I installed express, for routing, it added tens of dependencies and 2.5MB of code files. They call it "minimal". No thanks. I'll skip the router, for now or write my own.
- hinkley 7y ago
  I don’t think anyone is suggesting downloading a library is better than writing a single short function. It’s a false dilemma. People who are salty about this subject have dealt with NIH situations where people think they can do better, and are either wrong immediately, or become so when their attention shifts to some other problem. I have written and maintained internal libraries. Many times it’s the wrong choice, sometimes it’s the only choice. But I’m also more responsive to feedback than the NIH bozos my coworkers typically complain about, which may in part be why I get to hear about their grievances. There’s an old half-joke, half-theory that we should elect a President who doesn’t want the job, instead of people who do. You should have the individual contributor who doesn’t want snowflake code write the snowflake code. They’ll keep it no-nonsense.
SnowflakeOnIce 7y ago
RIP joe We miss you
cogman10 7y ago
Too many devs don't understand that dependencies add fragility to a system. The larger and more complex a system you design, the more likely it is that you end up with something breaking because two dependencies are incompatible or even your code and a dependant break due to updating the dependant. Further, security is rarely considered. The more code you have, the more likely you are to have a security vulnerability. You have to worry not only about code you maintain, but also the code of your dependencies. If you can write the functionality you crave rapidly, do it. You'll save yourself (and others) headaches in the future.
- jonnycomputer 7y ago
  Or worse, the developer gets bored and stops supporting the project, and it stagnates until it breaks everything else. VisionEgg I'm looking at you.
  - caymanjim 7y ago
    Except that happens far more often when you roll your own code internally. For anything of even moderate complexity, only the author will know how it works.
    - clarry 7y ago
      > For anything of even moderate complexity, only the author will know how it works. I think that's as much true of external dependencies as it is of ones that you wrote for yourself. With the exception that your own solution is probably smaller, much smaller, because it only ever needed to solve your immediate needs and not everything else a third party had to deal with. So it's quite possible that the internal solution is more approachable when the time comes to debug it.
    - deleted 7y ago
      [deleted]
    - hinkley 7y ago
      If this person is bad at knowledge transfer, they end up being the only expert and thereby gaining status for having done so, when objectively this should cost them status instead.
airstrike 7y ago
https://xkcd.com/1205/ https://xkcd.com/1205/
- clarry 7y ago
  That's a nice chart to keep in mind, but kinda tangential. Joe's point is that with the right tools, it is sometimes faster to write your own solution than it is to pick one from all the available canned solutions and learn to build & install & configure & use it (and figure out what to do when it doesn't work like it's supposed to).
  - zimpenfish 7y ago
    > Joe's point I think his point is more it's faster to solve the problem you have ("I want to get files from machine X") rather than the problem you imagine you have ("I need an FTP server") - it's just wrapped up in "I solved it myself using Erlang".
    - clarry 7y ago
      Well both Joe's program and an FTP server & client are possible solutions to the problem "I want to get files from machine X." But I very much agree that the speed of the custom solution can in large part be attributed to the fact that you can reject the cost and complexity of existing solutions like FTP (which set out to solve more than just Joe's immediate problem). Similarly, I often transfer files by piping them to netcat. EDIT: the point about tools still stands. For example: host1$ nc -l 1234 > my.file host2$ cat my.file | nc host1 1234 This is very easy to do given that unix is my toolbox. It'd be much more involved to write from scratch in (say) C. On the other hand, Joe's tool is much more powerful; it doesn't require him to manually interact with the shell on two different hosts, and his program can list files on the remote host and automatically download them under the correct name whereas I'm stuck manually redirecting nc's output to the right file (and cursing myself when I accidentally forget to change the name and overwrite a file I wasn't supposed to fetch). Those features were very easy to add to his program because Erlang is a powerful tool. On unix shell, it's much harder.
hirako2000 7y ago
Writing from scratch costs more, but provides full scope for creativity. Reusing increases productivity, at the costs of reduced scope for creativity, along with other hidden costs..
enriquto 7y ago
There's a clear-cut "scissor statement" that you can use to divide programmers into two opposite sides. You ask the following question: "--If you need to invert a 2x2 matrix in your program, do you write down the algebraic formula for the solution by hand or do you call a linear algebra library? (that you would not need otherwise)" The answer to this question is obvious to most programmers. Also, it is obvious to them that the opposite answer is clearly wrong and absurd. Unfortunately, not everybody gives the same answer!
- deleted 7y ago
  [deleted]
- seisvelas 7y ago
  Okay, you got me. I am in the camp that says calling the library is (in the general case) obviously the better choice. My (camp's) reasoning: 1. Your handrolled solution is likely worse than the library anyway 2. You're wasting effort reinventing the wheel 3. You risk adding unnecessary opacity by littering the program with dense math 4. Other programmers will assume there is a pertinent reason you handrolled the algorithm instead of using the library. Now everytime someone touches that code they have to reverse engineer your approach to see what it does different from the library such that you had to roll it yourself (surprise: nothing!). The two exceptions I imagine are 1) if you really do absolutely need some slightly modified microoptimization of what the library does or 2) if you are working for a platform where there is good reason to avoid even slight bloat (embedded). For the people who read OP's comment and believe I am (obviously) wrong, what is your camp's way of looking at this?
  - enriquto 7y ago
    > I am in the camp that says calling the library is (in the general case) You misunderstood the question. The question is not about the general case. It is about solving a linear system of two equations and two variables. The solution is not found by "an algorithm", it is a simple formula that fits in one line.
    - 7y ago
hirako2000 7y ago
I wouldn't call the article biased, but I don't know many people who can write erlang while I do know many who can write Java, c#, c++ The code and steps needed to compile/bundle/start/maintain for many other languages would be far more than in erlang or other beauties. Where are the tests? Even in erlang, unit tests are useful. Why would I trust my code? What about security? You might not want this server to allow nasty things to happen if unintended use takes place. So even in this particular case, I think it is worth evaluating opensource solutions, their maintenance, their license, and have a proper server in place that will be far easier to maintain and trust its robustness (security and availability)
- yxhuvud 7y ago
  I think it is fair to assume that Joe, the original author of Erlang, was working on a project that already was using Erlang. Sadly, he is not around anymore so it is not possible to bring clarity into the particulars.
jackcviers3 7y ago
The biggest reason to introduce dependencies, in my mind, isn't immediate productivity boost, but: > With enough eyes, all bugs become shallow.
caymanjim 7y ago
While the author's main point has some merit and is worth discussing, the example chosen is wrong, and the reason it's wrong is the reason you should take a beat before you consider ever rolling your own solution. So now we've got some simple code to copy files between machines, because we hired someone who was too academic to figure out how to set up a simple FTP server. Here's what comes next: 1. We need to open firewall ports so these two machines can communicate. What are they? Who knows. If it were FTP, the sysadmin would know. 2. We need authentication; we can't just allow any old machines to connect. If we were using FTP, we'd have that built-in. And it would have been audited, so we can have at least baseline confidence in its safety and accuracy. 3. We can't go having the disk overflow. We need something to manage quotas, because even though we trust our users, sometimes software goes rogue. Maybe the default FTP server can't handle this, but some other product can. 4. We need logging so there's an audit trail, in case something goes wrong and we need to track it down. And on and on and hey look you've reinvented FTP and now no one else on your team knows how it works, it's full of bugs, you've hardcoded a bunch of inefficiencies, there's no documentation, we have to explain what it is and how it works to the operations team, and hey here comes the boss asking us to make sure it works over SSL. If you think you can do it better, make sure you think about what you're doing.
- clarry 7y ago
  How can you say the example chosen is wrong? You came up with a bunch of things to complicate the scenario that do not apply to the author's problem, and those things are precisely the kind of things that make a general solution more complex. Here, by writing their own tool, Joe could avoid all the complexity that does not apply to his scenario. I don't think the point of the example was to demonstrate a complete and general solution to the problem of transferring files across computers, but a specific solution that solves the immediate problem in the specific environment Joe was operating in, faster than you'd get it done by trying to apply a more complex (and more general) solution. Another point I'd like to make (that the original post doesn't) is that they could easily extend this custom solution with additional logic that doesn't exist in any FTP server off the shelf. In fact I've had to reject libraries and programs on similar grounds; they almost kinda do what we need, but then a vital piece of the puzzle is missing and forking the project & implementing our special sauce (and then maintaining it) is going to be much more painful than rolling a custom solution from scratch.
smitty1e 7y ago
Everything is easy when you know how to do it. It's that learning curve. . .
- convolvatron 7y ago
  right. the other side of that learning curve is that if you implement some X instead of integrating a third party library..even if you didn't before, you now know a lot more about X. enough that you might be able to debug issues with X, or have an informed opinion as to the quality of other implementations of X
jchook 7y ago
Once upon a time there was a systems programmer who wanted to use a readymade 3rd party solution to an specific problem. He evaluated 3 options, then downloaded, compiled, configured and integrated the best choice into the project over the course of a full day. Its limited database integration configuration necessitated some odd quirks like duplicating a db column, but oh well. When finally done, integration tests revealed that it didn't actually solve the entire problem. It only solved 90% of the problem. Not only that, it had a significant but elusive bug that only occurred in 5% of situations. He filed a bug report, but the project hadn't received any updates in over a year. Damn. When it became clear neither of the other two readymade libraries would solve 100% of the problem either, the programmer wrote his own solution. It took 3 days of work but he learned a LOT about exactly how it works, and the result solved 100% of the problem with no major bugs. All of the quirks disappeared. The solution fit the problem exactly, and remained flexible and maintainable. It eventually served as a solid foundation for future developments which had no readymade solution.
- _bxg1 7y ago
  This parable is heavily dependent on the quality of a given ecosystem. When I bring in a third-party solution for something that might be feasible for me to implement myself, it must 1) be actively maintained, 2) be decoupled from my code at the function-interface boundary, and 3) fully cover my use-case, at least as it's known at the time of introduction. In the ecosystems I work with, finding all of these qualities is not uncommon.
- hinkley 7y ago
  There’s a questionnaire out there for anyone writing a library and the first couple of questions are what ones did you try, why don’t they work, and why can’t they be fixed. Sounds like your coworker wrote a library the “right” way. We had our own snowflake wrapper around the saucelabs client. I eventually figured out how to file a PR to do the same. Now I get to use that solution on every other project I ever work on. I have similarly used my own Stack Overflow answer to solve a problem I also had three years prior. And I recently found my SO answer to another question verbatim in our own codebase. Ironically, from a guy who often ignores my advice. Get as many things out of your head and into the group memory as you can. Things get forgotten otherwise. Speaking of, did you guys ever consider open sourcing his solution?
bluedino 7y ago
Imagine you have a project where you need to draw a fairly complex object. You have found a file in a standard format that you can use. However, there are 10 dependencies, some of which are non-trivial. You spend 4 hours changing versions of libraries, downloading a newer compiler than the one you are using, and never get the thing running. You could have slapped some code together to read the file, and then draw it yourself. It would have taken a half hour. However, later on you are tasked with with representing many more objects, which have corner cases you didn't account for. This eats up hours and hours of development. Also, you're then tasked with drawing those objects in many different ways, more and more development has been added. The two take aways are choose based on what you have to do, and make that choice based on knowing 100% what you have to do. If a customer can't make up their mind, they're hurting development whether they know it or not. BTW, the objects were GIS shapefiles.
- ldng 7y ago
  If there is a domain I would not DIY, it is GIS. It is very often much harder than it looks. Was by any chance one of the dependencies GDAL ?
  - bluedino 7y ago
    basemap, maplotlib, geos... The easy solution was to just use Conda.
danShumway 7y ago
> If you have the right tools it's often quicker to implement something from scratch than going to all the trouble of downloading compiling and installing something that somebody else has written. Heavily agree. Shameless self plug, this is the philosophy behind Distilled[0]: provide a good-enough, low-abstraction tool so you can build on top of it without it getting in your way or forcing you to memorize configuration options. Adding unnecessary black-boxes to your code can occasionally come back to bite you. I try not to be religious about this, but I am a little biased towards avoiding the abstractions of the 3rd-party libraries unless they come with a lot of benefits. A big "aha" moment for me was trying to teach interns how to use Grunt/Gulp, and eventually realizing that just teaching them Bash was easier. Then I started noticing other stuff -- for example, that the documentation for our inline documentation library was longer than some of the documentation pages we were generating with it. There's a principle here that's true for abstractions in general[1], but I find it is especially true for some 3rd-party dependencies. I regularly need to read the source code of 3rd-party dependencies that I use, so I no longer treat them as free. Obviously this stuff isn't a hard rule. I still use 3rd-party dependencies. But there's a balance here. You should be at least a tiny bit cautious about ready-made solutions. [0]: https://distilledjs.com https://distilledjs.com [1]: https://peertube.danshumway.com/videos/watch/2daaee22-4d92-47cf-a211-9d44a5e3e50a https://peertube.danshumway.com/videos/watch/2daaee22-4d92-4...
nartz 7y ago
This is extremely context dependent. For instance, a dev has to consider: - guess whether or not this is an isolated case, or whether or not this will become core functionality - a self-assessment of the true difficulty of the problem - a self-assessment of their own skills and knowledge in the area - security reasoning - API access/readabilty for other developers to use this code - maintainability of new code I have personally seen personal implementations that lead to bug, after bug, that have already been reasoned about in equivalent libraries. Often for the simple fact that other devs who have to work on this code, its likely that the abstraction and readability of a third party library is probably greater than the 'quick-and-dirty' implementation.
- hinkley 7y ago
  That’s a good list. I’d throw “facility at evaluating existing solutions” on there, too. The person who reaches for the “New File” button is often not used to asking if the thing they need already exists. They can end up duplicating or triplicating business logic which results in other engineers confidently declaring that a problem has been fixed when they only fixed the obvious occurrence.
thijsvandien 7y ago
An important reason to try it from scratch first is that it helps your understanding of the problem. That way, you can either confirm that it's better not to take on a dependency, or leverage the experience to find, evaluate, and appreciate the right dependency when one turns out to be desirable. Just like people build their own web frameworks before they settle for an established one. That's a good thing, in my opinion.

144 points by clarry 7y ago

Sending data between two machines should not be difficult even in 2006, and this post shows that it isn't. However the author seems to want to imply that this would hold for any task, which is of course totally false.

pvtmert 7y ago

if any given task is trivial (eg. can be done within minutes manually) author's method is the way I go too. Instead of downloading node.js application with numerous dependencies, I also prefer implementing it in few lines of code. Some can claim running a 'proper' application (like nodejs/express with JWT authentication) is better. But if your business not solely depend on your solution, trivial stuff wins on the long run IMHO. Of course security/access-control is debatable...

codazoda 7y ago
I am working on a pet project to build a slot machine. I'm looking at it from a regulatory angle as well. As such, I want to understand the code as much as possible. I did some initial tests in Node. When I installed express, for routing, it added tens of dependencies and 2.5MB of code files. They call it "minimal". No thanks. I'll skip the router, for now or write my own.
hinkley 7y ago
I don’t think anyone is suggesting downloading a library is better than writing a single short function. It’s a false dilemma. People who are salty about this subject have dealt with NIH situations where people think they can do better, and are either wrong immediately, or become so when their attention shifts to some other problem. I have written and maintained internal libraries. Many times it’s the wrong choice, sometimes it’s the only choice. But I’m also more responsive to feedback than the NIH bozos my coworkers typically complain about, which may in part be why I get to hear about their grievances. There’s an old half-joke, half-theory that we should elect a President who doesn’t want the job, instead of people who do. You should have the individual contributor who doesn’t want snowflake code write the snowflake code. They’ll keep it no-nonsense.

SnowflakeOnIce 7y ago

RIP joe We miss you

cogman10 7y ago

Too many devs don't understand that dependencies add fragility to a system. The larger and more complex a system you design, the more likely it is that you end up with something breaking because two dependencies are incompatible or even your code and a dependant break due to updating the dependant. Further, security is rarely considered. The more code you have, the more likely you are to have a security vulnerability. You have to worry not only about code you maintain, but also the code of your dependencies. If you can write the functionality you crave rapidly, do it. You'll save yourself (and others) headaches in the future.

jonnycomputer 7y ago
Or worse, the developer gets bored and stops supporting the project, and it stagnates until it breaks everything else. VisionEgg I'm looking at you.
- caymanjim 7y ago
  Except that happens far more often when you roll your own code internally. For anything of even moderate complexity, only the author will know how it works.
  - clarry 7y ago
    > For anything of even moderate complexity, only the author will know how it works. I think that's as much true of external dependencies as it is of ones that you wrote for yourself. With the exception that your own solution is probably smaller, much smaller, because it only ever needed to solve your immediate needs and not everything else a third party had to deal with. So it's quite possible that the internal solution is more approachable when the time comes to debug it.
  - deleted 7y ago
    [deleted]
  - hinkley 7y ago
    If this person is bad at knowledge transfer, they end up being the only expert and thereby gaining status for having done so, when objectively this should cost them status instead.

airstrike 7y ago

https://xkcd.com/1205/ https://xkcd.com/1205/

clarry 7y ago
That's a nice chart to keep in mind, but kinda tangential. Joe's point is that with the right tools, it is sometimes faster to write your own solution than it is to pick one from all the available canned solutions and learn to build & install & configure & use it (and figure out what to do when it doesn't work like it's supposed to).
- zimpenfish 7y ago
  > Joe's point I think his point is more it's faster to solve the problem you have ("I want to get files from machine X") rather than the problem you imagine you have ("I need an FTP server") - it's just wrapped up in "I solved it myself using Erlang".
  - clarry 7y ago
    Well both Joe's program and an FTP server & client are possible solutions to the problem "I want to get files from machine X." But I very much agree that the speed of the custom solution can in large part be attributed to the fact that you can reject the cost and complexity of existing solutions like FTP (which set out to solve more than just Joe's immediate problem). Similarly, I often transfer files by piping them to netcat. EDIT: the point about tools still stands. For example: host1$ nc -l 1234 > my.file host2$ cat my.file | nc host1 1234 This is very easy to do given that unix is my toolbox. It'd be much more involved to write from scratch in (say) C. On the other hand, Joe's tool is much more powerful; it doesn't require him to manually interact with the shell on two different hosts, and his program can list files on the remote host and automatically download them under the correct name whereas I'm stuck manually redirecting nc's output to the right file (and cursing myself when I accidentally forget to change the name and overwrite a file I wasn't supposed to fetch). Those features were very easy to add to his program because Erlang is a powerful tool. On unix shell, it's much harder.

hirako2000 7y ago

Writing from scratch costs more, but provides full scope for creativity. Reusing increases productivity, at the costs of reduced scope for creativity, along with other hidden costs..

enriquto 7y ago

There's a clear-cut "scissor statement" that you can use to divide programmers into two opposite sides. You ask the following question: "--If you need to invert a 2x2 matrix in your program, do you write down the algebraic formula for the solution by hand or do you call a linear algebra library? (that you would not need otherwise)" The answer to this question is obvious to most programmers. Also, it is obvious to them that the opposite answer is clearly wrong and absurd. Unfortunately, not everybody gives the same answer!

deleted 7y ago
[deleted]
seisvelas 7y ago
Okay, you got me. I am in the camp that says calling the library is (in the general case) obviously the better choice. My (camp's) reasoning: 1. Your handrolled solution is likely worse than the library anyway 2. You're wasting effort reinventing the wheel 3. You risk adding unnecessary opacity by littering the program with dense math 4. Other programmers will assume there is a pertinent reason you handrolled the algorithm instead of using the library. Now everytime someone touches that code they have to reverse engineer your approach to see what it does different from the library such that you had to roll it yourself (surprise: nothing!). The two exceptions I imagine are 1) if you really do absolutely need some slightly modified microoptimization of what the library does or 2) if you are working for a platform where there is good reason to avoid even slight bloat (embedded). For the people who read OP's comment and believe I am (obviously) wrong, what is your camp's way of looking at this?
- enriquto 7y ago
  > I am in the camp that says calling the library is (in the general case) You misunderstood the question. The question is not about the general case. It is about solving a linear system of two equations and two variables. The solution is not found by "an algorithm", it is a simple formula that fits in one line.
  - 7y ago

hirako2000 7y ago

I wouldn't call the article biased, but I don't know many people who can write erlang while I do know many who can write Java, c#, c++ The code and steps needed to compile/bundle/start/maintain for many other languages would be far more than in erlang or other beauties. Where are the tests? Even in erlang, unit tests are useful. Why would I trust my code? What about security? You might not want this server to allow nasty things to happen if unintended use takes place. So even in this particular case, I think it is worth evaluating opensource solutions, their maintenance, their license, and have a proper server in place that will be far easier to maintain and trust its robustness (security and availability)

yxhuvud 7y ago
I think it is fair to assume that Joe, the original author of Erlang, was working on a project that already was using Erlang. Sadly, he is not around anymore so it is not possible to bring clarity into the particulars.

jackcviers3 7y ago

The biggest reason to introduce dependencies, in my mind, isn't immediate productivity boost, but: > With enough eyes, all bugs become shallow.

caymanjim 7y ago

While the author's main point has some merit and is worth discussing, the example chosen is wrong, and the reason it's wrong is the reason you should take a beat before you consider ever rolling your own solution. So now we've got some simple code to copy files between machines, because we hired someone who was too academic to figure out how to set up a simple FTP server. Here's what comes next: 1. We need to open firewall ports so these two machines can communicate. What are they? Who knows. If it were FTP, the sysadmin would know. 2. We need authentication; we can't just allow any old machines to connect. If we were using FTP, we'd have that built-in. And it would have been audited, so we can have at least baseline confidence in its safety and accuracy. 3. We can't go having the disk overflow. We need something to manage quotas, because even though we trust our users, sometimes software goes rogue. Maybe the default FTP server can't handle this, but some other product can. 4. We need logging so there's an audit trail, in case something goes wrong and we need to track it down. And on and on and hey look you've reinvented FTP and now no one else on your team knows how it works, it's full of bugs, you've hardcoded a bunch of inefficiencies, there's no documentation, we have to explain what it is and how it works to the operations team, and hey here comes the boss asking us to make sure it works over SSL. If you think you can do it better, make sure you think about what you're doing.

clarry 7y ago
How can you say the example chosen is wrong? You came up with a bunch of things to complicate the scenario that do not apply to the author's problem, and those things are precisely the kind of things that make a general solution more complex. Here, by writing their own tool, Joe could avoid all the complexity that does not apply to his scenario. I don't think the point of the example was to demonstrate a complete and general solution to the problem of transferring files across computers, but a specific solution that solves the immediate problem in the specific environment Joe was operating in, faster than you'd get it done by trying to apply a more complex (and more general) solution. Another point I'd like to make (that the original post doesn't) is that they could easily extend this custom solution with additional logic that doesn't exist in any FTP server off the shelf. In fact I've had to reject libraries and programs on similar grounds; they almost kinda do what we need, but then a vital piece of the puzzle is missing and forking the project & implementing our special sauce (and then maintaining it) is going to be much more painful than rolling a custom solution from scratch.

smitty1e 7y ago

Everything is easy when you know how to do it. It's that learning curve. . .

convolvatron 7y ago
right. the other side of that learning curve is that if you implement some X instead of integrating a third party library..even if you didn't before, you now know a lot more about X. enough that you might be able to debug issues with X, or have an informed opinion as to the quality of other implementations of X

jchook 7y ago

Once upon a time there was a systems programmer who wanted to use a readymade 3rd party solution to an specific problem. He evaluated 3 options, then downloaded, compiled, configured and integrated the best choice into the project over the course of a full day. Its limited database integration configuration necessitated some odd quirks like duplicating a db column, but oh well. When finally done, integration tests revealed that it didn't actually solve the entire problem. It only solved 90% of the problem. Not only that, it had a significant but elusive bug that only occurred in 5% of situations. He filed a bug report, but the project hadn't received any updates in over a year. Damn. When it became clear neither of the other two readymade libraries would solve 100% of the problem either, the programmer wrote his own solution. It took 3 days of work but he learned a LOT about exactly how it works, and the result solved 100% of the problem with no major bugs. All of the quirks disappeared. The solution fit the problem exactly, and remained flexible and maintainable. It eventually served as a solid foundation for future developments which had no readymade solution.

_bxg1 7y ago
This parable is heavily dependent on the quality of a given ecosystem. When I bring in a third-party solution for something that might be feasible for me to implement myself, it must 1) be actively maintained, 2) be decoupled from my code at the function-interface boundary, and 3) fully cover my use-case, at least as it's known at the time of introduction. In the ecosystems I work with, finding all of these qualities is not uncommon.
hinkley 7y ago
There’s a questionnaire out there for anyone writing a library and the first couple of questions are what ones did you try, why don’t they work, and why can’t they be fixed. Sounds like your coworker wrote a library the “right” way. We had our own snowflake wrapper around the saucelabs client. I eventually figured out how to file a PR to do the same. Now I get to use that solution on every other project I ever work on. I have similarly used my own Stack Overflow answer to solve a problem I also had three years prior. And I recently found my SO answer to another question verbatim in our own codebase. Ironically, from a guy who often ignores my advice. Get as many things out of your head and into the group memory as you can. Things get forgotten otherwise. Speaking of, did you guys ever consider open sourcing his solution?

bluedino 7y ago

Imagine you have a project where you need to draw a fairly complex object. You have found a file in a standard format that you can use. However, there are 10 dependencies, some of which are non-trivial. You spend 4 hours changing versions of libraries, downloading a newer compiler than the one you are using, and never get the thing running. You could have slapped some code together to read the file, and then draw it yourself. It would have taken a half hour. However, later on you are tasked with with representing many more objects, which have corner cases you didn't account for. This eats up hours and hours of development. Also, you're then tasked with drawing those objects in many different ways, more and more development has been added. The two take aways are choose based on what you have to do, and make that choice based on knowing 100% what you have to do. If a customer can't make up their mind, they're hurting development whether they know it or not. BTW, the objects were GIS shapefiles.

ldng 7y ago
If there is a domain I would not DIY, it is GIS. It is very often much harder than it looks. Was by any chance one of the dependencies GDAL ?
- bluedino 7y ago
  basemap, maplotlib, geos... The easy solution was to just use Conda.

danShumway 7y ago

> If you have the right tools it's often quicker to implement something from scratch than going to all the trouble of downloading compiling and installing something that somebody else has written. Heavily agree. Shameless self plug, this is the philosophy behind Distilled[0]: provide a good-enough, low-abstraction tool so you can build on top of it without it getting in your way or forcing you to memorize configuration options. Adding unnecessary black-boxes to your code can occasionally come back to bite you. I try not to be religious about this, but I am a little biased towards avoiding the abstractions of the 3rd-party libraries unless they come with a lot of benefits. A big "aha" moment for me was trying to teach interns how to use Grunt/Gulp, and eventually realizing that just teaching them Bash was easier. Then I started noticing other stuff -- for example, that the documentation for our inline documentation library was longer than some of the documentation pages we were generating with it. There's a principle here that's true for abstractions in general[1], but I find it is especially true for some 3rd-party dependencies. I regularly need to read the source code of 3rd-party dependencies that I use, so I no longer treat them as free. Obviously this stuff isn't a hard rule. I still use 3rd-party dependencies. But there's a balance here. You should be at least a tiny bit cautious about ready-made solutions. [0]: https://distilledjs.com https://distilledjs.com [1]: https://peertube.danshumway.com/videos/watch/2daaee22-4d92-47cf-a211-9d44a5e3e50a https://peertube.danshumway.com/videos/watch/2daaee22-4d92-4...

nartz 7y ago

This is extremely context dependent. For instance, a dev has to consider: - guess whether or not this is an isolated case, or whether or not this will become core functionality - a self-assessment of the true difficulty of the problem - a self-assessment of their own skills and knowledge in the area - security reasoning - API access/readabilty for other developers to use this code - maintainability of new code I have personally seen personal implementations that lead to bug, after bug, that have already been reasoned about in equivalent libraries. Often for the simple fact that other devs who have to work on this code, its likely that the abstraction and readability of a third party library is probably greater than the 'quick-and-dirty' implementation.

hinkley 7y ago
That’s a good list. I’d throw “facility at evaluating existing solutions” on there, too. The person who reaches for the “New File” button is often not used to asking if the thing they need already exists. They can end up duplicating or triplicating business logic which results in other engineers confidently declaring that a problem has been fixed when they only fixed the obvious occurrence.

thijsvandien 7y ago

An important reason to try it from scratch first is that it helps your understanding of the problem. That way, you can either confirm that it's better not to take on a dependency, or leverage the experience to find, evaluate, and appreciate the right dependency when one turns out to be desirable. Just like people build their own web frameworks before they settle for an established one. That's a good thing, in my opinion.