5 ms·

The "not invented here" pendulum has swung a long way, and I think that there's now a real, hidden anti-pattern in industry: "never invent here". At some point

by vosper 11y ago

The "not invented here" pendulum has swung a long way, and I think that there's now a real, hidden anti-pattern in industry: "never invent here". At some point, it's important for us as programmers to indulge that "I could do this better" instinct. If you're right you could make a real contribution to your employer or the community. If you're wrong then you'll still be a better engineer (and possibly a domain expert) for the effort. If everyone embraced "not invented here" as a philosophy then the whole industry would stagnate.

wvenable 11y ago
Most of the time I can do it better because my problem space is so much smaller. And sometimes I look at components and I'm so disgusted by the code or the API that I merely use it as an example and re-write the code for my own purposes. Just because somebody put it on Github or built a package doesn't automatically make the code good. Most of the "not invented here" I come across is from developers who never look for 3rd party solutions. It doesn't even occur to them. And it's usually obvious stuff like XML or JSON parsing! Which is why the message of "Not Invented here" is a good one to repeat over and over.
- kybernetikos 11y ago
  > Which is why the message of "Not Invented here" is a good one to repeat over and over. So, there is a problem where many developers would rather work on abstractions rather than concrete value-providing things. I suffer from this myself - if abstractions weren't interesting to us, we wouldn't be working in software. However, over the years I've discovered that many software projects are gaining negative value from some of their libraries, and I'm not sure which is the larger problem. http://yosefk.com/blog/redundancy-vs-dependencies-which-is-worse.html http://yosefk.com/blog/redundancy-vs-dependencies-which-is-w... describes the essential problem.
  - jcoffland 11y ago
    So true. You always have to weight the cost of carrying around a dependency and what that will mean for others who will have to use and build your code in the future and on platforms like Windows where dependency resolution is far from trivial. Seen this way sometimes it's worth spending a few of hours writing a JSON parser to avoid adding a dependency. Probably not woth an XML parser though.
- xamuel 11y ago
  >obvious stuff like XML or JSON parsing! There is one case when those wheels can be worth (partially) reinventing: if you only need them to parse some files which use XML or JSON in a very restricted way. Say you've got a bunch of XML files which only go one level deep, only use ASCII, don't use namespaces, etc. A full XML parser comes with a whole ton of bloat you don't need for that! Plus, if there's ever a problem with those files, the outsourced XML parser will tend to give the end-user absolutely useless error messages.
  - yo-code-sucks 11y ago
    Nope nope nope. Your not a special flower, you cannot do it better, at least not on my dime.
    - xamuel 11y ago
      Did you even read what I wrote? Of course I can't do a better XML or JSON parser. But if what you need parsed is ten thousand lines of <key1>value1</key1> then I'm pretty sure I can write something a lot more efficient and portable, with better error reporting, than W3CBBQ, and I can do it in 10 minutes. That said, I'm very flattered you created a throwaway account just to respond to me.
      - MaulingMonkey 11y ago
        > then I'm pretty sure I can write something a lot more efficient Sure. > and portable ...it's hard to be "a lot more portable" than a simple C library that compiles out of the box on all your preferred compilers and platforms. Care to elaborate how you end up being "a lot more portable"? > with better error reporting Plenty of room to do your own error reporting with off the shelf parsers. > and I can do it in 10 minutes The bug reports, on the other hand, will last a lifetime. Because that format will change, your underlying assumptions will be invalidated, and your parser will eventually grow into one of the 4+ fully featured redundant XML parsers in your project, each with their own set of weird assumptions, inconsistencies, and "supported bugs" (for backwards compatibility), and all of them will need to be maintained. At least until someone takes the time to make a breaking change to the data format, and rip them out to replace them with a proper off-the-shelf JSON parser that has orders of magnitudes more eyeballs fixing the corner cases. Then you'll only have 3+. > A full XML parser comes with a whole ton of bloat you don't need for that! I'll take minor binary bloat over major maintenance bloat any day. And you're wrong: It turns out I actually need a lot of that "bloat" in the next revision. > Plus, if there's ever a problem with those files, the outsourced XML parser will tend to give the end-user absolutely useless error messages. If you have a blindly auto-generated XML-to-binary-format reflection based converter, sure. That's not a good reason to write off "off the shelf" XML parsers outright though, when there's plenty of ways to use them that will give you perfectly fine error messages. To keep with your theme of generalizations: in-house XML parsers tend to give the end-user no error messages, which is far worse ;)
- digi_owl 11y ago
  I suspect that the current patent cold war results in a number of programmers, never mind managers, being vary of glancing at third party solutions...
makeitsuckless 11y ago
I would say that over the past 10 years or so, especially with the rise of frameworks, PaaS and web API's, "never invent here" has become a much bigger and more troublesome anti-pattern. Especially since the latter seems to have a much deeper and irreversible impact on the engineering culture.
- digi_owl 11y ago
  Just grab 20 different containers, slapdash them together with silly string, and prey to your deity of choice that none of them hold some glaring security hole?
k3n 11y ago
I really like Spolsky's take on it: > If it's a core business function -- do it yourself, no matter what. http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html
agumonkey 11y ago
NIH is part lack of communication and understanding. Sometimes it's really hard to get the scope, qualities and failures of an external project.
ArkyBeagle 11y ago
And then there's that cold, dark morning when you find the profound, deeply embedded assumption at the core of a library that means you now have irreconcilable differences...
dragonwriter 11y ago
> At some point, it's important for us as programmers to indulge that "I could do this better" instinct. Sure, and that point is pretty easily defined: the point at which the existing solution has produces real, tangible, definable problems with achieving your current goals is the time to invent a replacement that "does it better", with better defined in exactly the terms of the problem that you are addressing. Its also okay to do it as something exploratory, off the critical path of a real project for understanding/skill-building. What's not okay -- and is instead a dangerous, expensive (both in the short-term and in maintenance terms), ultimately unproductive diversion of effort -- is making a reinvention of something that works adequately a dependency for some other effort.

by vosper 11y ago

wvenable 11y ago

Most of the time I can do it better because my problem space is so much smaller. And sometimes I look at components and I'm so disgusted by the code or the API that I merely use it as an example and re-write the code for my own purposes. Just because somebody put it on Github or built a package doesn't automatically make the code good. Most of the "not invented here" I come across is from developers who never look for 3rd party solutions. It doesn't even occur to them. And it's usually obvious stuff like XML or JSON parsing! Which is why the message of "Not Invented here" is a good one to repeat over and over.

kybernetikos 11y ago
> Which is why the message of "Not Invented here" is a good one to repeat over and over. So, there is a problem where many developers would rather work on abstractions rather than concrete value-providing things. I suffer from this myself - if abstractions weren't interesting to us, we wouldn't be working in software. However, over the years I've discovered that many software projects are gaining negative value from some of their libraries, and I'm not sure which is the larger problem. http://yosefk.com/blog/redundancy-vs-dependencies-which-is-worse.html http://yosefk.com/blog/redundancy-vs-dependencies-which-is-w... describes the essential problem.
- jcoffland 11y ago
  So true. You always have to weight the cost of carrying around a dependency and what that will mean for others who will have to use and build your code in the future and on platforms like Windows where dependency resolution is far from trivial. Seen this way sometimes it's worth spending a few of hours writing a JSON parser to avoid adding a dependency. Probably not woth an XML parser though.
xamuel 11y ago
>obvious stuff like XML or JSON parsing! There is one case when those wheels can be worth (partially) reinventing: if you only need them to parse some files which use XML or JSON in a very restricted way. Say you've got a bunch of XML files which only go one level deep, only use ASCII, don't use namespaces, etc. A full XML parser comes with a whole ton of bloat you don't need for that! Plus, if there's ever a problem with those files, the outsourced XML parser will tend to give the end-user absolutely useless error messages.
- yo-code-sucks 11y ago
  Nope nope nope. Your not a special flower, you cannot do it better, at least not on my dime.
  - xamuel 11y ago
    Did you even read what I wrote? Of course I can't do a better XML or JSON parser. But if what you need parsed is ten thousand lines of <key1>value1</key1> then I'm pretty sure I can write something a lot more efficient and portable, with better error reporting, than W3CBBQ, and I can do it in 10 minutes. That said, I'm very flattered you created a throwaway account just to respond to me.
    - MaulingMonkey 11y ago
      > then I'm pretty sure I can write something a lot more efficient Sure. > and portable ...it's hard to be "a lot more portable" than a simple C library that compiles out of the box on all your preferred compilers and platforms. Care to elaborate how you end up being "a lot more portable"? > with better error reporting Plenty of room to do your own error reporting with off the shelf parsers. > and I can do it in 10 minutes The bug reports, on the other hand, will last a lifetime. Because that format will change, your underlying assumptions will be invalidated, and your parser will eventually grow into one of the 4+ fully featured redundant XML parsers in your project, each with their own set of weird assumptions, inconsistencies, and "supported bugs" (for backwards compatibility), and all of them will need to be maintained. At least until someone takes the time to make a breaking change to the data format, and rip them out to replace them with a proper off-the-shelf JSON parser that has orders of magnitudes more eyeballs fixing the corner cases. Then you'll only have 3+. > A full XML parser comes with a whole ton of bloat you don't need for that! I'll take minor binary bloat over major maintenance bloat any day. And you're wrong: It turns out I actually need a lot of that "bloat" in the next revision. > Plus, if there's ever a problem with those files, the outsourced XML parser will tend to give the end-user absolutely useless error messages. If you have a blindly auto-generated XML-to-binary-format reflection based converter, sure. That's not a good reason to write off "off the shelf" XML parsers outright though, when there's plenty of ways to use them that will give you perfectly fine error messages. To keep with your theme of generalizations: in-house XML parsers tend to give the end-user no error messages, which is far worse ;)
digi_owl 11y ago
I suspect that the current patent cold war results in a number of programmers, never mind managers, being vary of glancing at third party solutions...

makeitsuckless 11y ago

I would say that over the past 10 years or so, especially with the rise of frameworks, PaaS and web API's, "never invent here" has become a much bigger and more troublesome anti-pattern. Especially since the latter seems to have a much deeper and irreversible impact on the engineering culture.

digi_owl 11y ago
Just grab 20 different containers, slapdash them together with silly string, and prey to your deity of choice that none of them hold some glaring security hole?

k3n 11y ago

I really like Spolsky's take on it: > If it's a core business function -- do it yourself, no matter what. http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html

agumonkey 11y ago

NIH is part lack of communication and understanding. Sometimes it's really hard to get the scope, qualities and failures of an external project.

ArkyBeagle 11y ago

And then there's that cold, dark morning when you find the profound, deeply embedded assumption at the core of a library that means you now have irreconcilable differences...

dragonwriter 11y ago

> At some point, it's important for us as programmers to indulge that "I could do this better" instinct. Sure, and that point is pretty easily defined: the point at which the existing solution has produces real, tangible, definable problems with achieving your current goals is the time to invent a replacement that "does it better", with better defined in exactly the terms of the problem that you are addressing. Its also okay to do it as something exploratory, off the critical path of a real project for understanding/skill-building. What's not okay -- and is instead a dangerous, expensive (both in the short-term and in maintenance terms), ultimately unproductive diversion of effort -- is making a reinvention of something that works adequately a dependency for some other effort.