10 ms·
I wish I could give you more upvotes. You are describing a somewhat hidden psychology, which I think provides a rational basis for much of "Not Invented Here" p
by lkrubner 9y ago
I wish I could give you more upvotes. You are describing a somewhat hidden psychology, which I think provides a rational basis for much of "Not Invented Here" psychology. We tend to think that "Not Invented Here" psychology is irrational, but in fact, the loss of control over possibly crucial technology is an important cost, which makes all of us stop and re-consider whether we really want to use some software developed by an external team.
- jstimpfle 9y agoAnd it's not only that - Time spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding. It's a sad state when otherwise very intelligent people think it's bad practice to use plain C and a clean OS API that has been stable for decades (because that was somehow "magic" und impossible to understand and error-prone), and advise you to use the Boost filesystem module just to concatenate two paths.
- kakarot 9y agoThe most valuable skill is knowing when to externalize your tools. You don't always want to reinvent the wheel every time you need something, when you have deadlines to consider.
- jstimpfle 9y agoIs that in support of using the Boost filesystem API?
- konschubert 9y agoIt depends on what you're doing with it, I'd guess.
- VHRanger 9y agoIMO, only if you're using g++ or older versions of the standard. MSVC, clang and ICC have all supported the experimental::filesystem module for years at this point.
- kakarot 9y agoI don't use C++ outside of games programming, I'm not familiar with the Boost filesystem API.
- islanderfun 9y agoExactly this. There's a ton of gray here. Have to pick your battles as best as you can. Another example are game engines. Hard deny the value engines like Unreal and Unity provide. They are hard to ignore and have thousands of expert hours put into them.
- jstimpfle 9y agoA good way to pick battles is not to fight most of them. There are so many solutions that don't have any real problems...
- paulryanrogers 9y agoCould you provide an example of solutions without real problems, game engines or otherwise?
- jstimpfle 9y agoBasically anything overengineered? Concrete example: C++ "manager" objects. Why not finally learn how to structure applications and keep allocations and resource use in check? The program will be so much simpler, compile quickly, be easy to understand (execution threads stop jumping around like crazy), and typically have less memory leaks / use-after-free etc. RAII, garbage collectors and other fancy inventions for freeing resources from the call stack automatically? Not needed. Use global resource managers. Globals are needed and semantically the right thing. Not a problem. Any crazy programming language with all the features they could conceive? The one you need is guaranteed to be missing. Better, express your problem yourself and write a simple generator script that translates your concepts (mostly as plain old data) to the minimal language. Another: XML (and even JSON), possibly for anything except true document markup. It's slow and doesn't buy us anything. Yes, there are ready-to-use parsers, but after parsing you still hold a mess in your hands (and only strings / floats). Finally learn database basics and manage data in tuples. Make a text file parser that takes simple table (= list of column types) descriptions and then parses tuples from single lines and puts them as typed data in arrays of structs, done!
- robocat 9y agoConcatenating two paths is difficult if you care about any of the following: security, multiple OS's, Unicode (multiple code units, validity, combining characters), file system restrictions, etc. Your "two decades" maybe holds for Linux, but what about Windows or MacOS???! I have seen too many people use string concatenation. I think an intelligent person would recommend to use the normal library (appropriate for your language, assuming it is well written) since usually your program will be doing many other filename/path manipulations too.
- kckkdkd929292 9y agoseems like the sort of thing people should know how to spot then otherwise you’re getting security through obscurity of a magic framework no one has time to vet looking at you, openssl, etc, etc
- jstimpfle 9y agohttps://msdn.microsoft.com/de-de/library/windows/desktop/bb773571(v=vs.85).aspx https://msdn.microsoft.com/de-de/library/windows/desktop/bb7...
- robocat 9y agoExactly my point - that function doesn't deal with modern windows. MAX_PATH is 260 characters. Yet Windows now supports longer paths (\\?\ prefix). So I presume there is another Windows function to combine long path names (and maybe canonicalise Unicode better).
- jstimpfle 9y ago> Yet Windows now supports longer paths (\\?\ prefix). I know. And this is the attitude that leads to complex software. It's a self-fulfilling prophecy: "I can't do it on my own since the problem is so complex". In which case the problem does get complex. I recommend not wasting time supporting this crazy feature (unless you are writing infrastructure code for tools and you're required to - in which case I'm sorry). 260 character paths are more than enough for any project. And I certainly recommend against using the crazy abstractions from Boost::filesystem. (Personally I think it's an unfortunate example. I'd prefer to avoid paths from the start, since nobody understands the semantics of hierarchical filesystems. Alas, typically you need to deal with them to some degree).
- pjmlp 9y agoThe amount of security exploits due to memory corruption in software written in C proves that during the last 50 years, it isn't that well understood.
- cottonseed 9y ago> Time spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding. I can't upvote this enough. I used to do mathematics, and there was the story of a professor would would hold up a book and say, "You should know everything in this book. But don't read it!" Which is to say, you have to go through the process of discovering mathematics to really understand it (maybe with a bit of guidance when you get really stuck). The skill of building complex software systems is no different.
- collyw 9y agoSure and those of us experienced in writing software tend to avoid reinventing the wheel (in a buggy, untested way). I see the value in learning by building yourself, but from a software engineering point of view, using a tried and tested framework is likely to give you higher quality product in less time.
- jstimpfle 9y agoWell, the framework I was to use last (Qt) had bugs that simply can't be fixed by users (memory leaks, double free leading to segfault when exiting after reloading QML engine) and immature modules (for example translation) and forces complicated types to the user and forces bad architectural decisions to the user and significantly increases compile times... > using a tried and tested framework is likely to give you higher quality product in less time. This is a common sentiment, but note that a framework has huge handicaps - not knowing your business requirements and concepts - must be suitable for many software projects that need features you'll never need. - there is a clear maintenance boundary (framework vs your own code), which requires a complex interface with maintenance overhead and typically forces you to use concepts that don't really match your requirements If you're experienced in the relevant domain it's almost always simpler to do it yourself / reuse your own work.
- crdoconnor 9y agoI don't think reinventing the wheel is the right approach. I think careful analysis of bugs and design deficiencies the likes of which you experienced before jumping in and using the framework is the way to do it. This is why I absolutely love seeing hate filled developer rants about technology with deep dive analyses and links to bug trackers. That's how I was saved from learning Ruby on Rails back in 2008 when most developers gushed about how awesome it after reacting to its slick marketing and building a tiny website in 5 minutes.
- deleted 9y ago[deleted]
- astral303 9y agoContinued plain C usage is an embarrassment to the software industry and has cost the world untold money.
- onion2k 9y agoTime spent not doing our own designs (and instead spent memorizing how to use magical frameworks) is time not spent advancing our technical understanding. That's a false dichotomy. The time it takes to "memorize a magical framework" is far less than the time it'd take to learn how to write code to do what the framework does. Consequently you can learn a framework and some other technical understanding in the same time it'd take you to only learn enough to implement your own version of the framework. In most circumstances that's actually more beneficial to do that. You'll be further forward in your understanding of the technical stuff. You're also assuming that frameworks are written by individuals. They're not. In the case of some large frameworks it'd be practically impossible to implement what they cover on your own. You simply can't learn the underlying principles and then implement them all in code yourself. It's definitely worthwhile learning the basics of the languages you use, and you should be working on things that improve your code and understanding as much as possible, but it's very likely in most cases that will mean building on top of someone else's existing code rather than implementing everything yourself.
- jstimpfle 9y agoThese are still claims without any context. It mostly depends on how experienced the programmer is. And most importantly, note that one never needs all the functionality from a framework. Typically it's only a very small part, and often the existing functionality in the framework does not match the requirements 100%.
- damagednoob 9y agoI would think this claim is self-evident > In the case of some large frameworks it'd be practically impossible to implement what they cover on your own. You simply can't learn the underlying principles and then implement them all in code yourself. Not even DHH would claim to have been able to build the whole of Rails by himself.
- Xylakant 9y agoVery often the match is good enough that it pays off to slightly align the requirements instead of patching the framework. The amount by of man-hours poured into the very core parts of rails for example, just processing and dispatching requests, safely decoding the input from a webserver to a useful set of parameters, routing the request to the proper handler and rendering and returning the response is huge. Certainly, you could take something slightly more modular, such as padrino, but that’s still a mind-melting amount of code if you look at all the libraries and dependencies. You could reimplement most of the basics, but that would be month or years of work and probably still buggy as hell. I’ve seen my share of “oh, we’ll just build our own framework” and they all turned out to be much more complex than the initiatiator expected.
- collyw 9y agoI would disagree about that. Learning Django taught me a lot about the proper way to things. When I first started using it I implemented a lot of my own stuff myself (as I wasn't aware that the framework had certain features). I basically wrote my own equivalent of class based views before I understood Django's own. (http://ccbv.co.uk/ http://ccbv.co.uk/ helps understand them a lot) Reinventing Django's class based views is something that I have seen in a few inherited apps (like where I currently work). If your application is going to be something long lived and gets a number of developers while in maintenance mode, then memorizing a standard framework is a good thing - new developers should be familiar with how a framework works - as opposed to needing to spend time going through someone else home grown code. Then there is the issue that writing your own code will be untested relative to a framework that has some level of popularity. Documentation is likely to be better with a framework as well. (This is my perspective coming from a Python / Django background - I have noticed that JavaScript frameworks and libraries often have a lot more problems).
- SatvikBeri 9y agoRecently, I spent some time trying to run several machine learning jobs simultaneously across AWS machines. This was a fairly simple use case: all the jobs were totally independent of each other, you could run them by calling a Python function with different parameters. I'm mostly a stats guy and not much of a programmer. I got a hacky, do-it-myself version using Python scripts up and running with about two hours of work, and learned about Python threads for the first time in the process, a tool that I can reuse in many different areas. I then tried to do this the "right" way, which according to Amazon is to use Docker and the AWS tools ECR, Batch, and SQS. It took me about 10 times as long to get that working. Yes, this offers much, much more functionality - but most of it is stuff I didn't need. The only real gain I got was my models running about 20% faster, and the knowledge I learned is ephemeral.
- enraged_camel 9y agoSure, NIH syndrome can be rational for technology that is crucial/central to your system. The problem is it is often used to justify re-inventing even mundane stuff. I once worked with a client who wanted to implement their own bug tracking system. The client’s main product was something totally unrelated.
- marcosdumay 9y agoIf even mundane stuff keeps breaking your workflow and demands all of your time to keep up with, then all the power for reinventing it. All the better if it is simple.
- flukus 9y agoBug trackers are one of those areas where a lot of companies should build there own. Everyone has there own workflows and information to capture and end up either conforming their process to the bug tracker or spending more time configuring the bug tracker than they would to build a new system from scratch. Those uber configurable systems always suck to use. Ones like JIRA can takes weeks to setup for your org and include their own query language. All of this complexity just to do something so simple is not rational. As long as you don't over engineer it then it's only a days work to get something up and running and it's a great project for interns.
- ionforce 9y ago> Bug trackers are one of those areas where a lot of companies should build there own. This sounds insane. I have not worked at a place where the workflow was so holy and important that it couldn't be captured in a near default JIRA install. Most of the customization asks I've seen with JIRA come from dysfunctional organizations that demand new swimlanes like "QA" and "product approval" and "spec design".
- kckkdkd929292 9y agodont spend money building spend it on vendor lock-in i guess if you want to change the world with todo app that probably makes sense
- cryptonector 9y agoI don't think that's the whole picture. You could copy an interface (and even implementation) and fork it. But often with NIH we see a reinvention of a technology without even looking at alternatives; often this ends badly (in Linux land much more often than not it seems).
- jstimpfle 9y agoThe problem is that reading (and learning from) code is hard. The other problem is that there is so much bad code out there (most of my own is certainly not an exception) that it gets even harder. And it's not about the code. Writing code is easy. It's about finding the right problems first. And then it's about finding the right abstractions. The easiest way to build a clean conceptual world is to start with a clean slate and ask yourself before introducing new code, "does this code solve a concrete problem? Do I really need it?" Unix had simple and clear ideas, and it has many reimplementations (NIH!), and most are not that bad - are they?
- cottonseed 9y agoI also like to ask myself, "Do I want to become an expert at working with external software X, or do I want to become the kind of person who can build software like X?"
- an_account_name 9y agoThat's a great heuristic to keep in mind, thanks for that.
- mooreds 9y agoThat's great. However, how does the time to solve the business problems fit in?
- simonebrunozzi 9y agoI gave him mine on your behalf :)
- mooreds 9y agoWorth dropping a link to this Joel Spolsky article, where he discusses this concept and talks about the fact that the Excel team (in the 1990s) had their own c compiler: https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-invented-here-syndrome/ https://www.joelonsoftware.com/2001/10/14/in-defense-of-not-...
- protoplant 9y agoI am glad I read this just now, I was jumping from hoop to hoop. and not only hoops but craft paradigms, reminds me of the Fred Brooks idea "There is no silver bullet". Brooks argues that "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity." He also states that "we cannot expect ever to see two-fold gains every two years" in software development, as there is in hardware development (Moore's law) from wikipedia
- mooreds 9y agoGlad it helped. Brooks should be required reading for every developer: software is hard.