5 ms·

NIH is amazing as long as you are realistic about what you are taking ownership of. For example, the cost of maintaining a bespoke web frontend "framework" tha

NIH is amazing as long as you are realistic about what you are taking ownership of. For example, the cost of maintaining a bespoke web frontend "framework" that is specific to your problem domain is probably justifiable in many cases. The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc. If you have a problem so hard that no existing RDBMS or engine can support, you should seriously question the practicality of solving your problem at all. There's probably some theoretical constraint or view of the problem space you haven't discovered yet. Reorganizing the problem is a lot cheaper than reinventing the entire SQLite test suite from zero.

abbotcabbot 1y ago
> Reorganizing the problem is a lot cheaper than reinventing the entire SQLite test suite from zero. Sure, but if you aren't happy with existing DBs you are probably wrong in thinking you need a general DB instead of a file layout on a filesystem.
- AstralStorm 1y ago
  It seems easy, but actually managing files as a database safely is actually very hard. Especially ensuring atomicity. Portability is a problem as well. You do not need an SQL database however, there are others available.
  - abbotcabbot 1y ago
    Of course these questions are hard and if you are asking them you may need a general db. The whole industry is wrapped up in these bizarre ideas that for the slightest requirement they are going to add just one more abstraction layer and it will do everything and the kitchen sink. But sometimes the requirement is simple and doesn't deserve solutions to anticipated problems and other times it does and is worth looking for an existing solution. Rarely the problem is complex and deserves a custom solution and then one should probably make a new startup for the one yak shave, etc..
- bob1029 1y ago
  The technology is only part of the problem. If you aren't happy with any existing database, throwing even more technology at it is definitely not going to help - statistically speaking. You're up against millions of engineering hours in this space. At some point, we need to get up from the computer and go talk to the business and customer about their problem in terms that don't involve anything more sophisticated than excel spreadsheets. There is definitely some aspect you missed if you think you need to build a fully custom technology vertical to get them to pay for a solution. Of course, all of this advice is pointless if what you are doing is for amusement or as a hobby. I think a lot of arguments about technology would be diffused if we prefaced with our intended applications.
  - immibis 1y ago
    What I want is sqlite but in native Java.
  - sgarland 1y ago
    > You're up against millions of engineering hours in this space. Tangential, but the fact that many orgs choose to not use native DB features like foreign key constraints, saying that “they’ll handle it in application code,” has always seemed like the pinnacle of hubris to me. You’re going to ignore some of the most battle-tested pieces of technology, and recreate it poorly instead? Quite the decision. While there are valid reasons to not want to use FKCs, IME much of the time when you peel apart the layers of why someone chose not to, you find that they made several erroneous assumptions that led them to that conclusion. Other times, it’s that they simply didn’t know something existed, because they never read the docs. Windowing functions are a good example: RDBMS can do quite a bit of manipulation to data, rather than just sending huge result sets and having you parse them. There are trade-offs, of course, but you should at least know that it’s even possible before choosing to ignore it.
- dontlaugh 1y ago
  Using files instead of SQLite very much is NIH. Good luck not corrupting or losing your data.
  - sgarland 1y ago
    The Gang Learns About Fsync.
    - dontlaugh 1y ago
      If only that’s all it took. It’s certainly something you can invent yourself in a cross-platform way, sure.
      - abbotcabbot 1y ago
        I thought we were talking about NIH, H is for here. If you can't scope your requirements then obviously you need to import something a bit larger than the whole world to solve the whole world's problems.
f4c39012 1y ago
i can think of two reasons for using a third-party dependency 1) a dependency on a third-party service provider that publishes the dependency. So long as that service provider is current, the dependency should be maintained 2) short-cut to code i don't want to write I have no arguments with (1), there's a business reason and the lifecycles should match. However, I should still expect major version breaking changes in order to keep my application running. For (2), the wins are less clear, more dependenent on the perceived complexity of what I can avoid writing. Taking on any kind of dependency means that someone else can dictate when I need to spend time updating and testing changes that don't add anything to my product. Taking on a third-party dependency is always taking on a responsibility to maintain a codebase or the risk of not doing so.
- iforgotpassword 1y ago
  Most important reasons imo: 1) even though reality has proven us wrong time and time again, we can just not look at the dependency too closely and just act as if it's written and maintained by competent, caring people and is of highest quality. No worries! 2) in case shit hits the fan, let's assume worst case and there is a vuln in the dep and you get hacked... It's somebody else's fault! \o/
  - jpc0 1y ago
    > 2) in case shit hits the fan... It's somebody else's fault! A contrived example but, good luck explaining to the lawyers that openssl had this bug that caused all your customer data to leak and your company is being sued. If your motto for dependencies are “we can just not look at the dependency too closely and just act as if it's written and maintained by competent…” I’m reasonably sure someone is getting fired in that situation if it doesn’t sink the entire company. Move fast and break things isn’t exactly a viable strategy in all industries. Now as I said openssl was a contrived example but what if instead it was your ORM that didn’t use templated queries but rather just did string interpolation and there was an SQL injection attack. Considering this is still one if the most popular vulnerabilities on the web someone is messing stuff up somewhere and you are blindly like hoping it isn’t in the 10k lines of ORM library you pulled in instead.
thesz 1y ago
That RDBMS example of yours is fascinating. There are plenty of RDBMS here (wikipedia lists some 100+ of them), there are plenty of problems most of them can not solve, but some of them do solve. These people considered practicality of their solution and went forward doing the implementation.
- chha 1y ago
  Absolutely. A bunch of them also predated the www, or started just when the www was gaining popularity, meaning that information on possible products might be more limited than it is today. Some have narrow use-cases, and some might have started as really narrow in terms of scope, but then ballooned after gaining popularity, sales (and feature requests), and some probably started because the alternative was deemed too expensive compared to just making it in-house. I think the main point bob1029 was trying to make is that it can be worthwhile doing somehting in-house if the alternatives doesn't match the use-case, are too expensive or whatever else - but that you seriously need to consider if your architecture is the best way to solve the problem before going down that route.
DrScientist 1y ago
Why are there so many different database engines then? Is each instance of a different database out their an instance of NIH? The answer is of course that every computer system of any complexity has to make trade-offs - do you need constraints, do you need scalability, do you need concurrency, security? Is your data of a specific form that benefits from a particular compressible storage form, is it optimised for write once read many or read/write etc etc.. In terns of the thrust of the main article - which is about the cost of dependencies - I find that I'm typically much happier to take a dependency if that dependency is very keen on minimising it's own dependencies! - ie I don't want to take on a whole forest of stuff. In my view many automatic dependency management systems have helped create the mess that they claim to try and solve, and using careful manual dependency management can help cut down the burden.
- SoftTalker 1y ago
  > do you need constraints, do you need scalability, do you need concurrency, security Don't forget, do you need someone you can call for support (or throw under the bus) when problems come up?
  - theshrike79 1y ago
    This is the thing. If the database is essential to your business - as in if it goes down your revenue drops to zero - do you want to be the one debugging it at 3am? Or would it be more cost effective to have a well-known database or even a managed DB
    - DrScientist 1y ago
      If you are running into a problem that is bringing your site down and it's truely a problem/bug with a commercial database then you are in trouble and unlikely to see a quick fix as these products are complicated and while the problem might be existential to you, it's not to the bigger vendor. However, it's much more likely it's a configuration issue - ie you didn't understand the product properly and have configured it incorrectly - and here support can really help. The point I'm making above is not that you should be writing fully fledged all singing and dancing databases - but that you might be able to get away with something very simple, more reliable and faster in your specific use case. If on the other hand your use case is complex and a good fit for something pre-existing - then sure you shouldn't be writing it yourself.
marginalia_nu 1y ago
You fairly quickly run into problems a RDBMS can't solve. They're above all incredibly hamstringed by being built around supporting concurrent mutation of the data set and mutable datasets in general. If you don't need that, even a fairly naive index implementation will get orders of magnitude more performance out of a computer than you will with a RDBMS if you can assume the data is immutable.
- SoftTalker 1y ago
  I'd tend to argue the opposite. Unless you have a problem that very explicitly is a poor match for an RDBMS, they are the easiest, most reliable, and most flexible way to manage data, separate your data from your application, and adapt or add functionality as new requirements come along.
account42 1y ago
> The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc. It absolutely can in many cases. - Many applications don't need much more of a database than files plus perhaps a runtime index. Spawning a database server or even embedding SQLite can be overkill in many cases. - Most games would be better off with a bespoke game engine IMO, especially those made by small teams. The only real advantage to using an established engine is the familiar asset pipeline for replaceable artists but you pay for that with a huge overhead. Remember that custom engines or at least heavily modified ones maintained by the game developer used to be the norm - "just use Unity^WUnreal" is a relatively recent trend. - There is no real difference in complexity between a simple web server and FastCGI application. - Not all uses of cryptographic primitives are a security issues. If you calculate a hash to check against corruption you can implement that yourself (or copy an existing implementation) instead of pulling in a huge library. If your use cases is decrypting existing data offline then again you don't really care about cryptographic security only about getting the correctly decrypted output. And there are many more cases like these. Learned helplessness when it comes to "hard" subjects helps no one. Just because some existing solution "solves" your problem doesn't mean its the best or most efficient way to do it.
- Cthulhu_ 1y ago
  > files plus perhaps a runtime index. The latter already adds a lot of complexity, especially if the application has to scale up or out. If scaling up, the index will increase in complexity and re-implement some clever algorithms that have already been solved by the other. If scaling out, you'll need some locking mechanism if multiple processes can write. It quickly becomes logical to switch to an established solution then. But the choice becomes more logical then; if one picks a scalable database or whatever without knowing they're going to need it, that's a lot of wasted effort. > - Most games would be better off with a bespoke game engine IMO, especially those made by small teams. I disagree; writing a game engine, even for simpler "indie" graphics, requires specialized knowledge and a dedicated engineer, whereas with off-the-shelf engines, non-programmer / creatives can build a good game without in-depth knowledge. A small team needs to be especially careful with what they choose to spend their time and money on, and solving a solved problem like building a game engine is a huge investment. Even big studios struggle with this, see also the dramatic release of Cyberpunk 20something by long-time own-engine-building veterans CDPR. They're switching to Unreal for Witcher 4.
michaelcampbell 1y ago
> as long as you are realistic about what you are taking ownership of Including training and onboarding. There's a reason "popular" libraries, languages, patterns, etc. are popular. One "win" is the network effect when looking for talent - they can hit the ground at least jogging.
a3w 1y ago
What is NIH? Skimmed the article. Still don't understand, after doing so twice. Google says national institute of health.
- deleted 1y ago
  [deleted]
- sgarland 1y ago
  Not Invented Here; the desire to always build your own rather than buy / borrow existing tools.

abbotcabbot 1y ago

> Reorganizing the problem is a lot cheaper than reinventing the entire SQLite test suite from zero. Sure, but if you aren't happy with existing DBs you are probably wrong in thinking you need a general DB instead of a file layout on a filesystem.

AstralStorm 1y ago
It seems easy, but actually managing files as a database safely is actually very hard. Especially ensuring atomicity. Portability is a problem as well. You do not need an SQL database however, there are others available.
- abbotcabbot 1y ago
  Of course these questions are hard and if you are asking them you may need a general db. The whole industry is wrapped up in these bizarre ideas that for the slightest requirement they are going to add just one more abstraction layer and it will do everything and the kitchen sink. But sometimes the requirement is simple and doesn't deserve solutions to anticipated problems and other times it does and is worth looking for an existing solution. Rarely the problem is complex and deserves a custom solution and then one should probably make a new startup for the one yak shave, etc..
bob1029 1y ago
The technology is only part of the problem. If you aren't happy with any existing database, throwing even more technology at it is definitely not going to help - statistically speaking. You're up against millions of engineering hours in this space. At some point, we need to get up from the computer and go talk to the business and customer about their problem in terms that don't involve anything more sophisticated than excel spreadsheets. There is definitely some aspect you missed if you think you need to build a fully custom technology vertical to get them to pay for a solution. Of course, all of this advice is pointless if what you are doing is for amusement or as a hobby. I think a lot of arguments about technology would be diffused if we prefaced with our intended applications.
- immibis 1y ago
  What I want is sqlite but in native Java.
- sgarland 1y ago
  > You're up against millions of engineering hours in this space. Tangential, but the fact that many orgs choose to not use native DB features like foreign key constraints, saying that “they’ll handle it in application code,” has always seemed like the pinnacle of hubris to me. You’re going to ignore some of the most battle-tested pieces of technology, and recreate it poorly instead? Quite the decision. While there are valid reasons to not want to use FKCs, IME much of the time when you peel apart the layers of why someone chose not to, you find that they made several erroneous assumptions that led them to that conclusion. Other times, it’s that they simply didn’t know something existed, because they never read the docs. Windowing functions are a good example: RDBMS can do quite a bit of manipulation to data, rather than just sending huge result sets and having you parse them. There are trade-offs, of course, but you should at least know that it’s even possible before choosing to ignore it.
dontlaugh 1y ago
Using files instead of SQLite very much is NIH. Good luck not corrupting or losing your data.
- sgarland 1y ago
  The Gang Learns About Fsync.
  - dontlaugh 1y ago
    If only that’s all it took. It’s certainly something you can invent yourself in a cross-platform way, sure.
    - abbotcabbot 1y ago
      I thought we were talking about NIH, H is for here. If you can't scope your requirements then obviously you need to import something a bit larger than the whole world to solve the whole world's problems.

f4c39012 1y ago

i can think of two reasons for using a third-party dependency 1) a dependency on a third-party service provider that publishes the dependency. So long as that service provider is current, the dependency should be maintained 2) short-cut to code i don't want to write I have no arguments with (1), there's a business reason and the lifecycles should match. However, I should still expect major version breaking changes in order to keep my application running. For (2), the wins are less clear, more dependenent on the perceived complexity of what I can avoid writing. Taking on any kind of dependency means that someone else can dictate when I need to spend time updating and testing changes that don't add anything to my product. Taking on a third-party dependency is always taking on a responsibility to maintain a codebase or the risk of not doing so.

iforgotpassword 1y ago
Most important reasons imo: 1) even though reality has proven us wrong time and time again, we can just not look at the dependency too closely and just act as if it's written and maintained by competent, caring people and is of highest quality. No worries! 2) in case shit hits the fan, let's assume worst case and there is a vuln in the dep and you get hacked... It's somebody else's fault! \o/
- jpc0 1y ago
  > 2) in case shit hits the fan... It's somebody else's fault! A contrived example but, good luck explaining to the lawyers that openssl had this bug that caused all your customer data to leak and your company is being sued. If your motto for dependencies are “we can just not look at the dependency too closely and just act as if it's written and maintained by competent…” I’m reasonably sure someone is getting fired in that situation if it doesn’t sink the entire company. Move fast and break things isn’t exactly a viable strategy in all industries. Now as I said openssl was a contrived example but what if instead it was your ORM that didn’t use templated queries but rather just did string interpolation and there was an SQL injection attack. Considering this is still one if the most popular vulnerabilities on the web someone is messing stuff up somewhere and you are blindly like hoping it isn’t in the 10k lines of ORM library you pulled in instead.

thesz 1y ago

That RDBMS example of yours is fascinating. There are plenty of RDBMS here (wikipedia lists some 100+ of them), there are plenty of problems most of them can not solve, but some of them do solve. These people considered practicality of their solution and went forward doing the implementation.

chha 1y ago
Absolutely. A bunch of them also predated the www, or started just when the www was gaining popularity, meaning that information on possible products might be more limited than it is today. Some have narrow use-cases, and some might have started as really narrow in terms of scope, but then ballooned after gaining popularity, sales (and feature requests), and some probably started because the alternative was deemed too expensive compared to just making it in-house. I think the main point bob1029 was trying to make is that it can be worthwhile doing somehting in-house if the alternatives doesn't match the use-case, are too expensive or whatever else - but that you seriously need to consider if your architecture is the best way to solve the problem before going down that route.

DrScientist 1y ago

Why are there so many different database engines then? Is each instance of a different database out their an instance of NIH? The answer is of course that every computer system of any complexity has to make trade-offs - do you need constraints, do you need scalability, do you need concurrency, security? Is your data of a specific form that benefits from a particular compressible storage form, is it optimised for write once read many or read/write etc etc.. In terns of the thrust of the main article - which is about the cost of dependencies - I find that I'm typically much happier to take a dependency if that dependency is very keen on minimising it's own dependencies! - ie I don't want to take on a whole forest of stuff. In my view many automatic dependency management systems have helped create the mess that they claim to try and solve, and using careful manual dependency management can help cut down the burden.

SoftTalker 1y ago
> do you need constraints, do you need scalability, do you need concurrency, security Don't forget, do you need someone you can call for support (or throw under the bus) when problems come up?
- theshrike79 1y ago
  This is the thing. If the database is essential to your business - as in if it goes down your revenue drops to zero - do you want to be the one debugging it at 3am? Or would it be more cost effective to have a well-known database or even a managed DB
  - DrScientist 1y ago
    If you are running into a problem that is bringing your site down and it's truely a problem/bug with a commercial database then you are in trouble and unlikely to see a quick fix as these products are complicated and while the problem might be existential to you, it's not to the bigger vendor. However, it's much more likely it's a configuration issue - ie you didn't understand the product properly and have configured it incorrectly - and here support can really help. The point I'm making above is not that you should be writing fully fledged all singing and dancing databases - but that you might be able to get away with something very simple, more reliable and faster in your specific use case. If on the other hand your use case is complex and a good fit for something pre-existing - then sure you shouldn't be writing it yourself.

marginalia_nu 1y ago

You fairly quickly run into problems a RDBMS can't solve. They're above all incredibly hamstringed by being built around supporting concurrent mutation of the data set and mutable datasets in general. If you don't need that, even a fairly naive index implementation will get orders of magnitude more performance out of a computer than you will with a RDBMS if you can assume the data is immutable.

SoftTalker 1y ago
I'd tend to argue the opposite. Unless you have a problem that very explicitly is a poor match for an RDBMS, they are the easiest, most reliable, and most flexible way to manage data, separate your data from your application, and adapt or add functionality as new requirements come along.

account42 1y ago

> The same cannot be said for databases, game engines, web servers, cryptographic primitives, etc. It absolutely can in many cases. - Many applications don't need much more of a database than files plus perhaps a runtime index. Spawning a database server or even embedding SQLite can be overkill in many cases. - Most games would be better off with a bespoke game engine IMO, especially those made by small teams. The only real advantage to using an established engine is the familiar asset pipeline for replaceable artists but you pay for that with a huge overhead. Remember that custom engines or at least heavily modified ones maintained by the game developer used to be the norm - "just use Unity^WUnreal" is a relatively recent trend. - There is no real difference in complexity between a simple web server and FastCGI application. - Not all uses of cryptographic primitives are a security issues. If you calculate a hash to check against corruption you can implement that yourself (or copy an existing implementation) instead of pulling in a huge library. If your use cases is decrypting existing data offline then again you don't really care about cryptographic security only about getting the correctly decrypted output. And there are many more cases like these. Learned helplessness when it comes to "hard" subjects helps no one. Just because some existing solution "solves" your problem doesn't mean its the best or most efficient way to do it.

Cthulhu_ 1y ago
> files plus perhaps a runtime index. The latter already adds a lot of complexity, especially if the application has to scale up or out. If scaling up, the index will increase in complexity and re-implement some clever algorithms that have already been solved by the other. If scaling out, you'll need some locking mechanism if multiple processes can write. It quickly becomes logical to switch to an established solution then. But the choice becomes more logical then; if one picks a scalable database or whatever without knowing they're going to need it, that's a lot of wasted effort. > - Most games would be better off with a bespoke game engine IMO, especially those made by small teams. I disagree; writing a game engine, even for simpler "indie" graphics, requires specialized knowledge and a dedicated engineer, whereas with off-the-shelf engines, non-programmer / creatives can build a good game without in-depth knowledge. A small team needs to be especially careful with what they choose to spend their time and money on, and solving a solved problem like building a game engine is a huge investment. Even big studios struggle with this, see also the dramatic release of Cyberpunk 20something by long-time own-engine-building veterans CDPR. They're switching to Unreal for Witcher 4.

michaelcampbell 1y ago

> as long as you are realistic about what you are taking ownership of Including training and onboarding. There's a reason "popular" libraries, languages, patterns, etc. are popular. One "win" is the network effect when looking for talent - they can hit the ground at least jogging.

a3w 1y ago

What is NIH? Skimmed the article. Still don't understand, after doing so twice. Google says national institute of health.

deleted 1y ago
[deleted]
sgarland 1y ago
Not Invented Here; the desire to always build your own rather than buy / borrow existing tools.