To me the primary thing that matters is how strategic the software is to your core business. This may sound like just evaluating whether it's your core business

To me the primary thing that matters is how strategic the software is to your core business. This may sound like just evaluating whether it's your core business or not, but it's a nuance on that which is important. It's easiest described by an example: we use open source Gitlab extensively. Numerous times it has been proposed that we migrate to $expensive alternative because they all have advanced functionality that we don't get from Gitlab (in many cases, even the enterprise version). Yet we are sticking with Gitlab, and making extensive customisations (mostly using the API) and workflows adapted to it. Is our core business developing software for CI/CD/Issue tracking/source code management? Absolutely not. But is that critical to us? Absolutely. Because Gitlab is open source and because it has an extensive API, we can eek out efficiencies all over the place that aren't possible with closed source software. So I think it's really important that people use the right frame of mind when evaluating "core business" or not. You don't have to be in the business of a selling particular type of software for it to be worth doing "in-house". You do have to be good at it though.