I'm assuming "this domain" refers to scientific computing. My theory is that the typical scientist crunching data is just proficient enough to write specialized

I'm assuming "this domain" refers to scientific computing. My theory is that the typical scientist crunching data is just proficient enough to write specialized code for themselves but doesn't have time to develop a scalable, "professional" solution. Very large organizations tend to have capable internal systems but they may not make it into public view or they are too specific to their niche. On the other hand you have premade, proprietary packages that can become quite popular (Matlab, JMP, etc.) for non-programmers, which dilutes the user base. That leaves us with things like "R" (horrible, like you say) or Julia (eternally promising) or Python (stitch together low-level Fortran and C code). The latter solution is winning, it seems.