3 ms·

That might be true about some pharma software, but that does not mean there's no appetite for innovation. A lot of the computational work is done with decades o

That might be true about some pharma software, but that does not mean there's no appetite for innovation. A lot of the computational work is done with decades old Fortran libraries, plugged into R (yup, no Python in this world) and spreadsheets. It is very hard to upgrade these old Fortran libraries to leverage capabilities of modern numerical stacks. The reason why companies have to hold on to their SAS and COBOL is that once a drug is out in the real world, and there is a lawsuit, or a recall - you need to be able to have everything available for review. These things can happen 20 or 30 years after a drug becomes available. That would naturally induce risk aversion. New drugs today will use newer tools, and 30 years out, we'll be saying the same things about today's tools. In fact pharma companies have no choice but to adapt. Technology moves fast, and anything that gives an edge, if not adopted, will leave you behind.

fock 6y ago
What are these old Fortran libraries doing, which requires "new numerical stacks"? From my naive point of view, all what you are describing should be trivially done on a workstation. As I'm sure that's probably me underestimating the field: can you point me to some algorithm/technique which would benefit from new numerical stacks? Also is your stack open source? I somehow doubt the long-term advantage of one proprietary stack over another.
- DrScientist 6y ago
  Winnonlin is one such tool - been around forever, written in fortran. One of the reasons it is still popular is for regulatory use it's very well understood. I'm not enough of an expert to answer the second part of your question - on why new numerical stacks are beneficial. It may be around building more complex models of PK/PD but I'm not up to speed on what winnonlin can and can't do.
- ChrisRackauckas 6y ago
  Pharmacometrics is built around small stiff ODE models. You might think that this might mean it's "simple" computationally, but what it actually means is that every little detail matters a lot for performance. The way you do lu-factorization for 4 ODEs vs 6 ODEs needs to change in an architecture-dependent fashion, and BLAS libraries are not efficient in this range. Standard floating point exponentiation can be too expensive in some spots, while at the same time the standard stiff ODE solvers make assumptions about regularity which are regularly violated by pharmacometric models (due to how dosing works). There is more than a little bit of "by-hand SIMD" in this software stack. Specializing on all of these features is an interplay between algorithms and JIT compilation, where not utilizing statically compiled optimizations will hurt you at this size. We see this in benchmarks like the SciML Hires [1] which is able to outperform the classic Fortran libraries like LSODA with newly developed Rodas methods. As a less direct and more illustrative example, these benchmarks vs PyTorch [2] demonstrate what happens when you compare against an optimizing JIT compiler which doesn't specialize on small sized interactions (>30x performance difference!), showing how this kind of application is very much not in the regime of large kernels which a lot of recent compiler work tends to optimize for and instead heavily relies on being able to cross-compile and inline computations to remove every little overhead, all while improving the ODE algorithms themselves. We will have a paper that goes into more detail on this fairly soon. [1] https://benchmarks.sciml.ai/html/StiffODE/Hires.html [2] https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e0584a2bfd320
  - fock 6y ago
    Thanks, super interesting. So it's really high throughput for simple stuff (and probably people don't mind that it's too slow).
    - ChrisRackauckas 6y ago
      Like any big computation, it's lots of simple stuff smashed together, so if the simple stuff is slightly slow it can cost a lot. Nonlinear mixed effects model fitting with a few thousand patients (like in a later stage clinical trial) can take 2 weeks to one month to run. Being slow at this stage of the computation can delay clinical trials. Because of the extreme cost of these trials, people very much care if it's too slow.

fock 6y ago

What are these old Fortran libraries doing, which requires "new numerical stacks"? From my naive point of view, all what you are describing should be trivially done on a workstation. As I'm sure that's probably me underestimating the field: can you point me to some algorithm/technique which would benefit from new numerical stacks? Also is your stack open source? I somehow doubt the long-term advantage of one proprietary stack over another.

DrScientist 6y ago
Winnonlin is one such tool - been around forever, written in fortran. One of the reasons it is still popular is for regulatory use it's very well understood. I'm not enough of an expert to answer the second part of your question - on why new numerical stacks are beneficial. It may be around building more complex models of PK/PD but I'm not up to speed on what winnonlin can and can't do.
ChrisRackauckas 6y ago
Pharmacometrics is built around small stiff ODE models. You might think that this might mean it's "simple" computationally, but what it actually means is that every little detail matters a lot for performance. The way you do lu-factorization for 4 ODEs vs 6 ODEs needs to change in an architecture-dependent fashion, and BLAS libraries are not efficient in this range. Standard floating point exponentiation can be too expensive in some spots, while at the same time the standard stiff ODE solvers make assumptions about regularity which are regularly violated by pharmacometric models (due to how dosing works). There is more than a little bit of "by-hand SIMD" in this software stack. Specializing on all of these features is an interplay between algorithms and JIT compilation, where not utilizing statically compiled optimizations will hurt you at this size. We see this in benchmarks like the SciML Hires [1] which is able to outperform the classic Fortran libraries like LSODA with newly developed Rodas methods. As a less direct and more illustrative example, these benchmarks vs PyTorch [2] demonstrate what happens when you compare against an optimizing JIT compiler which doesn't specialize on small sized interactions (>30x performance difference!), showing how this kind of application is very much not in the regime of large kernels which a lot of recent compiler work tends to optimize for and instead heavily relies on being able to cross-compile and inline computations to remove every little overhead, all while improving the ODE algorithms themselves. We will have a paper that goes into more detail on this fairly soon. [1] https://benchmarks.sciml.ai/html/StiffODE/Hires.html [2] https://gist.github.com/ChrisRackauckas/cc6ac746e2dfd285c28e0584a2bfd320
- fock 6y ago
  Thanks, super interesting. So it's really high throughput for simple stuff (and probably people don't mind that it's too slow).
  - ChrisRackauckas 6y ago
    Like any big computation, it's lots of simple stuff smashed together, so if the simple stuff is slightly slow it can cost a lot. Nonlinear mixed effects model fitting with a few thousand patients (like in a later stage clinical trial) can take 2 weeks to one month to run. Being slow at this stage of the computation can delay clinical trials. Because of the extreme cost of these trials, people very much care if it's too slow.