5 ms·

It seems like the author isn't really defending against the bulk of the criticisms, which revolve around correctness and reliability, and instead highlights 'co

by t_mann 4y ago

It seems like the author isn't really defending against the bulk of the criticisms, which revolve around correctness and reliability, and instead highlights 'cool' language features. If you can't trust gradients or probability distributions / sampling to be correct, that's a pretty damning verdict for a language aimed at numerical calculations. The fact that it has some traction in academia (where the link from incorrect computation results to real-world consequences is extremely long-winded and hard to attribute) is worrying from that perspective. Imagine Boeing were to use Julia for its computations (not unlikely given that it has a great repuation for differential equation solvers).

dekhn 4y ago
I'm sure Boeing uses Julia somewhere for its computations... and they have extensive testing to support that. I recall when the PR folks at SAS, a statistical tool, said that nobody would trust a Boeing plane that had been developed using R (an open source alternative to SAS), and Boeing replied that they used R to design airplanes.
- t_mann 4y ago
  Well, it's also not like Boeing didn't have some major engineering f*ups in recent years, but we're going on a tangent here.
- notagoodidea 4y ago
  Besides that the SAS vs R is something still ongoing. I heard the aame point made in favour of SAS in Public Health, Clinical trials analysis and set up and different public administration. They trust the algorithms provided by a closed source certified vendors and not an open source alternative. At the same time, correct implementation of statistics is hard and testing it to be correct is harder.
  - dekhn 4y ago
    the story you heard about health is exactly backward. I had also heard that FDA approvals depended on SAS, but I spoke more recently with somebody who does this and it turns out both the submittors and the FDA use R, not SAS. There have definitely been communities of stats people who "trust" only certain codes or routines, but generally, the trend has been towards open implementations of the code, as it allows more experts to inspect the workings when they detect problems.
    - HenrikB 4y ago
      There are working groups under the R Consortium umbrella that work on FDA-related tasks with a focus on R [0]. Folks from the pharma industry and FDA are involved. One of the working groups is dedicated toward submissions: R Submissions Working Group [1]. They recently announced: "The R Consortium is happy to announce that on Nov 22nd, 2021, the R Submissions Working Group successfully submitted an R-based test submission package through the FDA eCTD gateway! The submission package has been received by the FDA staff who were able to reproduce the numerical results." [2] [0] https://www.r-consortium.org/all-projects/isc-working-groups https://www.r-consortium.org/all-projects/isc-working-groups [1] https://rconsortium.github.io/submissions-wg/ https://rconsortium.github.io/submissions-wg/ [2] https://www.r-consortium.org/blog/2021/12/08/successful-r-based-test-package-submitted-to-fda https://www.r-consortium.org/blog/2021/12/08/successful-r-ba...
    - notagoodidea 4y ago
      Preach but I should have added than my experience was not in the US but in Europe.
gugagore 4y ago
> If you can't trust gradients or probability distributions / sampling to be correct, that's a pretty damning verdict for a language aimed at numerical calculations. If you draw the distinction between Julia the language and Julia the ecosystem, then that's not a damning verdict for the language, even if we accept the premise that you can't trust the computations you mentioned. The way that Julia composes is unprecedented. Typical examples include computing with measurement uncertainty, or big floats through functions that were not explicitly designed to support those types. This is great, but the correctness of this composition cannot be assumed. One good proxy for how this composition occurs is in many type combinations. As Here is a concrete, made-up example. There is a function you've been using without any problem that takes an argument of type `DenseArray{Float64, 2}`. (Meaning, the function gets called with an argument of that type. The function definition is not constrained to work on that type.) If you've never focused on the correctness (e.g. reviewing the code, unit tests) of this function to see what it does on `SparseArray{BigFloat, 2}`, I would not take it for granted that it works. This is perhaps too big a burden on a user. Some might say they rather not have the compositionality unless it's guaranteed to be correct. But, if you're actually asking me to imagine using Julia in a situation where I really want to be sure about the correctness of a Julia program, I would at the very attempt to characterize all the expected types to go into any function. If at runtime a function is called with a new type, I would log that, and possibly error out. Note this is related to, but not equivalent to what's necessary to do "static" compilation in Julia.
- ModernMech 4y ago
  > If you draw the distinction between Julia the language and Julia the ecosystem Language and ecosystem are so intertwined that I almost never do this. Maybe when there are multiple well developed ecosystems, but when there’s only one, then it and the language are inseparable to me (for any language, not just Julia).
  - gugagore 4y ago
    I wonder if you feel that way as equally about C and Lisp as you do about Julia.
ChrisRackauckas 4y ago
Can you point to a concrete example of one that someone would run into when using the differential equation solvers with the default and recommended Enzyme AD for vector-Jacobian products? I'd be happy to look into it, but there do not currently seem to be any correctness issues in the Enzyme issue tracker that are current (3 issues are open but they all seem to be fixed, other than https://github.com/EnzymeAD/Enzyme.jl/issues/278 https://github.com/EnzymeAD/Enzyme.jl/issues/278 which is actually an activity analysis bug in LLVM). So please be more specific. The issue with Enzyme right now seems to moreso be about finding functional forms that compile, and it throws compile-time errors in the event that it cannot fully analyze the program and if it has too much dynamic behavior (example: https://github.com/EnzymeAD/Enzyme.jl/issues/368 https://github.com/EnzymeAD/Enzyme.jl/issues/368). Additional note, we recently did a overhaul of SciMLSensitivity (https://sensitivity.sciml.ai/dev/ https://sensitivity.sciml.ai/dev/, as part of the new https://docs.sciml.ai/dev/ https://docs.sciml.ai/dev/) and setup a system which amounts to 15 hours of direct unit tests doing a combinatoric check of arguments with 4 hours of downstream testing (https://github.com/SciML/SciMLSensitivity.jl/actions/runs/2563251123 https://github.com/SciML/SciMLSensitivity.jl/actions/runs/25...). What that identified is that any remaining issues that can arise are due to the implicit parameters mechanism in Zygote (Zygote.params). To counteract this upstream issue, we (a) try to default to never default to Zygote VJPs whenever we can avoid it (hence defaulting to Enzyme and ReverseDiff first as previously mentioned), and (b) put in a mechanism for early error throwing if Zygote hits any not implemented derivative case with an explicit error message (https://github.com/SciML/SciMLSensitivity.jl/blob/v7.0.1/src/derivative_wrappers.jl#L474-L500 https://github.com/SciML/SciMLSensitivity.jl/blob/v7.0.1/src...). We have alerted the devs of the machine learning libraries, and from this there has been a lot of movement. In particular, a globals-free machine learning library, Lux.jl, was created with fully explicit parameters https://lux.csail.mit.edu/dev/ https://lux.csail.mit.edu/dev/, and thus by design it cannot have this issue. In addition, the Flux.jl library itself is looking to do a redesign that eliminates implicit parameters (https://github.com/FluxML/Flux.jl/issues/1986 https://github.com/FluxML/Flux.jl/issues/1986). Which design will be the one in the end, that's uncertain right now, but it's clear that no matter what the future designs of the deep learning libraries will fully cut out that part of Zygote.jl. And additionally, the other AD libraries (Enzyme and Diffractor for example) do not have this "feature", so it's an issue that can only arise from a specific (not recommended) way of using Zygote (which now throws explicit error messages early and often if used anywhere near SciML because I don't tolerate it). tl;dr: don't use Zygote.params, the ML AD devs know this its usage is being removed ASAP. So from this, SciML should be rather safe and if not, please share some details and I'd be happy to dig in.
ViralBShah 4y ago
A detailed discussion is on Julia discourse: https://discourse.julialang.org/t/discussion-on-why-i-no-longer-recommend-julia-by-yuri-vishnevsky/81151 https://discourse.julialang.org/t/discussion-on-why-i-no-lon... Bugs are everywhere - including Python and R libraries. You would not design a safety critical system in Python using libraries and codepaths that are not heavily tested. Even then, you still have to carefully design testsuites that give you confidence that what you are computing is what is expected. See these talks on how a collision avoidance system was designed with Julia. https://www.youtube.com/watch?v=rj-WhTL_VXE https://www.youtube.com/watch?v=rj-WhTL_VXE https://www.youtube.com/watch?v=19zm1Fn0S9M https://www.youtube.com/watch?v=19zm1Fn0S9M
leke 4y ago
Just wondering...If gradients or probability distributions are an issue for Julia, can't they just be fixed some day in the future?
- gugagore 4y ago
  Yes, those fixes don't inherently require the Julia language to change at all.

by t_mann 4y ago

dekhn 4y ago

I'm sure Boeing uses Julia somewhere for its computations... and they have extensive testing to support that. I recall when the PR folks at SAS, a statistical tool, said that nobody would trust a Boeing plane that had been developed using R (an open source alternative to SAS), and Boeing replied that they used R to design airplanes.

t_mann 4y ago
Well, it's also not like Boeing didn't have some major engineering f*ups in recent years, but we're going on a tangent here.
notagoodidea 4y ago
Besides that the SAS vs R is something still ongoing. I heard the aame point made in favour of SAS in Public Health, Clinical trials analysis and set up and different public administration. They trust the algorithms provided by a closed source certified vendors and not an open source alternative. At the same time, correct implementation of statistics is hard and testing it to be correct is harder.
- dekhn 4y ago
  the story you heard about health is exactly backward. I had also heard that FDA approvals depended on SAS, but I spoke more recently with somebody who does this and it turns out both the submittors and the FDA use R, not SAS. There have definitely been communities of stats people who "trust" only certain codes or routines, but generally, the trend has been towards open implementations of the code, as it allows more experts to inspect the workings when they detect problems.
  - HenrikB 4y ago
    There are working groups under the R Consortium umbrella that work on FDA-related tasks with a focus on R [0]. Folks from the pharma industry and FDA are involved. One of the working groups is dedicated toward submissions: R Submissions Working Group [1]. They recently announced: "The R Consortium is happy to announce that on Nov 22nd, 2021, the R Submissions Working Group successfully submitted an R-based test submission package through the FDA eCTD gateway! The submission package has been received by the FDA staff who were able to reproduce the numerical results." [2] [0] https://www.r-consortium.org/all-projects/isc-working-groups https://www.r-consortium.org/all-projects/isc-working-groups [1] https://rconsortium.github.io/submissions-wg/ https://rconsortium.github.io/submissions-wg/ [2] https://www.r-consortium.org/blog/2021/12/08/successful-r-based-test-package-submitted-to-fda https://www.r-consortium.org/blog/2021/12/08/successful-r-ba...
  - notagoodidea 4y ago
    Preach but I should have added than my experience was not in the US but in Europe.

gugagore 4y ago

> If you can't trust gradients or probability distributions / sampling to be correct, that's a pretty damning verdict for a language aimed at numerical calculations. If you draw the distinction between Julia the language and Julia the ecosystem, then that's not a damning verdict for the language, even if we accept the premise that you can't trust the computations you mentioned. The way that Julia composes is unprecedented. Typical examples include computing with measurement uncertainty, or big floats through functions that were not explicitly designed to support those types. This is great, but the correctness of this composition cannot be assumed. One good proxy for how this composition occurs is in many type combinations. As Here is a concrete, made-up example. There is a function you've been using without any problem that takes an argument of type `DenseArray{Float64, 2}`. (Meaning, the function gets called with an argument of that type. The function definition is not constrained to work on that type.) If you've never focused on the correctness (e.g. reviewing the code, unit tests) of this function to see what it does on `SparseArray{BigFloat, 2}`, I would not take it for granted that it works. This is perhaps too big a burden on a user. Some might say they rather not have the compositionality unless it's guaranteed to be correct. But, if you're actually asking me to imagine using Julia in a situation where I really want to be sure about the correctness of a Julia program, I would at the very attempt to characterize all the expected types to go into any function. If at runtime a function is called with a new type, I would log that, and possibly error out. Note this is related to, but not equivalent to what's necessary to do "static" compilation in Julia.

ModernMech 4y ago
> If you draw the distinction between Julia the language and Julia the ecosystem Language and ecosystem are so intertwined that I almost never do this. Maybe when there are multiple well developed ecosystems, but when there’s only one, then it and the language are inseparable to me (for any language, not just Julia).
- gugagore 4y ago
  I wonder if you feel that way as equally about C and Lisp as you do about Julia.

ChrisRackauckas 4y ago

Can you point to a concrete example of one that someone would run into when using the differential equation solvers with the default and recommended Enzyme AD for vector-Jacobian products? I'd be happy to look into it, but there do not currently seem to be any correctness issues in the Enzyme issue tracker that are current (3 issues are open but they all seem to be fixed, other than https://github.com/EnzymeAD/Enzyme.jl/issues/278 https://github.com/EnzymeAD/Enzyme.jl/issues/278 which is actually an activity analysis bug in LLVM). So please be more specific. The issue with Enzyme right now seems to moreso be about finding functional forms that compile, and it throws compile-time errors in the event that it cannot fully analyze the program and if it has too much dynamic behavior (example: https://github.com/EnzymeAD/Enzyme.jl/issues/368 https://github.com/EnzymeAD/Enzyme.jl/issues/368). Additional note, we recently did a overhaul of SciMLSensitivity (https://sensitivity.sciml.ai/dev/ https://sensitivity.sciml.ai/dev/, as part of the new https://docs.sciml.ai/dev/ https://docs.sciml.ai/dev/) and setup a system which amounts to 15 hours of direct unit tests doing a combinatoric check of arguments with 4 hours of downstream testing (https://github.com/SciML/SciMLSensitivity.jl/actions/runs/2563251123 https://github.com/SciML/SciMLSensitivity.jl/actions/runs/25...). What that identified is that any remaining issues that can arise are due to the implicit parameters mechanism in Zygote (Zygote.params). To counteract this upstream issue, we (a) try to default to never default to Zygote VJPs whenever we can avoid it (hence defaulting to Enzyme and ReverseDiff first as previously mentioned), and (b) put in a mechanism for early error throwing if Zygote hits any not implemented derivative case with an explicit error message (https://github.com/SciML/SciMLSensitivity.jl/blob/v7.0.1/src/derivative_wrappers.jl#L474-L500 https://github.com/SciML/SciMLSensitivity.jl/blob/v7.0.1/src...). We have alerted the devs of the machine learning libraries, and from this there has been a lot of movement. In particular, a globals-free machine learning library, Lux.jl, was created with fully explicit parameters https://lux.csail.mit.edu/dev/ https://lux.csail.mit.edu/dev/, and thus by design it cannot have this issue. In addition, the Flux.jl library itself is looking to do a redesign that eliminates implicit parameters (https://github.com/FluxML/Flux.jl/issues/1986 https://github.com/FluxML/Flux.jl/issues/1986). Which design will be the one in the end, that's uncertain right now, but it's clear that no matter what the future designs of the deep learning libraries will fully cut out that part of Zygote.jl. And additionally, the other AD libraries (Enzyme and Diffractor for example) do not have this "feature", so it's an issue that can only arise from a specific (not recommended) way of using Zygote (which now throws explicit error messages early and often if used anywhere near SciML because I don't tolerate it). tl;dr: don't use Zygote.params, the ML AD devs know this its usage is being removed ASAP. So from this, SciML should be rather safe and if not, please share some details and I'd be happy to dig in.

ViralBShah 4y ago

A detailed discussion is on Julia discourse: https://discourse.julialang.org/t/discussion-on-why-i-no-longer-recommend-julia-by-yuri-vishnevsky/81151 https://discourse.julialang.org/t/discussion-on-why-i-no-lon... Bugs are everywhere - including Python and R libraries. You would not design a safety critical system in Python using libraries and codepaths that are not heavily tested. Even then, you still have to carefully design testsuites that give you confidence that what you are computing is what is expected. See these talks on how a collision avoidance system was designed with Julia. https://www.youtube.com/watch?v=rj-WhTL_VXE https://www.youtube.com/watch?v=rj-WhTL_VXE https://www.youtube.com/watch?v=19zm1Fn0S9M https://www.youtube.com/watch?v=19zm1Fn0S9M

leke 4y ago

Just wondering...If gradients or probability distributions are an issue for Julia, can't they just be fixed some day in the future?

gugagore 4y ago
Yes, those fixes don't inherently require the Julia language to change at all.