5 ms·

Correctness and composability bugs in the Julia ecosystem (2022)

105 points by cs702 1y ago

jerf 1y ago
@dang: I'm not sure exactly when this was posted since it seems to have no date, but it's at least (2022) per HN's link from that year: https://news.ycombinator.com/item?id=31396861 https://news.ycombinator.com/item?id=31396861 I mention this because this is definitely the sort of content that can age poorly. I have no direct experience, I've never so much as touched Julia.
- tokai 1y ago
  It cites an example from 2024 so the text has definitely been updated since 2022.
  - dang 1y ago
    Usually when an article is substantively the same but has been updated, we use the original year. I've put 2022 in the title now. The previous HN thread: Correctness and composability bugs in the Julia ecosystem - https://news.ycombinator.com/item?id=31396861 https://news.ycombinator.com/item?id=31396861 - May 2022 (407 comments) Edit: since there are (again? I seem to remember this last time) complaints about the title being a bit too baity, I've pilfered that previous title for this thread as well
    - deleted 1y ago
      [deleted]
    - cs702 1y ago
      @dang, thank you for doing that. When I posted the OP, I considered changing the title, but decided not to editorialize it, per the guidelines.
  - Alexander-Barth 1y ago
    The incorrect example of julia's documentation was fixed 2021: https://github.com/JuliaLang/julia/commit/f31ef767ef9cb0eb1de04d23e2e9e1ac765869b3 https://github.com/JuliaLang/julia/commit/f31ef767ef9cb0eb1d... As an experiment, I would be interested to see if somebody would make a 1-based python list-like data structure (or a 0-based R array), to check how many 3rd party (or standard library) function would no longer work.
- add-sub-mul-div 1y ago
  We really don't need all the hall monitoring here, it's lame when you see a submission has comments but then it's meta stuff like complaints about dates and titles.
  - jerf 1y ago
    "This Is A Really Cool Insect" doesn't really need (1998) in it, but for things like deficiencies of programming language it's helpful to know when the article comes from. "C++ Really Sucks (1995)" is a very different article from "C++ Really Sucks (2025)", and has very different takeaways for the reader.
  - refulgentis 1y ago
    Jerf's been here for 17 years, me, 16 years. I've seen this article several times, and I'm sure Jerf has as well. Our instinct, with years of being here, is it isn't a good fit for HN, at least at its current age and as labelled. It is not conducive to healthy discussion to have an aged[1] blanket dismissal of a language coupled to an assertion that saying "the issues looked fixed?" is denial of people's lived experience. [1] We can infer it was written in 2021, as the newest issue they created is from then, and they avowed never to use the language again.
  - dang 1y ago
    I don't think 'hall monitoring' is fair here - it's standard for HN titles to include the year that an article dates from, and it's standard for users to point out when such a year hasn't been added to the title yet.
  - wk_end 1y ago
    The culture of "hall monitoring" is one of the best things about HN, IMO. It's one of the few places on the internet where people - including/not just the mods - care about maintaining a high quality of discourse.
- deleted 1y ago
  [deleted]
TheRealPomax 1y ago
I think I may have missed what alternative is being recommended instead, after scrolling through the whole article.
- wbolt 1y ago
  None
- taikahessu 1y ago
  Hinted as Python not having these issues.
  - TimorousBestie 1y ago
    Python also suffers from such problems. For example, here’s scipy’s issue tracker, filtered for bugs only: https://github.com/scipy/scipy/issues?q=is%3Aissue%20state%3Aopen%20BUG https://github.com/scipy/scipy/issues?q=is%3Aissue%20state%3... Scrolling through this list, it’s clear that many are “correctness issues.” I do not link this to argue that scipy bugs are more serious or more frequent. I don’t think that kind of statistical comparison is meaningful. However, I think a motivated reasoner could write a very similar blog post to the OP, but about $arb_python_lib instead of $arb_julia_lib. I suppose my position is closer to “only Kahan, Boyd and Higham write correct numerical algorithms” (a hyperbole, but illustrative of the difficulty level).
    - Grikbdl 1y ago
      Explicit errors are not correctness issues. Some numerical instability issues in certain algorithms in certain corner cases might be considered as such though. Regardless, overall, these are grossly of another complexity and seriousness than the base sum function being just wrong, or cultural issues among developers with not verifying inputs "for performance", or things of that nature. The scientific Python community has, in my experience, a much higher adherence to good standards than that.
      - TimorousBestie 1y ago
croes 1y ago
> My conclusion after using Julia for many years is that there are too many correctness and composability bugs throughout the ecosystem to justify using it in just about any context where correctness matters. Does this have any impact on the cosmological emulator written in Julia? https://news.ycombinator.com/item?id=45346538 https://news.ycombinator.com/item?id=45346538
- SkiFire13 1y ago
  I wonder if you can even distinguish correctness issues caused by these bugs in Julia from just the underlaying ML model behaving weirdly.
- leephillips 1y ago
  It certainly would if it were a timely and justified conclusion. Since it’s not, no, it has no impact.
CyberDildonics 1y ago
I like the design of the language, but I eventually went through too many cycles of "fast compilation and/or module caching is coming in the next release" and "ahead of time compilation is coming soon" and got burned out. I remember believing the same stuff from java for years until forgetting about it.
- mgkuhn 1y ago
  The pre-compilation speed/caching performance ("time to first plot") has practically been solved since 2024, when Julia 1.10 became the current LTS version. The current focus is on improving the generation of reasonably-sized stand-alone binaries.
  - CyberDildonics 1y ago
    I heard it for 10 years, I gave it too many chances. Each time it was solved, then it was going to be solved in a new release right around the corner, again and again. Maybe it is now, I don't care anymore.
    - StefanKarpinski 1y ago
      Sad to hear that, CyberDildonics
maxbond 1y ago
I'm reminded of the recent post about R's (CRAN's) somewhat radical approach to integration testing [1] and I wonder if something like that would help with the composition issues described here. [1] https://news.ycombinator.com/item?id=45259623 https://news.ycombinator.com/item?id=45259623
- nsajko 1y ago
  The Julia world is already quite careful with testing and CI. Apart from the usual unit testing, many packages do employ integration testing. The Julia project itself (compiler, etc) is tested against the package ecosystem quite often (regularly and for select pull requests).
  - maxbond 1y ago
    I certainly didn't mean to imply that Julia's community was incompetent or that they were not doing integration testing. CRAN's approach (which is mandatory integration testing against all known dependents enforced by the packaging authority - the global and mandatory nature being what makes it different) is genuinely innovative and radical. I don't think that's an approach that should be adopted lightly or by most ecosystems, but I do observe that a.) these languages have similar goals and b.) it's an approach intended to solve problems of much the same shape as described in the article. Again I think this approach is too radical for most ecosystems, but Julia is pursuing a similarly radical level of composability/reusability and evidently encountering difficulties with it, so I think there may be a compatibility there.
    - nsajko 1y ago
      I don't think testing against every existing dependent would make sense currently. The issue is the lack of tooling for mechanically checking whether the dependent accesses implementation details of the dependency, in which case it would be valid for the dependency to break the dependent. There are some proposals to forbid the registration of a package release which trespasses on the internals of another package, though. I hope someone tackles the above sooner or later, but another issue is the approach of testing every known dependent package might be very costly, both in terms of compute and manual labor, the latter because someone would have to do the work of maintaining a blacklist for packages with flaky unit tests. The good news is that this work might considerably overlap with the already existing PkgEval infrastructure. We'll see.
dhampi 1y ago
I quit Julia after running into serious bugs in basic CSV package functionality a few years back. The language is elegant, intuitive and achieves what it promises 99% of the time, but that’s not enough compared to other programming languages.
elcritch 1y ago
Which is a bummer, however the basics are always critical. Numerical stability and known correctness is still (or was) a reason a lot of old fortran libraries were used for so long. I'm really surprised by the list of issues as some of those are pretty recent (2024) and pretty important parts of the ecosystem like ordereddict.
deleted 1y ago
[deleted]
max_ 1y ago
I hear similar bugs exist in python libraries. Any recommended libraries (or languages) that have thoroughly verified libraries?
- mmacvicarprett 1y ago
  Do you mean any specific library?
  - max_ 1y ago
    Any really, library, language etc
- TimorousBestie 1y ago
  In my professional experience, the older numerics libraries tend to be more reliable, with the notable exception of Intel’s MKL.
mmacvicarprett 1y ago
How to quickly kill a language primarily meant for technical computing.
mgkuhn 1y ago
Julia is a very powerful and flexible language. With very powerful tools you can get a lot done quickly, including shooting yourself into the foot. Julia's type-system allows you to easily compose different elements of Julia's vast package ecosystem in ways that possibly were never tested or even intended or foreseen by the authors of these packages to be used that way. If you don't do that, you may have a much better experience than the author. My own Julia code generally does not feed the custom type of one package into the algorithms of another package.
- jacobolus 1y ago
  One of the basic marketing claims of the language developers is that one author's algorithm can be composed with another author's custom data type. If that's not really true in general, even for some of the most popular libraries and data types, maybe the claims should be moderated a bit.
  - hatmatrix 1y ago
    > one author's algorithm can be composed with another author's custom data type This is true, and it's a powerful part of the language - but you can implement it incorrectly when you compose elements together that expect some attributes from the custom data type. There is no way to formally enforce that, so you can end up with correctness bugs.
  - ekjhgkejhgk 1y ago
    My experience is that it is true, if you throughly implement the interfaces that your types are supposed to respect. If you dont, well, thats not the languages fault.
- bobbylarrybobby 1y ago
  One can hardly call using the canonical autograd library and getting incorrect gradients or using arrays whose indices aren't 1:len and getting OOB errors “shooting oneself in the foot” — these things are supposed to Just Work, but they don't. Interfaces would go a long way towards codifying interoperability expectations (although wouldn't help with plain old correctness bugs). With regard to power and flexibility, homoiconicity and getting to hook into compiler passes does make Julia powerful and flexible in a way that most other languages aren't. But I'm not sure if that power is what results in bugs — more likely it's the function overloading/genericness, whose power and flexibility I think is a bit overstated.
ForceBru 1y ago
Has anything changed since then? What are y'all's thoughts about correctness in Julia in 2025?
- postflopclarity 1y ago
  improving, still not perfect. it was true then, and is even more true now, that a large fraction of "correctness bugs" (maybe even the majority) arise from `OffsetArrays.jl`, so a simple solution besides "avoid Julia" is "avoid that package"
  - hatmatrix 1y ago
    The greater issue that there is still no way to prevent those types of composability bugs.
    - postflopclarity 1y ago
      sure there are ways. they're just not employed as diligently as they should be. that's more of a social problem than a technical problem.
    - krull10 1y ago
      The simplest approach is to always read the interface of packages one wants to use, and if one isn't provided look at the code / open an issue to interact with the developers about their input assumptions. One should also make tests to ensure the interface behaves in the expected manner when working with your code. Using this approach since 2017 I've never really encountered the types of issues mentioned in Yuri's blog post. The biggest issue I've had is if some user-package makes a change that is effectively breaking but they don't flag it the associated release as breaking. But this isn't really a Julia issue so much as a user-space issue, and can happen in any language when relying on others' libraries.
      - hatmatrix 1y ago
        It doesn't happen so frequently in practice, which is maybe why it's not felt equally by everyone in the community. I've followed some discussions on this topic and the current solution is, as you say, to place the onus entirely on the user. There is another faction that wants to support the user through the IDE to warn them of errors they might be making, and others that want to implement formal specifications that catch the errors during specifications. It's not an easy problem to fix and therefore the "simplest approach" as you describe remains the leading solution.
yurivish 1y ago
I'm the author of the original post. I still like and miss my friends in the Julia community, but the technical and cultural issues persist. I may refresh the post with more recent information at some point. In the meantime, those curious can find a short story of one newer correctness bug here: https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/40 https://discourse.julialang.org/t/why-is-it-reliable-to-use-... The person who eventually fixed the issue, mkitti, had to push through a lot of "institutional" friction to do so, and the eventual fix is the result of his determined efforts. While his part of the story mostly played out in venues outside of the Discourse forum some of it is on display in this thread: https://discourse.julialang.org/t/csv-jl-findmax-and-argmax-are-still-broken-please-help/115366 https://discourse.julialang.org/t/csv-jl-findmax-and-argmax-...
- postflopclarity 1y ago
  while I certainly agree that bug was a bit frustrating, in all that * it happened in the first place * it took so long to get noticed * it took so long to merge the fix after being reported I do feel like I should push back on the term "institutional friction." it was more of a bus factor problem; there were not enough (aka zero) maintainer eyeballs on the proposed fix. but there wasn't exactly anybody saying not to fix it, which is what I think of as friction.
sgaure 1y ago
I use julia intensively and have done so for 5 years or so. I have never encountered anything I would call a correctness bug. I guess it depends on what you count as a correctness bug, and what you mean by "julia". The core language has no obvious bugs, but there are packages of dubious quality. Say you use some package for numerical integration. One day you cook up your own floating point type, and use the same package with success. Then you change your floating point type subtly, and suddenly weird things start to happen. Is it a correctness bug? Whose bug? Surely, the author of the integration package didn't have your weird floating type in mind, but it still worked. Until you made it even weirder. These are the things some people think are correctness bugs in julia. It's mostly poor coding.
- csvance 1y ago
  I've found that I'm most productive in Julia when minimizing the number of third party dependencies for this reason, even more so than other languages. That's not to say there are not many high quality packages available but rather the benefits of the type system align better when I have a strong understanding or control over the most pertinent interfaces. As a language Julia definitely rewards you heavily for this type of thing. Coming from Python my first instinct was to try to solve as many problems as possible with third party packages and filling in between the lines. Unsurprisingly this was the worst of both worlds. If there was one thing I could change about Julia it most certainly wouldn't be correctness issues in my own experience. Filling in the ecosystem in terms of boring glue type stuff like a production grade gRPC client would be amazing. This was the type of problem that almost got me to give up on the language.
nsajko 1y ago
Julia is not without warts, but this blog post is kinda rubbish. The post claims vague but scary "correctness issues", trying to support this with a collection of unrelated issue tickets from all across Julia and the Julia package ecosystem. Not all of which were even bugs in the first place, and many of which have long been resolved. The fact that bugs happen in software should not surprise anyone. Even software of critical importance, such as GCC or LLVM, whose correctness is relied upon by the implementations of many programming languages (including C, C++ and Julia itself), are buggy. Instead the post could have focused more on actual design issues, such as some of the Base interfaces being underspecified: > the nature of many common implicit interfaces has not been made precise (for example, there is no agreement in the Julia community on what a number is) The underspecified nature of Number (or Real, or IO) is an issue, albeit not related with the rest of the blog post. It does not excuse the scaremongering in the blog post, however.
- StefanKarpinski 1y ago
  Number isn’t an interface—there are no operations common to all numbers. Subtyping Number is a way to opt into numeric promotion and a few other useful generic behaviors. That’s it. The fact that some abstract types are interfaces with expected behaviors, while others are dispatch points to opt into behaviors is a double edged sword: powerful and flexible, but only explicitly expressed/explained in documentation.
  - nsajko 1y ago
    > Number isn’t an interface—there are no operations common to all numbers. When creating a new type, it should be more clear cut when is subtyping Number (or Real, etc.) valid. Should unitful quantities be numbers? Should intervals be numbers? Related: I think there are some attempts by Tim Holy and others to create/document "thick numbers". Furthermore, I believe it might be good to align the Number type hierarchy with math/abstract algebra as much as possible without breaking backwards compatibility, which might making Number, or some subtypes of it, actual interfaces. > Subtyping Number is a way to opt into numeric promotion and a few other useful generic behaviors. That’s it. OK, but I think that's not documented either.

105 points by cs702 1y ago

jerf 1y ago

@dang: I'm not sure exactly when this was posted since it seems to have no date, but it's at least (2022) per HN's link from that year: https://news.ycombinator.com/item?id=31396861 https://news.ycombinator.com/item?id=31396861 I mention this because this is definitely the sort of content that can age poorly. I have no direct experience, I've never so much as touched Julia.

tokai 1y ago
It cites an example from 2024 so the text has definitely been updated since 2022.
- dang 1y ago
  Usually when an article is substantively the same but has been updated, we use the original year. I've put 2022 in the title now. The previous HN thread: Correctness and composability bugs in the Julia ecosystem - https://news.ycombinator.com/item?id=31396861 https://news.ycombinator.com/item?id=31396861 - May 2022 (407 comments) Edit: since there are (again? I seem to remember this last time) complaints about the title being a bit too baity, I've pilfered that previous title for this thread as well
  - deleted 1y ago
    [deleted]
  - cs702 1y ago
    @dang, thank you for doing that. When I posted the OP, I considered changing the title, but decided not to editorialize it, per the guidelines.
- Alexander-Barth 1y ago
  The incorrect example of julia's documentation was fixed 2021: https://github.com/JuliaLang/julia/commit/f31ef767ef9cb0eb1de04d23e2e9e1ac765869b3 https://github.com/JuliaLang/julia/commit/f31ef767ef9cb0eb1d... As an experiment, I would be interested to see if somebody would make a 1-based python list-like data structure (or a 0-based R array), to check how many 3rd party (or standard library) function would no longer work.
add-sub-mul-div 1y ago
We really don't need all the hall monitoring here, it's lame when you see a submission has comments but then it's meta stuff like complaints about dates and titles.
- jerf 1y ago
  "This Is A Really Cool Insect" doesn't really need (1998) in it, but for things like deficiencies of programming language it's helpful to know when the article comes from. "C++ Really Sucks (1995)" is a very different article from "C++ Really Sucks (2025)", and has very different takeaways for the reader.
- refulgentis 1y ago
  Jerf's been here for 17 years, me, 16 years. I've seen this article several times, and I'm sure Jerf has as well. Our instinct, with years of being here, is it isn't a good fit for HN, at least at its current age and as labelled. It is not conducive to healthy discussion to have an aged[1] blanket dismissal of a language coupled to an assertion that saying "the issues looked fixed?" is denial of people's lived experience. [1] We can infer it was written in 2021, as the newest issue they created is from then, and they avowed never to use the language again.
- dang 1y ago
  I don't think 'hall monitoring' is fair here - it's standard for HN titles to include the year that an article dates from, and it's standard for users to point out when such a year hasn't been added to the title yet.
- wk_end 1y ago
  The culture of "hall monitoring" is one of the best things about HN, IMO. It's one of the few places on the internet where people - including/not just the mods - care about maintaining a high quality of discourse.
deleted 1y ago
[deleted]

TheRealPomax 1y ago

I think I may have missed what alternative is being recommended instead, after scrolling through the whole article.

wbolt 1y ago
None
taikahessu 1y ago
Hinted as Python not having these issues.
- TimorousBestie 1y ago
  Python also suffers from such problems. For example, here’s scipy’s issue tracker, filtered for bugs only: https://github.com/scipy/scipy/issues?q=is%3Aissue%20state%3Aopen%20BUG https://github.com/scipy/scipy/issues?q=is%3Aissue%20state%3... Scrolling through this list, it’s clear that many are “correctness issues.” I do not link this to argue that scipy bugs are more serious or more frequent. I don’t think that kind of statistical comparison is meaningful. However, I think a motivated reasoner could write a very similar blog post to the OP, but about $arb_python_lib instead of $arb_julia_lib. I suppose my position is closer to “only Kahan, Boyd and Higham write correct numerical algorithms” (a hyperbole, but illustrative of the difficulty level).
  - Grikbdl 1y ago
    Explicit errors are not correctness issues. Some numerical instability issues in certain algorithms in certain corner cases might be considered as such though. Regardless, overall, these are grossly of another complexity and seriousness than the base sum function being just wrong, or cultural issues among developers with not verifying inputs "for performance", or things of that nature. The scientific Python community has, in my experience, a much higher adherence to good standards than that.
    - TimorousBestie 1y ago

croes 1y ago

> My conclusion after using Julia for many years is that there are too many correctness and composability bugs throughout the ecosystem to justify using it in just about any context where correctness matters. Does this have any impact on the cosmological emulator written in Julia? https://news.ycombinator.com/item?id=45346538 https://news.ycombinator.com/item?id=45346538

SkiFire13 1y ago
I wonder if you can even distinguish correctness issues caused by these bugs in Julia from just the underlaying ML model behaving weirdly.
leephillips 1y ago
It certainly would if it were a timely and justified conclusion. Since it’s not, no, it has no impact.

CyberDildonics 1y ago

I like the design of the language, but I eventually went through too many cycles of "fast compilation and/or module caching is coming in the next release" and "ahead of time compilation is coming soon" and got burned out. I remember believing the same stuff from java for years until forgetting about it.

mgkuhn 1y ago
The pre-compilation speed/caching performance ("time to first plot") has practically been solved since 2024, when Julia 1.10 became the current LTS version. The current focus is on improving the generation of reasonably-sized stand-alone binaries.
- CyberDildonics 1y ago
  I heard it for 10 years, I gave it too many chances. Each time it was solved, then it was going to be solved in a new release right around the corner, again and again. Maybe it is now, I don't care anymore.
  - StefanKarpinski 1y ago
    Sad to hear that, CyberDildonics

maxbond 1y ago

I'm reminded of the recent post about R's (CRAN's) somewhat radical approach to integration testing [1] and I wonder if something like that would help with the composition issues described here. [1] https://news.ycombinator.com/item?id=45259623 https://news.ycombinator.com/item?id=45259623

nsajko 1y ago
The Julia world is already quite careful with testing and CI. Apart from the usual unit testing, many packages do employ integration testing. The Julia project itself (compiler, etc) is tested against the package ecosystem quite often (regularly and for select pull requests).
- maxbond 1y ago
  I certainly didn't mean to imply that Julia's community was incompetent or that they were not doing integration testing. CRAN's approach (which is mandatory integration testing against all known dependents enforced by the packaging authority - the global and mandatory nature being what makes it different) is genuinely innovative and radical. I don't think that's an approach that should be adopted lightly or by most ecosystems, but I do observe that a.) these languages have similar goals and b.) it's an approach intended to solve problems of much the same shape as described in the article. Again I think this approach is too radical for most ecosystems, but Julia is pursuing a similarly radical level of composability/reusability and evidently encountering difficulties with it, so I think there may be a compatibility there.
  - nsajko 1y ago
    I don't think testing against every existing dependent would make sense currently. The issue is the lack of tooling for mechanically checking whether the dependent accesses implementation details of the dependency, in which case it would be valid for the dependency to break the dependent. There are some proposals to forbid the registration of a package release which trespasses on the internals of another package, though. I hope someone tackles the above sooner or later, but another issue is the approach of testing every known dependent package might be very costly, both in terms of compute and manual labor, the latter because someone would have to do the work of maintaining a blacklist for packages with flaky unit tests. The good news is that this work might considerably overlap with the already existing PkgEval infrastructure. We'll see.

dhampi 1y ago

I quit Julia after running into serious bugs in basic CSV package functionality a few years back. The language is elegant, intuitive and achieves what it promises 99% of the time, but that’s not enough compared to other programming languages.

elcritch 1y ago

Which is a bummer, however the basics are always critical. Numerical stability and known correctness is still (or was) a reason a lot of old fortran libraries were used for so long. I'm really surprised by the list of issues as some of those are pretty recent (2024) and pretty important parts of the ecosystem like ordereddict.

deleted 1y ago

[deleted]

max_ 1y ago

I hear similar bugs exist in python libraries. Any recommended libraries (or languages) that have thoroughly verified libraries?

mmacvicarprett 1y ago
Do you mean any specific library?
- max_ 1y ago
  Any really, library, language etc
TimorousBestie 1y ago
In my professional experience, the older numerics libraries tend to be more reliable, with the notable exception of Intel’s MKL.

mmacvicarprett 1y ago

How to quickly kill a language primarily meant for technical computing.

mgkuhn 1y ago

Julia is a very powerful and flexible language. With very powerful tools you can get a lot done quickly, including shooting yourself into the foot. Julia's type-system allows you to easily compose different elements of Julia's vast package ecosystem in ways that possibly were never tested or even intended or foreseen by the authors of these packages to be used that way. If you don't do that, you may have a much better experience than the author. My own Julia code generally does not feed the custom type of one package into the algorithms of another package.

jacobolus 1y ago
One of the basic marketing claims of the language developers is that one author's algorithm can be composed with another author's custom data type. If that's not really true in general, even for some of the most popular libraries and data types, maybe the claims should be moderated a bit.
- hatmatrix 1y ago
  > one author's algorithm can be composed with another author's custom data type This is true, and it's a powerful part of the language - but you can implement it incorrectly when you compose elements together that expect some attributes from the custom data type. There is no way to formally enforce that, so you can end up with correctness bugs.
- ekjhgkejhgk 1y ago
  My experience is that it is true, if you throughly implement the interfaces that your types are supposed to respect. If you dont, well, thats not the languages fault.
bobbylarrybobby 1y ago
One can hardly call using the canonical autograd library and getting incorrect gradients or using arrays whose indices aren't 1:len and getting OOB errors “shooting oneself in the foot” — these things are supposed to Just Work, but they don't. Interfaces would go a long way towards codifying interoperability expectations (although wouldn't help with plain old correctness bugs). With regard to power and flexibility, homoiconicity and getting to hook into compiler passes does make Julia powerful and flexible in a way that most other languages aren't. But I'm not sure if that power is what results in bugs — more likely it's the function overloading/genericness, whose power and flexibility I think is a bit overstated.

ForceBru 1y ago

Has anything changed since then? What are y'all's thoughts about correctness in Julia in 2025?

postflopclarity 1y ago
improving, still not perfect. it was true then, and is even more true now, that a large fraction of "correctness bugs" (maybe even the majority) arise from `OffsetArrays.jl`, so a simple solution besides "avoid Julia" is "avoid that package"
- hatmatrix 1y ago
  The greater issue that there is still no way to prevent those types of composability bugs.
  - postflopclarity 1y ago
    sure there are ways. they're just not employed as diligently as they should be. that's more of a social problem than a technical problem.
  - krull10 1y ago
    The simplest approach is to always read the interface of packages one wants to use, and if one isn't provided look at the code / open an issue to interact with the developers about their input assumptions. One should also make tests to ensure the interface behaves in the expected manner when working with your code. Using this approach since 2017 I've never really encountered the types of issues mentioned in Yuri's blog post. The biggest issue I've had is if some user-package makes a change that is effectively breaking but they don't flag it the associated release as breaking. But this isn't really a Julia issue so much as a user-space issue, and can happen in any language when relying on others' libraries.
    - hatmatrix 1y ago
      It doesn't happen so frequently in practice, which is maybe why it's not felt equally by everyone in the community. I've followed some discussions on this topic and the current solution is, as you say, to place the onus entirely on the user. There is another faction that wants to support the user through the IDE to warn them of errors they might be making, and others that want to implement formal specifications that catch the errors during specifications. It's not an easy problem to fix and therefore the "simplest approach" as you describe remains the leading solution.

yurivish 1y ago

I'm the author of the original post. I still like and miss my friends in the Julia community, but the technical and cultural issues persist. I may refresh the post with more recent information at some point. In the meantime, those curious can find a short story of one newer correctness bug here: https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/40 https://discourse.julialang.org/t/why-is-it-reliable-to-use-... The person who eventually fixed the issue, mkitti, had to push through a lot of "institutional" friction to do so, and the eventual fix is the result of his determined efforts. While his part of the story mostly played out in venues outside of the Discourse forum some of it is on display in this thread: https://discourse.julialang.org/t/csv-jl-findmax-and-argmax-are-still-broken-please-help/115366 https://discourse.julialang.org/t/csv-jl-findmax-and-argmax-...

postflopclarity 1y ago
while I certainly agree that bug was a bit frustrating, in all that * it happened in the first place * it took so long to get noticed * it took so long to merge the fix after being reported I do feel like I should push back on the term "institutional friction." it was more of a bus factor problem; there were not enough (aka zero) maintainer eyeballs on the proposed fix. but there wasn't exactly anybody saying not to fix it, which is what I think of as friction.

sgaure 1y ago

I use julia intensively and have done so for 5 years or so. I have never encountered anything I would call a correctness bug. I guess it depends on what you count as a correctness bug, and what you mean by "julia". The core language has no obvious bugs, but there are packages of dubious quality. Say you use some package for numerical integration. One day you cook up your own floating point type, and use the same package with success. Then you change your floating point type subtly, and suddenly weird things start to happen. Is it a correctness bug? Whose bug? Surely, the author of the integration package didn't have your weird floating type in mind, but it still worked. Until you made it even weirder. These are the things some people think are correctness bugs in julia. It's mostly poor coding.

csvance 1y ago
I've found that I'm most productive in Julia when minimizing the number of third party dependencies for this reason, even more so than other languages. That's not to say there are not many high quality packages available but rather the benefits of the type system align better when I have a strong understanding or control over the most pertinent interfaces. As a language Julia definitely rewards you heavily for this type of thing. Coming from Python my first instinct was to try to solve as many problems as possible with third party packages and filling in between the lines. Unsurprisingly this was the worst of both worlds. If there was one thing I could change about Julia it most certainly wouldn't be correctness issues in my own experience. Filling in the ecosystem in terms of boring glue type stuff like a production grade gRPC client would be amazing. This was the type of problem that almost got me to give up on the language.

nsajko 1y ago

Julia is not without warts, but this blog post is kinda rubbish. The post claims vague but scary "correctness issues", trying to support this with a collection of unrelated issue tickets from all across Julia and the Julia package ecosystem. Not all of which were even bugs in the first place, and many of which have long been resolved. The fact that bugs happen in software should not surprise anyone. Even software of critical importance, such as GCC or LLVM, whose correctness is relied upon by the implementations of many programming languages (including C, C++ and Julia itself), are buggy. Instead the post could have focused more on actual design issues, such as some of the Base interfaces being underspecified: > the nature of many common implicit interfaces has not been made precise (for example, there is no agreement in the Julia community on what a number is) The underspecified nature of Number (or Real, or IO) is an issue, albeit not related with the rest of the blog post. It does not excuse the scaremongering in the blog post, however.

StefanKarpinski 1y ago
Number isn’t an interface—there are no operations common to all numbers. Subtyping Number is a way to opt into numeric promotion and a few other useful generic behaviors. That’s it. The fact that some abstract types are interfaces with expected behaviors, while others are dispatch points to opt into behaviors is a double edged sword: powerful and flexible, but only explicitly expressed/explained in documentation.
- nsajko 1y ago
  > Number isn’t an interface—there are no operations common to all numbers. When creating a new type, it should be more clear cut when is subtyping Number (or Real, etc.) valid. Should unitful quantities be numbers? Should intervals be numbers? Related: I think there are some attempts by Tim Holy and others to create/document "thick numbers". Furthermore, I believe it might be good to align the Number type hierarchy with math/abstract algebra as much as possible without breaking backwards compatibility, which might making Number, or some subtypes of it, actual interfaces. > Subtyping Number is a way to opt into numeric promotion and a few other useful generic behaviors. That’s it. OK, but I think that's not documented either.