9 ms·

Is Fortran easier to optimize than C for heavy calculations?

94 points by jokoon 3y ago

pipo234 3y ago
> The performance difference comes from the fact that Fortran says aliasing is not allowed [...] This is one of the legacies modern C++ is still wrestling with today, supposedly solved by Rust, Zig, Val and the likes.
- fsckboy 3y ago
  fortran is call by reference, it's all aliasing, you can even in some implementations change the value of literals c is call by value, you have to at least try to alias
  - gpderetta 3y ago
    my understanding is that in fortran, if two arguments alias each other, it is UB. So it is not that fortran doesn't have aliasing, it is just that the optimizer assume that it doesn't (but don't quote me on this!).
    - Bostonian 3y ago
      Arguments are allowed to alias if they are both inputs that will not be changed in the function -- technically if they are both "intent(in)".
      - pklausler 3y ago
        INTENT(IN) is not required to allow aliasing of unmodified dummy arguments.
        kergonath 3y ago
        Yes, but in practice no compiler checks whether an argument with no INTENT is modified or not, or could alias with something else or not. So the compilers tend to trust the programmer. With INTENTS, it’s obvious and much easier to check.
        pklausler 3y ago
        Dummy argument aliasing has nothing to do with INTENT.
        TheRealKing 3y ago
        I believe the NAG Fortran compiler does check for some aliasing scenarios if not all. The argument intent is irrelevant.
  - gnufx 3y ago
    Fortran does not call by reference, and the term aliasing isn't used by the standard. It calls by what's usually called value-return or copy-in/copy-out, according to the "storage association" rules in the standard. While it may be true that the rules aren't generally checked by compilers, Toolpack did it for pure Fortran77 in the 1980s, though I don't know how comprehensive it actually was.
  - bregma 3y ago
    It took me days one time to figure out why 0 had adopted the value four. The technicality is that Fortran is call by descriptor (not by reference, not by value). That's a powerful difference but a sharp two-edged sword.
    - kaba0 3y ago
      Could you please explain how is it differs from the other two? Never looked into Fortran.
      - kergonath 3y ago
        The Fortran standard does not specify anything about passing by references or value. Instead, it specifies the expected results. The compilers do it however they want or can, which means that they can pass pointers, values, copies, or pointers to copies depending on the situation. Sometimes (most of the time, to be fair), the compiler can work out the best solution. Sometimes, the developer has to jump through some hoops to avoid unnecessary copies.
- zik 3y ago
  It's true that there's no "restrict" inthe C++ standard, however all the major C++ compilers have extensions for it ("__restrict__") and have had for a long time.
  - patrec 3y ago
    Except that it doesn't work and is basically WONTFIX since no C or C++ programmer ever uses it. This is only very slightly exaggerated.
    - zik 3y ago
      That's utterly incorrect. It's used thousands of times in the linux kernel.
    - Conscat 3y ago
      I've never had issues with it, personally. In what way does it not work?
      - lmkg 3y ago
        Rust has aliasing guarantees. In theory, Rust code can pass noalias annotations to LLVM and become more optimized. In practice, it took years for Rust access these optimizations because the noalias annotations kept hitting codegen bugs in LLVM. It was an area of the compiler that had not seen much active use before Rust, and they were causing a lot of dormant bugs to become issues. The feedback cycle was also very long because LLVM didn't fix the problems quickly. I didn't pay close attention so I don't know if this is prioritization or because those fixes required major surgery, but Rust sometimes had to wait for major version bumps in LLVM before seeing if "this time was the last time."
        vlovich123 3y ago
        Yup. But it is true now though so hopefully they’ve added the test cases to LLVM to make sure things continue to work correctly in the face of more and more optimizations being thrown at code.
- _flux 3y ago
  Hmm, I thought basically only Rust solves it. How do Zig and Val solve it?
  - Conscat 3y ago
    Multiple-aliasing cannot exist in a language without reference semantics, such as Val (or whatever its new name is).
  - dralley 3y ago
    Zig doesn't solve it. I don't remember where but I know Andrew has mentioned that directly.
saagarjha 3y ago
The difference is mostly not relevant these days. Yes, restrict is important, but anyone who is optimizing numerical computing hard is probably already using it judiciously, and for them what is more important is what is familiar to them and makes them most productive.
- pjmlp 3y ago
  The keyword is judiciously, without triggering UB, and I bet most don't.
- gnufx 3y ago
  Unfortunately I see rather few people doing serious optimizing knowledgeably at all in research computing. (I know how and where it's done well, of course.) That has global effects on productivity of users, though some of the biggest gains can come from more global optimization, like in the use of MPI.
Alifatisk 3y ago
This might be a bit irrelevant but, my professor who teaches computer programming (who previously worked as a nuclear physicist) loved mentioning Fortran whenever possible. He truly enjoy the language and believes it is well suited for heavy computation. He said its used a lot for weather forecasting. He is quite old now so I don’t know how true it is today, but he made me more interested in Fortran and I might give it a shot instead of C sometime soon.
- Joel_Mckay 3y ago
  [flagged]
  - therealcamino 3y ago
    "Most academic institutions have been sued many times due to the volume of nonsense they feed kids." I've never heard of that happening, let alone many times to most institutions.
    - Joel_Mckay 3y ago
      [flagged]
      - bananapub 3y ago
        so your weird unfounded assertion is also explicitly unprovable? awesome, thanks for sharing.
        Joel_Mckay 3y ago
        In general lexis nexus holds the records of former students suing for BS employment promises. You would be fairly surprised who showed up besides trump u in the states. It was a lot more common than most like to admit. =)
      - jcranmer 3y ago
        > Part of the settlements is an NDA. If someone is sued, that creates a court record. Even if the terms of the settlement are not disclosed, that there was a settlement is lodged in the court record, and publicly accessible. Even if court records are sealed, the existence of the case is not, and initial complaints are rarely sealed. In short, if suing universities because of what they're teaching is routine, you should be able to point to court cases. If you can't find them, then it's not because it's secret--secret court cases don't exist--it's because they don't exist.
bregma 3y ago
Fortran is column-major and C is row-major. That difference alone makes vectorization in Fortran much much more frequently used.
- defrost 3y ago
  As a 60 year old C | Fortran coder since ~1980 or so, that difference makes near zero practical difference to numerical computing. Pretty much any vector|matrix operations will see row against column (or colum against row) operations in equal proportion - what you gain on the swings you lose on the roundabouts whichever way you choose to orientate storage. Indeed, given large sparse arrays, language specific default array storage methods might not even be used - and here C is (arguably) more flexible at building task specific data structures - a column(?) of pointers to row fragments (perhaps). The big advantage early FORTRAN had was baked in anti alias assumptions and a truckload of pre existing relatively accessible NIST | NASA grade quality numerical algorithms that had been crawled over, eyeballed, tested, and pounded in simulations by many qualified applied scientists.
  - AnimalMuppet 3y ago
    > The big advantage early FORTRAN had was baked in anti alias assumptions and a truckload of pre existing relatively accessible NIST | NASA grade quality numerical algorithms that had been crawled over, eyeballed, tested, and pounded in simulations by many qualified applied scientists. Yeah. That Fortran library that has a solver for sparse complex matrices that won't produce nonsense if you give it a stiff problem, that has 50 years of people pounding on it to make sure that it's rock solid? I'm not sure that there's a C/C++ equivalent for it. I absolutely cannot write an equivalent. Yes, I can write a matrix solver that takes complex data. No, I can't write one that is as efficient, as numerically stable, or as bug-free as what already exists in Fortran.
  - bregma 3y ago
    As a 60 year old C and Fortran coder since ~1980 or so, and who has spent the better part of the last decade supporting compilers and their internals, I say it makes a big difference when it comes to autovectorization optimizations. But that's my experience based on most users being naive and know-enough-to-be-dangerous programmers, and your experience may differ.
andy99 3y ago
In terms of being easier (for the programmer, not the compiler) one thing worth mentioning is that Fortran already has optimized intrinsics for matmul and dot_product. I was recently doing a small comparison between C and Fortran for matrix-vector multiplication and was surprised how fast the matmul is "out of the box" vs handwritten loops. Personally I also think that for code that is linear algebra heavy Fortran is easier to work with because of the intrinsics - it's analogous to numpy, whereas C takes more effort, mainly keeping track of sizes of things. https://gist.github.com/rbitr/3b86154f78a0f0832e8bd1716152365d https://gist.github.com/rbitr/3b86154f78a0f0832e8bd171615236...
- physicsguy 3y ago
  True for small matrices but above certain size you wouldn’t want to use these methods or loops anyway, and they’re not optimised if your matrix is sparse either. So practically you end up using a library and you’re back to basically being equivalent with C.
  - leereeves 3y ago
    The Fortran matmul intrinsic doesn't use an optimal algorithm?
    - duped 3y ago
      Even theoretically this is only possible if the dimensions and sparseness are known at compile time, which can be pretty heavy assumptions.
      - jcranmer 3y ago
        The implementation of every BLAS implementation I'm aware of basically looks like this: if (dims in range A) gemm_implA else if (dims in range B) gemm_implB else if (dims in range C) gemm_implC // several more implementations... The immediate call function is invariably a switch over several implementations, and by the time you reach that call, you have actual knowledge of dimensions.
mschaef 3y ago
Back in school (mid-90's), I made a point of taking a FORTRAN introductory class to go along with all the other work I was doing in Pascal, C++, Lisp, etc. For most of the class, FORTRAN felt mostly like a penalty box. It was full of behaviors and limitations that dated straight back to the early 1960's, if not earlier. It was at least easy to make it do the things it could do. Where my opinion of FORTRAN changed was in the last time that class met. By that point, we 'knew' the language and the instructor was on to more advanced topics, including optimization. He made a point of how an optimizing FORTRAN compiler could re-nest three nested loops for the purpose of automatically improving memory access patterns. Vector Cray machines (which implemented common looping patterns in hardware) shipped with FORTRAN compilers that could take normal-looking loops and turn them into code that optimized for the hardware. I don't know if it's still the case (thanks to restrict and better compilers), but he made a compelling point at the time for the simplicity and restrictions of the language leading to more options for optimization.
- dralley 3y ago
  >I don't know if it's still the case (thanks to restrict and better compilers) Nobody (really) uses restrict, certainly not at scale. This is especially obvious because when Rust came along (which can effectively use restrict in most places, much like Fortran) it uncovered a lot of serious bugs in LLVM's implementation for restrict. That whole situation only really stabilized in the past year or so. Maybe GCC does a better job as a result of having gfortran.
- 8bitsrule 3y ago
  "as of 2022 programs have been written in Fortran for over six decades and there is a vast body of Fortran software in daily use throughout the scientific and engineering communities.... It is the primary language for some of the most intensive super-computing tasks, such as in astronomy, climate modeling, computational chemistry, computational economics, computational fluid dynamics, computational physics.... Since the early 2000s, many of the widely used support libraries have also been implemented in C and more recently, in C++... For this reason, facilities for inter-operation with C [1] were added to Fortran 2003 and enhanced by the ISO/IEC technical specification 29113, which was incorporated into Fortran 2018... " [0] [0] https://en.wikipedia.org/wiki/Fortran#Science_and_engineering https://en.wikipedia.org/wiki/Fortran#Science_and_engineerin... [1] https://en.wikipedia.org/wiki/Foreign_function_interface https://en.wikipedia.org/wiki/Foreign_function_interface
jcranmer 3y ago
Fortran has a lot more (N-dimensional) array logic intrinsic to the language than C or C++ does, and in general, Fortran has a fair number of "little" things that can add up for optimization: * Fortran arrays do not alias by default. (This is the most well-known, and is indeed repeated in the answers on StackOverflow) * Fortran N-D array indexes are UB if they go out-of-bounds in a way that isn't true for most other languages. In C, if you have a declaration `int x[2][3];`, the expression `x[0][4]` is a legal way to access one of the elements--it's only UB to go outside the bounds of x as a whole, not individual rows. In Fortran, the equivalent expression (well, modified because Fortran is column-major) is illegal. * Fortran has built in array expressions--you can express a vector addition just by adding the arrays themselves. This makes it easier to autovectorize.
- pklausler 3y ago
  Fortran's dummy arguments can (generally) be assumed in the scope of a subprogram to not alias, whether arrays or not. The burden is on the programmer of calls to procedures to not associate any of its dummy arguments that are modified during the call with data that might be associated with any other dummy argument.
- KeplerBoy 3y ago
  > In C, if you have a declaration `int x[2][3];`, the expression `x[0][4]` is a legal way to access one of the elements Is it really? Does the C-compiler guarantee that your 6 ints will be stored in a contiguous memory area?
  - jcranmer 3y ago
    Yes. A[i] in C is exactly equivalent to *(A + i). (Incidentally, this means that 2[x] is a legal expression in C).
    - KeplerBoy 3y ago
      Of course, that part is clear. The question is, if the two rows are guaranteed to be contiguous in memory. If using heap allocated memory, one could easily construct a counterexample (i'm not sure how the stack allocated array behaves, i guess it works because it's the sane choice, but not strictly guaranteed): int *x; int numRows = 2; int numCols = 3; x = (int *)malloc(numRows * sizeof(int )); for (int i = 0; i < numRows; i++) { x[i] = (int )malloc(numCols * sizeof(int)); } // This could segfault x[0][4] = 42;
theodorethomas 3y ago
People mention the no-aliasing, the compilers, the intrinsics, the libraries, and the expressivity but one aspect of the difference is ignored and it is this: C is a language for specifying behaviour of hardware (time sequence of detailed states), Fortran is a language for specifying computation of values. Fortran abstracts far more of the hardware than C and consequently, a Fortran compiler can benefit from quantum processors, mind-readers or time-machines, should they ever be invented. A Fortran program that reads its input, calculates and finally writes out its output does not have to execute any particular instruction at all. As long as the answer is "AS IF" it had done the user-specified computation, the Fortran compiler has done its job. In between I/O, it submerges into the ineffable like a Cold War SSBN. C is about the instruments, the players and the conductor, Fortran is about the music.
- dTal 3y ago
  Maybe this was once true, but the hardware that C was designed for specifying the behavior of was a PDP-11. Nowadays you are programming an abstract C-ish virtual machine that provides certain semantic guarantees that don't necessarily map terribly well to the physical hardware. For example, if you write to a memory address, and then read from the memory address in the "next instruction", you expect the change to be immediate, even though the code is actually running on a pipeline that could be dozens of instructions deep, with several layers of cache between the core and system memory. So in a sense there's not really a qualitative difference between C and Fortran - they are both for specifying a sequence of operations on an abstract machine, relying on the compiler to implement that machine - and indeed modern optimizing C compilers actually provide very few guarantees about specific assembly instructions to be executed, happily rewriting or omitting code so long as it executes "as if" it ran on the C virtual machine. See "C is not a low-level language" - https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479
  - theodorethomas 3y ago
    I don't see how C can match Fortran's abstraction level and still reliably control hardware that uses memory-mapped I/O. C, as an operating system implementation language, is trying to do something fundamentally different than Fortran. You live by memory address, you die by memory address.
pklausler 3y ago
Subroutine and function "dummy arguments" are basically restricted references. When analyzing data dependences in the body of a subprogram, a compiler doesn't have to worry about writes to one dummy argument affecting any reads to or writes from another dummy argument. The burden is on the programmer of a procedure's caller to ensure that any dummy argument that is modified during the call does not alias with any other dummy argument. This is the big one. Fortran >= 90 has ALLOCATABLE variables (and now components), which are dynamically managed objects that are akin to std::unique_ptr<> or std::vector<> smart objects. A compiler can assume that an ALLOCATABLE object is free from aliasing, unlike a POINTER. Fortran does have POINTERs, but they can only point to objects that have been explicitly tagged with the TARGET attribute. Objects that are not TARGETs are safe from aliasing with POINTERs. Fortran does still not have the concept of a pointer to immutable data. Do any of these still yield performance advantages over C/C++? Probably not.
redandblack 3y ago
I worked at an Airline in late 80s working on numerical optimization code in fortan / IBM mainframes - integer optimization with maybe 10k rows and 10m+ columns. I remember they having specialized vector processors to run those optimizations. We used to run those schedule optimizers daily until it was time to hand it over to the next planning department. The firm also had a in-house team from IBM for help with it. Anyway the urban myth in the IT dept was, they had a custom fortan compiler written in early 70s that saved the firm from buying 2 additional IBM mainframes with the compiler writer compensated well

94 points by jokoon 3y ago

> The performance difference comes from the fact that Fortran says aliasing is not allowed [...] This is one of the legacies modern C++ is still wrestling with today, supposedly solved by Rust, Zig, Val and the likes.

fsckboy 3y ago
fortran is call by reference, it's all aliasing, you can even in some implementations change the value of literals c is call by value, you have to at least try to alias
- gpderetta 3y ago
  my understanding is that in fortran, if two arguments alias each other, it is UB. So it is not that fortran doesn't have aliasing, it is just that the optimizer assume that it doesn't (but don't quote me on this!).
  - Bostonian 3y ago
    Arguments are allowed to alias if they are both inputs that will not be changed in the function -- technically if they are both "intent(in)".
    - pklausler 3y ago
      INTENT(IN) is not required to allow aliasing of unmodified dummy arguments.
      - kergonath 3y ago
        Yes, but in practice no compiler checks whether an argument with no INTENT is modified or not, or could alias with something else or not. So the compilers tend to trust the programmer. With INTENTS, it’s obvious and much easier to check.
        pklausler 3y ago
        Dummy argument aliasing has nothing to do with INTENT.
        TheRealKing 3y ago
        I believe the NAG Fortran compiler does check for some aliasing scenarios if not all. The argument intent is irrelevant.
- gnufx 3y ago
  Fortran does not call by reference, and the term aliasing isn't used by the standard. It calls by what's usually called value-return or copy-in/copy-out, according to the "storage association" rules in the standard. While it may be true that the rules aren't generally checked by compilers, Toolpack did it for pure Fortran77 in the 1980s, though I don't know how comprehensive it actually was.
- bregma 3y ago
  It took me days one time to figure out why 0 had adopted the value four. The technicality is that Fortran is call by descriptor (not by reference, not by value). That's a powerful difference but a sharp two-edged sword.
  - kaba0 3y ago
    Could you please explain how is it differs from the other two? Never looked into Fortran.
    - kergonath 3y ago
      The Fortran standard does not specify anything about passing by references or value. Instead, it specifies the expected results. The compilers do it however they want or can, which means that they can pass pointers, values, copies, or pointers to copies depending on the situation. Sometimes (most of the time, to be fair), the compiler can work out the best solution. Sometimes, the developer has to jump through some hoops to avoid unnecessary copies.
zik 3y ago
It's true that there's no "restrict" inthe C++ standard, however all the major C++ compilers have extensions for it ("__restrict__") and have had for a long time.
- patrec 3y ago
  Except that it doesn't work and is basically WONTFIX since no C or C++ programmer ever uses it. This is only very slightly exaggerated.
  - zik 3y ago
    That's utterly incorrect. It's used thousands of times in the linux kernel.
  - Conscat 3y ago
    I've never had issues with it, personally. In what way does it not work?
    - lmkg 3y ago
      Rust has aliasing guarantees. In theory, Rust code can pass noalias annotations to LLVM and become more optimized. In practice, it took years for Rust access these optimizations because the noalias annotations kept hitting codegen bugs in LLVM. It was an area of the compiler that had not seen much active use before Rust, and they were causing a lot of dormant bugs to become issues. The feedback cycle was also very long because LLVM didn't fix the problems quickly. I didn't pay close attention so I don't know if this is prioritization or because those fixes required major surgery, but Rust sometimes had to wait for major version bumps in LLVM before seeing if "this time was the last time."
      - vlovich123 3y ago
        Yup. But it is true now though so hopefully they’ve added the test cases to LLVM to make sure things continue to work correctly in the face of more and more optimizations being thrown at code.
_flux 3y ago
Hmm, I thought basically only Rust solves it. How do Zig and Val solve it?
- Conscat 3y ago
  Multiple-aliasing cannot exist in a language without reference semantics, such as Val (or whatever its new name is).
- dralley 3y ago
  Zig doesn't solve it. I don't remember where but I know Andrew has mentioned that directly.

saagarjha 3y ago

The difference is mostly not relevant these days. Yes, restrict is important, but anyone who is optimizing numerical computing hard is probably already using it judiciously, and for them what is more important is what is familiar to them and makes them most productive.

pjmlp 3y ago
The keyword is judiciously, without triggering UB, and I bet most don't.
gnufx 3y ago
Unfortunately I see rather few people doing serious optimizing knowledgeably at all in research computing. (I know how and where it's done well, of course.) That has global effects on productivity of users, though some of the biggest gains can come from more global optimization, like in the use of MPI.

Alifatisk 3y ago

This might be a bit irrelevant but, my professor who teaches computer programming (who previously worked as a nuclear physicist) loved mentioning Fortran whenever possible. He truly enjoy the language and believes it is well suited for heavy computation. He said its used a lot for weather forecasting. He is quite old now so I don’t know how true it is today, but he made me more interested in Fortran and I might give it a shot instead of C sometime soon.

Joel_Mckay 3y ago
[flagged]
- therealcamino 3y ago
  "Most academic institutions have been sued many times due to the volume of nonsense they feed kids." I've never heard of that happening, let alone many times to most institutions.
  - Joel_Mckay 3y ago
    [flagged]
    - bananapub 3y ago
      so your weird unfounded assertion is also explicitly unprovable? awesome, thanks for sharing.
      - Joel_Mckay 3y ago
        In general lexis nexus holds the records of former students suing for BS employment promises. You would be fairly surprised who showed up besides trump u in the states. It was a lot more common than most like to admit. =)
    - jcranmer 3y ago
      > Part of the settlements is an NDA. If someone is sued, that creates a court record. Even if the terms of the settlement are not disclosed, that there was a settlement is lodged in the court record, and publicly accessible. Even if court records are sealed, the existence of the case is not, and initial complaints are rarely sealed. In short, if suing universities because of what they're teaching is routine, you should be able to point to court cases. If you can't find them, then it's not because it's secret--secret court cases don't exist--it's because they don't exist.

bregma 3y ago

Fortran is column-major and C is row-major. That difference alone makes vectorization in Fortran much much more frequently used.

defrost 3y ago
As a 60 year old C | Fortran coder since ~1980 or so, that difference makes near zero practical difference to numerical computing. Pretty much any vector|matrix operations will see row against column (or colum against row) operations in equal proportion - what you gain on the swings you lose on the roundabouts whichever way you choose to orientate storage. Indeed, given large sparse arrays, language specific default array storage methods might not even be used - and here C is (arguably) more flexible at building task specific data structures - a column(?) of pointers to row fragments (perhaps). The big advantage early FORTRAN had was baked in anti alias assumptions and a truckload of pre existing relatively accessible NIST | NASA grade quality numerical algorithms that had been crawled over, eyeballed, tested, and pounded in simulations by many qualified applied scientists.
- AnimalMuppet 3y ago
  > The big advantage early FORTRAN had was baked in anti alias assumptions and a truckload of pre existing relatively accessible NIST | NASA grade quality numerical algorithms that had been crawled over, eyeballed, tested, and pounded in simulations by many qualified applied scientists. Yeah. That Fortran library that has a solver for sparse complex matrices that won't produce nonsense if you give it a stiff problem, that has 50 years of people pounding on it to make sure that it's rock solid? I'm not sure that there's a C/C++ equivalent for it. I absolutely cannot write an equivalent. Yes, I can write a matrix solver that takes complex data. No, I can't write one that is as efficient, as numerically stable, or as bug-free as what already exists in Fortran.
- bregma 3y ago
  As a 60 year old C and Fortran coder since ~1980 or so, and who has spent the better part of the last decade supporting compilers and their internals, I say it makes a big difference when it comes to autovectorization optimizations. But that's my experience based on most users being naive and know-enough-to-be-dangerous programmers, and your experience may differ.

andy99 3y ago

In terms of being easier (for the programmer, not the compiler) one thing worth mentioning is that Fortran already has optimized intrinsics for matmul and dot_product. I was recently doing a small comparison between C and Fortran for matrix-vector multiplication and was surprised how fast the matmul is "out of the box" vs handwritten loops. Personally I also think that for code that is linear algebra heavy Fortran is easier to work with because of the intrinsics - it's analogous to numpy, whereas C takes more effort, mainly keeping track of sizes of things. https://gist.github.com/rbitr/3b86154f78a0f0832e8bd1716152365d https://gist.github.com/rbitr/3b86154f78a0f0832e8bd171615236...

physicsguy 3y ago
True for small matrices but above certain size you wouldn’t want to use these methods or loops anyway, and they’re not optimised if your matrix is sparse either. So practically you end up using a library and you’re back to basically being equivalent with C.
- leereeves 3y ago
  The Fortran matmul intrinsic doesn't use an optimal algorithm?
  - duped 3y ago
    Even theoretically this is only possible if the dimensions and sparseness are known at compile time, which can be pretty heavy assumptions.
    - jcranmer 3y ago
      The implementation of every BLAS implementation I'm aware of basically looks like this: if (dims in range A) gemm_implA else if (dims in range B) gemm_implB else if (dims in range C) gemm_implC // several more implementations... The immediate call function is invariably a switch over several implementations, and by the time you reach that call, you have actual knowledge of dimensions.

mschaef 3y ago

Back in school (mid-90's), I made a point of taking a FORTRAN introductory class to go along with all the other work I was doing in Pascal, C++, Lisp, etc. For most of the class, FORTRAN felt mostly like a penalty box. It was full of behaviors and limitations that dated straight back to the early 1960's, if not earlier. It was at least easy to make it do the things it could do. Where my opinion of FORTRAN changed was in the last time that class met. By that point, we 'knew' the language and the instructor was on to more advanced topics, including optimization. He made a point of how an optimizing FORTRAN compiler could re-nest three nested loops for the purpose of automatically improving memory access patterns. Vector Cray machines (which implemented common looping patterns in hardware) shipped with FORTRAN compilers that could take normal-looking loops and turn them into code that optimized for the hardware. I don't know if it's still the case (thanks to restrict and better compilers), but he made a compelling point at the time for the simplicity and restrictions of the language leading to more options for optimization.

dralley 3y ago
>I don't know if it's still the case (thanks to restrict and better compilers) Nobody (really) uses restrict, certainly not at scale. This is especially obvious because when Rust came along (which can effectively use restrict in most places, much like Fortran) it uncovered a lot of serious bugs in LLVM's implementation for restrict. That whole situation only really stabilized in the past year or so. Maybe GCC does a better job as a result of having gfortran.
8bitsrule 3y ago
"as of 2022 programs have been written in Fortran for over six decades and there is a vast body of Fortran software in daily use throughout the scientific and engineering communities.... It is the primary language for some of the most intensive super-computing tasks, such as in astronomy, climate modeling, computational chemistry, computational economics, computational fluid dynamics, computational physics.... Since the early 2000s, many of the widely used support libraries have also been implemented in C and more recently, in C++... For this reason, facilities for inter-operation with C [1] were added to Fortran 2003 and enhanced by the ISO/IEC technical specification 29113, which was incorporated into Fortran 2018... " [0] [0] https://en.wikipedia.org/wiki/Fortran#Science_and_engineering https://en.wikipedia.org/wiki/Fortran#Science_and_engineerin... [1] https://en.wikipedia.org/wiki/Foreign_function_interface https://en.wikipedia.org/wiki/Foreign_function_interface

jcranmer 3y ago

Fortran has a lot more (N-dimensional) array logic intrinsic to the language than C or C++ does, and in general, Fortran has a fair number of "little" things that can add up for optimization: * Fortran arrays do not alias by default. (This is the most well-known, and is indeed repeated in the answers on StackOverflow) * Fortran N-D array indexes are UB if they go out-of-bounds in a way that isn't true for most other languages. In C, if you have a declaration `int x[2][3];`, the expression `x[0][4]` is a legal way to access one of the elements--it's only UB to go outside the bounds of x as a whole, not individual rows. In Fortran, the equivalent expression (well, modified because Fortran is column-major) is illegal. * Fortran has built in array expressions--you can express a vector addition just by adding the arrays themselves. This makes it easier to autovectorize.

pklausler 3y ago
Fortran's dummy arguments can (generally) be assumed in the scope of a subprogram to not alias, whether arrays or not. The burden is on the programmer of calls to procedures to not associate any of its dummy arguments that are modified during the call with data that might be associated with any other dummy argument.
KeplerBoy 3y ago
> In C, if you have a declaration `int x[2][3];`, the expression `x[0][4]` is a legal way to access one of the elements Is it really? Does the C-compiler guarantee that your 6 ints will be stored in a contiguous memory area?
- jcranmer 3y ago
  Yes. A[i] in C is exactly equivalent to *(A + i). (Incidentally, this means that 2[x] is a legal expression in C).
  - KeplerBoy 3y ago
    Of course, that part is clear. The question is, if the two rows are guaranteed to be contiguous in memory. If using heap allocated memory, one could easily construct a counterexample (i'm not sure how the stack allocated array behaves, i guess it works because it's the sane choice, but not strictly guaranteed): int *x; int numRows = 2; int numCols = 3; x = (int *)malloc(numRows * sizeof(int )); for (int i = 0; i < numRows; i++) { x[i] = (int )malloc(numCols * sizeof(int)); } // This could segfault x[0][4] = 42;

theodorethomas 3y ago

People mention the no-aliasing, the compilers, the intrinsics, the libraries, and the expressivity but one aspect of the difference is ignored and it is this: C is a language for specifying behaviour of hardware (time sequence of detailed states), Fortran is a language for specifying computation of values. Fortran abstracts far more of the hardware than C and consequently, a Fortran compiler can benefit from quantum processors, mind-readers or time-machines, should they ever be invented. A Fortran program that reads its input, calculates and finally writes out its output does not have to execute any particular instruction at all. As long as the answer is "AS IF" it had done the user-specified computation, the Fortran compiler has done its job. In between I/O, it submerges into the ineffable like a Cold War SSBN. C is about the instruments, the players and the conductor, Fortran is about the music.

dTal 3y ago
Maybe this was once true, but the hardware that C was designed for specifying the behavior of was a PDP-11. Nowadays you are programming an abstract C-ish virtual machine that provides certain semantic guarantees that don't necessarily map terribly well to the physical hardware. For example, if you write to a memory address, and then read from the memory address in the "next instruction", you expect the change to be immediate, even though the code is actually running on a pipeline that could be dozens of instructions deep, with several layers of cache between the core and system memory. So in a sense there's not really a qualitative difference between C and Fortran - they are both for specifying a sequence of operations on an abstract machine, relying on the compiler to implement that machine - and indeed modern optimizing C compilers actually provide very few guarantees about specific assembly instructions to be executed, happily rewriting or omitting code so long as it executes "as if" it ran on the C virtual machine. See "C is not a low-level language" - https://queue.acm.org/detail.cfm?id=3212479 https://queue.acm.org/detail.cfm?id=3212479
- theodorethomas 3y ago
  I don't see how C can match Fortran's abstraction level and still reliably control hardware that uses memory-mapped I/O. C, as an operating system implementation language, is trying to do something fundamentally different than Fortran. You live by memory address, you die by memory address.

pklausler 3y ago

Subroutine and function "dummy arguments" are basically restricted references. When analyzing data dependences in the body of a subprogram, a compiler doesn't have to worry about writes to one dummy argument affecting any reads to or writes from another dummy argument. The burden is on the programmer of a procedure's caller to ensure that any dummy argument that is modified during the call does not alias with any other dummy argument. This is the big one. Fortran >= 90 has ALLOCATABLE variables (and now components), which are dynamically managed objects that are akin to std::unique_ptr<> or std::vector<> smart objects. A compiler can assume that an ALLOCATABLE object is free from aliasing, unlike a POINTER. Fortran does have POINTERs, but they can only point to objects that have been explicitly tagged with the TARGET attribute. Objects that are not TARGETs are safe from aliasing with POINTERs. Fortran does still not have the concept of a pointer to immutable data. Do any of these still yield performance advantages over C/C++? Probably not.

redandblack 3y ago

I worked at an Airline in late 80s working on numerical optimization code in fortan / IBM mainframes - integer optimization with maybe 10k rows and 10m+ columns. I remember they having specialized vector processors to run those optimizations. We used to run those schedule optimizers daily until it was time to hand it over to the next planning department. The firm also had a in-house team from IBM for help with it. Anyway the urban myth in the IT dept was, they had a custom fortan compiler written in early 70s that saved the firm from buying 2 additional IBM mainframes with the compiler writer compensated well