5 ms·

A more meaningful adventure into microbenchmarking than my last. I look at why we no longer need to P/Invoke memcmp to efficiently compare arrays in C# / .NET.

A more meaningful adventure into microbenchmarking than my last. I look at why we no longer need to P/Invoke memcmp to efficiently compare arrays in C# / .NET. Old stackoverflow answers are a dangerous form of bit-rot. They get picked up by well-meaning developers and LLMs alike and recreated years after they are out of date.

neonsunset 2y ago
For loop regression in .NET 9, please submit an issue at dotnet/runtime. It’s yet another loop tearing miscompilation caused by suboptimal loop lowering changes if my guess is correct.
- xnorswap 2y ago
  No problem, I've raised the issue as https://github.com/dotnet/runtime/issues/114047 https://github.com/dotnet/runtime/issues/114047 .
  - neonsunset 2y ago
    Thanks!
    - jve 2y ago
      19 Hours in and that PR has already hands on from multiple people at MS. Incredible.
- neonsunset 2y ago
  UPD: For those interested, it was an interaction between microbenchmark algorithm and tiered compilation and not a regression. https://github.com/dotnet/runtime/issues/114047#issuecomment-2764768173 https://github.com/dotnet/runtime/issues/114047#issuecomment...
  - Dylan16807 2y ago
    This is a ten line function that takes half a second to run. Why do you have to call it more than 50 times before it gets fully optimized?? Is the decision-maker completely unaware of the execution time?
    - lozenge 2y ago
      The optimiser doesn't know how long optimisation will take or how much time it will save before starting the work, therefore it has to hold off on optimising not frequently called functions. There are also often multiple concrete types that can be passed in, optimising for one will not help if it is also getting called with other concrete types.
      - Dylan16807 2y ago
        > The optimiser doesn't know how long optimisation will take or how much time it will save before starting the work, therefore it has to hold off on optimising not frequently called functions. I don't buy that logic. It can use the length of the function to estimate how long it will take. It can estimate the time savings by the total amount of time the function uses. Time used is a far better metric than call count. And the math to track it is not significantly more complicated than a counter.
        gavinray 2y ago
        > It can use the length of the function to estimate how long it will take. Ah, yes, because a function that defines and then prints a 10,000 line string will take x1,000 longer to run than a 10 line function which does matrix multiplication over several billion elements.
        high_na_euv 2y ago
        I think he meant how long it will take to optimize it It is naive eitherway
timewizard 2y ago
The number of times I've caught developers wholesale copying stack overflow posts, errors and all, is far too high.
- guerrilla 2y ago
  Indeed, the problems of LLMs are not new. We just automated what people who have no idea what they are doing were doing anyway. We... optimized incompetence.
  - eddythompson80 2y ago
    This is a bit elitist isn’t it. It highly depends on the type of code copied and it’s huge part of software engineer bullishness approach to LLMs compared to most other professions. Regardless of how competent as a programmer you are, you don’t necessarily possess the knowledge/answer to “How to find open ports on Linux” or “How to enumerate child pids of a parent pid” or “what is most efficient way to compare 2 byte arrays in {insert language}” etc. A search engine or an LLM is a fine solution for those problems. You know that the answer to that question if what you’re after. I’d generally consider you knowing the right question to ask is all that matters. The answer is not interesting. It’s most likely a deeply nested knowledge about how Linux networking stack works, or how process management works on a particular OS. If that was the central point of the software we’re build (like for example we’re a Linux Networking Stack company) then by all means. It’s silly to find a lead engineer in our company who is confused about how open ports work in Linux.
    - timewizard 2y ago
      In this case it was code to generate an "oauth2 code_challenge" and the correctly URLEncode it. Instead of using replaceAll the example used replace. So only the first character in the string was getting converted. When pressed the developer said they thought their code was "too fast for the oauth server" and that's why it failed about 25% of the time. The level of disappointment I had when I found the problem was enough to be memorable, but to find the post he flat out copied on stack overflow, along with a comment below it highlighting the bug AND the fix, nearly brought be to apoplexy.

neonsunset 2y ago

For loop regression in .NET 9, please submit an issue at dotnet/runtime. It’s yet another loop tearing miscompilation caused by suboptimal loop lowering changes if my guess is correct.

xnorswap 2y ago
No problem, I've raised the issue as https://github.com/dotnet/runtime/issues/114047 https://github.com/dotnet/runtime/issues/114047 .
- neonsunset 2y ago
  Thanks!
  - jve 2y ago
    19 Hours in and that PR has already hands on from multiple people at MS. Incredible.
neonsunset 2y ago
UPD: For those interested, it was an interaction between microbenchmark algorithm and tiered compilation and not a regression. https://github.com/dotnet/runtime/issues/114047#issuecomment-2764768173 https://github.com/dotnet/runtime/issues/114047#issuecomment...
- Dylan16807 2y ago
  This is a ten line function that takes half a second to run. Why do you have to call it more than 50 times before it gets fully optimized?? Is the decision-maker completely unaware of the execution time?
  - lozenge 2y ago
    The optimiser doesn't know how long optimisation will take or how much time it will save before starting the work, therefore it has to hold off on optimising not frequently called functions. There are also often multiple concrete types that can be passed in, optimising for one will not help if it is also getting called with other concrete types.
    - Dylan16807 2y ago
      > The optimiser doesn't know how long optimisation will take or how much time it will save before starting the work, therefore it has to hold off on optimising not frequently called functions. I don't buy that logic. It can use the length of the function to estimate how long it will take. It can estimate the time savings by the total amount of time the function uses. Time used is a far better metric than call count. And the math to track it is not significantly more complicated than a counter.
      - gavinray 2y ago
        > It can use the length of the function to estimate how long it will take. Ah, yes, because a function that defines and then prints a 10,000 line string will take x1,000 longer to run than a 10 line function which does matrix multiplication over several billion elements.
        high_na_euv 2y ago
        I think he meant how long it will take to optimize it It is naive eitherway

timewizard 2y ago

The number of times I've caught developers wholesale copying stack overflow posts, errors and all, is far too high.

guerrilla 2y ago
Indeed, the problems of LLMs are not new. We just automated what people who have no idea what they are doing were doing anyway. We... optimized incompetence.
- eddythompson80 2y ago
  This is a bit elitist isn’t it. It highly depends on the type of code copied and it’s huge part of software engineer bullishness approach to LLMs compared to most other professions. Regardless of how competent as a programmer you are, you don’t necessarily possess the knowledge/answer to “How to find open ports on Linux” or “How to enumerate child pids of a parent pid” or “what is most efficient way to compare 2 byte arrays in {insert language}” etc. A search engine or an LLM is a fine solution for those problems. You know that the answer to that question if what you’re after. I’d generally consider you knowing the right question to ask is all that matters. The answer is not interesting. It’s most likely a deeply nested knowledge about how Linux networking stack works, or how process management works on a particular OS. If that was the central point of the software we’re build (like for example we’re a Linux Networking Stack company) then by all means. It’s silly to find a lead engineer in our company who is confused about how open ports work in Linux.
  - timewizard 2y ago
    In this case it was code to generate an "oauth2 code_challenge" and the correctly URLEncode it. Instead of using replaceAll the example used replace. So only the first character in the string was getting converted. When pressed the developer said they thought their code was "too fast for the oauth server" and that's why it failed about 25% of the time. The level of disappointment I had when I found the problem was enough to be memorable, but to find the post he flat out copied on stack overflow, along with a comment below it highlighting the bug AND the fix, nearly brought be to apoplexy.