4 ms·

AI leaderboards are no longer useful. It's time to switch to Pareto curves

40 points by jobbagy 2y ago

ukuina 2y ago
This is the most applicable part of the article: Strategies to improve LLM accuracy: Retry: We repeatedly invoke a model with the temperature set to zero, up to five times, if it fails the test cases provided with the problem description. Retrying makes sense because LLMs aren’t deterministic even at temperature zero. Warming: This is the same as the retry strategy, but we gradually increase the temperature of the underlying model with each run, from 0 to 0.5. This increases the stochasticity of the model and, we hope, increases the likelihood that at least one of the retries will succeed. Escalation: We start with a cheap model (Llama-3 8B) and escalate to more expensive models (GPT-3.5, Llama-3 70B, GPT-4) if we encounter a test case failure.
- smaddox 2y ago
  > Retrying makes sense because LLMs aren’t deterministic even at temperature zero. This is news to me. I'm trying to think where non-determinism would come in at temperature zero, but coming up with nothing. What am I missing?
  - wongarsu 2y ago
    It can happen due to a number of reasons, but in the case of GPT-4 it's probably because of their MoE implementation https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
  - nicklecompte 2y ago
    It's because floating-point arithmetic isn't deterministic, which becomes salient when (speaking loosely) the difference between likelihood of two different tokens is less than the precision of the FPU. I am not sure to what extent this effect has been quantified.
    - exe34 2y ago
      They can be made to be deterministic on CPU, but not on GPU (unless you want to give up on the speedup). With floating points, things like addition are not associative: a + (b + c) is not the same as (a + b) + c. So on CPU, you can make sure the order is always the same and the result is deterministic. On GPU, the order is not guaranteed, and thus the output is not deterministic. This is because of the
    - chessgecko 2y ago
      Having played with this stuff its definitely spots in the expert buffers (the other comment in the thread has the link to explanation) and not the extremely small differences in floating point arithmetic. The effect from this is much much less than any change in quantization, i.e. almost impossible to see from the outputs.
      - nicklecompte 2y ago
        I guess the root cause of my claim is that OpenAI won't tell us whether or not GPT-3.5 is an MoE model, and I assumed it wasn't. Since GPT-3.5 is clearly nondeterministic at temp=0, I believed the nondeterminism was due to FPU stuff, and this effect was amplified with GPT-4's MoE. But if GPT-3.5 is also MoE then that's just wrong. What makes this especially tricky is that small models are truly 100% deterministic at temp=0 because the relative likelihoods are too coarse for FPU issues to be a factor. I had thought 3.5 was big enough that some of its token probabilities were too fine-grained for the FPU. But that's probably wrong. On the other hand, it's not just GPT, there are currently floating-point difficulties in vllm which significantly affect the determinism of any model run on it: https://github.com/vllm-project/vllm/issues/966 https://github.com/vllm-project/vllm/issues/966 Note that a suggested fix is upcasting to float32. So it's possible that GPT-3.5 is using an especially low-precision float and introducing nondeterminism by saving money on compute costs. Sadly I do not have the money[1] to actually run a test to falsify any of this. It seems like this would be a good little research project. [1] Or the time, or the motivation :) But this stuff is expensive.
    - memhole 2y ago
      I’m so glad to see LLMs spark these conversations lately. It’s been a huge gripe of mine that we don’t question the underlying precision in other areas of AI/ML
- vok 2y ago
  These strategies seem immediately practical. If you want to go beyond zero-shot for LLM coding, you may not need a complicated agent architecture - just start with escalation, retry, and warming.
wongarsu 2y ago
Alternative title: AI leaderboards would be useful if they didn't blindly believe the author's benchmarks, included good baselines, and factored in real cost to run the model (parameter count can be misleading). Pareto curves are a good tool to decide which model is the best for a given price/performance tradeoff and should be used more But that's not quite as catchy. Great article
deleted 2y ago
[deleted]
sgt101 2y ago
I have advocated and used pareto fronts as a model selection method for ML for a long while. It's really useful to construct two tests - hard but important, and run of the mill - and plot model performance against each one and draw a pareto front so you can see which of your models are off the edge. In fact if you were to look at Figure 8.3 in "Managing Machine Learning Projects" then you would see this kind of thing! But, I'm just an old bot shilling for my product.
nicklecompte 2y ago
I had a very similar comment last month - albeit more ignorant and less helpful: https://news.ycombinator.com/item?id=39957153 https://news.ycombinator.com/item?id=39957153 Basically, none of these agentic / MoE / etc papers have actually compared their results to the naive baseline: since these are nondeterministic programs, Randomized Algortihms 101 tells you that if the probability of success is sufficiently high, you can improve performance simply by running the algorithm multiple times and taking the majority/plurality result. So is MoE or agents actually more effective than doing it the dumb way? AI Snake Oil says "no." Truly bizarre that dozens of researchers didn't even ask! It made me feel like I was missing something.
sergeyk 2y ago
[dead]

40 points by jobbagy 2y ago

This is the most applicable part of the article: Strategies to improve LLM accuracy: Retry: We repeatedly invoke a model with the temperature set to zero, up to five times, if it fails the test cases provided with the problem description. Retrying makes sense because LLMs aren’t deterministic even at temperature zero. Warming: This is the same as the retry strategy, but we gradually increase the temperature of the underlying model with each run, from 0 to 0.5. This increases the stochasticity of the model and, we hope, increases the likelihood that at least one of the retries will succeed. Escalation: We start with a cheap model (Llama-3 8B) and escalate to more expensive models (GPT-3.5, Llama-3 70B, GPT-4) if we encounter a test case failure.

smaddox 2y ago
> Retrying makes sense because LLMs aren’t deterministic even at temperature zero. This is news to me. I'm trying to think where non-determinism would come in at temperature zero, but coming up with nothing. What am I missing?
- wongarsu 2y ago
  It can happen due to a number of reasons, but in the case of GPT-4 it's probably because of their MoE implementation https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
- nicklecompte 2y ago
  It's because floating-point arithmetic isn't deterministic, which becomes salient when (speaking loosely) the difference between likelihood of two different tokens is less than the precision of the FPU. I am not sure to what extent this effect has been quantified.
  - exe34 2y ago
    They can be made to be deterministic on CPU, but not on GPU (unless you want to give up on the speedup). With floating points, things like addition are not associative: a + (b + c) is not the same as (a + b) + c. So on CPU, you can make sure the order is always the same and the result is deterministic. On GPU, the order is not guaranteed, and thus the output is not deterministic. This is because of the
  - chessgecko 2y ago
    Having played with this stuff its definitely spots in the expert buffers (the other comment in the thread has the link to explanation) and not the extremely small differences in floating point arithmetic. The effect from this is much much less than any change in quantization, i.e. almost impossible to see from the outputs.
    - nicklecompte 2y ago
      I guess the root cause of my claim is that OpenAI won't tell us whether or not GPT-3.5 is an MoE model, and I assumed it wasn't. Since GPT-3.5 is clearly nondeterministic at temp=0, I believed the nondeterminism was due to FPU stuff, and this effect was amplified with GPT-4's MoE. But if GPT-3.5 is also MoE then that's just wrong. What makes this especially tricky is that small models are truly 100% deterministic at temp=0 because the relative likelihoods are too coarse for FPU issues to be a factor. I had thought 3.5 was big enough that some of its token probabilities were too fine-grained for the FPU. But that's probably wrong. On the other hand, it's not just GPT, there are currently floating-point difficulties in vllm which significantly affect the determinism of any model run on it: https://github.com/vllm-project/vllm/issues/966 https://github.com/vllm-project/vllm/issues/966 Note that a suggested fix is upcasting to float32. So it's possible that GPT-3.5 is using an especially low-precision float and introducing nondeterminism by saving money on compute costs. Sadly I do not have the money[1] to actually run a test to falsify any of this. It seems like this would be a good little research project. [1] Or the time, or the motivation :) But this stuff is expensive.
  - memhole 2y ago
    I’m so glad to see LLMs spark these conversations lately. It’s been a huge gripe of mine that we don’t question the underlying precision in other areas of AI/ML
vok 2y ago
These strategies seem immediately practical. If you want to go beyond zero-shot for LLM coding, you may not need a complicated agent architecture - just start with escalation, retry, and warming.

wongarsu 2y ago

Alternative title: AI leaderboards would be useful if they didn't blindly believe the author's benchmarks, included good baselines, and factored in real cost to run the model (parameter count can be misleading). Pareto curves are a good tool to decide which model is the best for a given price/performance tradeoff and should be used more But that's not quite as catchy. Great article

deleted 2y ago

[deleted]

sgt101 2y ago

I have advocated and used pareto fronts as a model selection method for ML for a long while. It's really useful to construct two tests - hard but important, and run of the mill - and plot model performance against each one and draw a pareto front so you can see which of your models are off the edge. In fact if you were to look at Figure 8.3 in "Managing Machine Learning Projects" then you would see this kind of thing! But, I'm just an old bot shilling for my product.

nicklecompte 2y ago

I had a very similar comment last month - albeit more ignorant and less helpful: https://news.ycombinator.com/item?id=39957153 https://news.ycombinator.com/item?id=39957153 Basically, none of these agentic / MoE / etc papers have actually compared their results to the naive baseline: since these are nondeterministic programs, Randomized Algortihms 101 tells you that if the probability of success is sufficiently high, you can improve performance simply by running the algorithm multiple times and taking the majority/plurality result. So is MoE or agents actually more effective than doing it the dumb way? AI Snake Oil says "no." Truly bizarre that dozens of researchers didn't even ask! It made me feel like I was missing something.

sergeyk 2y ago

[dead]