4 ms·

14% better using a model from late 2020. I’d expect this to increase materially with most recent models and better tooling to pull in relevant data, particularl

• jiggawatts 3y ago
  One issue with that is that more capable models are slower and more expensive, negating some of the benefits. I could do insane things with a hypothetical version of GPT 4 that’s private, effectively free, and a few orders of magnitude faster. “Find a security vulnerability in this” thrown at a million lines of code could be fun.
  • buildbot 3y ago
    It’ll happen someday! Given computings history of exponential increases, this will come to pass.
    • dboreham 3y ago
      No exponential performance increase for some time. Probably 20 years. Linear at best, toward flat.
      • buildbot 3y ago
        Why not?
      • avereveard 3y ago
        https://i0.wp.com/timdettmers.com/wp-content/uploads/2023/01/GPUs_Ada_performance_per_dollar6.png?ssl=1 https://i0.wp.com/timdettmers.com/wp-content/uploads/2023/01... It's linear per dollar but the price is also increasing, and that's specifically for customer hardware. Top end is still growing.
    • swexbe 3y ago
      Could it be possible! This old saint has not heard in his forest that Moore's law is dead?
      • buildbot 3y ago
        Sure, but that’s just one factor effecting this area. New algorithms, ASICs, models, etc.
        • swexbe 3y ago
          Could you really expect ASICs to increase performance exponentially if Moore's law is dead? As for new algorithms, surely this would be more of a logarithmic increase for any specific problem as it gets harder and harder to improve on existing knowledge. Has, say, sorting become exponentially faster since Tony Hoare invented quicksort in 1959? My guess is no.
          • jiggawatts 3y ago
            the NVIDIA A100 and H100 GPUs being used to run the GPT models are general purpose devices. If they had video ports, you could run 3D games on them! They're like a CPU: highly programmable, flexible, multi-purpose. This means that when any single "part" of the GPT inference code is running, it's probably using only 10-40% of the die area while the rest is not used and idle. Imagine a future where a nearly-optimal architecture for LLMs has been figured out and stops changing every few months. At that point, it would be possible to make a dedicated chip that does "nothing else but that". It would be 100% utilised and faster because it would have more units dedicated to the specific task at hand. The data paths could also be optimised to minimise wasteful trips to-and-from external memory, etc... The ideal "inference chip" would be something akin to an FPGA. Huge numbers of simple cells that can do just the dozen-or-so instructions required, with local memory for the weights and biases. Make these on a cheaper process, run them at 500 MHz for power efficiency, and tile them out by their hundreds in a hybrid memory+compute architecture. It would be relatively easy to get a factor of 10x, and I suspect 100x is possible.
          • buildbot 3y ago
  • usaar333 3y ago
    I've personally found on net it negates ALL the benefits (on net) for areas I'm at least "modestly skilled" in. That is if I could only use GPT-4 vs. GPT-3, I'd take GPT-3 because the faster speed outweighs the higher accuracy on some tasks for GPT-4.
  • brucethemoose2 3y ago
    > I could do insane things with a hypothetical version of GPT 4 that’s private, effectively free, and a few orders of magnitude faster. You are describing a privately hosted LLaMA, Falcon or Starcoder instance (albeit with lower quality than GPT4). You pay for the speed you want, though TBH they are already very fast on modest GPUs.