6 ms·

Nvidia’s Vera Whitepaper Has a Thread Loose

208 points by pella 2mo ago

foota 2mo ago
This is neat. I'm only halfway through but some really cool discussion about the internals of the chips and what they offer.
davoneus 2mo ago
I'm glad to see the competition; nothing provokes meaningful change without it. Not surprised at Nvidia's blatant fabrications though; more of the same we've seen time and again (Superchip anyone, with 2+ year old designs).
twoodfin 2mo ago
I don’t think picking a handful of SPEC benchmarks that approximate today’s most common agentic workloads (compiling code, interpreting Python) and then calling them “agentic benchmarks” is misleading at all. That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.
- jcranmer 2mo ago
  The four benchmarks selected are cppcheck, llvm, cpython, and gcc [1]. These are all essentially compiler benchmarks... and all of the compiler benchmarks in SPEC cpu2026! This makes the benchmark selection somewhat suspicious to me, since it's not particularly representative of a diverse set of workloads. I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows. [1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
  - fibonacci112358 2mo ago
    The code in compilers is the closest to your typical app you can get in a benchmark like SPEC, eveerything else is actually far more specialized. Compiler code is full of small basic blocks, lots of branches, indirect memory access; it's actually harder to get good performance for such code, both for CPUs and compilers (that was part of the death of Itanium too).
    - jcranmer 2mo ago
      That's true of most of the applications in SPECint (SPECfp is a different matter); there's nothing special about compilers there. Where compiler code is going to get really unusual, I suspect, is that compilers tend to be a little mono-focused on relatively few data structures. I know I was able to get measurable (single-digit percent!) performance differences in LLVM making very small tweaks to layout in llvm::Value. By contrast, when I was working on Thunderbird, the only similarly small change I could think to make that kind of difference would be to "oops, all string functions are now a cross-DLL call" (and even then, only because string handling is so dominant in that kind of application). Another kind of difference is that the compiler-based benchmarks are going to be quite light in virtual or indirect function calls (there's more of an emphasis on switch-based dispatching than vtable-based dispatching in most compiler implementations), which is going to make it a poorer proxy for some kinds of applications.
      - 14113 2mo ago
        Seconding this - having worked on LLVM and Firefox, the performance tuning of each application was very different. Even measuring the performance of an application like Firefox (in a meaningful way) is non-trivial, wheras compilers are much more approachable with traditional profilers (either tracing or sampling).
    - lstodd 2mo ago
      Death of Itanium was that it was a) VLIW and b) Intel was too arrogant. So it went to the same destination as later Larrabee and ATI/AMD attempts at VLIW GPUs. That is, nowhere. Also you are wrong and anyone sizing up an arch to put their loads onto must first try that load on it and not rely on "bah, compilers compile on it".
- rbanffy 2mo ago
  Benchmarks are benchmarks. You can base your decisions on them, because good numbers on them will give you good numbers on other related things you do, but nobody who has been in this industry for more than a year will take a benchmark as a guarantee those same numbers for your own workloads. Benchmarking is tricky. The only one that counts is your software running the way you run it. I often run a profiler while running unit/integration tests, but I know the results will not replicate actual use - it's just a proxy, because I don't want to profile everything in production unless the profiler has almost zero cost.
brcmthrowaway 2mo ago
So, is this rip AMD
- wmf 2mo ago
  AMD announced a faster CPU two days later.
  - allenrb 2mo ago
    It’s a time-honored tradition to compare your upcoming product to the competition’s old news. That said, looks like an impressive core and complete CPU built with it. Would love to see a smaller, affordable version.
- jnaina 2mo ago
  Olympus’s raison d’être is to orchestrate GPUs—handling the control-heavy, latency-sensitive workloads that keep Rubin fed and the AI factory running without bottlenecks. Including tool calling and data marshaling. They are not competing in the CPU space. Different markets.
transcriptase 2mo ago
Let’s not forget the fact that despite their new market cap and being the beneficiary of having a near monopoly on making incredibly complex pickaxes during a gold rush… Nvidia is still the company with a long and consistent history of misleading their customers via marketing. Their greatest hits include: - The digital equivalent of the VW emissions scandal where drivers detected when they were being benchmarked and altering rendering for better results - Selling GPUs as having 4gb vram when it was only 3.5gb usable, the remaining 0.5 being absurdly slower and causing performance loss when used - Using intentionally misleading naming schemes to obfuscate things like memory bus width being drastically different between what superficially appeared to be similarly spec’d cards A dozen other less egregious but similarly disingenuous decisions But to be clear, I’m a huge fan and continue to run Nvidia because their products are generally incredible regardless
- lostmsu 2mo ago
  You forgot their tensor core performance numbers "with sparsity".
- podocarp 2mo ago
  That 4gb vram deal sounds ahead of its time. CPU caches are tiered, why not ram? Looking forward to future systems with 8gb ddr6 and 8gb ddr5. "Swapping to ram" would become a thing.
  - kjs3 2mo ago
    There are lots of platforms where RAM is tiered, and operating systems that support it. For example, CLX allows for a "NUMA node without CPU" abstraction; there was a HN article recently about Facebook cooking up their own silicon and Linux patches to implement this so they could expand DDR5 platforms with retired DDR4 memory. Not a new idea...20+ years ago I used mc68k & ns32k BSD systems that had a couple of meg of fast memory onboard and more, slower memory hung off a VMEBus. System tried to keep hot pages in fast memory and it worked reasonably well.
  - Dylan16807 2mo ago
    Xbox has that right now. Some memory channels have 1GB chips while some have 2GB chips. So part of your memory space uses all channels and part uses more like half. That GPU was much worse though. If that .5GB had been moderately slower it wouldn't have gotten the same attention. But because it was a weird backup path to that segment of memory, on a design that normally runs all segments in parallel, it ran at 1/7 the speed of everything else. Overflowing into it was devastating.
amluto 2mo ago
In today’s “agentic” world, everyone seems to have forgotten approximately everything we used to know about security. And this new CPU is going all in on value speculation. Delightful. Maybe if “cyber” models get good enough at exploiting speculation attacks, people will start demanding equipment that is less prone to these attacks.
- archipelago123 2mo ago
  Attacker: I can run any code on this machine? Time for speculation attacks! Attacker: Oh wait, I can run any code? I already own the machine...
  - debugnik 2mo ago
    Not really, Spectre showcased an attack from within a JavaScript sandbox, which isn't considered to own the machine. Any side effects from speculation machinery can easily become a side-channel to infer values across security boundaries.
  - samrus 2mo ago
    Did you forget about spectre and meltdown?
- dzaima 2mo ago
  aarch64 has a CPU mode, DIT (Data Independent Timing), specifically for allowing software to request all fancy value prediction stuff to be disabled for the duration of processing of sensitive data. (doesn't help when the attack target is general-purpose/user-controlled code leaking things, but if you're relying on a process not leaking memory plainly available to it without full careful control of what the process runs, you've already been fully-SOL on that for decades and nothing has nor will nor can change about that)
  - amluto 2mo ago
    No way, ARM screwed this up less than Intel and at least allows user code to access the control bit. Intel’s equivalent, DOITM, is not accessible at CPL3.
pjmlp 2mo ago
So typical big corp marketing material disguised as whitepaper.
KeplerBoy 2mo ago
AMDs marketing is just as sketchy. Their new Zen6 server page claims 3.3x performance per watt over vera on "agentic workloads" for a 100 kW rack. Maybe they are comparing a CPU heavy rack to an nvidia vera rack with 50 kW of GPUs sitting idle, who knows? https://www.amd.com/en/products/processors/server/epyc/9006-series.html https://www.amd.com/en/products/processors/server/epyc/9006-...
- wtallis 2mo ago
  > who knows? You could try reading the footnotes, which include a link to https://www.amd.com/content/dam/amd/en/documents/solutions/ai/methodology-description.pdf https://www.amd.com/content/dam/amd/en/documents/solutions/a...
  - KeplerBoy 2mo ago
    That doesn't clarify it and AMD says as much "Because these estimates rely on published results, internal measurements and projection-based scaling factors, they are intended to provide directional comparison rather than direct measured rack benchmarks." It's marketing after all and nobody should make buying decisions based on that.
titzer 2mo ago
Ohhh look, value prediction. Exactly the kind of thing that led to Spectre. There will be a cottage industry of information leaks and mitigations for a decade.
- JohnMakin 2mo ago
  I've been around now long enough I'm becoming more convinced that the tech industry in a nutshell is just relearning the same things on a 10-15 year cadence
  - rbanffy 2mo ago
    Was it that long ago?
- honkostani 2mo ago
  [dead]
dannyw 2mo ago
I did appreciate the article and it's not AI slop by any means, but did anyone else notice how the language and grammar felt very LLM-written? Or at least edited from a LLM draft? I used to really enjoy Chips and Cheese's writing, not sure if they made a change.
- Lvl999Noob 2mo ago
  I haven't read any of their previous articles but I had to jump out and go through the comments here just because of how LLM-ist it felt. Right in first paragraph or two, it started feeling weird. If you say it isn't just slop, I suppose I'll push past and read it. The topic itself did seem interesting.

208 points by pella 2mo ago

foota 2mo ago

This is neat. I'm only halfway through but some really cool discussion about the internals of the chips and what they offer.

davoneus 2mo ago

I'm glad to see the competition; nothing provokes meaningful change without it. Not surprised at Nvidia's blatant fabrications though; more of the same we've seen time and again (Superchip anyone, with 2+ year old designs).

twoodfin 2mo ago

I don’t think picking a handful of SPEC benchmarks that approximate today’s most common agentic workloads (compiling code, interpreting Python) and then calling them “agentic benchmarks” is misleading at all. That you need a whole lot of “ordinary” compute to benefit from the scaling properties of agents is the reason Nvidia is making this chip in the first place.

jcranmer 2mo ago
The four benchmarks selected are cppcheck, llvm, cpython, and gcc [1]. These are all essentially compiler benchmarks... and all of the compiler benchmarks in SPEC cpu2026! This makes the benchmark selection somewhat suspicious to me, since it's not particularly representative of a diverse set of workloads. I also don't buy that it's a particularly representative set of tasks you might do with agents. Also included in the SPEC benchmarks are multimedia codecs, lossless data compression codecs, sqlite (i.e., database), all of which are going to be things you should easily throw into the sets of tasks an agentic workload might do. Cherry-picking just the compiler benchmarks instead of all of SPECint... again, it just raises a couple of eyebrows. [1] To be honest, I'm kinda surprised that both gcc and llvm are in SPEC cpu2026.
- fibonacci112358 2mo ago
  The code in compilers is the closest to your typical app you can get in a benchmark like SPEC, eveerything else is actually far more specialized. Compiler code is full of small basic blocks, lots of branches, indirect memory access; it's actually harder to get good performance for such code, both for CPUs and compilers (that was part of the death of Itanium too).
  - jcranmer 2mo ago
    That's true of most of the applications in SPECint (SPECfp is a different matter); there's nothing special about compilers there. Where compiler code is going to get really unusual, I suspect, is that compilers tend to be a little mono-focused on relatively few data structures. I know I was able to get measurable (single-digit percent!) performance differences in LLVM making very small tweaks to layout in llvm::Value. By contrast, when I was working on Thunderbird, the only similarly small change I could think to make that kind of difference would be to "oops, all string functions are now a cross-DLL call" (and even then, only because string handling is so dominant in that kind of application). Another kind of difference is that the compiler-based benchmarks are going to be quite light in virtual or indirect function calls (there's more of an emphasis on switch-based dispatching than vtable-based dispatching in most compiler implementations), which is going to make it a poorer proxy for some kinds of applications.
    - 14113 2mo ago
      Seconding this - having worked on LLVM and Firefox, the performance tuning of each application was very different. Even measuring the performance of an application like Firefox (in a meaningful way) is non-trivial, wheras compilers are much more approachable with traditional profilers (either tracing or sampling).
  - lstodd 2mo ago
    Death of Itanium was that it was a) VLIW and b) Intel was too arrogant. So it went to the same destination as later Larrabee and ATI/AMD attempts at VLIW GPUs. That is, nowhere. Also you are wrong and anyone sizing up an arch to put their loads onto must first try that load on it and not rely on "bah, compilers compile on it".
rbanffy 2mo ago
Benchmarks are benchmarks. You can base your decisions on them, because good numbers on them will give you good numbers on other related things you do, but nobody who has been in this industry for more than a year will take a benchmark as a guarantee those same numbers for your own workloads. Benchmarking is tricky. The only one that counts is your software running the way you run it. I often run a profiler while running unit/integration tests, but I know the results will not replicate actual use - it's just a proxy, because I don't want to profile everything in production unless the profiler has almost zero cost.

brcmthrowaway 2mo ago

So, is this rip AMD

wmf 2mo ago
AMD announced a faster CPU two days later.
- allenrb 2mo ago
  It’s a time-honored tradition to compare your upcoming product to the competition’s old news. That said, looks like an impressive core and complete CPU built with it. Would love to see a smaller, affordable version.
jnaina 2mo ago
Olympus’s raison d’être is to orchestrate GPUs—handling the control-heavy, latency-sensitive workloads that keep Rubin fed and the AI factory running without bottlenecks. Including tool calling and data marshaling. They are not competing in the CPU space. Different markets.

transcriptase 2mo ago

Let’s not forget the fact that despite their new market cap and being the beneficiary of having a near monopoly on making incredibly complex pickaxes during a gold rush… Nvidia is still the company with a long and consistent history of misleading their customers via marketing. Their greatest hits include: - The digital equivalent of the VW emissions scandal where drivers detected when they were being benchmarked and altering rendering for better results - Selling GPUs as having 4gb vram when it was only 3.5gb usable, the remaining 0.5 being absurdly slower and causing performance loss when used - Using intentionally misleading naming schemes to obfuscate things like memory bus width being drastically different between what superficially appeared to be similarly spec’d cards A dozen other less egregious but similarly disingenuous decisions But to be clear, I’m a huge fan and continue to run Nvidia because their products are generally incredible regardless

lostmsu 2mo ago
You forgot their tensor core performance numbers "with sparsity".
podocarp 2mo ago
That 4gb vram deal sounds ahead of its time. CPU caches are tiered, why not ram? Looking forward to future systems with 8gb ddr6 and 8gb ddr5. "Swapping to ram" would become a thing.
- kjs3 2mo ago
  There are lots of platforms where RAM is tiered, and operating systems that support it. For example, CLX allows for a "NUMA node without CPU" abstraction; there was a HN article recently about Facebook cooking up their own silicon and Linux patches to implement this so they could expand DDR5 platforms with retired DDR4 memory. Not a new idea...20+ years ago I used mc68k & ns32k BSD systems that had a couple of meg of fast memory onboard and more, slower memory hung off a VMEBus. System tried to keep hot pages in fast memory and it worked reasonably well.
- Dylan16807 2mo ago
  Xbox has that right now. Some memory channels have 1GB chips while some have 2GB chips. So part of your memory space uses all channels and part uses more like half. That GPU was much worse though. If that .5GB had been moderately slower it wouldn't have gotten the same attention. But because it was a weird backup path to that segment of memory, on a design that normally runs all segments in parallel, it ran at 1/7 the speed of everything else. Overflowing into it was devastating.

amluto 2mo ago

In today’s “agentic” world, everyone seems to have forgotten approximately everything we used to know about security. And this new CPU is going all in on value speculation. Delightful. Maybe if “cyber” models get good enough at exploiting speculation attacks, people will start demanding equipment that is less prone to these attacks.

archipelago123 2mo ago
Attacker: I can run any code on this machine? Time for speculation attacks! Attacker: Oh wait, I can run any code? I already own the machine...
- debugnik 2mo ago
  Not really, Spectre showcased an attack from within a JavaScript sandbox, which isn't considered to own the machine. Any side effects from speculation machinery can easily become a side-channel to infer values across security boundaries.
- samrus 2mo ago
  Did you forget about spectre and meltdown?
dzaima 2mo ago
aarch64 has a CPU mode, DIT (Data Independent Timing), specifically for allowing software to request all fancy value prediction stuff to be disabled for the duration of processing of sensitive data. (doesn't help when the attack target is general-purpose/user-controlled code leaking things, but if you're relying on a process not leaking memory plainly available to it without full careful control of what the process runs, you've already been fully-SOL on that for decades and nothing has nor will nor can change about that)
- amluto 2mo ago
  No way, ARM screwed this up less than Intel and at least allows user code to access the control bit. Intel’s equivalent, DOITM, is not accessible at CPL3.

pjmlp 2mo ago

So typical big corp marketing material disguised as whitepaper.

KeplerBoy 2mo ago

AMDs marketing is just as sketchy. Their new Zen6 server page claims 3.3x performance per watt over vera on "agentic workloads" for a 100 kW rack. Maybe they are comparing a CPU heavy rack to an nvidia vera rack with 50 kW of GPUs sitting idle, who knows? https://www.amd.com/en/products/processors/server/epyc/9006-series.html https://www.amd.com/en/products/processors/server/epyc/9006-...

wtallis 2mo ago
> who knows? You could try reading the footnotes, which include a link to https://www.amd.com/content/dam/amd/en/documents/solutions/ai/methodology-description.pdf https://www.amd.com/content/dam/amd/en/documents/solutions/a...
- KeplerBoy 2mo ago
  That doesn't clarify it and AMD says as much "Because these estimates rely on published results, internal measurements and projection-based scaling factors, they are intended to provide directional comparison rather than direct measured rack benchmarks." It's marketing after all and nobody should make buying decisions based on that.

titzer 2mo ago

Ohhh look, value prediction. Exactly the kind of thing that led to Spectre. There will be a cottage industry of information leaks and mitigations for a decade.

JohnMakin 2mo ago
I've been around now long enough I'm becoming more convinced that the tech industry in a nutshell is just relearning the same things on a 10-15 year cadence
- rbanffy 2mo ago
  Was it that long ago?
honkostani 2mo ago
[dead]

dannyw 2mo ago

I did appreciate the article and it's not AI slop by any means, but did anyone else notice how the language and grammar felt very LLM-written? Or at least edited from a LLM draft? I used to really enjoy Chips and Cheese's writing, not sure if they made a change.

Lvl999Noob 2mo ago
I haven't read any of their previous articles but I had to jump out and go through the comments here just because of how LLM-ist it felt. Right in first paragraph or two, it started feeling weird. If you say it isn't just slop, I suppose I'll push past and read it. The topic itself did seem interesting.