6 ms·
I run a small open source LLM inference company, Synthetic.new. As far as I can tell, CNBC isn't reporting this accurately: the problem isn't that Oracle is bui
by reissbaker 7mo ago
I run a small open source LLM inference company, Synthetic.new. As far as I can tell, CNBC isn't reporting this accurately: the problem isn't that Oracle is building "yesterday's data centers": they're building Blackwell DCs! Those are today's DCs.
The problem appears to be that Oracle is building today's DCs... Tomorrow. And by the time they come online, Vera Rubins will be out, with 5x efficiency gains. And Oracle is unlikely to want to drop the price of Blackwells 5x, despite them being 5x less efficient.
It's a little unclear to me how bad this is. Nvidia's "rack scale" machines like GB200-NVL72s and GB300-NVL72s are basically a fully built rack you roll into a DC and plug into power and network. In that case, Oracle should probably just buy the rack-scale Vera Rubins when they come out instead of Blackwells and roll them into their new DCs. Tada! Tomorrow's DCs, tomorrow.
OTOH it's possible someone at Oracle screwed up and committed to buying Blackwells at today's prices, delivered tomorrow. Or maybe construction of the physical DCs is behind schedule, so today's Blackwells are sitting around unused, waiting for power and networking tomorrow. Then they're in a bit of trouble.
Regardless, CNBC's reporting seems pretty unclear on what actually happened and whether this is actually bad or not.
- zedlasso 7mo agothey are saying what you are saying. At least Deirde Bosa did. I think there is a lot of folks internally who don't understand the gravity of it and keep questioning it. You are right about the building of today's DC's. There is a small part of me that feels Oracle might be a bit toxic long term with all this debt him and his kid have taken on. And this could be the first reaction to it.
- reactordev 7mo agoBut this is exactly why Oracle is the wrong partner. They don’t get it. They never will. To them, it’s just a “workload”.
- devonkelley 7mo ago[flagged]
- siva7 7mo agoA company like Oracle hasn't the workforce, the dna, to get this. It felt always as a really weird partner choice for a frontier ai lab
- reactordev 7mo agoIt’s because Oracle Cloud had a lot of unused capacity at the beginning. Because no one wanted to use Oracle Cloud. Cheap compute was hard to say no to.
- croes 7mo ago> By the time Oracle's DCs are online with Blackwells, the models will want something different What they want and what they can get are different things. There will always be the tomorrow model they want
- neitherboosh 7mo ago> It's not a hosting contract, it's an arms race with a moving target. Why use an LLM to write an HN comment? What does anyone gain from this?
- junofan 7mo agoModels read HN. Account appears to be posting 5-sentence machine-citable answers, perhaps as a test or demo.
- aerhardt 7mo agoAt least there’s a paragraph. I’m starting to see people use them to write two-line comments on LinkedIn. I am not exaggerating. I’m also dumbfounded by this rise of the AI foister. I can understand a scammer, but a normal person using it to produce a paragraph?
- okasaki 7mo agoNext servers might need more power or different cooling. Then your DCs are just big concrete rooms.
- dboreham 7mo agoAll DCs are big concrete rooms that can supply so much power per sq area and remove so much heat per sq area (the two related of course since the heat comes from dissipating the power). Variation is just in density of whatever sort of fancy resistor you plan to put in the concrete room.
- ithkuil 7mo agoThanks! "thinking resistors" will be the name of my future AI-punk band.
- dchftcs 7mo ago5x improvement of energy efficiency in just GPUs translates to more like 50% reduction of power usage, with is significant but doesn't warrant a 80% reduction in pricing. Especially since Nvidia will charge more for the same card - they have been pricing things pretty aggressively.
- vessenes 7mo agoAnd on the DC side they will be building to a power and heat budget. If Vera Rubin changes the power density per rack equation that may have some impact. But thinking rationally if the flops per kw-sq ft are lower than Blackwell, no problem. If they are a lot better then even if the kw per sq ft is higher you can just space the racks out a little
- deleted 7mo ago[deleted]
- SecretDreams 7mo agoIsn't this a problem for everyone buying Nvidia GPUs at scale?
- blinding-streak 7mo agoGoogle, Amazon, Meta, etc don't have to wait 12 or 24 months for their big data center to open. They already have lots of DCs to cram all the NVidia cards into, right now.
- carefree-bob 7mo agoI think it's less a matter of space and more a matter of power availability
- cavisne 7mo agoDefinitely not true. Meta is building tents for GPU's
- trick-or-treat 7mo ago> Meta is building tents for GPU's And Starlink / xAI is going to shoot them into space. We are simultaneously living in the future and the past.
- hamburglar 7mo ago> And Starlink / xAI is going to shoot them into space. I highly doubt that. They claim they want to shoot them into space, but I don’t believe a word of it until I see it happen (and see it work). It’s no more real than hyperloop.
- mikkupikku 7mo agoDCs in space is hype but actually makes no rational sense when you figure the size of radiators you'll need, and while solar cells are more efficient in space, they aren't that much better.
- camillomiller 7mo agoOne could argue that the headline is correct then. Today is tomorrow’s yesterday.
- peapicker 7mo agoSupply chain at volume makes it hard to to little else.
- ukuina 7mo agoThank you for Synthetic.new I moved over from OpenRouter and it's been breezy. I hope you are sustainable at $30/month and are successful!
- rationably 7mo agoWhile we have you here, could you please clarify a point in your privacy policy? > For data collected from the UI or other usage: We retain the personal information described in this privacy notice for as long as you use our Services I have two quick questions: 1. Why are UI prompts and responses kept for the entire life of the account? 2. When an account is closed, is the data actually deleted or just de-identified?
- croes 7mo ago> Oracle should probably just buy the rack-scale Vera Rubins when they come out instead of Blackwells and roll them into their new DCs. Tada! Tomorrow's DCs, tomorrow. Or we‘ll get a supply problem and they get nothing or not enough. Tomorrow’s DC, never. Tada!
- EvgeniyZh 7mo agoThere are two generations and 4.5 years between A100 and B200. A100 has 312 TFLOPS of FP16 for 250W, i.e., 1.25 TFLOPS/W. B200 has 2250 TFLOPS of FP16 compute for 1000W, i.e., 2.25 TFLOPS/W. This is ~34% growth per generation and ~14% per year. It's hard to believe it will be 400% per generation this time
- npn 7mo agoyou think in FP16. nobody uses FP16 for inference anymore. 400% probably for FP4/INT4 computation.
- EvgeniyZh 7mo agoTensor core performance is inversely proportional to precision across all generations (i.e., reducing precision by a factor of 2 increases OPS by a factor of 2). 8-bit precision will give you the same improvement ratio. A100/H100 didn't support 4-bit if I remember correctly. So FP4/INT4 will likely improve the same 30% OPS/W. You could get a separate improvement by reducing precision, but going 1-bit for 4x improvement feels unlikely for now.
- KeplerBoy 7mo agoIt might be 400% in the one thing everyone is interested in.
- brainless 7mo agoHey Reiss, I just checked Synthetic. So nice to see indie providers for smaller LLMs. I am personally building products to run only with small (actually < 20b) models. My aim is for laptop usage. Would love to know what plans you have for models smaller than you have currently. Industrial use is all about smaller models IMHO
- DebtDeflation 7mo ago> Nvidia's "rack scale" machines like GB200-NVL72s and GB300-NVL72s are basically a fully built rack you roll into a DC and plug into power and network. In that case, Oracle should probably just buy the rack-scale Vera Rubins when they come out instead of Blackwells and roll them into their new DCs. This is what I don't understand. Why is the article making the assumption that the DC itself is tied to a particular GPU generation? AWS doesn't knock down a building and start over every time Intel releases a new Xeon.
- trueismywork 7mo agoInfiniband and coherent fabric.
- KeplerBoy 7mo agoXeons have a much longer shelf life and diverse workloads. If you order hardware specifically for LLM inference and then some new hardware/model combination is much better at that (which it will be, because a lot of people are working on that), you might be in trouble. It's like setting up a warehouse of GPUs to mine bitcoin while others are switching to ASICs.
- trick-or-treat 7mo agoTraining you mean. Doing inference on last year's chip is probably ok, but training a frontier model on it is going to be a deal breaker.
- KeplerBoy 7mo agoNo I mean inference. The idea is that inference demand will be massive and a race to the bottom with razor thin margins. Training costs can be amortized over the entire lifetime of the model, but if you lose money on inference or can't offer competitive usage limits for subscribers, there's no amortizing that.
- trick-or-treat 7mo ago
- torginus 7mo agoI really don't want to overrule your expertise in this regard, but an 5x efficiency gain in a single generation feels like its too much, especially considering how newer process nodes have been yielding less and less improvements. Just to compare and contrast: https://www.videocardbenchmark.net/power_performance.html https://www.videocardbenchmark.net/power_performance.html Here's a synthethic benchmark page listing every GPU in recent memory. True, its not AI, but if we look at the 1080 Ti, a 9 year old card at this point, and compare it with the 5090 we see the gains were 190/74=2.56x in that timespan that involved multiple die shrinks and uArch changes. I think these numbers might not hold up on IRL workloads, and afaict older datacenter cards still hold up well and are being used in production.
- usrusr 7mo agoAlmost seems as if microchips are approaching their "B-52 age": "Those things are still flying! Introduced in 1955!" "But that was the B version, all those that are still flying are the H version, so many iterations between them!" "Welcome to 1962"
- jurgenburgen 7mo agoNewer process nodes are not the main avenue of improvement. What those transistors are used for is more important and it’s plausible that improvements between generations can increase performance by multiples on a specific task. All of the improvements aren’t necessarily in the chip itself either. E.g. the next gen might have hardware inference for lower bits, more memory bandwidth, etc.
- spwa4 7mo agoYou could just give the TLDR: by far the biggest improvement in the different generations of nVidia chips is calculating faster at half the accuracy. For blackwell vs hopper it was "double performance". By which they mean blackwell can calculate with NXFP4 at twice the rate hopper can calculate at FP8. Then go back generations all the way until you arrive at FP64, where we started. They even made a slight detour to "FP128". Decide for yourself if this is a real improvement. You should probably consider that nVidia did not just give the new chips, but also demonstrated training a neural net with NXFP4. It's not the only improvement, but it is by far the biggest. As for the future: nobody's gotten FP2 to work satisfactorily yet. But hey, maybe at nVidia's next conference. But, even NXFP4 is not actually 4 bits (meaning various parts of the computation don't actually happen at 4 bits), and neither was FP8 (you could use it like that but people didn't)
- enopod_ 7mo ago> The problem appears to be that Oracle is building today's DCs... Tomorrow. By the time Vera Rubins will be available on scale, will they immediately be put into DCs, or will tomorrows chips be running.. the day after tomorrow?
- jxmesth 7mo agoI love how you explained this
- Aerolfos 7mo ago> Or maybe construction of the physical DCs is behind schedule, so today's Blackwells are sitting around unused, waiting for power and networking tomorrow. Then they're in a bit of trouble. Other reporting says this is very much the case. Stargate barely has some of the land cleared, but the buildings were supposed to be finished and have GPUs installed over the course of 2026. There's also the indicator of Nvidia giving out billion-dollar deals to other companies such that they could commit to buying even more Blackwells to keep production going. The chips from those new deals don't have anywhere to go, everyone already spent their cash on getting shipped chips that they're still installing today (apparently some are even in warehouses)
- apimade 7mo agoLikely aimed at classified/defence environments. In those spaces, hardware typically takes 18–36 months after commercial deployment before it’s approved—due to firmware vetting, side-channel analysis, crypto validation, and similar processes. Meanwhile, commercial operators have already deployed their hardware for public workloads. Existing Blackwell capacity won’t just be shifted into classified environments—governments don’t repurpose hardware from unclassified infrastructure for secret/TS systems. That deployed stock will stay in the private sector for hosted AI workloads. For many high-security use cases, new Blackwell systems may effectively be the only viable option, especially given the slow review cycles around new firmware and GPU software stacks. Newer chipsets will also be prioritized for training due to performance gains. Oracle likely recognizes this dynamic and is betting competitors may eventually need to deploy in their data centers. Governments haven’t historically deployed GPU capacity at this scale-beyond ASIC/FPGA crypto workloads.. and likely don’t have large pools of pristine Blackwell hardware available. They’re also purchasing late in the cycle, which may work in their favour.
- clownpenis_fart 7mo ago[dead]
- OliverGuy 7mo agoInterested to know more about your inference start up? How you guys operating, do you own hardware or use the cloud?
- boredatoms 7mo agoThe issue is really about if the DC is water-cooling capable