8 ms·
AmpereOne: Cores Are the New MHz
- phendrenad2 2y ago> Cores are the new megahertz, at least for enterprise servers I agree. It's too bad the MHz race ended though. Us personal PC users benefit from raw performance more.
- BizarroLand 2y agoI wonder if one day we will figure out a method by which the kernel can analyze the software it is running and then automatically distribute the workload in an ideal manner based on resources. Probably a moot point but most computers leave so much performance on the table due to poor software optimizations and not taking advantage of the systems full capabilities due to having to try to make software work on essentially all computers rather than laser focusing on a specific hardware set.
- cainxinth 2y ago> High-core-count servers are the cutting edge in datacenters, and they're so insane, most software doesn't even know how to handle it. Why not? We hit the single thread wall more than a decade ago. Why isn’t more software parallelized by now?
- bzzzt 2y agoAlmost all service middleware is capable of running tasks in multiple threads so when serving lots of users that's a very easy way to get parallelism. If you're just copying data around (like in an HTTP API implementation) you're mostly I/O bound anyway and adding more compute won't help that much so there's no return on investment.
- nocsi 2y agoThere's overhead with parallelization. Security issues, race conditions and resource contention. I think the big one is its ass to debug.
- zifpanachr23 2y agoNot all software that people want to run can be efficiently parallelized. There's still a place for larger caches / fatter cores instead of blowing up core count. In traditional web serving tasks and particularly for commercial cloud providers (usually renting out VM time), this isn't the case, and so I'm sure Ampere will be successful in that space.
- penguin_booze 2y agoTraditionally, one of the salient features of ARM systems was that it didn't have fans. Here we are now.
- BobbyTables2 2y agoTrue in more ways than one…
- penguin_booze 2y agoNow only fans, huh?
- zackmorris 2y agoI'm late to the party due to work and holiday gatherings, but just wanted to say that this is the first glimmer of hope on the horizon in 25 years. The slowing down of Moore's law was well-understood by the time I was graduating from college with my Electrical and Computer Engineering (ECE) degree in 1999. The DEC Alpha was 700 MHz and they had GHz chips in labs, but most of us assumed that getting past 3-4 GHz was going to be difficult or impossible due to the switching limit of silicon with respect to pipeline stages. Also, on-chip interconnect had grown from a square profile to a ribbon shape that was taller than wide, which requires waveguide analysis. And features weren't shrinking anymore, they were just adding more layers, which isn't N^2 scaling. The end arrived just after 2007 when smartphones arrived, and the world chose affordability and efficiency over performance. Everyone hopped on the GPU bandwagon and desktop computing was largely abandoned for the next 20 years. But I've been hearing more about High-Performance Computing (HPC) recently. It's multicore computing with better languages, which is the missing link between CPU and GPU. What happened was, back in the 90s, game companies pushed for OpenGL on GPUs without doing the real work of solving auto-parallelization of code (with no pragmas/intrinsics/compiler hints etc) for multicore computers first. We should have had parallel multicore, then implemented matrix and graphics libraries over that, with OpenGL as a thin wrapper over that DSP framework. But we never got it, so everyone has been manually transpiling their general solutions to domain-specific languages (DSLs) like GLSL, HLSL and CUDA, at exorbitant effort and cost with arbitrary limitations. Established players like Nvidia don't care about this, because the status quo presents barriers to entry to competitors who might disrupt their business model. Anyway, since HPC is a general computing approach, it can be the foundation for the propriety and esoteric libraries like Vulkan, Metal, etc. Which will democratize programming and let hobbyists more easily dabble in the dozen or so alternatives to Neural Nets. Especially Genetic Algorithms (GAs), which can automatically derive the complex hand-rolled implementations like Large Language Models (LLMs) and Stable Diffusion. We'll also be able to try multiple approaches in simulation and it will be easier to run a high number of agents simultaneously so we can study their interactions and learning models in a transparent and repeatable fashion akin to using reproducible builds and declarative programming. That's how far John Koza and others got with GAs in the mid-2000s before they had to be abandon them due to cost. Now, this AmpereOne costs about an order of magnitude more than would be required for a hobbyist 256 core computer. But with inflation, $5-10,000 isn't too far off from the price of desktop computers in the 80s. So I expect that prices will come down over 3-5 years as more cores and memory are moved on-chip. Note that we never really got a System on a Chip (SoC) either, so that's some more real work that remains. We also never got reconfigurable hardware with on-chip FPGA units that could solve problems with a divide and conquer strategy, a bit like Just in Time (JIT) compilers. I had detailed plans in my head for all of this by around 2010. Especially on the software side, with ideas for better languages that combine the power of Octave/MATLAB with the approachability of Go/Erlang. The hardware's pretty easy other than needing a fab, which presents a $1-10 billion barrier. That's why it's so important that AmpereOne exists as a model to copy. The real lesson for me is that our imaginations can manifest anything, but the bigger the dream, the longer it takes for the universe to deliver it. On another timeline, I might have managed it in 10 years if I had won the internet lottery, but instead I had to wait 25 as I ran the rat race to make rent. Now I'm too old and busted so it's probably too late for me. But if HPC takes off, I might find renewed passion for tech. Since this is the way we deliver Universal Basic Income (UBI) and moneyless resources through 3D printing etc outside of subscription-based AI schemes.
- znpy 2y agoWeird, but this makes me thing X86-64 might actually be better? It isn't mentioned anywhere in __this__ article, but the power draw of that chip is 276W. I got this from Phoronix [1]: > The AmpereOne A192-32X boasts 192 AmpereOne cores, a 3.2GHz clock frequency, a rated 276 Watt usage power Which is interesting because it's almost half of AMD's 192-core offering [2]. Why is this interesting? the AMD offering draws a little less than double the wattage but has hyper-threading (!!!) meaning you get 384 threads... So this means that AMD is essentially on par with ARM cpus (at least with Ampere's ARM cpus) in terms of power efficiency... Maybe a little better. I'd be more inclined to think that AMD is the king/queen of power efficiency in the datacenter rather than ARM/Ampere [3]. notes: [1]: https://www.phoronix.com/review/ampereone-a192-32x https://www.phoronix.com/review/ampereone-a192-32x [2]: https://www.youtube.com/watch?v=S-NbCPEgP1A https://www.youtube.com/watch?v=S-NbCPEgP1A [3]: regarding Graviton/Axiom or "alleged" server-class apple silicon... Their power draw is essentially irrelevant as they're all claim that cannot be tested and evaluated independently, so they don't count in my opinion.
- alfiedotwtf 2y agoLast time I looked at Ampere it was way less power hub but then the competition. EPYC is over 250W at idle! Off the top of my head, the Ampere 128 core was 250W at max load. Big difference. Only issue I had was cache size, and you could only buy them from NewEgg. Oh… and the dual socket gigabyte board had the sockets too close together that I’d you put a big ass heat sink and fan, the outlet of one fan will go straight into the inlet of the other!
- zamadatix 2y agoI think you might be comparing your memories of Ampere's CPU power draw to either a full Epyc server's wall power draw or stated TDP. The 128 core EPYC 9754 in the chart linked above has a min of 10 W and a max draw of 397 W even though it outperforms the new 192 core variant AmpereOne A192-32X which had a min of 102 W and a max of 401 W. The min on the A192 does seem anonymously high though. That said, even the M128-30 has a min of 21 W or about twice that of the AmpereOne.
- amelius 2y ago> Cores are great, but it's all about how you slice them. Don't think of this as a single 192-core server. Think of it more like 48 dedicated 4-core servers in one box. And each of those servers has 10 gigs of high-speed RAM and consistent performance. Suddenly it sounds less impressive ...
- kevingadd 2y agoTo be fair, utilizing 192 cores for a single process or operation is often exceedingly difficult. Scheduling and coordination and resource sharing are all really hard with thread counts that high, so you're probably best operating in terms of smaller clusters of 4-16 threads instead. Lots of algorithms stop scaling well around the 8-32 range.
- stonemetal12 2y agoWhen was the last time you saw a server with 10GB Ram no matter the number of cores\threads?
- Suppafly 2y ago>When was the last time you saw a server with 10GB Ram no matter the number of cores\threads? Is that a lot or a little? I have a bunch that only have 8gb, it just depends on what they are being used for.
- geerlingguy 2y agoMy two primary webservers are running on 2GB of RAM still... it depends on the needs of your application :)
- com2kid 2y agoI've run plenty of microservices with 256 or 512 GB of RAM, and they were handling large loads. So long as each request is short lived, and using a runtime with low per request overhead (e.g. Node), memory is not really a problem for many types of workloads.
- ThinkBeat 2y agoHow much does one of these servers' cost? Is the estimate that it is cheaper based on comparable server with the same or similar core count?
- voxadam 2y agoFrom the article: > the AmpereOne A192-32X is $5,555, while the EPYC 9965 is almost $15,000!
- cbmuser 2y agoSounds like x86-64 is starting to lose some of it’s market share very soon.
- wmf 2y agoHalf of all CPUs in AWS are ARM already. Update: "Over the last two years, more than 50 percent of all the CPU capacity landed in our datacenters was on AWS Graviton." So not half of all but getting there. https://www.nextplatform.com/2024/12/03/aws-reaps-the-benefits-of-the-custom-silicon-it-has-sown/ https://www.nextplatform.com/2024/12/03/aws-reaps-the-benefi...
- SahAssar 2y agoThat quote says that half of all new CPUs are Graviton. Very different.
- droideqa 2y agoI just looked it up - that is a mistaken statistic. 50% of their CPUs are not Arm, but AWS has 50% of all server-side Arm CPUs. "But that total is beaten by just one company – Amazon – which has slightly above 50 percent of all Arm server CPUs in the world deployed in its Amazon Web Services (AWS) datacenters, said the analyst."[0] [0]: https://www.theregister.com/2023/08/08/amazon_arm_servers/ https://www.theregister.com/2023/08/08/amazon_arm_servers/
- loudmax 2y agoI found this part particularly interesting: > Also, with 512 gigs of RAM and a massive CPU, it can run a 405 billion parameter Large Language Model. It's not fast, but it did run, giving me just under a token per second. If you're serious about running LLMs and you can afford it, you'll of course want GPUs. But this might be a relatively affordable way to run really huge models like Llama 405B on your own hardware. This could be even more plausible on Ampere's upcoming 512-core CPU, though RAM bandwidth might be more of a bottleneck than CPU cores. Probably a niche use case, but intriguing.
- revnode 2y agoIt's really slow. Like, unusably slow. For those interested in self-hosting, this is a really good resource: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...
- lostmsu 2y agoThis resource looks very bad to me as they don't check batched inference at all. This might make sense now when most people a just running single query at once, but pretty soon almost everything will be running queries in parallel to take advantage of the compute.
- menaerus 2y agoHow do you run multiple queries from multiple clients simultaneously on the same HW without affecting each other context?
- lostmsu 2y agoIt depends on the framework. Here's a LlamaSharp example: https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Examples/Examples/BatchedExecutorBoolQ.cs https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Exa...
- astrodust 2y agoThe article states $5,550USD or so.
- binary132 2y agonot when your throughput is dictated by how fast you can process a single task
- cbmuser 2y agoReally a pity that Oracle killed off SPARC. They already had 32-core CPUs almost a decade ago but Oracle never really understood the value that SPARC and Solaris brought to the table.
- mrbluecoat 2y agoIronically, Oracle seems to be the only cloud compute offering Ampere currently.
- geerlingguy 2y agoAzure seems to be offering Ampere-based offerings[1], as well as Hetzner[2]. [1] https://azure.microsoft.com/en-us/blog/azure-virtual-machines-with-ampere-altra-arm-based-processors-generally-available/ https://azure.microsoft.com/en-us/blog/azure-virtual-machine... [2] https://www.hetzner.com/press-release/arm64-cloud/ https://www.hetzner.com/press-release/arm64-cloud/
- mrbluecoat 2y agoOh cool, I thought they had discontinued them.
- Tuna-Fish 2y ago... SPARC was terrible. Really just utter excrement. Having a lot of cores is not a good thing when the cores are too slow. This is an interesting CPU because the cores are actually usably fast. Nearly all programs have a constant memory usage regardless of how fast the core running them is. If you deliberately make your cores half the speed, and double the amount of them, you have approximately doubled the cost of the memory you need. In aggregate, memory already costs so much more than CPUs that this is rarely if ever useful, even if it meant your cpu is free. The starting point of a design that tiles a lot of cores just needs to be ones of the fastest cores available, or it is not commercially viable.
- UltraSane 2y agoSPARC was getting quite slow compared to other architectures and it would have cost a fortune to keep it competitive.
- torginus 2y agoNow, what percentage of companies could get away with sticking a pair of these (for redundancy) and run their entire operation off of it?
- alecco 2y agoThe problem is most current systems are not designed for these new architectures. It's like distributed programming but in local silicon.
- throwaway984393 2y ago[dead]
- drmpeg 2y agoHot Chips 2024 talk on AmpereOne. https://www.youtube.com/watch?v=kCXcZf4glcM https://www.youtube.com/watch?v=kCXcZf4glcM
- bluedino 2y ago> Using the 6 U.2 slots, you could cache a hundred terabytes of data at the edge, all at ridiculously high speeds. Who's going to try this? Not Netflix but maybe Cloudflare or a CDN?
- deleted 2y ago[deleted]
- aldonius 2y ago> I'm also happy to see systems like this that run without exotic water cooling and 240V power. Always funny to see 240V power described as exotic (or at least exotic-adjacent). It's the standard across practically the entire world except North and Central America!
- unscaled 2y agoThat would actually be 230V in most countries to be nitpicky. And the split-phase nature of 240V in the US (or 200V here in Japan) does make it somewhat more exotic. I guess building codes may be more expensive when you've got two hot wires?
- UltraSane 2y agoEvery home in the US with an electric stove or water heater is using 240v electricity. It isn't really very exotic. 3 phase power to homes IS very exotic though and I'm not sure why it would be useful in homes.
- ssl-3 2y agoIn Norway, I understand that electric ranges are commonly cabled up with 3-phase wire that's about the size of a not-special extension cord here in the States. They accomplish this by using 400v. Meanwhile, also here in the States: The wiring for my very normal (240v) electric range is the size of a baby's arm and might be even stiffer. (That last claim may be a bit hyperbolic: I've bent at least my share of 6/3 wire, but I don't think I've ever bent a baby's arm.)
- UltraSane 2y agoThe 400v is the important part. Not common at all in US homes but is very common in US factories.
- kiney 2y ago3 phase power is near universal inncentral and northern europe. And it's useful for powering saunas, electric warm water or bigger tools
- NotSammyHagar 2y agoWhy do these new machines have 192 cores, not a power of 2? (3* 2^6) = 192. The article mentions 256 and 512 cores coming. I get that various components can limit scaling like say bus speed or capacity. But 192, that's wacky.
- leebriskcyrano 2y ago[flagged]
- polynomial 2y agoDon't ask GPT: > Architectural design: The physical layout of the chip may favor certain core arrangements. For example, a 48-core processor could be designed as a 6x8 grid, which efficiently utilizes the square shape of the silicon die6.[1] Major Deep Thought vibes. [1] https://www.perplexity.ai/search/why-would-a-cpu-have-48-cores-VV_S0g._TbG30CKq8A1r5Q https://www.perplexity.ai/search/why-would-a-cpu-have-48-cor...
- ksec 2y agoI worried that AmpereOne isn't competitive with ARM core design. Its 512 CPU is only equivalent to x86-64 / Zen 6 256 Core with 2 threads. Which means AmpereOne may still not be competitive against AMD Zen6c unless they have their Core IPC massively improved. But other than that we are only 2 years away from getting this into our hand a single server / node with 512 vCPU. Or may be 1024 vCPU if we do Dual Socket. Along with PCI-e 6.0 SSD. We can rent one or two of these server and forget about scaling issues for 95%+ of most use cases. Not to mention nearly all dynamic languages has gotten a lot faster with JIT.