10 ms·
US Air Force connects 1,760 Playstation 3's to build supercomputer (2010)
- srvmshr 4y agoThe challenge was not rigging this up - but making things usable & efficient. The Cell Broadband Engine was notoriously hard in architecture & needed lots of optimization. Part of the reason why PS4 & Xbox later pivoted towards x86-64 ISA
- bilekas 4y agoI would have thought that the CBE was one of the reasons they chose the ps3 over conventional servers though ?
- srvmshr 4y agoIn my naive understanding, instead of modern SIMD type execution they were using some form of token passing between cores to execute instructions async. That made programming such multicore APUs pretty hard for the programmer. They had a SDK for mitigating some of these pain points. I could be somewhat wrong on the specifics (and the fact I last touched any architecture related topic about a decade ago).Please correct me if I am wrong.
- masklinn 4y agoIf you were building a supercomputer the CBE’s peculiarities (specifically the SPEs) were what you wanted. And the 360 didn’t use SPEs, it has a pretty standard 3-core CPU built out of the PPE. The weird-ass structure of the CBE was not why it switched (although PPC might be).
- chrisma0 4y agoThis is one of those headlines that initially reads like a joke until you think of the economics of it. "cost is about 5-10% of the cost of an equivalent system built with off-the-shelf computer parts." I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units?
- throwaway0a5e 4y ago>I wonder if this is due to the fact that the Playstation hardware is (was?) competitively priced to encourage revenue generation through games? Or was Sony simply very good at mass-producing these units? By 2010? Likely both.
- manishsharan 4y agoI thought PlayStation and XBoxes are subsidized by the Sony and Microsoft respectively as they make most of the money off the games and the network subscription. Nintendo is the only one that does not do that.
- vel0city 4y agoWhen it comes to network subscriptions, at the time of this article Playstation's online services were pretty much all free. It was one of the differentiators compared to Xbox's paid online services at the time. That said though, the original PS3 hardware was definitely sold at a negative margin and wouldn't be profitable until four years into the console's life. Article from 2010 talking about how the PS3 hardware had only just become profitable: https://www.pcworld.com/article/512740/article-4244.html https://www.pcworld.com/article/512740/article-4244.html The console released in 2006.
- chomp 4y agoThis is my understanding as well. A big part of this was the HD-DVD v Bluray war around that time - Sony winning with Bluray would be worth millions of dollars more, so they were willing to take a loss to get a Bluray player in each person's home.
- cestith 4y agoI bought my first XBox because the XBox One Series S was at one time one of the very best 4k Blu-ray players on the market at any price and also one of the less expensive ones. The fact it could also play games was merely a bonus to me at the time.
- paulmd 4y ago
- mywittyname 4y agoThis used to be fairly common, I think. I remember in college we had a lab full of PS3s because they were cheap and powerful and there was a course you could take to learn to develop on them. The lab later expanded after a big donation from nvidia of high end gaming machines to teach students CUDA. The courses sucked though, because everyone would take them just to get access to the labs.
- trimbo 4y agoRelated: Los Alamos' Roadrunner was based on the CELL as a co-processor, and was #1 in TOP500 in November 2008. https://en.wikipedia.org/wiki/Roadrunner_(supercomputer) https://en.wikipedia.org/wiki/Roadrunner_(supercomputer)
- schaefer 4y agoI used to crash their (Roadrunner Team's) lunch-time Linux kernel study group. So much fun.
- INTPenis 4y agoI remember reading about the air force buying a lot of ps3's back in the 2000s, and that just cemented to me that information is power and any government will do all they can to own as much of it as possible. IBM just published some quantum breakthrough[1] and my first thought was that it will be used by the military or intelligence agencies. Hopefully by the time this technology trickles down to more corrupt agencies the people will have caught up. You might as well be talking about magic when it comes to quantum computing and me, I have no idea if that new openssh standard really does protect against a quantum computer trying to break it. 1. https://www.techradar.com/news/ibm-claims-to-have-mapped-out-a-route-to-quantum-advantage https://www.techradar.com/news/ibm-claims-to-have-mapped-out...
- zaptrem 4y agoWouldn't seeing the Air Force clamoring for commodity gaming hardware usually destined for a kid's bedroom be an inspiring example of equal access to technology?
- smm11 4y agoJust think what they're doing with all our phones now.
- hypeatei 4y agoI believe they had to jailbreak these to get Linux installed on them. Just imagining an Air Force employee (?) browsing community modding forums and such for older firmware and tools, kinda funny.
- flutas 4y ago> I believe they had to jailbreak these to get Linux installed on them. OtherOS was a fully supported feature on the PS3, that enabled you to install Linux on a secondary partition on the built in HDD. At least until George Hotz used it to get hypervisor access and exploit the PS3. This lead to the feature being removed and him being sued by Sony. https://en.wikipedia.org/wiki/George_Hotz#Sony_lawsuit https://en.wikipedia.org/wiki/George_Hotz#Sony_lawsuit
- humanistbot 4y agoNope. PS3s were initially released with a built-in feature to install other OSes. This Air Force project got a ton of publicity. It directly led to Sony deciding to lock down the hardware and firmware on future releases.
- paulmd 4y agoSony actually officially supported it at the time... and then realized that since PS3 was sold at a massive loss, selling thousands of units to bulk customers who would never buy any games was costing them millions. Subsequent consoles dropped the "sell the hardware at a loss, make it up on games" model, they only lose money during a brief window at launch and then economies of scale take over and most of the console life it's profitable. It's likely that other kinds of hardware also share this model but we just don't hear about it - steam deck is running tight margins and day-1 sales are more expensive than the "average" unit sold mid-gen or late-gen, so similarly they are probably selling day-1 units at a loss even if they make a profit on the expected cost. https://www.extremetech.com/gaming/325504-sony-finally-turns-a-profit-on-499-ps5 https://www.extremetech.com/gaming/325504-sony-finally-turns... Of course... even though sony is selling the hardware at a profit, they still maintain the lock-in, just like apple/etc. The linux feature never came back even though the reason for the removal went away.
- AdmiralAsshat 4y ago> The PS3s are the older, larger variety, since the newer slim models don't allow for the installation of Linux. Big caveat there. Sony later removed the ability to install "OtherOS" (a.k.a. Linux or FreeBSD) on PS3s from those models. [0] I wonder if that firmware update basically neutered the supercomputer or if they were able to keep them from updating automatically. [0] https://en.wikipedia.org/wiki/OtherOS https://en.wikipedia.org/wiki/OtherOS
- eslaught 4y ago1. No one updates supercomputers automatically. Instead you firewall the compute nodes and have a team dedicated to maintenance and support to do periodic upgrades. 2. I'm sure they had a support contract, which may have included running an entirely different firmware. I have no specific knowledge, but based on other supercomputers I've worked with, these are often very custom machines, despite the "commodity" hardware. The odds that they were running the consumer version of the firmware are low to nil.
- thatguy0900 4y agoI can't imagine Sony really gave them custom firmware for just 2k units, right? Especially since all the money from a console is from games sales. Seems like a lot of work for nothing. Maybe they hacked something in on their own though
- yellowapple 4y ago> Especially since all the money from a console is from games sales. Seems to me like that'd be more reason to offer a support contract and recoup some of the profits lost from the lack of game sales.
- thetinguy 4y agoThey did not have a support contract. https://www.syracuse.com/news/2011/03/rome_labs_supercomputer_is_mad.html https://www.syracuse.com/news/2011/03/rome_labs_supercompute... > Rome Lab asked the Department of Defense for $2.5 million to assemble its supercomputer. By the time money to buy that many was approved in 2009, PlayStation 3s were hard to find. Rome Lab bought as many as they could — 1,700.
- system16 4y ago> that cost is about 5-10% of the cost of an equivalent system I bet 1,760 chickens cost roughly 5-10% of the cost of a Clydesdale, but I know which one I'd prefer to pull a cart.
- humanistbot 4y agoLiterally every single "supercomputer" is a distributed cluster like this.
- Communitivity 4y agoI was there, but not on that project. It worked well. The reasons it was attractive were the cost savings, but more importantly the unique aspects of the Cell processor in the PS3s.
- PartiallyTyped 4y agoWhy is that? What was it about the Cell processor at the time that was desirable? Afaik PS3s were sold at a loss for Sony, so it seems likely that they were very beefy computationally wise, but I am use USAF could have gotten an incredible deal with Intel, IBM or AMD, so what is it?
- Communitivity 4y agoIBM, Sony, and Toshiba worked together to develop STI, the architecture behind the Cell and PS3. What was really unique about the Cell was two things: it was the first dual-core processor machine, and it had 8 specialized streamlined cores as well: the SPEs (Synergistic Processing Elements). Michael Van de Luur of Gorilla Games said in an interview [1] “Even desktop chips nowadays, the fastest Intel stuff you can buy, is not by far as powerful as the Cell CPU, but it’s very difficult to get power out of the Cell. I think it was ahead of its age, because it was a little bit more like how GPUs work nowadays, but it was maybe not balanced nicely and it was too hard to use. It overshot a little bit in power and undershot in usability, but it was definitely visionary.” [1] https://www.gtplanet.net/playstation-3-cell-more-powerful-modern-chips/#:~:text=interview%20with%20Gameinformer https://www.gtplanet.net/playstation-3-cell-more-powerful-mo...
- danso 4y agoWhat's the best lengthy write-up of the Air Force's PS3 cluster (if any exist)? From the headlines, I had always thought it to be some marketing gimmick.
- dc-programmer 4y agoThe most surprising aspect of this story to me is that the Air Force was allowed to exercise this level of resourcefulness. I just assumed that the procurement process would stipulate 10 year support contracts or something similar.
- fdr 4y agome too. Large, even medium-large private bureaucracies may have also made it a challenge.
- smallmouth 4y agoMost would be quite surprised by how resourceful the military can get when a need arises. It's simultaneously impressive and scary.
- tomrod 4y agoThere is a lot of innovation in the contracting/procurement space with Government. As an example, check out AFWERX.
- bitwize 4y agoIf some officer can cobble together a proof of concept from COTS parts and show that it is just as reliable and effective as the boutique stuff from Raytheon, et al., at a way cheaper price, you bet that it will be met with considerable attention if not approval. Most ground-based military drones these days are controlled with Xbox controllers because those controllers are cheap and ubiquitous. In the 90s, Doom (with mods) was pressed into service as a simulator to help teach Marines fireteam tactics. About the most stiff-necked branch of the U.S. military is the Navy. I'd be more surprised if the Navy approved a PS3 supercomputer, especially if it were to be run aboard ship.
- blueboo 4y agoIs there any evidence of the resulting “supercomputer” being used? It’s one thing to buy two thousand PS3s. Another to deploy them into a data center. And yet another to build processes and software that uses it. The challenges involved seem discouraging.
- thetinguy 4y agoIt wasn’t just the air force. https://en.wikipedia.org/wiki/PlayStation_3_cluster https://en.wikipedia.org/wiki/PlayStation_3_cluster
- deleted 4y ago[deleted]
- Scuds 4y agoUGH! All this infrastructure and all this work into a software platform and it's something that will be obsoleted in a few years by off the shelf GPUs built on an standard development environment backed by industry That's the problem of building something super-cutting-edge on a grand scale - you run the risk of making a super evolved version of an evolutionary dead end. Sure it sounds like a cool idea at the time, the Cell optimized SETI At Home demo around the PS3 launch ripped through work units far more quickly than any Intel system. I wonder what became of the cluster once it was decommissioned? Military surplus PS3s anyone?
- latchkey 4y agoIn semi-related news... https://www.techspot.com/news/93980-14800-asrock-mining-rig-powered-12-playstation-5.html https://www.techspot.com/news/93980-14800-asrock-mining-rig-...
- kitplummer 4y agoI worked on a DARPA project a few years before this - where we were using CBE as the core for a polymorphic processor (one with an FPGA attached to every IO). We were also gutting PS3s to make mission computers for early unmanned systems - running Ubuntu on top. USAF wasn't the only one - not only were there commercial supply challenges with the PS3, various components were being horded by various nation states. We were pretty sure they didn't even no what to do with the parts, but was a basic attempt to prevent projects like this from getting off the ground.
- kitplummer 4y agoAnother interesting aspect of the PS3 is it was one of the first OTS device platforms that integrated hardware DRM.
- omershapira 4y agoI'm old enough to remember the rumors about Saddam Hussein fusing clusters of 15 PS2s to make a UAV realtime controller [1]. To be fair, he could have been more upfront about it. Some marketing managers would have shown that at E3. [1] https://www.reddit.com/r/AskHistorians/comments/l3hp2i/did_saddam_hussein_really_build_a_supercomputer/ https://www.reddit.com/r/AskHistorians/comments/l3hp2i/did_s...
- jlink 4y agoMake me think of this cluster of 200 PS3 that was used in 2009 by EPFL to solve the elliptic curve discrete logarithm problem: https://www.epfl.ch/labs/lacal/articles/112bit_prime/ https://www.epfl.ch/labs/lacal/articles/112bit_prime/
- marcodiego 4y agoIn the PS2 era I considering buying it and using the "linux kit" instead of buying a desktop. I was prepared for the differences in performance but then I knew that sony didn't release all the drivers or specs and graphic acceleration was not available. Just gave up the idea. The PS3 could be used as a computer, this allowed sony to pay less taxes in the EU. Since it is such a closed platform the "install other OS" feature could be disable remotely by the vendor automatically and, I think, without any user intervention. When it became economically "better" for sony, they disable the feature. These are good examples of the problems with such closed systems.
- Barrin92 4y agoHuh I wonder if that's where Jonathan Nolan got the idea from for his show Person of Interest.
- textcortex 4y agoReminds me the scene in Chappy
- etempleton 4y agoSomeone must have had a pretty amazing internal pitch deck to get this approved.
- elkos 4y agoWhich would be a good alternative to build similar cluster supercomputers nowadays?
- torginus 4y agoThe Cell's SPEs were such an inspired design - if you look at CPU architecture, 99% of the complexity comes from pretending there's a flat memory space - data hazards, cache coherency, latency hiding, prediction etc. If you make a beefy processor that works like a microcontroller - reading and writing everything from SRAM, and making all main memory DMA explicit - you get all the speed a fraction of the logic budget. If you look at a modern CPU pipeline, it's 20ish stages, with most of them about handling the aforementioned complexities, with fetch-decode-execute taking up at most like 6 of them. In MCUs (and SPEs) it's basically all there is. Unfortunately they never did figure out how to write programs for such a curiosity in an accessible manner. I'm curious if it's worth taking another crack at this architecture, this time with better tooling.
- pyinstallwoes 4y agoHave you seen https://www.greenarraychips.com/home/products/index.php https://www.greenarraychips.com/home/products/index.php