12 ms·
“Meta spent almost as much as the Manhattan Project on GPUs in today's dollars”
- Arnt 3y agoI think the overwhelming complexity of modern GPUs, CPUs, SoCs don't... overwhelm us. Compared to the process of producing a 3nm chip with all those billions of transistors, the Manhattan project isn't huge. Groundbreaking, sure. Huge, compared to the development of EUV wafers, no.
- wmf 3y ago10% is overwhelming complexity and 90% is Nvidia's profit margin.
- bawolff 3y agoIf you really believe that, i would suggest you go into the GPU business.
- 123yawaworht456 3y agowhat do you believe? that a chip and 80GB of HBM memory (a H100) cost $40000 to develop and manufacture? 90% margin is likely the low end for Nvidia's AI offerings.
- talldayo 3y agoThey should charge double that, and teach the industry what happens when they abandon Open standards.
- bawolff 3y agoUmm, no. I believe it would cost in the hundreds of millions to develop and manufacture. I dont work in hardware, so i dont know what is involved. I do however work in software, where people are constantly saying things are easy or devs are just lazy. Until they put their money where their mouth is and try it themselves. Usually they are in for a rude awekening once it becomes obvious they are full of shit.
- wmf 3y agoI should go into the GPU firmware, driver, and compiler business.
- chx 3y agoPerhaps it's worth mentioning some lesser known big achievements of the Manhattan Project. It's not the scientists. It's Hanford and Oak Ridge. Hanford was not only nuclear reactors but a city of more than 43 000 people out of literally nothing. Oak Ridge had the largest building in the world, K-25. It was only surpassed by the Boeing Everett Factory more than twenty years later. These two sites were also the major cost centers of the project. Science is cheap. Enriching uranium and manufacturing plutonium -- especially when no one did it before -- is not.
- Gare 3y ago> Science is cheap. Indeed. Sustained nuclear chain reaction was achieved in Chicago Pile-1 beneath the stands of campus football stadium at the end of 1942. That's when, scientifically, it was proven that the bomb is possible.
- adastra22 3y agoPedantic point, that's when they proved nuclear power was possible. It was still unknown whether the cross section of U-235 was large enough to permit a small, transportable bomb to go critical, and whether that reaction would have occurred fast enough to generate a significant explosion. And in the case of Plutonium, the idea of implosion hadn't even been worked out yet, let alone tested.
- chx 3y agoBy 1942 no one really had doubts about that. I mean, the Einstein–Szilard letter was written in 1939 and Bohr mentioned in lectures also in 1939 the bomb is possible but he didn't think it was possible to produce enough weapons grade uranium within the timeframe of the war. He couldn't foresee, no one could, really, what the US could do when determined.
- adastra22 3y agoThese details were covered in depth in The Making of the Atomic Bomb by Richard Rhodes. The Oppenheimer movie glosses over a lot of the technical details. It’s worth a read.
- rich_sasha 3y agoTIL; I didn't know the Manhattan project used GPUs in the first place.
- Cyphase 3y agoOf course; how else were they going to make the bomb without a bunch of Genius Physics Utilizers?
- bewaretheirs 3y agoThey used the best 1940's equivalent: IBM card tabulators. This is discussed at some length in Surely You're Joking, Mr. Feynman. My favorite part: "When I went back to work on the calculation program, I found it in a mess: There were white cards, there were blue cards, there were yellow cards, and I started to say, "You're not supposed to do more than one problem -- only one problem!" They said, "Get out, get out, get out. Wait -- and we'll explain everything." "So I waited, and what happened was this. As the cards went through, sometimes the machine made a mistake, or they put a wrong number in. What we used to have to do when that happened was to go back and do it over again. But they noticed that a mistake made at some point in one cycle only affects the nearby numbers, the next cycle affects the nearby numbers, and so on. It works its way through the pack of cards. If you have fifty cards and you make a mistake at card number thirty-nine, it affects thirty-seven, thirty-eight, and thirty-nine. The next, card thirty-six, thirty-seven, thirty-eight, thirty-nine, and forty. The next time it spreads like a disease. "So they found an error back a way, and they got an idea. They would only compute a small deck of ten cards around the error. And because ten cards could be put through the machine faster than the deck of fifty cards, they would go rapidly through with this other deck while they continued with the fifty cards with the disease spreading. But the other thing was computing faster, and they would seal it all up and correct it. Very clever. That was the way those guys worked to get speed. There was no other way. If they had to stop to try to fix it, we'd have lost time. We couldn't have got it. That was what they were doing. Of course, you know what happened while they were doing that. They found an error in the blue deck. And so they had a yellow deck with a little fewer cards; it was going around faster than the blue deck. Just when they are going crazy because after they get this straightened out, they have to fix the white deck the boss comes walking in. "Leave us alone," they say. I left them alone and everything came out. We solved the problem in time and that's the way it was."
- jeffbee 3y agoHopefully people can appreciate that a modern GPU is an achievement of physics and technology on par with the Manhattan project. EUV lithography is way off the end of the complexity scale. They had to make a bomb that was perfectly round, these days we have to make assemblies that are perfectly flat.
- salawat 3y ago...So you need three plates? Machinists have known this for over a century.
- jeffbee 3y agoI think you are underestimating the necessary planarity.
- mcmoor 3y agoIf current society collapse sometimes I wonder how many centuries will be needed to recreate 3nm from scratch. Even 1000 years doesn't seem enough.
- moxvallix 3y agoIt would seem the Manhatten project got more bang for its buck.
- Def_Os 3y agoZing!
- khazhoux 3y agoThe Manhattan Project was notoriously wasteful, in the heady WWII rush to defeat the Axis. Congress approved any and every budget request with little oversight, and the project's secrecy ensured almost total lack of accountability. Some of the most notable offenders are considered to be the roughly 3% of the budget that was dedicated to the HPD --the Hats Procurement Department-- at Oppenheimer's personal request. Another ~4% of the total budget was for the AOACD --the Acronyms and Other Abbreviations Coordination Department-- though they earned their worth when they delivered the mission-critical PLSF and TSR/Em in record time.
- afruitpie 3y agoI’m surprised how “cheap” the Manhattan Project and Apollo Program were. It’s bizarre we put people on the Moon in the ‘60s for an amount of money similar to Apple’s recent stock buybacks.
- ChrisMarshallNY 3y agoI think this comic says it all: https://pbs.twimg.com/media/DuJMkjIXcAcRru9?format=jpg https://pbs.twimg.com/media/DuJMkjIXcAcRru9?format=jpg
- deleted 3y ago[deleted]
- dehrmann 3y agoOne takeaway could be that sending people to the moon wasn't as hard as we like to think it is.
- rvnx 3y agoIt's easier than self-driving cars, because toward the moon there are practically no variables and no other drivers.
- shmeeed 3y agoYeah, I guess the several hundred thousand people involved were just slacking off a lot on government money. Please don't forget to close such statements with an /s, lest some notorious software "engineers" on HN might be led to actually believe engineering the most powerful vehicle on earth mostly by hand was a negligible task.
- csomar 3y agoIt is not. The difficulty is that, mainly, we don't know lots of the variables there; and at the time the US was in a race to do it. But the moon is just a regular small planet that is much easier to escape its gravity than earth.
- 3y ago
- xeckr 3y agoThat's about 0.1 Apollo programs.
- GenerocUsername 3y agoI'm always amazed by the quality and craft of works produced in the periods before roughly the 1960s. Not sure what changed so drastically about society around that time to make modern times so wasteful, slow, expensive, and unambitious but if we could go back, I would
- ChrisMarshallNY 3y agoI have my theories, but it's probably best that I keep them to myself, and do my best on my projects.
- vundercind 3y agoProfessional management class. Various major reforms toward “more efficient” management, first in manufacturing and such, then applied to knowledge work. Reduced specialization when it comes specifically to bureaucratic tasks in white collar work (everyone’s their own secretary, among other things, now—even doctors) [edit] anecdotally, as related by people who watched the transition happen, one of the things this new management culture screwed up was management itself, especially lower and middle management. There used to be a gradient of how much time managers spent in meetings with peers and higher-ups, with the lowest ones doing very little of that and mostly focusing on the needs of their subordinates and keeping things running well—after the shift, all levels of management got sucked into a culture of meetings, meetings, meetings, most of them very low-value. Posturing and politics soaked down into lower levels of organizations so-afflicted to a much more extreme degree than before.
- 3y ago
- idunnoman1222 3y agoTSMC’s accomplishments are just greater than putting some TNT around enriched uranium
- jauntywundrkind 3y agoTSMC has had aot of help. America helping ASML get started to power TSMC the end of a long chain of events here, is enormous amounts of science done by many, over time. Who really knows, but I tend to think the nuclear physics were pretty involved. I enjoyed the What John Von Neumann Did At Los Alamos, from the evr excellent 3quarksdaily crew, which talks about how his brilliant mind combined with his explosives expertise was key to steering the Manhattan Project towards success. The article enhanced my appreciation for the challenge. https://3quarksdaily.com/3quarksdaily/2020/10/what-john-von-neumann-really-did-at-los-alamos.html https://3quarksdaily.com/3quarksdaily/2020/10/what-john-von-...
- pfdietz 3y agoEnriched uranium was exploded in a gun bomb for the Manhattan Project, not by implosion. The lenses in Fat Man used two different explosives, in each of which TNT was a minority component.
- jiggawatts 3y agoI was trying to explain to some "civilians" just how much computer power Facebook dumped into the training of their AI models. First off, it's worth mentioning that the GPUs they're using are on the same kind of "banned for export" list as parts used in nuclear weapons. Facebook bought enough of them that the equivalent in weaponry would be a decent sized fleet of ICBMs with multiple warheads (MIRV) each. They didn't spend this cash for laughs, they're buying it for a purpose, and that purpose existed before ChatGPT was a big hit. Both Zuckerberg and Yann LeCun have stated in interviews that they're invested this money into AI for Facebook feed recommendations and for competing directly with TikTok in their Reels app. In other words, Meta aimed the digital equivalent of a weapon of mass-destruction at your kids' brains, with the express aim of getting them addicted to their products just like TikTok, but even more. If you want a visualisation of the kind of raw power that your teenager is up against, I came up with this: - A billion can be visualised as as 1m x 1m x 1m block subdivided into cubic-millimetre tiny little cubes. Think very coarse sand or very small pebbles. Just picture yourself trying to count out the grains in a block that size! - A billion-billion is a cubic kilometre of the same sized grains. - Facebook used two-trillion-trillion computations to train one of the smaller Llama 3 models! That's 2 million times the amount above, about the same as 1000 km x 1000 km covered 2 km deep. That's an area the size Egypt. Alternatively, Greenland covered 1km deep, or Texas 3 km (2 miles) deep. That's what your kid is up against. Good luck.
- ByThyGrace 3y agoIs this all speculation? Got a source for some of these big claims? Admittedly, it's a fun read.
- zargon 3y agoZuckerberg said the reason he bought so many GPUs was to build Reels to compete with TikTok. Is that the big claim you're asking about? https://www.dwarkeshpatel.com/p/mark-zuckerberg https://www.dwarkeshpatel.com/p/mark-zuckerberg
- jiggawatts 3y agoThese are all facts coming from publicly available information, such as the Llama 3 paper and recent interviews. I'd dig up the specific timestamps, but I'm working at the moment and can't spend the next hour scrolling through long interviews. The interviews are: "Mark Zuckerberg - Llama 3, $10B Models, Caesar Augustus, & 1 GW Datacenters": https://www.youtube.com/watch?v=bc6uFV9CJGg https://www.youtube.com/watch?v=bc6uFV9CJGg "Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI | Lex Fridman Podcast #416 ": https://www.youtube.com/watch?v=5t1vTLU7s40 https://www.youtube.com/watch?v=5t1vTLU7s40 The total FLOPS used in training is in bunch of places, I can't find a good link at the moment.
- Uptrenda 3y ago>History defining amounts of money being spent on AI What could go wrong.
- CyberDildonics 3y agoProbably money gets wasted and GPUs go unused.
- maratc 3y agoBoth the Manhattan Project and the Apollo Program were 0.4% of GDP. With today's GDP of $27T, $108B would be equal — Meta have only spent less than a third of that on GPUs.
- petermcneeley 3y agoProbably would be better to look at is as compared to federal government spending. https://fred.stlouisfed.org/series/FYONGDA188S https://fred.stlouisfed.org/series/FYONGDA188S
- thrdbndndn 3y agoWhy? I think comparing to GDP percentage make more sense in this case, since the point is to compare their economical scale, not how invested the gov is on these projects. (Not to mention, one side (meta) has little to do with government funding to begin with.)
- eru 3y agoWhy would that be a better or worse indicator?
- willis936 3y agoI feel like this move's the goalposts a little bit. In the recent MFTF thread people compared the entire 70 years of fusion research funding (about 80 Bn USD) to the Apollo program. This is in the context that the research is heavily underfunded given its applications and potential viability. If we agree that society got an Apollo Program's worth of value out of the Apollo Program, what value has society gained from Facebook's resource usage? A more effective manipulation engine?
- sinuhe69 3y agoValid question. But one thing is public and the other thing is (kinds of) private, so no they aren’t comparable. Besides, Meta open sourced their results, so in my opinion they should deserve a better status, allow them to access resource and indemnity like libraries. Perhaps only if they spin off their AI research into a non-profit entity, but yeah open source is digital public good and deserve a special status.
- rgmerk 3y agoThis is less impressive than it sounds. Real (inflation adjusted) US GDP is almost nine times higher than it was in 1945.
- londons_explore 3y agoA big part of this is the value of walled gardens/IP. If laws were changed so IP/data wasn't protectable, then next year there would be lots of people making cloned/modded copies of Facebook. Likewise, there would be lots of people taking Nvidias GPU design and making variations on it. (this is assuming a world where everything currently closed source is made open source through law changes) And in both cases, competition would push prices and profit margins to near zero.
- oskarkk 3y agoFor GPUs it could work as you say (except that the incentive for development of expensive and risky technologies would be gone), but I think that wouldn't help that much with things like websites. Most of the value of Facebook is in the network effect. Messenger has more than 1B users and I don't think it's a good app, Telegram is much better IMO (and is also used by almost 1B people in some countries - network effect there), and all Telegram apps are open source - well, backend isn't, but at least you can have modified clones (communicating with Telegram servers) with different features etc. And there is a small open source server implementation made by someone.
- spacebanana7 3y agoFacebook’s value is in the social network, rather than IP/data. Even if the code was all open sourced, the data was portable (e.g like open banking), and the messaging standards were open (like XMPP); the supermajority of users would keep using FB. See the example of gmail.
- notatoad 3y agoIs it public how much meta is using these GPUs for internal operations, and what kind of savings they might be seeing from that? It seems like there must be a ton of things other than the underwhelming “meta ai” chat box that they could be throwing H100s at - anything from content recommendation engines to anti-abuse to optimizing network bandwidth and traffic flows.
- pests 3y agoThe @MetaAI as a group member in group chats has been great. Riff on images together, write stories, have it impersonate authors or write chapters of a (bad) book. Revitalized a few old group chats that went days without messages to constant messing around with the AI.
- mvdtnz 3y agoThis entire "AI" paradigm is perverse. Literal truckloads of money throw into the ether (not to speak of the tons of fossil fuels burned) so we can have shitty chatbots. What does Meta have to show for all of this? My friends sending me unbearably shitty fake photos they asked the Meta Bot to "imagine" for them. What a complete and utter waste.
- lern_too_spel 3y agoThe silly AI chatbots are what have been built with their existing compute. This spending is for bigger and even more powerful models.
- justanotherjoe 3y agowe are a while into spending energy to shuffle bits around
- __lbracket__ 3y agoAnd what has it got to show for in return? A hyper PC chatbot who's 50% response is "It is important to note racial injustices..." in response to every damn question.
- moralestapia 3y agoThe math doesn't make sense, 30B on a million GPUs = 30k per GPU. They're expensive but not that expensive, and definitely not in bulk.
- kristofferR 3y agoThat's actually less than the price they fetch during this shortage: https://www.cnbc.com/2023/04/14/nvidias-h100-ai-chips-selling-for-more-than-40000-on-ebay.html https://www.cnbc.com/2023/04/14/nvidias-h100-ai-chips-sellin... https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidias-h100-ai-gpus-cost-up-to-four-times-more-than-amds-competing-mi300x-amds-chips-cost-dollar10-to-dollar15k-apiece-nvidias-h100-has-peaked-beyond-dollar40000 https://www.tomshardware.com/tech-industry/artificial-intell... However, they are likely vastly cheaper in bulk at MSRP, that's true. And Meta apparently got them before the AI hype cycle.
- 999900000999 3y agoI'm not a hardware engineer, but what are the chances this gets Moores Law'ed out in about a decade. As in within 10 years we'll have the same power with a million dollars in GPUs. With that much money I'd imagine you could purchase a GPU manufacturer or something.
- oskarkk 3y agoI've just checked and surprise, right now Nvidia is worth two times more than Meta.
- 999900000999 3y agoNvidia licenses GPUs to manufacturers, I was thinking Meta could of purchased MSI or something.
- deleted 3y ago[deleted]
- maxglute 3y agoIt's wierd TSMC isn't trillion dollar company yet. Seems like they're in position to charge Nvidia/Apple tax.
- SJC_Hacker 3y agoNot quite. Intel has many fabs and if TSMC started overcharging, you would see e.g. Apple start to drift back to x86.
- zelphirkalt 3y agoIf that statement is true, I would like to add: All financed by human attention on ads and selling out their users' personal data. Congratulations, humanity. Now, what could we have achieved in useful areas, with that kind of money?
- qp11 3y agoMore like it gets hard to achieve anything else because all other heavy duty sectors of the economy are already monopolized. When Google entered the telco space you would think they would easily take down the ancient parasites that are AT&T/Verizon/Comcast, but what happened? Same story with Apple trying to get into auto/banking (credit cards)/content. So the money is only going to be used in this way. They can't touch Exxon or Monsanto or Pfizer or Boeing. They can't do a thing about housing or education or health or banks. So burning cash on novelty becomes their default path of least resistance. "Useful areas" have all been walled up.
- phyrex 3y agoMeta does not sell personal data, they just use it for ads
- synergy20 3y agoNone of Mark's new ideas take off: VR/AR, Metaverse, now AI. All of them just burnt a HUGE amount of money he made from his old adventures. If he is not the boss, he might have got laid off couple of times long time ago.
- sidcool 3y agoMeta lost billions on Metaverse. And now spending on this with an unclear business model. Where do they get all this money from?
- syntaxing 3y agoThey made about 40B net income in 2023 (that’s right, net) despite spending billions on the metaverse. They have more than enough money in their war chest to do moonshot ideas.
- bhouston 3y agoThere is a lot of money to be made selling stuff to Meta for their latest project apparently. Figure out where they want to go and $$$.
- uyzstvqs 3y agoMetaverse was an obvious failure, but Quest is a huge success in the VR/AR market. They managed to create high quality products without it becoming overpriced novelty items (eg Valve Index, Vive, Apple Vision), while being an open ecosystem that supports OpenXR and sideloading.
- stoicjumbotron 3y agoHow is it possible that even after spending so much, they still can't catch up to OpenAI? How is OpenAI so good?
- blululu 3y agoLLAMA 3 is pretty sweet and some of their vision stuff like segment anything is cutting edge. Facebook just don’t need to publish or make demos since they have a robust path to revenue.
- syntaxing 3y agoNot a ML expert but I think “can’t” is a strong word. There was a podcast with Mark Zuckerberg where I interpret his remarks as, Meta doesn’t want to directly complete with OpenAI, their existing AI services like recommendation already consume a lot of resources. Ads has and will continue to be a cash cow and LLMs has not generated large revenue yet. OpenAI generated about 2B revenue in 2023 (I’m sure it’ll grow significantly in 2024). Meta has made 40B net income in 2023 so you can see financially, there’s no reason to compete (yet).
- arthurofbabylon 3y agoSupporting your argument, Meta is competing with OpenAI, but specifically in market position. They placed “Ask AI” prominently across their platforms (for me, at least). Meta knows where to compete, where to keep a foot in the door, and where to be a bystander.
- snats 3y agoRemember that Llama400B is still in training and could end up crushing a lot of the use cases for GPT-4
- renonce 3y agoHave you tried meta.ai which they offer for free? It’s already very competitive in terms of its abilities, they just don’t care about it or advertise it as much yet.
- eesmith 3y agoSo, less than IBM spent on System/360? Quoting Dijkstra, https://cacm.acm.org/news/an-interview-with-edsger-w-dijkstra/ https://cacm.acm.org/news/an-interview-with-edsger-w-dijkstr... > I had no idea of the power of large companies. Only recently I learned that in constant dollars the development of the IBM 360 has been more expensive than the Manhattan Project. Hee-hee! Immediately following that is: > I was beginning to see American publications in the first issue of Communications of the ACM. I was shocked by the clumsy, immature way in which they talked about computing. There was a very heavy use of anthropomorphic terminology, the "electronic brain" or "machines that think." That is absolutely killing. The use of anthropomorphic terminology forces you linguistically to adopt an operational view. And it makes it practically impossible to argue about programs independently of their being executed. Anyway, to confirm the main point, "IBM spent 5 billion dollars in mid-1960s money" says https://thehistoryofcomputing.net/the-ibm-system360 https://thehistoryofcomputing.net/the-ibm-system360 while "the Manhattan Project, which resulted in the first atomic bomb, cost $2 B". https://en.wikipedia.org/wiki/Manhattan_Project https://en.wikipedia.org/wiki/Manhattan_Project says that was 1945 dollars, so about $3.5 billion 1965 dollars using the CPI Inflation Calculator at https://www.bls.gov/data/inflation_calculator.htm https://www.bls.gov/data/inflation_calculator.htm . Going backwards, this $30 billion on GPU training is only $3 billion in 1965 dollars, making it cheaper than the 360.
- ffhhj 3y agoWhy is everyone talking about inflation? Creating a device like a modern GPU in the times of the Manhattan Project would have required building a supercomputer way bigger and more expensive than the Project itself. Tech has simply become super cheap, requiring less materials and all the research has already been paid.
- smel 3y agoMeta open sourcing LLama is just like Google open source Kubernetes to counter AWS dominance.
- Palmik 3y agoIt's really amazing that they are releasing the results of all of that compute (the Llama models). Much more efficient and environmentally friendly than keeping things closed like Google or OpenAI!
- whearyou 3y agoI find these kinds of comparison difficult to parse, especially since even aside from inflation the economies in which these spends are embedded are growing. I think it would be clearer to express these costs as percentages of the global / societal (e.g. economy of Europe, of greater Chinese sphere, etc) / national economy. Maybe I'm missing something.
- _xnmw 3y agoImagine if someone spent this much money on solving real problems ordinary people face, like the housing crisis.
- SCM-Enthusiast 3y agoThe book the sovereign individual talks alot about this. Since multi-national corporations can "tax" the world, and the goverments of the world can only tax their citizens, corporarations will only increasingly outspend governments.