33 ms·

Meta Unveils New AI Supercomputer

156 points by davidstoker 5y ago

adamnemecek 5y ago
Wow, I hope that the surveillance state will be at last 30% more efficient.
- clows 5y ago
  or at least ads will be 2% less irrelevant.
  - tikimcfee 5y ago
    Nope, you’ll just get 10x as many ads with half the duration to minimize the amount of time your brain has to determine if something irrelevant or not. Those 5 second ads don’t cut short because they’re kind - it’s all they need to repeat to have the name, jingle, or sad-face burned into your neural net.
    - Permit 5y ago
      Can you elaborate? Since Facebook has built a large supercomputer, we should all expect to see more ads? I don't understand why the quantity of ads would increase...
  - Traubenfuchs 5y ago
    Anecdote time: For the first time, my new partner spent last week at my home, using my wifi. He is a car nerd. I am now receiving car ads that are absolutely not relevant to me. Adtech is still a bad joke.
- bee_rider 5y ago
  The cool thing about improving efficiency is that you can either keep doing what you were doing, but 30% cheaper, or you can just do 30% more of it! The best thing is, assuming the 'quality' of their product scales with the amount of work put into it, we'll get... 30% more accurate ads? Somehow they'll steal 30% of Google's lunch? Well, I don't know, but it sure looks like an incredible amount of engineering talent has been put toward getting us 30% more nothing.
  - eezurr 5y ago
    I think you're not considering the effective efficiency difference. This is what scares me about unreviewed (by society, government) advances in technology. If we increase the efficiency of something (lets say software) by 100%, all the good things that can be done with software gain a 100% efficiency. However, that does not equate to all the bad things that can be done with software gain a 100% efficiency. Many destructive actions are orders of magnitude more efficient than all things constructive (currently), so the net result is that the world gets more dangerous. For a more physical example, consider that a truck filled with powerful explosives could knock down a sky scraper. That is, for a handful of manhours, it is possible to undo the work of hundreds of thousands of manhours, plus the hundreds of thousands of manhours society would need to divert to managing the after effects of that disaster, and the emotional cost, etc. There's an underlying efficiency bonus that destructive actions have that is not being accounted for.
yosito 5y ago
How can they possibly keep the location of something like this a secret? There have to be thousands of people involved in building and maintaining it.
- changoplatanero 5y ago
  won't it just be a few racks of gpus in one one the existing giant data centers?
  - ssully 5y ago
    Super computers are much larger than a few racks of GPU's.
    - ceejayoz 5y ago
      You could fit any of the TOP500 machines into one of Facebook's datacenters, couldn't you? With room left over to spare? It's not like Facebook had to go hollow out the Moon to make this.
      - jeffbee 5y ago
        The largest machine in the top500 draws 30MW, which is getting to be close to the size of a Facebook or Google datacenter. All the rest are much smaller. Mostly people misunderstand the relationship between supercomputers and the cloud. Supercomputers are somewhat large and very specialized. Cloud datacenters are just enormous.
    - riantogo 5y ago
      Many years back the military connected some 1700 PS3s to create the world's 35th most powerful supercomputer. That needed few racks and could do 500 tflops. One latest XBox can do 12 tflops sitting in your living room. Of course the supercomputers would also have gotten magnitudes faster since. But hope this gives some sense of physical size.
- riffic 5y ago
bearjaws 5y ago
“The experiences we’re building for the metaverse require enormous compute power…and RSC will enable new AI models that can learn from trillions of examples, understand hundreds of languages, and more,” Meta CEO Mark Zuckerberg I don't really understand how AI processing is going to make the 'experiences' any better? This seems to me like investor fluff, saying they have some insane capability that other 'VR providers' don't have...
- plafl 5y ago
  There may be some possibilities. I'm not sure if it counts as AI but nevertheless a nice video: https://m.youtube.com/watch?v=BTETsm79D3A https://m.youtube.com/watch?v=BTETsm79D3A There is never enough compute power. Dwarf Fortress on a supercomputer?
- tikimcfee 5y ago
  It won’t make it better - it makes it more cost efficient to throw random numbers at a random number optimizer to increase the number of times they can report someone clicked or saw an ad. That’s it, end of story. The value ad is that the engineering community that they employ has a job, the stock stays higher because of their perceived value add to the tech, and the push to control data continues unburdened by something as trivial as a lack of compute power. Hooray. Progress.
- mark_l_watson 5y ago
  I am not sure either, but I worked on “game AI” over 20 years ago for Nintendo and Disney, and I am 100% sure that I could have used deep learning to good effect if it had been available. In the past seven years, I have been using mostly LSTM and GAN, and recommender models, BTW.
- zwaps 5y ago
  ît is not about making experiences better, it's about modeling behavior as to sell stuff
  - varelse 5y ago
    Those models are surprisingly tractable. You're nowhere near as interesting and unique as you might think you are at scale. Evidence: actual work experience at building latent representations to characterize customer behavior at FAANG. It's hard to come up with something that really gets you, but it's not hard to come up with something likely to make you spend more. You're surprisingly predictable on that axis and even if you aren't because you put the hours into being a crazy outlier, almost everyone else is, and you don't matter.
bno1 5y ago
I wonder if things like this are the real reason behind the GPU shortage. How many other AI super computers are being built right now?
- exdsq 5y ago
  I think these sorts of computers use special GPUs that are industrial and used specifically for AI/ML work. I don't believe they've powered the super computer with 3080s and I also don't think they use the same underlying chips either (albeit they are probably built with the same raw material that might be in short supply).
  - colechristensen 5y ago
    They take up chip fab capacity and that’s the bottleneck. The fact that it would be a custom die doesn’t really make a difference (and high level the features that go on the chip aren’t really all that different either, same stuff with various quantities and features tweaked)
    - terafo 5y ago
      They take up chip capacity on different fab. You can't produce gaming Ampere on TSMC. They are built on different architectures that have only name in common. The difference between "Ampere" and "Ampere" is bigger than between Volta and Turing, or maybe even than difference between Pascal and Turing.
  - capableweb 5y ago
    Good luck building special GPUs in just two years, especially with what's happening regarding chip production right now. Not sure how they could have achieved a project of this size/scope unless they use off-the-shelf components, since the backlogs are so long and have been for some time now.
    - exdsq 5y ago
      Facebook has been hiring FPGA engineers with ML experience since 2018 so I don't think this would be out of the question! But even so, Nvidia sell custom GPUs that aren't the same ones for gaming.
exdsq 5y ago
How far can we actually take current machine learning technologies by scaling the underlying hardware? Are we going to see some AI algorithms that are 20% better or an order of magnitude better? And what will that realistically look like to an end user? This will have cost a lot of money and maybe the news alone will push stock prices and mean its paid for itself but is it actually going to result in a substantially better product?
- jazzyjackson 5y ago
  I was just in a Twitter Spaces room and they have a live transcription feature, so as to be accessible and all, except the transcript was gibberish. If Facebook wants live translation in the Metaverse, they should hope this brings orders of magnitudes improvement to voice recognition, especially in languages other than english (by far the largest training set available)
  - zydex 5y ago
    I obviously don't know the parameters of the room you're referencing, but is it possible that the majority of the issue is on the side of poor user audio and a large number of simultaneous speakers? I find YouTube's transcription to be quite impressive with a handful of speakers and moderate audio quality.
- The_rationalist 5y ago
- arnaudsm 5y ago
  It's linear for now (check GPT-2 vs GPT-3), but we're close to the point of diminishing returns.
  - mindcrime 5y ago
    Part of the problem though, is that we don't know for sure what non-linearities may be lurking out there. Maybe we add 100 more "neurons" to the net and it "goes exponential" so to speak. Or maybe not. There's still a lot we don't know about the emergent properties of these systems as they scale up.
ctoth 5y ago
Anybody else get the sense that we're just totally frickin doomed? Even if Yudkowsky is off about AGI (which is a big maybe!) in what possible world will this technology be used to make our individual lives better (assuming you're not a FAIR researcher?)
- The_rationalist 5y ago
- meetups323 5y ago
  More people trapped inside on the metaverse = smaller crowds at outdoor recreation areas?
  - Traubenfuchs 5y ago
    That would be a win-win for everyone.
    - doublerabbit 5y ago
      Why?
      - Traubenfuchs 5y ago
        VR enjoyers can enjoy VR, reality enjoyers can enjoy reality. Vienna, where I am living, is completely overrun by people. Public spaces and transport at peak hours have become a mess in the last years. The first lockdown showed how beautiful the city can be if everyone stays home.
  - emerged 5y ago
    I think the real world will become populated with android avatars which are controlled by people from their VR headsets. So you’ll have small human crowds but loads of anonymous avatar androids taking all the good fishing spots, riding the trails backwards, etc. I’m joking hopefully
- 5y ago
colechristensen 5y ago
What is the difference between “a supercomputer” and “a bunch of racks of computers”? The actual difference between the two is quite diminished compared to years past and seems to reduce more to how a collection of computers is used and not what it is.
- tyingq 5y ago
  The big remaining one appears to be an unusually high speed interconnect. Infiniband, etc.
  - lmeyerov 5y ago
    Yep, hetero multigpu fleet mixing high ram GPUs (40-80GB each on each A100) as multigpus w smaller (ex: ~12-16 GB T4s) nodes, w crazy interconnects locally (nvlink) and across nodes. And storage gets fun as well, like parallel SSD arrays for 100GB+/s combined per node. Then whatever legacy+hybrid CPU stuff. Ex: for stuff like PCIe, new generations that ~10x the bandwidth you'd see in a gamer box, and like 1-2 per GPU. Varies a lot for say log mining vs NN training, and even for diff NNs. Ex: Graph NNs end up needing more balanced CPU side. Saturating a box with 500+ GB GPU RAM is fun. Only our gov users ask us for help on that typically: most of our users are commercial nowadays, but with much smaller/scaled down GPU rigs. I think that'll change as the fintechs keep improving and software gets easier, but they are still not there (outside of niches). Working on it :) (If you like writing shaders, we are hiring :D )
- cjbgkagh 5y ago
  I’d say mainly networking bandwidth.
- benstrumental 5y ago
  > What is the difference between “a supercomputer” and “a bunch of racks of computers”? In addition to the other responses, I like pointing people to this talk[1] by Jeff Hammond for a comprehensive answer to this question (you can skip to the 11:15 timestamp). [1] https://uchicago.hosted.panopto.com/Panopto/Pages/Embed.aspx?id=ce0c53bf-d799-4cac-a132-ad1901484286 https://uchicago.hosted.panopto.com/Panopto/Pages/Embed.aspx...
michelb 5y ago
All this to better predict behavior and present ads. What a waste.
- sxv 5y ago
  A waste is when you throw something into a landfill. This is weaponization by an enemy of the people.
rezonant 5y ago
Alternative headline: "Facebook patents Skynet"
pohl 5y ago
"You're going to be eaten by a bronteroc. We don't know what it means."
penjelly 5y ago
it feels like the future of companies is to increasingly give tasks to AI. So eventually we'll have massive corps that have a couple execs and a super ai making them all obscenely rich? I want to hate this idea, but it would be the same as hating machines replacing manual labor over the last 100 years. im not sure what to think, nor how to prepare myself for the next 20 years.
- paxys 5y ago
  20 years is optimistic. This future isn't something we'll have to worry about in our lifetime, if ever. People wildly overestimate the state of AI as it exists today.
- edgyquant 5y ago
  Not likely. Automation only increases productivity and companies are always looking to expand. The only thing automating jobs does it create new ones for people to work.
kleiba 5y ago
I used to work at a university where my professor had been in automatic speech recognition for a long time, but basically gave up on that line of research about 10 years ago because he figured that universities simply cannot compete budget wise with the big industry players. I suppose the same will soon be true for most ML-related areas of research sooner or later, at least as far as applied ML is concerned. Already, a substantial amount of research innovation in NLP and CV has been coming from big companies in recent years. Of course there is a discussion to be had about what that means for society at large. At this point, a lot of said companies to publish their results at conferences etc. But what if at some point they decide to be as "open" as OpenAI (ie., not)?
- stathibus 5y ago
  I hope a positive outcome of this will be that universities direct more of their research effort toward efficiency of network architectures and/or understandability.
  - ml_hardware 5y ago
    Unfortunately it will be hard to investigate properties of large, powerful neural networks without access to their trained weights. And industrial labs that spend millions of dollars training them will not be keen to share. If academics want to do research on expensive cutting-edge tech, they will have to join industrial labs or pool together resources, similar to particle physics or drug discovery research today.
- rococode 5y ago
  I think the academic side will start shifting towards research on efficiency and speed while companies will continue to push the cutting edge. In the NLP space there's been a lot of work recently around reducing model sizes, since they've started to reach the point where model weights sometimes don't fit in the memory of most GPUs. There's also projects like MarianNMT which completely abandon Python and write heavily optimized models with fast languages that can run quickly and accurately even without GPUs. I think we'll see a lot more of this, though of course there's a pretty big barrier in the sheer rarity of being good at both deep learning research and writing optimized low-level code.
karmasimida 5y ago
> Meta’s AI supercomputer houses 6,080 Nvidia graphics-processing units ..... By mid-summer, when the AI Research SuperCluster is fully built, it will house some 16,000 GPUs Honestly ... this is lot of GPUs ... but is it the biggest...? > Model training is done with mixed precision on the NVIDIA DGX SuperPOD-based Selene supercomputer powered by 560 DGX A100 servers networked with HDR InfiniBand in a full fat tree configuration. Each DGX A100 has eight NVIDIA A100 80GB Tensor Core GPUs So Nvidia used 4480 GPUs to train Megatron-Turing NLG 530B for example.
- vl 5y ago
  Honestly, this single GPU-based install is child's play compared to Google's multiple TPU exoflop supercomputers with hyper-cube optical interconnects. Google's ML setups allow synchronous weight update on thousand+ TPUs...
  - rawtxapp 5y ago
    TPUs are amazing, but in my experience, debugging issues with them can be a bit tricky. Since nvidia's gpus are more common place (especially outside gcp), you can find a lot more information when you get stuck, it's also more battle tested, etc.
    - 6gvONxR4sf7o 5y ago
      For what it's worth, jax is helpful to me here. You can drop out of the jit to debug it as if it were numpy. Of course that assumes your issues aren't with the jit itself or inside pmap, etc. That shit's hard.
  - alex_sf 5y ago
    Tbh I thought I was being trolled with 'hyper-cube optical interconnects'.
    - vl 5y ago
      Actually, you are right, I mistyped. Although hypercube interconnects exist, and were used, for example, in AS400, system in question uses hypertorus topology.
perilousacts 5y ago
Literally don't care. Facebook should not be the company with this. :/
busymom0 5y ago
Could this have been the reason for chip shortage?
- fennecfoxen 5y ago
  Do you think 6,080 GPUs — admittedly very large ones — are sufficient to explain the chip shortage?
- hmate9 5y ago
  No, this is way too small in scale vs the global demand.
- zydex 5y ago
  Nvidia and AMD shipped 12.7m cards collectively in 2021, I don't know what the breakdown is on consumer vs corporate but I find it hard to believe this had any impact. Correction: terafo pointed out they shipped 12.7m cards in Q3 2021 alone.
  - terafo 5y ago
    It was not 12.7 million in 2021. It was 12.7 million in Q3 2021.
    - zydex 5y ago
      Where did you see that? This was my source: https://www.digitaltrends.com/computing/gpu-shipments-increased-by-25-percent-despite-shortage/ https://www.digitaltrends.com/computing/gpu-shipments-increa... "Nvidia and AMD shipped 12.7 million cards in 2021" Please correct me if I'm misreading or clearly missing something.
      - terafo 5y ago
        If you look at the source article it is clearly stated. >Year over year, total AIB shipments increased by 25.7% this quarter compared to last year at 12.7 million units, and up quarter-to-quarter from 11.47 million units in Q2’21.
chundicus 5y ago
I can't shake the feelings that a trillion or a quadrillion parameters won't solve the fundamental shortcomings of ML models not being models of artificial intelligence. I guess there's no way of knowing until we reach AGI, but I've never heard a compelling argument for why pure ML would get us there. GPT3 seems more like an argument against that hypothesis (in my view) than for it. Even the best, most expensive models today are incredibly brittle for enterprise usecases that shouldn't necessarily require AGI. I've always imagined AGI (perhaps naively) as being achieved by clever usage of ML, plus some utilization of classical/symbolic AI from pre-AI winter days, plus probably some unknown elements.
- MR4D 5y ago
  I agree. I read Jeff Hawkins book On Intelligence [0] back when it came out, and it had a profound effect on my thinking. Chasing more data, aka "parameters" doesn't seem to be the right answer. I think more of a Bayes model like spam filtering, but cobbled together with other Bayes models looking at other things until something emerges that we call "intelligent". Heck, I'd consider Google's spam filtering pretty intelligent today. [0] - https://en.wikipedia.org/wiki/On_Intelligence https://en.wikipedia.org/wiki/On_Intelligence
  - jcims 5y ago
    Hawkins way of thinking really maps well for me also. It seems like that more parameters helps until it doesn't, then you need to encapsulate those networks and pin them to some reference frame, they create hierarchies of these networks and a system to generalize and compress those hierarchies (aka patterns), rinse and repeat. My brother just became a grandpa and I was watching his grandson navigate the world this past weekend. It's unbelievable how quickly the brain can extrapolate a new relationship between objects/actions/etc and then apply it elsewhere. Minimally you see it in the drinking action applied to all sorts of things, this sort of repetitive clenching/releasing of the fingers to find things to grip without looking, etc etc. Watching mom use a fork and very quickly understand how to grasp and manipulate it. The model of just training everything from exogenous data into a flat network seems like it will hit some asymptotic limit.
- 5y ago
ricardobeat 5y ago
https://archive.is/xdQtE https://archive.is/xdQtE
mawadev 5y ago
If this won't make people watch ads 24/7, then what will?
pinewurst 5y ago
Are we supposed to collectively feel, "Yay, Facebook!"?! It's a bigger tool for surveillance enablement, no different from how the CCP monitors Xinjiang cameras.
hetspookjee 5y ago
So much wh will one inference cost? I mean computing has come more efficient but I still struggle to find data on how much wh is used for some sentences of GPT3, for example.
bognition 5y ago
https://archive.md/xdQtE https://archive.md/xdQtE
reggieband 5y ago
I hate that I always end up referencing the Lex Friedman podcast but it is often relevant to discussion on HN. Recently Lex spoke with Yann LeCun and they had a brief chat about AI at Meta/Facebook [1] (where I believe Yann is currently Chief AI Scientist). He claims that AI is the core of Meta and that if you were to take ML out of Meta systems the company would literally crumble because it is completely built around AI. My feeling is this is a PR push by Facebook. All tech companies keep touting AI, especially Google but also Microsoft, Apple and Amazon. In some sense I believe these business want to control how their own success is defined. That is, they are right now convincing everyone that tech dominance is equivalent to AI dominance which is equivalent to ML dominance. In some sense this is turning into a purity test, like "which tech company is the most AI focused". I expect this kind of PR to accelerate as each company tries to prove its AI bona-fides to the market. 1. https://youtu.be/SGzMElJ11Cc?t=6597 https://youtu.be/SGzMElJ11Cc?t=6597
- acchow 5y ago
  These aren't startups trying to prove "AI Purity" for more funding. These are money printers that are optimizing how to print more money. And Facebook and Google are competing against each other for online advertising dollars (yes, the pie is also growing). Their revenues are up 60% and 40% over the past 2 years, so I doubt their AI plays are just about proving some purity game.
- throwaway423342 5y ago
  As a side note: I listened to the episode with Yann. Compared to other talks (e.g. the previous one with Brian Keating) it was a bit dull and uninteresting. The answers were not that insightful.
- mrkramer 5y ago
  There is no single AI company. These are all machine learning techniques. AI is spreadsheets on steroids not real intelligence.
- 6gvONxR4sf7o 5y ago
  I haven't listened to the podcast, but the way I read a statement like that is, "Our business needs to do things that aren't as simple as defining manual rules, but the economics of our business prevents us from just paying people to do those things at scale."
sydthrowaway 5y ago
I feel like the guy in A Canticle For Leibowitz who knew the history of the world.
keithnz 5y ago
So a company that specializes in targeted advertising to make money has invested in the most powerful AI supercomputer? Great.
deleted 5y ago
[deleted]
lemax 5y ago
It feels eerie that these trillions of parameters and exabyte sized training sets will come from harvested user data. 17 years of user activity all culminating into some.. supercomputer? I wonder how comments I wrote when I, and all the other people from my generation, were like 14 and using FB will feed into this and sort of be immortalized in this strange way.
Atlas667 5y ago
An AI that has free range to more profoundly study all the human data Facebook users generate? That sounds wicked evil. If ads, marketing and habit inducing platform designs are a problem now, imagine what this will lead to. To understand what drives your users more than the users understand it themselves and to use that understanding for profit. Intensified. And not to mention for surveillance, you know DARPA and the NSA want their hands all over this.

156 points by davidstoker 5y ago

adamnemecek 5y ago

Wow, I hope that the surveillance state will be at last 30% more efficient.

clows 5y ago
or at least ads will be 2% less irrelevant.
- tikimcfee 5y ago
  Nope, you’ll just get 10x as many ads with half the duration to minimize the amount of time your brain has to determine if something irrelevant or not. Those 5 second ads don’t cut short because they’re kind - it’s all they need to repeat to have the name, jingle, or sad-face burned into your neural net.
  - Permit 5y ago
    Can you elaborate? Since Facebook has built a large supercomputer, we should all expect to see more ads? I don't understand why the quantity of ads would increase...
- Traubenfuchs 5y ago
  Anecdote time: For the first time, my new partner spent last week at my home, using my wifi. He is a car nerd. I am now receiving car ads that are absolutely not relevant to me. Adtech is still a bad joke.
bee_rider 5y ago
The cool thing about improving efficiency is that you can either keep doing what you were doing, but 30% cheaper, or you can just do 30% more of it! The best thing is, assuming the 'quality' of their product scales with the amount of work put into it, we'll get... 30% more accurate ads? Somehow they'll steal 30% of Google's lunch? Well, I don't know, but it sure looks like an incredible amount of engineering talent has been put toward getting us 30% more nothing.
- eezurr 5y ago
  I think you're not considering the effective efficiency difference. This is what scares me about unreviewed (by society, government) advances in technology. If we increase the efficiency of something (lets say software) by 100%, all the good things that can be done with software gain a 100% efficiency. However, that does not equate to all the bad things that can be done with software gain a 100% efficiency. Many destructive actions are orders of magnitude more efficient than all things constructive (currently), so the net result is that the world gets more dangerous. For a more physical example, consider that a truck filled with powerful explosives could knock down a sky scraper. That is, for a handful of manhours, it is possible to undo the work of hundreds of thousands of manhours, plus the hundreds of thousands of manhours society would need to divert to managing the after effects of that disaster, and the emotional cost, etc. There's an underlying efficiency bonus that destructive actions have that is not being accounted for.

yosito 5y ago

How can they possibly keep the location of something like this a secret? There have to be thousands of people involved in building and maintaining it.

changoplatanero 5y ago
won't it just be a few racks of gpus in one one the existing giant data centers?
- ssully 5y ago
  Super computers are much larger than a few racks of GPU's.
  - ceejayoz 5y ago
    You could fit any of the TOP500 machines into one of Facebook's datacenters, couldn't you? With room left over to spare? It's not like Facebook had to go hollow out the Moon to make this.
    - jeffbee 5y ago
      The largest machine in the top500 draws 30MW, which is getting to be close to the size of a Facebook or Google datacenter. All the rest are much smaller. Mostly people misunderstand the relationship between supercomputers and the cloud. Supercomputers are somewhat large and very specialized. Cloud datacenters are just enormous.
  - riantogo 5y ago
    Many years back the military connected some 1700 PS3s to create the world's 35th most powerful supercomputer. That needed few racks and could do 500 tflops. One latest XBox can do 12 tflops sitting in your living room. Of course the supercomputers would also have gotten magnitudes faster since. But hope this gives some sense of physical size.
riffic 5y ago

bearjaws 5y ago

“The experiences we’re building for the metaverse require enormous compute power…and RSC will enable new AI models that can learn from trillions of examples, understand hundreds of languages, and more,” Meta CEO Mark Zuckerberg I don't really understand how AI processing is going to make the 'experiences' any better? This seems to me like investor fluff, saying they have some insane capability that other 'VR providers' don't have...

plafl 5y ago
There may be some possibilities. I'm not sure if it counts as AI but nevertheless a nice video: https://m.youtube.com/watch?v=BTETsm79D3A https://m.youtube.com/watch?v=BTETsm79D3A There is never enough compute power. Dwarf Fortress on a supercomputer?
tikimcfee 5y ago
It won’t make it better - it makes it more cost efficient to throw random numbers at a random number optimizer to increase the number of times they can report someone clicked or saw an ad. That’s it, end of story. The value ad is that the engineering community that they employ has a job, the stock stays higher because of their perceived value add to the tech, and the push to control data continues unburdened by something as trivial as a lack of compute power. Hooray. Progress.
mark_l_watson 5y ago
I am not sure either, but I worked on “game AI” over 20 years ago for Nintendo and Disney, and I am 100% sure that I could have used deep learning to good effect if it had been available. In the past seven years, I have been using mostly LSTM and GAN, and recommender models, BTW.
zwaps 5y ago
ît is not about making experiences better, it's about modeling behavior as to sell stuff
- varelse 5y ago
  Those models are surprisingly tractable. You're nowhere near as interesting and unique as you might think you are at scale. Evidence: actual work experience at building latent representations to characterize customer behavior at FAANG. It's hard to come up with something that really gets you, but it's not hard to come up with something likely to make you spend more. You're surprisingly predictable on that axis and even if you aren't because you put the hours into being a crazy outlier, almost everyone else is, and you don't matter.

bno1 5y ago

I wonder if things like this are the real reason behind the GPU shortage. How many other AI super computers are being built right now?

exdsq 5y ago
I think these sorts of computers use special GPUs that are industrial and used specifically for AI/ML work. I don't believe they've powered the super computer with 3080s and I also don't think they use the same underlying chips either (albeit they are probably built with the same raw material that might be in short supply).
- colechristensen 5y ago
  They take up chip fab capacity and that’s the bottleneck. The fact that it would be a custom die doesn’t really make a difference (and high level the features that go on the chip aren’t really all that different either, same stuff with various quantities and features tweaked)
  - terafo 5y ago
    They take up chip capacity on different fab. You can't produce gaming Ampere on TSMC. They are built on different architectures that have only name in common. The difference between "Ampere" and "Ampere" is bigger than between Volta and Turing, or maybe even than difference between Pascal and Turing.
- capableweb 5y ago
  Good luck building special GPUs in just two years, especially with what's happening regarding chip production right now. Not sure how they could have achieved a project of this size/scope unless they use off-the-shelf components, since the backlogs are so long and have been for some time now.
  - exdsq 5y ago
    Facebook has been hiring FPGA engineers with ML experience since 2018 so I don't think this would be out of the question! But even so, Nvidia sell custom GPUs that aren't the same ones for gaming.

exdsq 5y ago

How far can we actually take current machine learning technologies by scaling the underlying hardware? Are we going to see some AI algorithms that are 20% better or an order of magnitude better? And what will that realistically look like to an end user? This will have cost a lot of money and maybe the news alone will push stock prices and mean its paid for itself but is it actually going to result in a substantially better product?

jazzyjackson 5y ago
I was just in a Twitter Spaces room and they have a live transcription feature, so as to be accessible and all, except the transcript was gibberish. If Facebook wants live translation in the Metaverse, they should hope this brings orders of magnitudes improvement to voice recognition, especially in languages other than english (by far the largest training set available)
- zydex 5y ago
  I obviously don't know the parameters of the room you're referencing, but is it possible that the majority of the issue is on the side of poor user audio and a large number of simultaneous speakers? I find YouTube's transcription to be quite impressive with a handful of speakers and moderate audio quality.
The_rationalist 5y ago
arnaudsm 5y ago
It's linear for now (check GPT-2 vs GPT-3), but we're close to the point of diminishing returns.
- mindcrime 5y ago
  Part of the problem though, is that we don't know for sure what non-linearities may be lurking out there. Maybe we add 100 more "neurons" to the net and it "goes exponential" so to speak. Or maybe not. There's still a lot we don't know about the emergent properties of these systems as they scale up.

ctoth 5y ago

Anybody else get the sense that we're just totally frickin doomed? Even if Yudkowsky is off about AGI (which is a big maybe!) in what possible world will this technology be used to make our individual lives better (assuming you're not a FAIR researcher?)

The_rationalist 5y ago
meetups323 5y ago
More people trapped inside on the metaverse = smaller crowds at outdoor recreation areas?
- Traubenfuchs 5y ago
  That would be a win-win for everyone.
  - doublerabbit 5y ago
    Why?
    - Traubenfuchs 5y ago
      VR enjoyers can enjoy VR, reality enjoyers can enjoy reality. Vienna, where I am living, is completely overrun by people. Public spaces and transport at peak hours have become a mess in the last years. The first lockdown showed how beautiful the city can be if everyone stays home.
- emerged 5y ago
  I think the real world will become populated with android avatars which are controlled by people from their VR headsets. So you’ll have small human crowds but loads of anonymous avatar androids taking all the good fishing spots, riding the trails backwards, etc. I’m joking hopefully
5y ago

colechristensen 5y ago

What is the difference between “a supercomputer” and “a bunch of racks of computers”? The actual difference between the two is quite diminished compared to years past and seems to reduce more to how a collection of computers is used and not what it is.

tyingq 5y ago
The big remaining one appears to be an unusually high speed interconnect. Infiniband, etc.
- lmeyerov 5y ago
  Yep, hetero multigpu fleet mixing high ram GPUs (40-80GB each on each A100) as multigpus w smaller (ex: ~12-16 GB T4s) nodes, w crazy interconnects locally (nvlink) and across nodes. And storage gets fun as well, like parallel SSD arrays for 100GB+/s combined per node. Then whatever legacy+hybrid CPU stuff. Ex: for stuff like PCIe, new generations that ~10x the bandwidth you'd see in a gamer box, and like 1-2 per GPU. Varies a lot for say log mining vs NN training, and even for diff NNs. Ex: Graph NNs end up needing more balanced CPU side. Saturating a box with 500+ GB GPU RAM is fun. Only our gov users ask us for help on that typically: most of our users are commercial nowadays, but with much smaller/scaled down GPU rigs. I think that'll change as the fintechs keep improving and software gets easier, but they are still not there (outside of niches). Working on it :) (If you like writing shaders, we are hiring :D )
cjbgkagh 5y ago
I’d say mainly networking bandwidth.
benstrumental 5y ago
> What is the difference between “a supercomputer” and “a bunch of racks of computers”? In addition to the other responses, I like pointing people to this talk[1] by Jeff Hammond for a comprehensive answer to this question (you can skip to the 11:15 timestamp). [1] https://uchicago.hosted.panopto.com/Panopto/Pages/Embed.aspx?id=ce0c53bf-d799-4cac-a132-ad1901484286 https://uchicago.hosted.panopto.com/Panopto/Pages/Embed.aspx...

michelb 5y ago

All this to better predict behavior and present ads. What a waste.

sxv 5y ago
A waste is when you throw something into a landfill. This is weaponization by an enemy of the people.

rezonant 5y ago

Alternative headline: "Facebook patents Skynet"

pohl 5y ago

"You're going to be eaten by a bronteroc. We don't know what it means."

penjelly 5y ago

it feels like the future of companies is to increasingly give tasks to AI. So eventually we'll have massive corps that have a couple execs and a super ai making them all obscenely rich? I want to hate this idea, but it would be the same as hating machines replacing manual labor over the last 100 years. im not sure what to think, nor how to prepare myself for the next 20 years.

paxys 5y ago
20 years is optimistic. This future isn't something we'll have to worry about in our lifetime, if ever. People wildly overestimate the state of AI as it exists today.
edgyquant 5y ago
Not likely. Automation only increases productivity and companies are always looking to expand. The only thing automating jobs does it create new ones for people to work.

kleiba 5y ago

I used to work at a university where my professor had been in automatic speech recognition for a long time, but basically gave up on that line of research about 10 years ago because he figured that universities simply cannot compete budget wise with the big industry players. I suppose the same will soon be true for most ML-related areas of research sooner or later, at least as far as applied ML is concerned. Already, a substantial amount of research innovation in NLP and CV has been coming from big companies in recent years. Of course there is a discussion to be had about what that means for society at large. At this point, a lot of said companies to publish their results at conferences etc. But what if at some point they decide to be as "open" as OpenAI (ie., not)?

stathibus 5y ago
I hope a positive outcome of this will be that universities direct more of their research effort toward efficiency of network architectures and/or understandability.
- ml_hardware 5y ago
  Unfortunately it will be hard to investigate properties of large, powerful neural networks without access to their trained weights. And industrial labs that spend millions of dollars training them will not be keen to share. If academics want to do research on expensive cutting-edge tech, they will have to join industrial labs or pool together resources, similar to particle physics or drug discovery research today.
rococode 5y ago
I think the academic side will start shifting towards research on efficiency and speed while companies will continue to push the cutting edge. In the NLP space there's been a lot of work recently around reducing model sizes, since they've started to reach the point where model weights sometimes don't fit in the memory of most GPUs. There's also projects like MarianNMT which completely abandon Python and write heavily optimized models with fast languages that can run quickly and accurately even without GPUs. I think we'll see a lot more of this, though of course there's a pretty big barrier in the sheer rarity of being good at both deep learning research and writing optimized low-level code.

karmasimida 5y ago

> Meta’s AI supercomputer houses 6,080 Nvidia graphics-processing units ..... By mid-summer, when the AI Research SuperCluster is fully built, it will house some 16,000 GPUs Honestly ... this is lot of GPUs ... but is it the biggest...? > Model training is done with mixed precision on the NVIDIA DGX SuperPOD-based Selene supercomputer powered by 560 DGX A100 servers networked with HDR InfiniBand in a full fat tree configuration. Each DGX A100 has eight NVIDIA A100 80GB Tensor Core GPUs So Nvidia used 4480 GPUs to train Megatron-Turing NLG 530B for example.

vl 5y ago
Honestly, this single GPU-based install is child's play compared to Google's multiple TPU exoflop supercomputers with hyper-cube optical interconnects. Google's ML setups allow synchronous weight update on thousand+ TPUs...
- rawtxapp 5y ago
  TPUs are amazing, but in my experience, debugging issues with them can be a bit tricky. Since nvidia's gpus are more common place (especially outside gcp), you can find a lot more information when you get stuck, it's also more battle tested, etc.
  - 6gvONxR4sf7o 5y ago
    For what it's worth, jax is helpful to me here. You can drop out of the jit to debug it as if it were numpy. Of course that assumes your issues aren't with the jit itself or inside pmap, etc. That shit's hard.
- alex_sf 5y ago
  Tbh I thought I was being trolled with 'hyper-cube optical interconnects'.
  - vl 5y ago
    Actually, you are right, I mistyped. Although hypercube interconnects exist, and were used, for example, in AS400, system in question uses hypertorus topology.

perilousacts 5y ago

Literally don't care. Facebook should not be the company with this. :/

busymom0 5y ago

Could this have been the reason for chip shortage?

fennecfoxen 5y ago
Do you think 6,080 GPUs — admittedly very large ones — are sufficient to explain the chip shortage?
hmate9 5y ago
No, this is way too small in scale vs the global demand.
zydex 5y ago
Nvidia and AMD shipped 12.7m cards collectively in 2021, I don't know what the breakdown is on consumer vs corporate but I find it hard to believe this had any impact. Correction: terafo pointed out they shipped 12.7m cards in Q3 2021 alone.
- terafo 5y ago
  It was not 12.7 million in 2021. It was 12.7 million in Q3 2021.
  - zydex 5y ago
    Where did you see that? This was my source: https://www.digitaltrends.com/computing/gpu-shipments-increased-by-25-percent-despite-shortage/ https://www.digitaltrends.com/computing/gpu-shipments-increa... "Nvidia and AMD shipped 12.7 million cards in 2021" Please correct me if I'm misreading or clearly missing something.
    - terafo 5y ago
      If you look at the source article it is clearly stated. >Year over year, total AIB shipments increased by 25.7% this quarter compared to last year at 12.7 million units, and up quarter-to-quarter from 11.47 million units in Q2’21.

chundicus 5y ago

I can't shake the feelings that a trillion or a quadrillion parameters won't solve the fundamental shortcomings of ML models not being models of artificial intelligence. I guess there's no way of knowing until we reach AGI, but I've never heard a compelling argument for why pure ML would get us there. GPT3 seems more like an argument against that hypothesis (in my view) than for it. Even the best, most expensive models today are incredibly brittle for enterprise usecases that shouldn't necessarily require AGI. I've always imagined AGI (perhaps naively) as being achieved by clever usage of ML, plus some utilization of classical/symbolic AI from pre-AI winter days, plus probably some unknown elements.

MR4D 5y ago
I agree. I read Jeff Hawkins book On Intelligence [0] back when it came out, and it had a profound effect on my thinking. Chasing more data, aka "parameters" doesn't seem to be the right answer. I think more of a Bayes model like spam filtering, but cobbled together with other Bayes models looking at other things until something emerges that we call "intelligent". Heck, I'd consider Google's spam filtering pretty intelligent today. [0] - https://en.wikipedia.org/wiki/On_Intelligence https://en.wikipedia.org/wiki/On_Intelligence
- jcims 5y ago
  Hawkins way of thinking really maps well for me also. It seems like that more parameters helps until it doesn't, then you need to encapsulate those networks and pin them to some reference frame, they create hierarchies of these networks and a system to generalize and compress those hierarchies (aka patterns), rinse and repeat. My brother just became a grandpa and I was watching his grandson navigate the world this past weekend. It's unbelievable how quickly the brain can extrapolate a new relationship between objects/actions/etc and then apply it elsewhere. Minimally you see it in the drinking action applied to all sorts of things, this sort of repetitive clenching/releasing of the fingers to find things to grip without looking, etc etc. Watching mom use a fork and very quickly understand how to grasp and manipulate it. The model of just training everything from exogenous data into a flat network seems like it will hit some asymptotic limit.
5y ago

ricardobeat 5y ago

https://archive.is/xdQtE https://archive.is/xdQtE

mawadev 5y ago

If this won't make people watch ads 24/7, then what will?

pinewurst 5y ago

Are we supposed to collectively feel, "Yay, Facebook!"?! It's a bigger tool for surveillance enablement, no different from how the CCP monitors Xinjiang cameras.

hetspookjee 5y ago

So much wh will one inference cost? I mean computing has come more efficient but I still struggle to find data on how much wh is used for some sentences of GPT3, for example.

bognition 5y ago

https://archive.md/xdQtE https://archive.md/xdQtE

reggieband 5y ago

I hate that I always end up referencing the Lex Friedman podcast but it is often relevant to discussion on HN. Recently Lex spoke with Yann LeCun and they had a brief chat about AI at Meta/Facebook [1] (where I believe Yann is currently Chief AI Scientist). He claims that AI is the core of Meta and that if you were to take ML out of Meta systems the company would literally crumble because it is completely built around AI. My feeling is this is a PR push by Facebook. All tech companies keep touting AI, especially Google but also Microsoft, Apple and Amazon. In some sense I believe these business want to control how their own success is defined. That is, they are right now convincing everyone that tech dominance is equivalent to AI dominance which is equivalent to ML dominance. In some sense this is turning into a purity test, like "which tech company is the most AI focused". I expect this kind of PR to accelerate as each company tries to prove its AI bona-fides to the market. 1. https://youtu.be/SGzMElJ11Cc?t=6597 https://youtu.be/SGzMElJ11Cc?t=6597

acchow 5y ago
These aren't startups trying to prove "AI Purity" for more funding. These are money printers that are optimizing how to print more money. And Facebook and Google are competing against each other for online advertising dollars (yes, the pie is also growing). Their revenues are up 60% and 40% over the past 2 years, so I doubt their AI plays are just about proving some purity game.
throwaway423342 5y ago
As a side note: I listened to the episode with Yann. Compared to other talks (e.g. the previous one with Brian Keating) it was a bit dull and uninteresting. The answers were not that insightful.
mrkramer 5y ago
There is no single AI company. These are all machine learning techniques. AI is spreadsheets on steroids not real intelligence.
6gvONxR4sf7o 5y ago
I haven't listened to the podcast, but the way I read a statement like that is, "Our business needs to do things that aren't as simple as defining manual rules, but the economics of our business prevents us from just paying people to do those things at scale."

sydthrowaway 5y ago

I feel like the guy in A Canticle For Leibowitz who knew the history of the world.

keithnz 5y ago

So a company that specializes in targeted advertising to make money has invested in the most powerful AI supercomputer? Great.

deleted 5y ago

[deleted]

lemax 5y ago

It feels eerie that these trillions of parameters and exabyte sized training sets will come from harvested user data. 17 years of user activity all culminating into some.. supercomputer? I wonder how comments I wrote when I, and all the other people from my generation, were like 14 and using FB will feed into this and sort of be immortalized in this strange way.

Atlas667 5y ago

An AI that has free range to more profoundly study all the human data Facebook users generate? That sounds wicked evil. If ads, marketing and habit inducing platform designs are a problem now, imagine what this will lead to. To understand what drives your users more than the users understand it themselves and to use that understanding for profit. Intensified. And not to mention for surveillance, you know DARPA and the NSA want their hands all over this.