5 ms·

The AI Research SuperCluster

92 points by minimaxir 5y ago

zwieback 5y ago
What's the track record on "super computers"? It seems risky to pour so much money into a platform that will be superseded in a few years time. Then again, it's not clear that there are really good alternatives.
- halfeatenpie 5y ago
  Well it's partially the cost/benefit analysis of: 1. How much benefit will we get? 2. Will this benefit be higher than the cost of purchasing this right now? 3. What other alternatives will satisfy our needs? For most of these solutions, the answers are: 1. A lot. We need computing power to perform these analysis faster and to be competitive. 2. Yes, as continuing to innovate in this space will keep us competitive and give our scientists the resources to remain productive. 3. AWS/GCP/Azure are alternatives sure, but then (the rate at which Meta probably uses these resources) it probably cost them less to build this out than to pay AWS/GCP/Azure for access to these hardware.
- tedivm 5y ago
  Major GPU architecture changes only happen every few years. * k80 - 2014, and this really was not that great of a chip. * v100 in 2017, and I'd consider this the first "built from the ground up" ML chip. * A100 in late 2020, with the first major cloud general availability being in 2021. Even when new chips come out the old ones are still usable- you can rent k80s for fairly cheap on all cloud providers, and have kept a surprising amount of their resale value. The v100s are also very much still in demand. The A100 is also an amazing system- the new nvswitch archetecture means the A100s work together far far better than their V100 counterparts. I was part of an upgrade project setting up an A100 cluster with infiniband and it really is amazing how well these chips work together. That communication barrier was a pretty obvious next step though (the k80s had crap intergpu communication, the v100s introduced the nvlink, and the nvswitch was the obvious way to go). There isn't an obvious next step, and I imagine the A100s to be the standard for at least the next four years (with lots of continued use after that).
  - boulos 5y ago
    You skipped the T4 which was in between Volta and Ampere, as well as the P100 between K80 and V100. So I'd say "meaningful chip changes" is closer to every 18 months. The T4 though isn't a "big part", but for people who fit within its envelope, it's a huge win (since it's cost is so much lower). A lot of deep learning folks had built out Turing-based workstations in that time period, and I think they're still reasonable value for money.
    - tedivm 5y ago
      The T4 isn't the same- you'd never do training on it. It's a cute little inference chip.
abhinai 5y ago
I wish researchers outside Meta were allowed to rent this SuperCluster for maximum benefit to humanity.
- forgotmyoldacc 5y ago
  Why not just rent AWS/Azure/GCP instead? They're all about the same. Top of the line enterprise GPUs with fast interconnect.
  - tedivm 5y ago
    They are not the same at all. AWS has the best GPU instances right now but there's some huge differences in networking speed. The P3 instances have 400Gbps per machine, with 8 GPUs on each machine. If you were to self host a cluster using the standard DGX machines you get 200Gbps per GPU, for a total of 1600Gbps for just the GPUs. The DGX machine has another two infiniband ports that can be used to attach to storage at pretty intense speeds as well. This makes a huge difference when using more than a single machine. I've done the math and purchased the machines at a previous company- assuming you aren't leaving the machines idle most of the time you save a considerable amount of money and get a lot better performance when building your own cluster.
    - ska 5y ago
      On-prem for nearly anything is going to be at least a bit of a win if your utilization is uniform or predictable. The real win for not doing it is in adaptability.
      - tedivm 5y ago
        Yeah, definitely, but I think it's important to talk about scale of that win. I mentioned in another comment that the GPU generation is roughly three years between major architecture upgrades- this has held true for a bit now, and that time may even stretch out a little. When the average company builds one of these clusters it's safe to assume they'll either run it for three years or sell it back for some return. Going with the cloud and assuming you don't commit to several years (losing that adaptability) the yearly cost of a p4 is $287,087. Over three years that's $861,261 to run a single machine. For about $450k you can build out a solid two machine (16gpu) cluster (including infiniband networking gear and a solid NAS) that will easily last three years. There are datacenters which specialize in this and companies that can manage these machines. If you don't have the cash up front you can lease them on good terms and your yearly bill will still be much lower than AWS. Model training is basically the one use case where I'm really willing to purchase equipment instead of using the cloud. The money it saves is enough to hire one or two more staff members, and the maintenance is shockingly low if you get it setup right to start.
kn8a 5y ago
Are there any alternatives to gradient based learning that could make this less useful? Is there another type of compute unit that is the next evolution of CPU -> GPU -> ?
- winterismute 5y ago
  It's a tough question, it's not even back-propagation but even just sometimes the "parameters" of the models, for example [1] shows that models such as ResNeXt already perform better on a very different architecture such as Graphcore, for some sizes of convolutions. Older models, or models that get tuned for existing GPUs, do not perform as well. It's tough to come up with a new architecture that can have an advantage on current and future models, at least from a peak perf point of view, from a perf/watt for example instead the scaled-up Apple GPUs seem to show new interesting properties. But the Graphcore architecture is quite interesting, being able to act somehow both as a SIMD machine and a task-parallel machine. [1] https://arxiv.org/pdf/1912.03413v1.pdf https://arxiv.org/pdf/1912.03413v1.pdf
- randcraw 5y ago
  Based the following constraints that lie at the center of AI and parallelism, I'd say no -- stochastic gradient pursuit using vector processors like GPUs is inescapable in all future AI advances. 1) All AI is based in search (esp. non-convex, where heuristics are insufficient to provide a global convex solution), and thus is inevitably implemented using iteration, driven locally by gradient-pursuit and globally by... ways to efficiently gather information to optimize the loss function that measures how well that info gain is being refined and exploited. 2) Search that is inherently non-convex and inefficient requires as much compute power as possible, i.e. using supercomputers. 3) All supercomputer-based solutions to non-convex problems are implemented iteratively, where results are improved not using closed-form math or complete info, but by incremental optimization of partial results that aggregate with the iterations, like repeated stochastic gradient descent that creates and enhances 'resonant' clusters of 'neurons'. 4) The only form of supercomputing that has proven to scale up at anywhere near indefinitely is data-parallelism (a dataflow-specific form of SIMD) -- where the search space is spread as evenly (and naively) as possible across as many processing elements as possible. 5) Vector processing hardware like GPUs implement data-parallelism as well as any HPC architecture yet devised. Thus, I believe that AI is stuck with GPUs, or equivalent meshes of vector processors, indefinitely.
caaqil 5y ago
This thread should probably be merged with this one: https://news.ycombinator.com/item?id=30062019 https://news.ycombinator.com/item?id=30062019
buildbot 5y ago
It’s not larger than Microsoft’s? https://blogs.microsoft.com/ai/openai-azure-supercomputer/ https://blogs.microsoft.com/ai/openai-azure-supercomputer/
- boulos 5y ago
  That was a V100 cluster though. 10k V100s is less powerful (for ML stuff) than ~6k A100s.
- forgotmyoldacc 5y ago
  Facebook's > When RSC is complete, the InfiniBand network fabric will connect 16,000 GPUs as endpoints Microsoft's > The supercomputer developed for OpenAI is a single system with more than 285,000 CPU cores, 10,000 GPUs and 400 gigabits per second of network connectivity for each GPU server
etaioinshrdlu 5y ago
It is interesting that it only allows training on anonymized and encrypted data. I wonder how much these restrictions slow down their research? Although, they are definitely a good idea considering the data source.
mooneater 5y ago
> All this infrastructure must be extremely reliable, as we estimate some experiments could run for weeks and require thousands of GPUs Is it hardward-fault tolerant? Curious how well this will work otherwise as it scales.
codeulike 5y ago
The Terminator franchise is based around the folly of letting an AI control a nuclear arsenal. But here we are building the biggest AI ever and letting it analyse our social interactions. Think of the power this could have if it goes rogue! It could manipulate entire populatuions by minutely controlling what they see and read. Surely if manipulation at that scale and fidelity became possible it would be something to be concerned about? .... Oh wait ....
- georgeecollins 5y ago
  The premise of the science fiction novel "After On" is that the first AI to reach sentience is running a dating app. It's actually a good, well researched book.
- lawrenceyan 5y ago
  > Surely if manipulation at that scale and fidelity became possible it would be something to be concerned about?
- LargoLasskhyfv 5y ago
  Basil van Roko sez BLIT!
- Proven 5y ago
- rbanffy 5y ago
  In many senses, corporations already behave like artificial general intelligences, and Meta has been using its information and models to manipulate large populations to increase its profits without much regard for negative externalities (such as coups, degradation of democracies, and so on). We will not be able to control an AGI any more than we are able to rein in Meta. Unless the humans in charge are held responsible for the actions of the organization they control, there is little hope we will.
  - codeulike 5y ago
    Exactly! Meta and the other corporations it co-exists with. Corporations have the ethics and self control of immortal superpowered toddlers and they are using facebook's advertising algorithms as a tool to get what they want.
amelius 5y ago
They didn't build their own version of Google's TPU? Seems like they are a step behind.
- dekhn 5y ago
  That's arguable. It's entirely nontrivial to spin up a custom TPU and the ecosystem around it, and nvidia is fairly good at supporting the entire end-to-end picture (hardware, software, interconnect, higher level libs, etc). Easier to just spend $$$ on a first-rate system built by skilled integrators and replace it every few years. TPUs are also just catching up on sparse workloads.
guard0g 5y ago
Maybe great for research, but not very useful for application at inference time.
- danielcampos93 5y ago
  Maybe not direct inference but for generation of dense represenatations that can be used with their nearest neighbor search system this large cluster will probably be hyper userful.
estebarb 5y ago
> The data is then encrypted before it can be used to train AI models How can they train ML models with encrypted data?
- dekhn 5y ago
  presuming they aren't using homoemorphic encryption (they aren't), it just means the trainign data is encrypted on disk and a job needs special permissions to access the decryption key. It's still loaded into RAM, decrypted, and passed to the training algorithms as decrypted data.
dekhn 5y ago
Stupid question: can anybody out there "get" A100s in AWS right now? We found they are unavailable in all the regions we want to operate.
- danielcampos93 5y ago
  I have had little luck getting them on AWS. If you time it outside of some major paper deadline I have had really good success on GCP. If you are willing to break out of the big clouds CoreWeave has great ability and even better prices. Highly recommend.

92 points by minimaxir 5y ago

What's the track record on "super computers"? It seems risky to pour so much money into a platform that will be superseded in a few years time. Then again, it's not clear that there are really good alternatives.

halfeatenpie 5y ago
Well it's partially the cost/benefit analysis of: 1. How much benefit will we get? 2. Will this benefit be higher than the cost of purchasing this right now? 3. What other alternatives will satisfy our needs? For most of these solutions, the answers are: 1. A lot. We need computing power to perform these analysis faster and to be competitive. 2. Yes, as continuing to innovate in this space will keep us competitive and give our scientists the resources to remain productive. 3. AWS/GCP/Azure are alternatives sure, but then (the rate at which Meta probably uses these resources) it probably cost them less to build this out than to pay AWS/GCP/Azure for access to these hardware.
tedivm 5y ago
Major GPU architecture changes only happen every few years. * k80 - 2014, and this really was not that great of a chip. * v100 in 2017, and I'd consider this the first "built from the ground up" ML chip. * A100 in late 2020, with the first major cloud general availability being in 2021. Even when new chips come out the old ones are still usable- you can rent k80s for fairly cheap on all cloud providers, and have kept a surprising amount of their resale value. The v100s are also very much still in demand. The A100 is also an amazing system- the new nvswitch archetecture means the A100s work together far far better than their V100 counterparts. I was part of an upgrade project setting up an A100 cluster with infiniband and it really is amazing how well these chips work together. That communication barrier was a pretty obvious next step though (the k80s had crap intergpu communication, the v100s introduced the nvlink, and the nvswitch was the obvious way to go). There isn't an obvious next step, and I imagine the A100s to be the standard for at least the next four years (with lots of continued use after that).
- boulos 5y ago
  You skipped the T4 which was in between Volta and Ampere, as well as the P100 between K80 and V100. So I'd say "meaningful chip changes" is closer to every 18 months. The T4 though isn't a "big part", but for people who fit within its envelope, it's a huge win (since it's cost is so much lower). A lot of deep learning folks had built out Turing-based workstations in that time period, and I think they're still reasonable value for money.
  - tedivm 5y ago
    The T4 isn't the same- you'd never do training on it. It's a cute little inference chip.

abhinai 5y ago

I wish researchers outside Meta were allowed to rent this SuperCluster for maximum benefit to humanity.

forgotmyoldacc 5y ago
Why not just rent AWS/Azure/GCP instead? They're all about the same. Top of the line enterprise GPUs with fast interconnect.
- tedivm 5y ago
  They are not the same at all. AWS has the best GPU instances right now but there's some huge differences in networking speed. The P3 instances have 400Gbps per machine, with 8 GPUs on each machine. If you were to self host a cluster using the standard DGX machines you get 200Gbps per GPU, for a total of 1600Gbps for just the GPUs. The DGX machine has another two infiniband ports that can be used to attach to storage at pretty intense speeds as well. This makes a huge difference when using more than a single machine. I've done the math and purchased the machines at a previous company- assuming you aren't leaving the machines idle most of the time you save a considerable amount of money and get a lot better performance when building your own cluster.
  - ska 5y ago
    On-prem for nearly anything is going to be at least a bit of a win if your utilization is uniform or predictable. The real win for not doing it is in adaptability.
    - tedivm 5y ago
      Yeah, definitely, but I think it's important to talk about scale of that win. I mentioned in another comment that the GPU generation is roughly three years between major architecture upgrades- this has held true for a bit now, and that time may even stretch out a little. When the average company builds one of these clusters it's safe to assume they'll either run it for three years or sell it back for some return. Going with the cloud and assuming you don't commit to several years (losing that adaptability) the yearly cost of a p4 is $287,087. Over three years that's $861,261 to run a single machine. For about $450k you can build out a solid two machine (16gpu) cluster (including infiniband networking gear and a solid NAS) that will easily last three years. There are datacenters which specialize in this and companies that can manage these machines. If you don't have the cash up front you can lease them on good terms and your yearly bill will still be much lower than AWS. Model training is basically the one use case where I'm really willing to purchase equipment instead of using the cloud. The money it saves is enough to hire one or two more staff members, and the maintenance is shockingly low if you get it setup right to start.

kn8a 5y ago

Are there any alternatives to gradient based learning that could make this less useful? Is there another type of compute unit that is the next evolution of CPU -> GPU -> ?

winterismute 5y ago
It's a tough question, it's not even back-propagation but even just sometimes the "parameters" of the models, for example [1] shows that models such as ResNeXt already perform better on a very different architecture such as Graphcore, for some sizes of convolutions. Older models, or models that get tuned for existing GPUs, do not perform as well. It's tough to come up with a new architecture that can have an advantage on current and future models, at least from a peak perf point of view, from a perf/watt for example instead the scaled-up Apple GPUs seem to show new interesting properties. But the Graphcore architecture is quite interesting, being able to act somehow both as a SIMD machine and a task-parallel machine. [1] https://arxiv.org/pdf/1912.03413v1.pdf https://arxiv.org/pdf/1912.03413v1.pdf
randcraw 5y ago
Based the following constraints that lie at the center of AI and parallelism, I'd say no -- stochastic gradient pursuit using vector processors like GPUs is inescapable in all future AI advances. 1) All AI is based in search (esp. non-convex, where heuristics are insufficient to provide a global convex solution), and thus is inevitably implemented using iteration, driven locally by gradient-pursuit and globally by... ways to efficiently gather information to optimize the loss function that measures how well that info gain is being refined and exploited. 2) Search that is inherently non-convex and inefficient requires as much compute power as possible, i.e. using supercomputers. 3) All supercomputer-based solutions to non-convex problems are implemented iteratively, where results are improved not using closed-form math or complete info, but by incremental optimization of partial results that aggregate with the iterations, like repeated stochastic gradient descent that creates and enhances 'resonant' clusters of 'neurons'. 4) The only form of supercomputing that has proven to scale up at anywhere near indefinitely is data-parallelism (a dataflow-specific form of SIMD) -- where the search space is spread as evenly (and naively) as possible across as many processing elements as possible. 5) Vector processing hardware like GPUs implement data-parallelism as well as any HPC architecture yet devised. Thus, I believe that AI is stuck with GPUs, or equivalent meshes of vector processors, indefinitely.

caaqil 5y ago

This thread should probably be merged with this one: https://news.ycombinator.com/item?id=30062019 https://news.ycombinator.com/item?id=30062019

buildbot 5y ago

It’s not larger than Microsoft’s? https://blogs.microsoft.com/ai/openai-azure-supercomputer/ https://blogs.microsoft.com/ai/openai-azure-supercomputer/

boulos 5y ago
That was a V100 cluster though. 10k V100s is less powerful (for ML stuff) than ~6k A100s.
forgotmyoldacc 5y ago
Facebook's > When RSC is complete, the InfiniBand network fabric will connect 16,000 GPUs as endpoints Microsoft's > The supercomputer developed for OpenAI is a single system with more than 285,000 CPU cores, 10,000 GPUs and 400 gigabits per second of network connectivity for each GPU server

etaioinshrdlu 5y ago

It is interesting that it only allows training on anonymized and encrypted data. I wonder how much these restrictions slow down their research? Although, they are definitely a good idea considering the data source.

mooneater 5y ago

> All this infrastructure must be extremely reliable, as we estimate some experiments could run for weeks and require thousands of GPUs Is it hardward-fault tolerant? Curious how well this will work otherwise as it scales.

codeulike 5y ago

The Terminator franchise is based around the folly of letting an AI control a nuclear arsenal. But here we are building the biggest AI ever and letting it analyse our social interactions. Think of the power this could have if it goes rogue! It could manipulate entire populatuions by minutely controlling what they see and read. Surely if manipulation at that scale and fidelity became possible it would be something to be concerned about? .... Oh wait ....

georgeecollins 5y ago
The premise of the science fiction novel "After On" is that the first AI to reach sentience is running a dating app. It's actually a good, well researched book.
lawrenceyan 5y ago
> Surely if manipulation at that scale and fidelity became possible it would be something to be concerned about?
LargoLasskhyfv 5y ago
Basil van Roko sez BLIT!
Proven 5y ago
rbanffy 5y ago
In many senses, corporations already behave like artificial general intelligences, and Meta has been using its information and models to manipulate large populations to increase its profits without much regard for negative externalities (such as coups, degradation of democracies, and so on). We will not be able to control an AGI any more than we are able to rein in Meta. Unless the humans in charge are held responsible for the actions of the organization they control, there is little hope we will.
- codeulike 5y ago
  Exactly! Meta and the other corporations it co-exists with. Corporations have the ethics and self control of immortal superpowered toddlers and they are using facebook's advertising algorithms as a tool to get what they want.

amelius 5y ago

They didn't build their own version of Google's TPU? Seems like they are a step behind.

dekhn 5y ago
That's arguable. It's entirely nontrivial to spin up a custom TPU and the ecosystem around it, and nvidia is fairly good at supporting the entire end-to-end picture (hardware, software, interconnect, higher level libs, etc). Easier to just spend $$$ on a first-rate system built by skilled integrators and replace it every few years. TPUs are also just catching up on sparse workloads.

guard0g 5y ago

Maybe great for research, but not very useful for application at inference time.

danielcampos93 5y ago
Maybe not direct inference but for generation of dense represenatations that can be used with their nearest neighbor search system this large cluster will probably be hyper userful.

estebarb 5y ago

> The data is then encrypted before it can be used to train AI models How can they train ML models with encrypted data?

dekhn 5y ago
presuming they aren't using homoemorphic encryption (they aren't), it just means the trainign data is encrypted on disk and a job needs special permissions to access the decryption key. It's still loaded into RAM, decrypted, and passed to the training algorithms as decrypted data.

dekhn 5y ago

Stupid question: can anybody out there "get" A100s in AWS right now? We found they are unavailable in all the regions we want to operate.

danielcampos93 5y ago
I have had little luck getting them on AWS. If you time it outside of some major paper deadline I have had really good success on GCP. If you are willing to break out of the big clouds CoreWeave has great ability and even better prices. Highly recommend.