3 ms·

A year or two ago, I helped the Cloud TPU team fix an obscure error when creating more than 200 TPUs. Apparently no one had ever tried to create more than 200

A year or two ago, I helped the Cloud TPU team fix an obscure error when creating more than 200 TPUs. Apparently no one had ever tried to create more than 200 TPUs before (https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf). I was pretty proud of that. (As a bonus, I actually wanted to use all 200 TPUs. It wasn't merely a stress test. Big data indeed!)

ZeroCool2u 5y ago
Here I was having fun with my p3.16xlarge at work today, that's awesome!
- sillysaurusx 5y ago
  If you like playing with ridiculous amounts of hardware, definitely try out TPU VMs with TFRC: https://blog.gpt4.org/jaxtpu https://blog.gpt4.org/jaxtpu You can create up to 5 TPU v3-8's in europe-west4-a. Each core is roughly equivalent to a V100, so it's sort of like having 5 p3.16xlarge's. (The TPU has 96 CPUs and 330GB of system RAM, whereas a p3.16xlarge has 64 CPUs and 488GB of system RAM according to https://aws.amazon.com/ec2/instance-types/p3/ https://aws.amazon.com/ec2/instance-types/p3/). The best part IMO is that you don't have to figure out how to link the cores together. JAX does that for you automatically. You get 100GigE from TPU to TPU too, which you can also leverage: https://twitter.com/theshawwn/status/1406171487988498433 https://twitter.com/theshawwn/status/1406171487988498433
  - google234123 5y ago
    The program is for researchers, not for play...
    - antonvs 5y ago
      Are you so sure there's a difference?
      - ms4720 5y ago
        One is a strict subset of the other
        google234123 5y ago
        Really? A “strict subset”, come on…
CamperBob2 5y ago
Wow, that docdroid site is toxic waste. Don't click on anything but the back button if you follow that link.
- sillysaurusx 5y ago
  Sorry. As an AI outsider, I wasn't sure how to publish to arxiv, so I used the first PDF hosting site I could find. I've mirrored it to https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf and edited my original comment with that instead. Thanks for pointing that out.

ZeroCool2u 5y ago

Here I was having fun with my p3.16xlarge at work today, that's awesome!

sillysaurusx 5y ago
If you like playing with ridiculous amounts of hardware, definitely try out TPU VMs with TFRC: https://blog.gpt4.org/jaxtpu https://blog.gpt4.org/jaxtpu You can create up to 5 TPU v3-8's in europe-west4-a. Each core is roughly equivalent to a V100, so it's sort of like having 5 p3.16xlarge's. (The TPU has 96 CPUs and 330GB of system RAM, whereas a p3.16xlarge has 64 CPUs and 488GB of system RAM according to https://aws.amazon.com/ec2/instance-types/p3/ https://aws.amazon.com/ec2/instance-types/p3/). The best part IMO is that you don't have to figure out how to link the cores together. JAX does that for you automatically. You get 100GigE from TPU to TPU too, which you can also leverage: https://twitter.com/theshawwn/status/1406171487988498433 https://twitter.com/theshawwn/status/1406171487988498433
- google234123 5y ago
  The program is for researchers, not for play...
  - antonvs 5y ago
    Are you so sure there's a difference?
    - ms4720 5y ago
      One is a strict subset of the other
      - google234123 5y ago
        Really? A “strict subset”, come on…

CamperBob2 5y ago

Wow, that docdroid site is toxic waste. Don't click on anything but the back button if you follow that link.

sillysaurusx 5y ago
Sorry. As an AI outsider, I wasn't sure how to publish to arxiv, so I used the first PDF hosting site I could find. I've mirrored it to https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf and edited my original comment with that instead. Thanks for pointing that out.