6 ms·

I hope you succeed. TPU research cloud (TRC) tried this in 2019. It was how I got my start. In 2023 you can barely get a single TPU for more than an hour. Back

I hope you succeed. TPU research cloud (TRC) tried this in 2019. It was how I got my start. In 2023 you can barely get a single TPU for more than an hour. Back then you could get literally hundreds, with an s. I believed in TRC. I thought they’d solve it by scaling, and building a whole continent of TPUs. But in the end, TPU time was cut short in favor of internal researchers — some researchers being more equal than others. And how could it be any other way? If I made a proposal today to get these H100s to train GPT to play chess, people would laugh. The world is different now. Your project has a youthful optimism that I hope you won’t lose as you go. And in fact it might be the way to win in the long run. So whenever someone comes knocking, begging for a tiny slice of your H100s for their harebrained idea, I hope you’ll humor them. It’s the only reason I was able to become anybody.

nwoli 3y ago
> In 2023 you can barely get a single TPU for more than an hour Oh come on, colab gives TPU access in the free tier for a whole half day. No need to exaggerate the shortage
latchkey 3y ago
What Shawn says is absolutely right. The race right now is way too hot for this stuff. A single customer will eat up 512 gpus for 3 years.
LoganDark 3y ago
> In 2023 you can barely get a single TPU for more than an hour. Um. Can't you order them from coral.ai and put them in an NVMe slot? Or are the cloud TPUs more powerful?
- whimsicalism 3y ago
  TPU pod is not sold by google, edge tpu is different
  - LoganDark 3y ago
    So the cloud TPUs are more powerful...? Or what are you saying?
    - whimsicalism 3y ago
      yes
    - sillysaurusx 3y ago
      Yeah, it’s a silly branding thing. One TPU (not even a pod, just a regular old TPUv2) has 96 CPU cores with 1.4TB of RAM, and that’s not even counting their hardware acceleration. I’d love to buy one.
      - haldujai 3y ago
        Huh, this doesn't seem right. Based on #s you seem to be referring to pods but even then I'm not familiar with such a configuration existing. A single TPUv2 chip has 1 core and 8gb of memory. A single device comes in the v2-8 configuration with 8 cores and 64gb of memory. Pod variants come in v2-32 to v2-512 configurations.
        simonster 3y ago
        A single TPUv2 host has 8 TPU cores with 64GB of total HBM (8GB per core), but like GPUs, TPUs can't directly access a network, so the host also needs CPUs and standard RAM to send data to them. They are fast, and the host has to be fast enough to keep them fed with data, so the host is pretty beefy. But FWIW, a TPUv2 host has somewhere around 330GB of RAM, not 1.4TB.
        haldujai 3y ago
        Thanks for clarifying, I misinterpreted the commenter as referring to the accelerator as the conversation was about TPU availability for purchase. I know just enough about the architecture to facilitate using TPUs for research training runs but I'm not sure what's so special about the host? Sure it's beefy but there are much beefier servers readily available.
flaque 3y ago
> Your project has a youthful optimism that I hope you won’t lose as you go. And in fact it might be the way to win in the long run. This is the nicest thing anyone has said to us about this. We're gonna frame this and hang it out on our wall. > So whenever someone comes knocking, begging for a tiny slice of your H100s for their harebrained idea, I hope you’ll humor them. Absolutely! :D
- camhart 3y ago
  Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I'm not encouraging the false belief that everything you do will work out. Instead I'm encouraging the realization that the greatest accomplishments almost always feel like long shots, and require significant amounts of optimism. Fear and pessimism, while helpful in appropriate doses, will limit you greatly in life if you let them rule you too significantly. When I look back on my life, the greatest accomplishments I've achieved are ones where I was naive yet optimistic going into it. This was a good thing, because I would have been too scared to try had I really known the challenges that lay ahead.
  - johnthewise 3y ago
    I like this quote from Napoleon on taking risks: “If the art of war were nothing but the art of avoiding risks, glory would become the prey of mediocre minds.... I have made all the calculations; fate will do the the rest."
  - jacquesm 3y ago
    To me the payoff of failed projects is in what I learned. As long as that's the case I can carry my optimism over into new projects.
  - Frost1x 3y ago
    >Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I argue that realism trumps optimism. It's perfectly normal in a realist farming to see something difficult, acknowledge the high risk and failure potential, and still pursue something with intent to succeed. I've personally grown tired of over optimism everywhere because it creates unrealistic situations and passes consequences of failure in an inequitable way. The "visionary" is rewarded when the rare successes occur, while everyone else suffers the consequences for most failures. No contingency plans for failure, no discussion of failure, and so on. Optimism just takes any idea, pursues it and consequences be someone else's problem and be damned. Pessimism isn't much better, you essentially think everything is too risky or unlikely to succeed so you never do anything. You live in a state of inaction because any level of risk or uncertainty is too much. To me, realism is much better. You acknowledge the challenge. You acknowledge the risk. You make sure everyone involved understands it, but you still charge forward knowing you might succeed. Some think if you're not naively optimistic (what most people in my experience refer to as "optimism") you don't create enough pressure. I think that's non-sense.
ShamelessC 3y ago
Wow! I never thought you’d see the light. All I ever see from your posts is praise for TRC. As someone who got started way later on, I had infinitely more success with a gaming GPU I owned myself. Obviously not really comparable, but TRC was very very difficult to work with. I think I only ever had access to a TPUv3 once and that wasn’t nearly enough time to learn the ropes. My understanding was that this situation changed drastically depending on what sort of email you had or how popular your Twitter handle was.
haldujai 3y ago
My experience has been different. Considering how easy the application is I think they're still being fairly generous as I've been offered multiple v3-8s and v3-32s x 30days as well as pre-emptible v3-64s x 28 days for a few different projects within the last 6 months. Are you affiliated with an academic institution? Otherwise I'm not sure why they're been more generous with me, my projects have been mildly interesting at best. They're certainly a lot stingier with larger pods than they used to be though.
zak 3y ago
Actually, the TPU Research Cloud program is still going strong! We've expanded the compute pool significantly to include Cloud TPU v4 Pod slices, and larger projects still use hundreds of chips at a time. (TRC capacity has not been reclaimed for internal use.) Check out this list of recent TRC-supported publications: https://sites.research.google/trc/publications/ https://sites.research.google/trc/publications/ Demand for Cloud TPUs is definitely intense, so if you're using preemptible capacity, you're probably seeing more frequent interruptions, but reserved capacity is also available. Hope you email the TRC support team to say hello!
- sillysaurusx 3y ago
  Zak, I love you buddy, but you should have some of your researchers try to use the TRC program. They should pretend to be a nobody (like I was in 2019) and try to do any research with the resources they’re granted. I guarantee you those researchers will all tell you “we can’t start any training runs anymore because the TPUs die after 45 minutes.” This may feel like an anime betrayal, since you basically launched my career as a scientist. But it’s important for hobbyists and tinkerers to be able to participate in the AI ecosystem, especially today. And TRC just does not support them anymore. I tried, many times, over the last year and a half. You don’t need to take my word for it. Here’s some unfiltered DMs on the subject: https://imgur.com/a/6vqvzXs https://imgur.com/a/6vqvzXs Notice how their optimism dries up, and not because I was telling them how bad TRC has become. It’s because their TPUs kept dying. I held out hope for so long. I thought it was temporary. It ain’t temporary, Zak. And I vividly remember when it happened. Some smart person in google proposed a new allocation algorithm back near the end of 2021, and poof, overnight our ability to make TPUs went from dozens to a handful. It was quite literally overnight; we had monitoring graphs that flatlined. I can probably still dig them up. I’ve wanted to email you privately about this, but given that I am a small fish in a pond that’s grown exponentially bigger, I don’t think it would’ve made a difference. The difference is in your last paragraph: you allocate reserved instances to those who deserve it, and leave everybody else to fight over 45 minutes of TPU time when it takes 25 minutes just to create and fill your TPU with your research data. Your non-preemptible TPUs are frankly a lie. I didn’t want to drop the L word, but a TPUv3 in euw4a will literally delete itself — aka preempt — after no more than a couple hours. I tested this over many months. That was some time ago, so maybe things have changed, but I wouldn’t bet on it. There’s some serious “left hand doesn’t know that right hand detached from its body and migrated south for the winter” energy in the TRC program. I don’t know where it embedded itself, but if you want to elevate any other engineers from software devs to researchers, I urge you to make some big changes. One last thing. The support staff of TRC is phenomenal. Jonathan Colton has worked more miracles than I can count, along with the rest of his crew. Ultimately he had to send me an email like “by the way, TRC doesn’t delete TPUs. This distinction probably won’t be too relevant, but I wanted to let you know” (paraphrasing). Translation: you took the power away from the people who knew where to put it (Jonathan) and gave it to some really important researchers, probably in Brain or some other division of Google. And the rest is history. So I don’t want to hear that one of the changes is “ok, we’ve punished the support staff” - as far as I can tell, they’ve moved mountains with whatever tools they had available, and I definitely wouldn’t have been able to do any better in their shoes. Also, hello. Thanks for launching my career. Sorry that I had to leave this here, but my duty is to the open source community. The good news is that you can still recover, if only you’d revert this silly “we’ll slip you some reserved TPUs that don’t kamikaze themselves after 45 minutes if you ask in just the right way” stuff. That wasn’t how the program was in 2019, and I guarantee that I couldn’t have done the work I did then under the current conditions.

nwoli 3y ago

> In 2023 you can barely get a single TPU for more than an hour Oh come on, colab gives TPU access in the free tier for a whole half day. No need to exaggerate the shortage

latchkey 3y ago

What Shawn says is absolutely right. The race right now is way too hot for this stuff. A single customer will eat up 512 gpus for 3 years.

LoganDark 3y ago

> In 2023 you can barely get a single TPU for more than an hour. Um. Can't you order them from coral.ai and put them in an NVMe slot? Or are the cloud TPUs more powerful?

whimsicalism 3y ago
TPU pod is not sold by google, edge tpu is different
- LoganDark 3y ago
  So the cloud TPUs are more powerful...? Or what are you saying?
  - whimsicalism 3y ago
    yes
  - sillysaurusx 3y ago
    Yeah, it’s a silly branding thing. One TPU (not even a pod, just a regular old TPUv2) has 96 CPU cores with 1.4TB of RAM, and that’s not even counting their hardware acceleration. I’d love to buy one.
    - haldujai 3y ago
      Huh, this doesn't seem right. Based on #s you seem to be referring to pods but even then I'm not familiar with such a configuration existing. A single TPUv2 chip has 1 core and 8gb of memory. A single device comes in the v2-8 configuration with 8 cores and 64gb of memory. Pod variants come in v2-32 to v2-512 configurations.
      - simonster 3y ago
        A single TPUv2 host has 8 TPU cores with 64GB of total HBM (8GB per core), but like GPUs, TPUs can't directly access a network, so the host also needs CPUs and standard RAM to send data to them. They are fast, and the host has to be fast enough to keep them fed with data, so the host is pretty beefy. But FWIW, a TPUv2 host has somewhere around 330GB of RAM, not 1.4TB.
        haldujai 3y ago
        Thanks for clarifying, I misinterpreted the commenter as referring to the accelerator as the conversation was about TPU availability for purchase. I know just enough about the architecture to facilitate using TPUs for research training runs but I'm not sure what's so special about the host? Sure it's beefy but there are much beefier servers readily available.

flaque 3y ago

> Your project has a youthful optimism that I hope you won’t lose as you go. And in fact it might be the way to win in the long run. This is the nicest thing anyone has said to us about this. We're gonna frame this and hang it out on our wall. > So whenever someone comes knocking, begging for a tiny slice of your H100s for their harebrained idea, I hope you’ll humor them. Absolutely! :D

camhart 3y ago
Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I'm not encouraging the false belief that everything you do will work out. Instead I'm encouraging the realization that the greatest accomplishments almost always feel like long shots, and require significant amounts of optimism. Fear and pessimism, while helpful in appropriate doses, will limit you greatly in life if you let them rule you too significantly. When I look back on my life, the greatest accomplishments I've achieved are ones where I was naive yet optimistic going into it. This was a good thing, because I would have been too scared to try had I really known the challenges that lay ahead.
- johnthewise 3y ago
  I like this quote from Napoleon on taking risks: “If the art of war were nothing but the art of avoiding risks, glory would become the prey of mediocre minds.... I have made all the calculations; fate will do the the rest."
- jacquesm 3y ago
  To me the payoff of failed projects is in what I learned. As long as that's the case I can carry my optimism over into new projects.
- Frost1x 3y ago
  >Optimism is (almost) always required in order to accomplish anything of significance. Those who lose it, aren't living up to their potential. I argue that realism trumps optimism. It's perfectly normal in a realist farming to see something difficult, acknowledge the high risk and failure potential, and still pursue something with intent to succeed. I've personally grown tired of over optimism everywhere because it creates unrealistic situations and passes consequences of failure in an inequitable way. The "visionary" is rewarded when the rare successes occur, while everyone else suffers the consequences for most failures. No contingency plans for failure, no discussion of failure, and so on. Optimism just takes any idea, pursues it and consequences be someone else's problem and be damned. Pessimism isn't much better, you essentially think everything is too risky or unlikely to succeed so you never do anything. You live in a state of inaction because any level of risk or uncertainty is too much. To me, realism is much better. You acknowledge the challenge. You acknowledge the risk. You make sure everyone involved understands it, but you still charge forward knowing you might succeed. Some think if you're not naively optimistic (what most people in my experience refer to as "optimism") you don't create enough pressure. I think that's non-sense.

ShamelessC 3y ago

Wow! I never thought you’d see the light. All I ever see from your posts is praise for TRC. As someone who got started way later on, I had infinitely more success with a gaming GPU I owned myself. Obviously not really comparable, but TRC was very very difficult to work with. I think I only ever had access to a TPUv3 once and that wasn’t nearly enough time to learn the ropes. My understanding was that this situation changed drastically depending on what sort of email you had or how popular your Twitter handle was.

haldujai 3y ago

My experience has been different. Considering how easy the application is I think they're still being fairly generous as I've been offered multiple v3-8s and v3-32s x 30days as well as pre-emptible v3-64s x 28 days for a few different projects within the last 6 months. Are you affiliated with an academic institution? Otherwise I'm not sure why they're been more generous with me, my projects have been mildly interesting at best. They're certainly a lot stingier with larger pods than they used to be though.

zak 3y ago

Actually, the TPU Research Cloud program is still going strong! We've expanded the compute pool significantly to include Cloud TPU v4 Pod slices, and larger projects still use hundreds of chips at a time. (TRC capacity has not been reclaimed for internal use.) Check out this list of recent TRC-supported publications: https://sites.research.google/trc/publications/ https://sites.research.google/trc/publications/ Demand for Cloud TPUs is definitely intense, so if you're using preemptible capacity, you're probably seeing more frequent interruptions, but reserved capacity is also available. Hope you email the TRC support team to say hello!

sillysaurusx 3y ago
Zak, I love you buddy, but you should have some of your researchers try to use the TRC program. They should pretend to be a nobody (like I was in 2019) and try to do any research with the resources they’re granted. I guarantee you those researchers will all tell you “we can’t start any training runs anymore because the TPUs die after 45 minutes.” This may feel like an anime betrayal, since you basically launched my career as a scientist. But it’s important for hobbyists and tinkerers to be able to participate in the AI ecosystem, especially today. And TRC just does not support them anymore. I tried, many times, over the last year and a half. You don’t need to take my word for it. Here’s some unfiltered DMs on the subject: https://imgur.com/a/6vqvzXs https://imgur.com/a/6vqvzXs Notice how their optimism dries up, and not because I was telling them how bad TRC has become. It’s because their TPUs kept dying. I held out hope for so long. I thought it was temporary. It ain’t temporary, Zak. And I vividly remember when it happened. Some smart person in google proposed a new allocation algorithm back near the end of 2021, and poof, overnight our ability to make TPUs went from dozens to a handful. It was quite literally overnight; we had monitoring graphs that flatlined. I can probably still dig them up. I’ve wanted to email you privately about this, but given that I am a small fish in a pond that’s grown exponentially bigger, I don’t think it would’ve made a difference. The difference is in your last paragraph: you allocate reserved instances to those who deserve it, and leave everybody else to fight over 45 minutes of TPU time when it takes 25 minutes just to create and fill your TPU with your research data. Your non-preemptible TPUs are frankly a lie. I didn’t want to drop the L word, but a TPUv3 in euw4a will literally delete itself — aka preempt — after no more than a couple hours. I tested this over many months. That was some time ago, so maybe things have changed, but I wouldn’t bet on it. There’s some serious “left hand doesn’t know that right hand detached from its body and migrated south for the winter” energy in the TRC program. I don’t know where it embedded itself, but if you want to elevate any other engineers from software devs to researchers, I urge you to make some big changes. One last thing. The support staff of TRC is phenomenal. Jonathan Colton has worked more miracles than I can count, along with the rest of his crew. Ultimately he had to send me an email like “by the way, TRC doesn’t delete TPUs. This distinction probably won’t be too relevant, but I wanted to let you know” (paraphrasing). Translation: you took the power away from the people who knew where to put it (Jonathan) and gave it to some really important researchers, probably in Brain or some other division of Google. And the rest is history. So I don’t want to hear that one of the changes is “ok, we’ve punished the support staff” - as far as I can tell, they’ve moved mountains with whatever tools they had available, and I definitely wouldn’t have been able to do any better in their shoes. Also, hello. Thanks for launching my career. Sorry that I had to leave this here, but my duty is to the open source community. The good news is that you can still recover, if only you’d revert this silly “we’ll slip you some reserved TPUs that don’t kamikaze themselves after 45 minutes if you ask in just the right way” stuff. That wasn’t how the program was in 2019, and I guarantee that I couldn’t have done the work I did then under the current conditions.