7 ms·

What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that

What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that is what matters. If the TPU 3 further drops the cost fantastic. How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?

deepnotderp 8y ago
You care from a technical standpoint. But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
- jacksmith21006 8y ago
  Cost is to do some task.
  - p1esk 8y ago
    So how does cost compare to run a task 24/7 for a year if I buy Nvidia GPUs vs rent TPUs?
    - jacksmith21006 8y ago
      Would expect Nvidia GPUs run 24/7 would be far cheaper. But that is an unusual use case. Even more so with ML training. It is much more bumpy. But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded. Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease. But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example. Just one year and we have the 3.0.
      - p1esk 8y ago
        Let's see, last year I bought a 4x 1080Ti workstation for ~$6k. I agree that I haven't run it 24/7 all year long, the typical usage pattern has been training a model for 2-8 days, then it's 0-3 days idle time. The longest it ever sat idle was a week. Let's say 50% utilization rate for the last 12 months. That would be 180x24 = 4,300 hours. In a small startup, typically such workstations would be shared by multiple people, so the utilization would be higher. When I interned in a small startup people always waited to launch their experiments, so the utilization was actually close to 24/7. So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
        jacksmith21006 8y ago
        The thing is ML for training is much more bumpy. You need further capacity at certain times. But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware. The other is buying the hardware is much harder from a cash flow standpoint.
        p1esk 8y ago
        So you completely ignored my questions, and simply repeated your previous comment word for word? You provided some generic considerations, which don't apply to my situation, and I'm a fairly typical ML researcher. In fact, I know several startups doing DL research and they all have been buying hardware. Can you give me some examples where doing DL training in the cloud makes financial sense? Also, you seem to imply that TPU is some kind of a miracle chip, even though we know very little about TPU2 and we know nothing about TPU3. It's actually pretty embarrassing that a general purpose V100 GPU is competitive with TPU2, which is an ASIC made for DL. If Nvidia ever decides to make a pure DL chip it will destroy anything Google can design. I mean, you can't be serious comparing Google to Nvidia when it comes to designing chips, right? That would be like believing that Nvidia could make a competitive search engine.
  - shaklee3 8y ago
    No. You don't pay google to run a task. You pay google to run a TPU instance for some amount of time. The time you spend leaving it on, setting it up, tearing it down, etc, are all still time you pay for. When they have TPU lambda jobs, that's different, but they don't. From their page: Virtual machine pricing In order to connect to a TPU, you must provision a virtual machine (VM), which is billed separately. For details on pricing for VM instances, see Compute Engine pricing.
    - jacksmith21006 8y ago
      You are a bit much. You have some task to complete. You can chose to use AWS with Nvidia chips or can use Google with their TPUs. How much it cost you to complete that task is what matters. How it is done is here or there as long as get the precision. We can see right now Google with their TPUs is about 1/2 the cost of using Nvidia with AWS.
      - shaklee3 8y ago
        No, you can't. Under the best case circumstance where you have a V100 and TPU both using tensorflow for something that's optimized for both, the TPU is about 37% better: https://blog.riseml.com/comparing-google-tpuv2-against-nvidia-v100-on-resnet-50-c2bbb6a51e5e https://blog.riseml.com/comparing-google-tpuv2-against-nvidi... The 50% is a number you made up, and isn't based in any real benchmarks. For all other tasks that are not tensorflow tasks, the V100 is the only one that works.
        jacksmith21006 8y ago
        We can see the cost and they are signficantly less for TPU 2.0. But now Google has the 3.0. But if we look at WaveNet we can see Google must be taking much larger margins with the TPU 2.0 versus Amazon using Nvidia. Rolling out using a NN at 16k cycles a second and offering at a competitive price to the old way means Google TPUs have to be way more efficient than using anything from Nvidia. It is hard to believe Google pulled it off. But if we look at WaveNet it suggests that TTS is a solved problem. It will be how it is done for a very long time and just the NN will improve. Nvidia honestly needs to get on their game. Google is running a 1000 mph. Iterating to the TPU 3.0 in just a year was a big surprise. I suspect Capsule networks and dynamic routing which were invented by Google drove the TPU 3.0 but do not know. Hope Google will share a paper now on the TPU 2.0 and their secrets. Lucky for Nvidia they will share and Nvidia can copy. But just keeps Nvidia behind.
Voloskaya 8y ago
I care because assuming the cost of a single TPUv3 is roughly similar to the cost of a single TPUv2 (at launch), then paying for a "pod" of 256 or 512 (whatever a pod is this year) will be nowhere near the cost of 64 TPUv2 "pod". So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS. If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
- jacksmith21006 8y ago
  You can already hook together TPUs. So, IMO, it is here or there what is happening in the data center. What I care about is how much is the cost. Time to train is more of a function of how much resources I want to use.
  - Voloskaya 8y ago
    So what? I just want to know if the 8x speed increase was achieve at roughly the same cost (i.e same numbers of TPUs) or not. If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
    - jacksmith21006 8y ago
      I am also curious. But really it comes down to cost. Right now we see about 1/2 the cost of using TPUs versus Nvidia on the AWS cloud. Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.

deepnotderp 8y ago

You care from a technical standpoint. But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)

jacksmith21006 8y ago
Cost is to do some task.
- p1esk 8y ago
  So how does cost compare to run a task 24/7 for a year if I buy Nvidia GPUs vs rent TPUs?
  - jacksmith21006 8y ago
    Would expect Nvidia GPUs run 24/7 would be far cheaper. But that is an unusual use case. Even more so with ML training. It is much more bumpy. But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded. Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease. But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example. Just one year and we have the 3.0.
    - p1esk 8y ago
      Let's see, last year I bought a 4x 1080Ti workstation for ~$6k. I agree that I haven't run it 24/7 all year long, the typical usage pattern has been training a model for 2-8 days, then it's 0-3 days idle time. The longest it ever sat idle was a week. Let's say 50% utilization rate for the last 12 months. That would be 180x24 = 4,300 hours. In a small startup, typically such workstations would be shared by multiple people, so the utilization would be higher. When I interned in a small startup people always waited to launch their experiments, so the utilization was actually close to 24/7. So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
      - jacksmith21006 8y ago
        The thing is ML for training is much more bumpy. You need further capacity at certain times. But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware. The other is buying the hardware is much harder from a cash flow standpoint.
        p1esk 8y ago
        So you completely ignored my questions, and simply repeated your previous comment word for word? You provided some generic considerations, which don't apply to my situation, and I'm a fairly typical ML researcher. In fact, I know several startups doing DL research and they all have been buying hardware. Can you give me some examples where doing DL training in the cloud makes financial sense? Also, you seem to imply that TPU is some kind of a miracle chip, even though we know very little about TPU2 and we know nothing about TPU3. It's actually pretty embarrassing that a general purpose V100 GPU is competitive with TPU2, which is an ASIC made for DL. If Nvidia ever decides to make a pure DL chip it will destroy anything Google can design. I mean, you can't be serious comparing Google to Nvidia when it comes to designing chips, right? That would be like believing that Nvidia could make a competitive search engine.
- shaklee3 8y ago
  No. You don't pay google to run a task. You pay google to run a TPU instance for some amount of time. The time you spend leaving it on, setting it up, tearing it down, etc, are all still time you pay for. When they have TPU lambda jobs, that's different, but they don't. From their page: Virtual machine pricing In order to connect to a TPU, you must provision a virtual machine (VM), which is billed separately. For details on pricing for VM instances, see Compute Engine pricing.
  - jacksmith21006 8y ago
    You are a bit much. You have some task to complete. You can chose to use AWS with Nvidia chips or can use Google with their TPUs. How much it cost you to complete that task is what matters. How it is done is here or there as long as get the precision. We can see right now Google with their TPUs is about 1/2 the cost of using Nvidia with AWS.
    - shaklee3 8y ago
      No, you can't. Under the best case circumstance where you have a V100 and TPU both using tensorflow for something that's optimized for both, the TPU is about 37% better: https://blog.riseml.com/comparing-google-tpuv2-against-nvidia-v100-on-resnet-50-c2bbb6a51e5e https://blog.riseml.com/comparing-google-tpuv2-against-nvidi... The 50% is a number you made up, and isn't based in any real benchmarks. For all other tasks that are not tensorflow tasks, the V100 is the only one that works.
      - jacksmith21006 8y ago
        We can see the cost and they are signficantly less for TPU 2.0. But now Google has the 3.0. But if we look at WaveNet we can see Google must be taking much larger margins with the TPU 2.0 versus Amazon using Nvidia. Rolling out using a NN at 16k cycles a second and offering at a competitive price to the old way means Google TPUs have to be way more efficient than using anything from Nvidia. It is hard to believe Google pulled it off. But if we look at WaveNet it suggests that TTS is a solved problem. It will be how it is done for a very long time and just the NN will improve. Nvidia honestly needs to get on their game. Google is running a 1000 mph. Iterating to the TPU 3.0 in just a year was a big surprise. I suspect Capsule networks and dynamic routing which were invented by Google drove the TPU 3.0 but do not know. Hope Google will share a paper now on the TPU 2.0 and their secrets. Lucky for Nvidia they will share and Nvidia can copy. But just keeps Nvidia behind.

Voloskaya 8y ago

I care because assuming the cost of a single TPUv3 is roughly similar to the cost of a single TPUv2 (at launch), then paying for a "pod" of 256 or 512 (whatever a pod is this year) will be nowhere near the cost of 64 TPUv2 "pod". So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS. If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)

jacksmith21006 8y ago
You can already hook together TPUs. So, IMO, it is here or there what is happening in the data center. What I care about is how much is the cost. Time to train is more of a function of how much resources I want to use.
- Voloskaya 8y ago
  So what? I just want to know if the 8x speed increase was achieve at roughly the same cost (i.e same numbers of TPUs) or not. If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
  - jacksmith21006 8y ago
    I am also curious. But really it comes down to cost. Right now we see about 1/2 the cost of using TPUs versus Nvidia on the AWS cloud. Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.