3 ms·

I hope they'll also attach GPU's to those machines. We switched part of our operation to local hardware because wouldn't get both the fast GPU's and the fast CP

by tinco 7y ago

I hope they'll also attach GPU's to those machines. We switched part of our operation to local hardware because wouldn't get both the fast GPU's and the fast CPU's in the same node. A proper solution of course would be to have the CPU intensive algorithms run on different nodes, but it's an integrated solution we pay for so we don't control that.

ImJasonH 7y ago
You can already get 1-4 GPUs on VMS with 64-96 vCPUs in a number of regions: https://cloud.google.com/compute/docs/gpus/#gpus-list https://cloud.google.com/compute/docs/gpus/#gpus-list Is the problem that the CPU max scales with the number of GPUs, so you can't get 1 GPU with 96 vCPUs?
- tinco 7y ago
  No, we were looking for the high single thread performance machines, so the new compute optimized cascade lake ones, but those you can't get gpu's on.
KaoruAoiShiho 7y ago
> We switched part of our operation to local hardware because wouldn't get both the fast GPU's and the fast CPU's in the same node. And how did that work out? Upfront costs but should be significant savings overall right?
- tinco 7y ago
  Not sure yet, the machines should arrive next week. To be honest we might switch back and forth between own and cloud hardware as we scale up. Right now we process one dataset per week or so, but by December that should become multiple per day. The 3 machines we bought should put us at around 1 per 2 days, but it's more so we can do worry free experimentation. It's really annoying to be constantly worried about wasting money when experimenting. We 'wasted' thousands just experimenting with flags and rendering bad datasets, of course it might still come out as cost effective but it discourages experimentation. The big question for us is if we're going to be able to afford to do it locally, there's both the upfront cost and the cost of system administration. These 3 I could still do with my dev team amateur style, but when that becomes 30, we'll probably need some technician that has experience managing compute clusters.

by tinco 7y ago

ImJasonH 7y ago

You can already get 1-4 GPUs on VMS with 64-96 vCPUs in a number of regions: https://cloud.google.com/compute/docs/gpus/#gpus-list https://cloud.google.com/compute/docs/gpus/#gpus-list Is the problem that the CPU max scales with the number of GPUs, so you can't get 1 GPU with 96 vCPUs?

tinco 7y ago
No, we were looking for the high single thread performance machines, so the new compute optimized cascade lake ones, but those you can't get gpu's on.

KaoruAoiShiho 7y ago

> We switched part of our operation to local hardware because wouldn't get both the fast GPU's and the fast CPU's in the same node. And how did that work out? Upfront costs but should be significant savings overall right?

tinco 7y ago
Not sure yet, the machines should arrive next week. To be honest we might switch back and forth between own and cloud hardware as we scale up. Right now we process one dataset per week or so, but by December that should become multiple per day. The 3 machines we bought should put us at around 1 per 2 days, but it's more so we can do worry free experimentation. It's really annoying to be constantly worried about wasting money when experimenting. We 'wasted' thousands just experimenting with flags and rendering bad datasets, of course it might still come out as cost effective but it discourages experimentation. The big question for us is if we're going to be able to afford to do it locally, there's both the upfront cost and the cost of system administration. These 3 I could still do with my dev team amateur style, but when that becomes 30, we'll probably need some technician that has experience managing compute clusters.