8 ms·

The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffi

The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.

neysofu 7y ago
It absolutely is high traffic if each and every one of them is downloading your NN model
- bildung 7y ago
  I see the model size is 6G - a bit larger than I thought :) 100gbit/s upstream would cost an additional 1k€/month. The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.
  - orbifold 7y ago
    The server instances are free though, google only charged them for the datatransfer to those instances.
shandor 7y ago
For similar price point (€105.62), from e.g. Azure you can get their "E2 v3" VM, with 2 vCPUs and 16 GB RAM, plus 50GB temporary storage. That's...quite a difference.
jrockway 7y ago
They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption). That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU <-> video card, not disk <-> cpu, or cpu <-> network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game. The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.
- krick 7y ago
  > 60k downloads of 6GB models per day Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.
  - jsty 7y ago
    No, the models are running in Colab - but in the end-user's account - so each 'run' costs the lab 6GB of internet egress when the model is downloaded from GCS to the Colab VM.
  - lis 7y ago
    No, the computation is not running on the client in the browser. This is the traffic to transfer the model from GCS to Google Colab. This is what makes the price so surprising - you are copying data from one Google Service to another, but it's billed as egress.
    - jstanley 7y ago
      But if you were hosting it yourself you wouldn't transfer 6G of data around per user. You'd be a bit more intelligent about it.
      - dwild 7y ago
        They send it to Google Colab. It allow you to run model over a powerful server for free. 6 GB of download at this crazy bandwidth cost would still be cheap versus offering themselves that kind of beast of a server to 60k persons each day. I remember when I tried it I saw that it used 10 GB of memory, that was crazy!
      - jrockway 7y ago
        You'd save on transferring the bytes around, but now you would have to self-host jupyter and the GPUs it uses. That is going to be even more expensive than IP transit because now you have to have 60,000 12GB GPUs in your datacenter. Like I said before, this is one of those things that wouldn't exist without the Cloud. If you run things on your user's computers, you have to send them a lot of bits. If you run things on your own computers, you're spared that bandwidth, but now have to have enough "computers" to satisfy your users. It's simply something that's not super cheap to run these days. I will admit that it is surprising that Google <-> Google traffic is billed at the normal egress rates, but the reasoning does make sense -- a 30Gbps flow is nothing to sneeze at. That is using some tangible resources.
mark_l_watson 7y ago
About a year and a half ago, I rented a Hetzner server with a 1080 GPU, 16 core i7, etc. it was only about $105/month. I had a job queue for TensorFlow/Keras experiments and set it to mine ether when there were no ML jobs. I did this until I bought a System76 laptop with a 1070 GPU.

neysofu 7y ago

It absolutely is high traffic if each and every one of them is downloading your NN model

bildung 7y ago
I see the model size is 6G - a bit larger than I thought :) 100gbit/s upstream would cost an additional 1k€/month. The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.
- orbifold 7y ago
  The server instances are free though, google only charged them for the datatransfer to those instances.

shandor 7y ago

For similar price point (€105.62), from e.g. Azure you can get their "E2 v3" VM, with 2 vCPUs and 16 GB RAM, plus 50GB temporary storage. That's...quite a difference.

jrockway 7y ago

They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption). That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU <-> video card, not disk <-> cpu, or cpu <-> network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game. The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.

krick 7y ago
> 60k downloads of 6GB models per day Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.
- jsty 7y ago
  No, the models are running in Colab - but in the end-user's account - so each 'run' costs the lab 6GB of internet egress when the model is downloaded from GCS to the Colab VM.
- lis 7y ago
  No, the computation is not running on the client in the browser. This is the traffic to transfer the model from GCS to Google Colab. This is what makes the price so surprising - you are copying data from one Google Service to another, but it's billed as egress.
  - jstanley 7y ago
    But if you were hosting it yourself you wouldn't transfer 6G of data around per user. You'd be a bit more intelligent about it.
    - dwild 7y ago
      They send it to Google Colab. It allow you to run model over a powerful server for free. 6 GB of download at this crazy bandwidth cost would still be cheap versus offering themselves that kind of beast of a server to 60k persons each day. I remember when I tried it I saw that it used 10 GB of memory, that was crazy!
    - jrockway 7y ago
      You'd save on transferring the bytes around, but now you would have to self-host jupyter and the GPUs it uses. That is going to be even more expensive than IP transit because now you have to have 60,000 12GB GPUs in your datacenter. Like I said before, this is one of those things that wouldn't exist without the Cloud. If you run things on your user's computers, you have to send them a lot of bits. If you run things on your own computers, you're spared that bandwidth, but now have to have enough "computers" to satisfy your users. It's simply something that's not super cheap to run these days. I will admit that it is surprising that Google <-> Google traffic is billed at the normal egress rates, but the reasoning does make sense -- a 30Gbps flow is nothing to sneeze at. That is using some tangible resources.

mark_l_watson 7y ago

About a year and a half ago, I rented a Hetzner server with a 1080 GPU, 16 core i7, etc. it was only about $105/month. I had a job queue for TensorFlow/Keras experiments and set it to mine ether when there were no ML jobs. I did this until I bought a System76 laptop with a 1070 GPU.