5 ms·

Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up

Not blaming them, but still a nice example to illustrate the myth that 'the cloud' will let you operate without a sysadmin (someone that knows about setting up and managing infrastructure). Yes, there might not be servers you have to rack and electrical power to be provisioned, but the amount and granularity of the parameters and decisions on the virtual fabric to be decided on and continuously monitored for changes and opportunities that could have an enormous impact on performance and cost is orders of magnitude higher.

jdnenej 7y ago
The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.
- CriticalCathed 7y ago
  This isn't 2005. It really doesn't need to happen anymore.
  - londons_explore 7y ago
    Says the person not paying the $10k/day unexpected bill...
- bildung 7y ago
  The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.
  - neysofu 7y ago
    It absolutely is high traffic if each and every one of them is downloading your NN model
    - bildung 7y ago
      I see the model size is 6G - a bit larger than I thought :) 100gbit/s upstream would cost an additional 1k€/month. The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.
      - orbifold 7y ago
        The server instances are free though, google only charged them for the datatransfer to those instances.
  - shandor 7y ago
    For similar price point (€105.62), from e.g. Azure you can get their "E2 v3" VM, with 2 vCPUs and 16 GB RAM, plus 50GB temporary storage. That's...quite a difference.
  - jrockway 7y ago
    They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption). That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU <-> video card, not disk <-> cpu, or cpu <-> network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game. The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.
    - krick 7y ago
      > 60k downloads of 6GB models per day Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.
      - jsty 7y ago
        No, the models are running in Colab - but in the end-user's account - so each 'run' costs the lab 6GB of internet egress when the model is downloaded from GCS to the Colab VM.
      - lis 7y ago
        No, the computation is not running on the client in the browser. This is the traffic to transfer the model from GCS to Google Colab. This is what makes the price so surprising - you are copying data from one Google Service to another, but it's billed as egress.
        jstanley 7y ago
  - mark_l_watson 7y ago
    About a year and a half ago, I rented a Hetzner server with a 1080 GPU, 16 core i7, etc. it was only about $105/month. I had a job queue for TensorFlow/Keras experiments and set it to mine ether when there were no ML jobs. I did this until I bought a System76 laptop with a 1070 GPU.
edf13 7y ago
Also not blaming him/them but isn't this an indicator of lack of cost control at research facilities? Surely someone should of costed this out before making it public? Here in the UK running at 10k/day would of ruined a departments budget for an entire year!
- PeterStuer 7y ago
  Costing for cloud solutions is often not trivial, and a single overlooked parameter can do you in. I at least blame Google partially for having quotas and spending limits set to unlimited by default.
  - FussyZeus 7y ago
    Don't forget Amazon's incomprehensible billing practices for AWS. It seems to be designed to be impossible to figure out what you're going to spend ahead of time.
    - wongarsu 7y ago
      Amazon really seems to be doing this on purpose. Even tasks as simple as starting an EC2 instance require referencing multiple pages to find out how much this instance will cost you, while some other providers just show the predicted costs/month in the instance creation dialog.
- masukomi 7y ago
  in addition to how difficult costing cloud services is, what if you get it perfect but underestimate the success of your product? great, "it would have been good except we were too popular" do you then cut off your popular product while it's gaining traction, and thus almost guarantee you kill it in its infancy?
FooHentai 7y ago
Also, at least from what I've witnessed, the kind of sysadmin activities often stated as those that cloud eliminates was already becoming a vanishingly small element of the day-to-day work involved in being a sysadmin. It was already going away, and cloud was not the reason. But with the whole cloud = ditch sysadmins, and the weird way that 'devops' was interpreted to mean a similar thing, a whole lot of baby got thrown out alongside the bathwater. Virtualization, the maturity of modern networking, compute and storage systems, and the sheer capacity of each atomic piece of hardware means that kind of stuff had already become a once-every-couple-of-years kinda task for most sysadmins, except the few still stuck in some kind of backwards anachronistic environment.
- blaser-waffle 7y ago
  > Also, at least from what I've witnessed, the kind of sysadmin activities often stated as those that cloud eliminates was already becoming a vanishingly small element of the day-to-day work involved in being a sysadmin. This. Most of my work is not technical and consists of technical-ish meetings, and budget arguments. Automation is part of that, but also that a lot of stuff is fairly robust and works (mostly) after initial config. Three weeks of setup, plus some shakedown (and a few late nights when the new mission-critical system stutters), but then it's mostly autopilot, or scheduled maintenance.

jdnenej 7y ago

The non cloud version is your website goes offline for a day due to high traffic. Depending on the website that may or may not be preferable to variable costs.

CriticalCathed 7y ago
This isn't 2005. It really doesn't need to happen anymore.
- londons_explore 7y ago
  Says the person not paying the $10k/day unexpected bill...
bildung 7y ago
The Twitter feed says the $10k+ are for serving about 60k Visitors a day. This isn't exactly high traffic. A dedicated server from Hetzner with unlimited Traffic and 1gbit/s uplink, a 16 core Threadripper, 1TB SSD and 128G RAM costs €140/Month and would be absolute overkill for 60k Visitors. It doesn't have GPUs, though.
- neysofu 7y ago
  It absolutely is high traffic if each and every one of them is downloading your NN model
  - bildung 7y ago
    I see the model size is 6G - a bit larger than I thought :) 100gbit/s upstream would cost an additional 1k€/month. The larger problem is that people didn't actually download the model, but apparently got custom server instances that got a copy of the model plus a gpu to run them on.
    - orbifold 7y ago
      The server instances are free though, google only charged them for the datatransfer to those instances.
- shandor 7y ago
  For similar price point (€105.62), from e.g. Azure you can get their "E2 v3" VM, with 2 vCPUs and 16 GB RAM, plus 50GB temporary storage. That's...quite a difference.
- jrockway 7y ago
  They are seeing 60k downloads of 6GB models per day, which is about 33Gbps of bandwidth (assuming no burstiness in when people visit, which is a poor assumption). That is starting to get out of the range of what is easily available. 10Gbps circuits are commodities (I had one at my desk at my last job), but 100Gbps circuits are still pretty pricey. And, it's not necessarily trivial to get that kind of throughput on file serving out of the box; this bandwidth is something like CPU <-> video card, not disk <-> cpu, or cpu <-> network. Some tweaking is for sure going to be necessary if you are self-hosting this, and now you're tweaking network parameters and writing a custom file server instead of writing your game. The cloud here is making something possible that should never have been possible, which is pretty cool. Being able to go from 0 infrastructure to 30Gbps of file serving without lifting a finger is somewhat impressive... but with that fast iteration times, comes the entity that did all the work wanting their cut. It seems fair to me, though perhaps not economically viable. Such is life.
  - krick 7y ago
    > 60k downloads of 6GB models per day Wait... you mean the actual computation is running client in the browser? I didn't even open this "game", but I assumed such high cost is because there is a separate GPT-2 running on a GPUs for each and every user.
    - jsty 7y ago
      No, the models are running in Colab - but in the end-user's account - so each 'run' costs the lab 6GB of internet egress when the model is downloaded from GCS to the Colab VM.
    - lis 7y ago
      No, the computation is not running on the client in the browser. This is the traffic to transfer the model from GCS to Google Colab. This is what makes the price so surprising - you are copying data from one Google Service to another, but it's billed as egress.
      - jstanley 7y ago
- mark_l_watson 7y ago
  About a year and a half ago, I rented a Hetzner server with a 1080 GPU, 16 core i7, etc. it was only about $105/month. I had a job queue for TensorFlow/Keras experiments and set it to mine ether when there were no ML jobs. I did this until I bought a System76 laptop with a 1070 GPU.

edf13 7y ago

Also not blaming him/them but isn't this an indicator of lack of cost control at research facilities? Surely someone should of costed this out before making it public? Here in the UK running at 10k/day would of ruined a departments budget for an entire year!

PeterStuer 7y ago
Costing for cloud solutions is often not trivial, and a single overlooked parameter can do you in. I at least blame Google partially for having quotas and spending limits set to unlimited by default.
- FussyZeus 7y ago
  Don't forget Amazon's incomprehensible billing practices for AWS. It seems to be designed to be impossible to figure out what you're going to spend ahead of time.
  - wongarsu 7y ago
    Amazon really seems to be doing this on purpose. Even tasks as simple as starting an EC2 instance require referencing multiple pages to find out how much this instance will cost you, while some other providers just show the predicted costs/month in the instance creation dialog.
masukomi 7y ago
in addition to how difficult costing cloud services is, what if you get it perfect but underestimate the success of your product? great, "it would have been good except we were too popular" do you then cut off your popular product while it's gaining traction, and thus almost guarantee you kill it in its infancy?

FooHentai 7y ago

Also, at least from what I've witnessed, the kind of sysadmin activities often stated as those that cloud eliminates was already becoming a vanishingly small element of the day-to-day work involved in being a sysadmin. It was already going away, and cloud was not the reason. But with the whole cloud = ditch sysadmins, and the weird way that 'devops' was interpreted to mean a similar thing, a whole lot of baby got thrown out alongside the bathwater. Virtualization, the maturity of modern networking, compute and storage systems, and the sheer capacity of each atomic piece of hardware means that kind of stuff had already become a once-every-couple-of-years kinda task for most sysadmins, except the few still stuck in some kind of backwards anachronistic environment.

blaser-waffle 7y ago
> Also, at least from what I've witnessed, the kind of sysadmin activities often stated as those that cloud eliminates was already becoming a vanishingly small element of the day-to-day work involved in being a sysadmin. This. Most of my work is not technical and consists of technical-ish meetings, and budget arguments. Automation is part of that, but also that a lot of stuff is fairly robust and works (mostly) after initial config. Three weeks of setup, plus some shakedown (and a few late nights when the new mission-critical system stutters), but then it's mostly autopilot, or scheduled maintenance.