4 ms·

I rolled my own version of this for a side project. I ended up on a Digital Ocean droplet with RAM upgrades. https://lucidbeaming.com/blog/fine-tuning-a-gpt-2-

by ogou 6y ago

I rolled my own version of this for a side project. I ended up on a Digital Ocean droplet with RAM upgrades. https://lucidbeaming.com/blog/fine-tuning-a-gpt-2-language-model-and-generating-text-with-a-flask-web-app/ https://lucidbeaming.com/blog/fine-tuning-a-gpt-2-language-m... The extra monthly $40 for a novelty project has me searching for cheaper alternatives. Google preemptibles scared me off because they can scale dynamically and generate runaway costs if my project suddenly got popular on Reddit or something.

htahir111 6y ago
Nice! Thats a cool side project. Not sure how Google preemptible can scale though. They are not auto-scaling - so its a fixed cost
- ogou 6y ago
  I swear I saw something in the terms about charges from sustained uptime, which is related. A popular resource would trigger that. It starts and stops on demand so that automation is what I am leery of. I will look into it again though, maybe I misinterpreted the fine print.
  - htahir111 6y ago
    Ah no thats more for AWS spot instances I think. In Google, preemtible rates are constant, which makes it way more deterministic
zomglings 6y ago
Preemptible instances should not have any associated auto-scaling. It is common for people to run pre-emptible instances as part of a "Managed Instance Group", with which you can associate some kind of autoscaling policy. You can run preemptible instances without any kind of management or with only very simple management and you should be okay. I recommend a simple managed group with a policy of, "I want maximum one of these instances running at all times and, if it gets preempted, then please start a new one." If you're using Google Cloud, don't deny yourself the cost savings of preemptible instances. When I was using the same infrastructure, preemptions were very rare!
FL33TW00D 6y ago
What's the average inference time like for your model on DO?
- ogou 6y ago
  65 seconds
  - ogou 6y ago
    This is now 240 seconds on a Google Cloud N2D instance. I successfully recreated my inference worker on a Google preemptible instance. My monthly cost went from $40 on Digital Ocean to ~$16. It's much slower though.
    - FL33TW00D 6y ago
      Nice big cost reduction! Is this using BudgetML from the post? Have you tried optimizing the model (i.e quantization and converting to something like ONNX)? I know this can bring big speed gains on T5 on CPU (5x faster), another generative model. More info here: https://discuss.huggingface.co/t/speeding-up-t5-inference/1841/7 https://discuss.huggingface.co/t/speeding-up-t5-inference/18...

by ogou 6y ago

htahir111 6y ago

Nice! Thats a cool side project. Not sure how Google preemptible can scale though. They are not auto-scaling - so its a fixed cost

ogou 6y ago
I swear I saw something in the terms about charges from sustained uptime, which is related. A popular resource would trigger that. It starts and stops on demand so that automation is what I am leery of. I will look into it again though, maybe I misinterpreted the fine print.
- htahir111 6y ago
  Ah no thats more for AWS spot instances I think. In Google, preemtible rates are constant, which makes it way more deterministic

zomglings 6y ago

Preemptible instances should not have any associated auto-scaling. It is common for people to run pre-emptible instances as part of a "Managed Instance Group", with which you can associate some kind of autoscaling policy. You can run preemptible instances without any kind of management or with only very simple management and you should be okay. I recommend a simple managed group with a policy of, "I want maximum one of these instances running at all times and, if it gets preempted, then please start a new one." If you're using Google Cloud, don't deny yourself the cost savings of preemptible instances. When I was using the same infrastructure, preemptions were very rare!

FL33TW00D 6y ago

What's the average inference time like for your model on DO?

ogou 6y ago
65 seconds
- ogou 6y ago
  This is now 240 seconds on a Google Cloud N2D instance. I successfully recreated my inference worker on a Google preemptible instance. My monthly cost went from $40 on Digital Ocean to ~$16. It's much slower though.
  - FL33TW00D 6y ago
    Nice big cost reduction! Is this using BudgetML from the post? Have you tried optimizing the model (i.e quantization and converting to something like ONNX)? I know this can bring big speed gains on T5 on CPU (5x faster), another generative model. More info here: https://discuss.huggingface.co/t/speeding-up-t5-inference/1841/7 https://discuss.huggingface.co/t/speeding-up-t5-inference/18...