3 ms·

I wonder if a 1B model could be close to free to host. That's an eventuality, but I wonder how long it'll take for that to be real.

• giantrobot 1y ago
  A 1B model at 2-bit quantization is about the size of the average web page anymore. With some WebGPU support you could run such a model in a browser. I'm half joking. Web pages are ludicrously fat these days.
  • reverius42 1y ago
    Something like this? https://github.com/huggingface/transformers.js-examples/tree/main/llama-3.2-webgpu https://github.com/huggingface/transformers.js-examples/tree...
    • giantrobot 1y ago
      That was the exact thing I was thinking of but couldn't remember the project name. Thanks!
• codazoda 1y ago
  I’m planning to deploy a 1B model, feed it all the documents I’ve ever written, host it on a $149 mini-PC in my bedroom, and enable you to chat with it. I’ve released similar projects before. I’ll drop a post about my plans in the coming days and I’ll build and document it about two weeks later if there’s enough interest. joeldare.com
  • indigodaddy 1y ago
    Sounds cool!
  • codazoda 1y ago
    The post about my plan is now online: https://joeldare.com/my_plan_to_build_an_ai_chat_bot_in_my_bedroom https://joeldare.com/my_plan_to_build_an_ai_chat_bot_in_my_b...