4 ms·

Show HN: Run AI models on your PC instead of the cloud

fair-dikobraz 3y ago
Hi all, I'm currently addressing the challenge of high-cost GPU computations and trying to come up with a solution. I would appreciate your insights. Currently, there are a few options: 1. Rent a GPU from AWS, GCP, and Azure. It is the most popular but also the most expensive option. Renting A100 will cost about $3-4.5 per hour, which can rack up to $3000 monthly. If your workload can not utilize hardware fully, you can leverage services that charge per second or request instead of per hour, like Replicate. The typical cost is around $5 per hour; however, you'll save a lot of money if your computations need to run sporadically. 2. Work with less-known providers like Lambda, Coreweave, or runpod.io. You can get the same GPU for about $1.1-2.1 per hour. However, it is tough to get one at times. 3. Rent a cheaper GPU like A6000 for about $0.6 or consumer-grade RTX 4090 for approximately $0.4 from vast.ai. 4. Use your PC if you have one. The cost varies greatly depending on the electricity price in your area; we can assume around $0.1 per hour at $0.2 kWH and 500W power consumption. At full load, it will cost you around $70 per month. The last option is considerably cheaper than the others. However, managing a cluster at home or in the office requires qualifications and effort. Also, existing tools don't target this use case to the best of my knowledge. For example, we need to ensure that multiple workloads are not using the resources simultaneously. This is the problem that I’m trying to address with FairCompute by providing tools and API for managing your home cluster and workloads. Here is a demo leveraging my gaming PC at home to serve an AI model - https://huggingface.co/spaces/faircompute/stable-diffusion https://huggingface.co/spaces/faircompute/stable-diffusion. The demo is hosted on HuggingFace, but a home PC is used for inference. I would appreciate your feedback on how to make Fair more useful or your thoughts on how to solve the problem of AI model training and deployment cost. If you want to try Fair - check out the tutorial at https://faircompute.com/docs/docs/getting_started https://faircompute.com/docs/docs/getting_started
orliesaurus 3y ago
Great idea, can you leverage other people's computers too?
- fair-dikobraz 3y ago
  That's the idea in the long run. Currently, the MVP allows you to leverage only computers that are logged in using your credentials. So only your machines and your friend's if they trust you. The main challenge for leveraging other people's machines is the economy. Need to estimate how much computation time is worth given differences in hardware, reliability, location and connection speed to make the system fair for all paricipants. Privacy and safety is another typical concern, but I believe modern sandboxing technologies give enough protection. Any browser is running the code from the internet in the end. An interesting problem to solve.

Hi all, I'm currently addressing the challenge of high-cost GPU computations and trying to come up with a solution. I would appreciate your insights. Currently, there are a few options: 1. Rent a GPU from AWS, GCP, and Azure. It is the most popular but also the most expensive option. Renting A100 will cost about $3-4.5 per hour, which can rack up to $3000 monthly. If your workload can not utilize hardware fully, you can leverage services that charge per second or request instead of per hour, like Replicate. The typical cost is around $5 per hour; however, you'll save a lot of money if your computations need to run sporadically. 2. Work with less-known providers like Lambda, Coreweave, or runpod.io. You can get the same GPU for about $1.1-2.1 per hour. However, it is tough to get one at times. 3. Rent a cheaper GPU like A6000 for about $0.6 or consumer-grade RTX 4090 for approximately $0.4 from vast.ai. 4. Use your PC if you have one. The cost varies greatly depending on the electricity price in your area; we can assume around $0.1 per hour at $0.2 kWH and 500W power consumption. At full load, it will cost you around $70 per month. The last option is considerably cheaper than the others. However, managing a cluster at home or in the office requires qualifications and effort. Also, existing tools don't target this use case to the best of my knowledge. For example, we need to ensure that multiple workloads are not using the resources simultaneously. This is the problem that I’m trying to address with FairCompute by providing tools and API for managing your home cluster and workloads. Here is a demo leveraging my gaming PC at home to serve an AI model - https://huggingface.co/spaces/faircompute/stable-diffusion https://huggingface.co/spaces/faircompute/stable-diffusion. The demo is hosted on HuggingFace, but a home PC is used for inference. I would appreciate your feedback on how to make Fair more useful or your thoughts on how to solve the problem of AI model training and deployment cost. If you want to try Fair - check out the tutorial at https://faircompute.com/docs/docs/getting_started https://faircompute.com/docs/docs/getting_started

orliesaurus 3y ago

Great idea, can you leverage other people's computers too?

fair-dikobraz 3y ago
That's the idea in the long run. Currently, the MVP allows you to leverage only computers that are logged in using your credentials. So only your machines and your friend's if they trust you. The main challenge for leveraging other people's machines is the economy. Need to estimate how much computation time is worth given differences in hardware, reliability, location and connection speed to make the system fair for all paricipants. Privacy and safety is another typical concern, but I believe modern sandboxing technologies give enough protection. Any browser is running the code from the internet in the end. An interesting problem to solve.