3 ms·

Nice I built something similar https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calculator https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calc..

Nice I built something similar https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calculator https://huggingface.co/spaces/Whiteshadow12/llm-pricing-calc... I like your charting, many have taken this task and then lose interest. similar other tools for inspiration https://llmprices.dev/ https://llmprices.dev/ https://www.llmpricing.app/ https://www.llmpricing.app/ What no one is doing is focusing on GPUs, what is the cost of running L3-8B on an A100 or H100 per second.

alextttty 2y ago
Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot! There are many inference engines which can be tested and updated to find best inference methods
- Whiteshadow12 2y ago
  Goodluck, companies would love that. Don't get depressed unlike my tool I think you should charge, that might keep you motivated to keep doing the work. It's a lot of work, your target users is companies that use Runpod and AWS/GCP/Azure, not Fireworks and Together, they are in the game of selling tokens, you are selling the cost of running seconds on GPUs.
  - alextttty 2y ago
    Thank you!
  - agcat 2y ago
    This is true especially if you are deploying custom or fine-tuned models. Infact, for my company i also ran benchmark tests where we tested cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers - https://www.inferless.com/learn/the-state-of-serverless-gpus-part-2 https://www.inferless.com/learn/the-state-of-serverless-gpus... Can save months of evaluation time. Do give it a read. P.S: I am from Inferless.
maeil 2y ago
Going to add my personal favorite: https://artificialanalysis.ai/models/llama-3-instruct-70b/providers#latency https://artificialanalysis.ai/models/llama-3-instruct-70b/pr... What sets them apart is that they have speed and latency as well.
mcbetz 2y ago
Thanks for bringing llmprices.dev to my attention. I have also a comparison page for models hosted on OpenRouter (https://minthemiddle.github.io/openrouter-model-comparison/ https://minthemiddle.github.io/openrouter-model-comparison/), I do comparison via regex (so "claude-3-haiku(?!:beta)|flash" will show you haiku, but not haiku-beta vs flash. I wish that OpenRouter would also expose the amount of output tokens via API as this is also an important criteria.

alextttty 2y ago

Yeah we want to do exactly this, benchmark and add more data from differnt gpus/cloud providers, will appreciate your help a lot! There are many inference engines which can be tested and updated to find best inference methods

Whiteshadow12 2y ago
Goodluck, companies would love that. Don't get depressed unlike my tool I think you should charge, that might keep you motivated to keep doing the work. It's a lot of work, your target users is companies that use Runpod and AWS/GCP/Azure, not Fireworks and Together, they are in the game of selling tokens, you are selling the cost of running seconds on GPUs.
- alextttty 2y ago
  Thank you!
- agcat 2y ago
  This is true especially if you are deploying custom or fine-tuned models. Infact, for my company i also ran benchmark tests where we tested cold-starts, performance consistency, scalability, and cost-effectiveness for models like Llama2 7Bn & Stable Diffusion across different providers - https://www.inferless.com/learn/the-state-of-serverless-gpus-part-2 https://www.inferless.com/learn/the-state-of-serverless-gpus... Can save months of evaluation time. Do give it a read. P.S: I am from Inferless.

maeil 2y ago

Going to add my personal favorite: https://artificialanalysis.ai/models/llama-3-instruct-70b/providers#latency https://artificialanalysis.ai/models/llama-3-instruct-70b/pr... What sets them apart is that they have speed and latency as well.

mcbetz 2y ago

Thanks for bringing llmprices.dev to my attention. I have also a comparison page for models hosted on OpenRouter (https://minthemiddle.github.io/openrouter-model-comparison/ https://minthemiddle.github.io/openrouter-model-comparison/), I do comparison via regex (so "claude-3-haiku(?!:beta)|flash" will show you haiku, but not haiku-beta vs flash. I wish that OpenRouter would also expose the amount of output tokens via API as this is also an important criteria.