5 ms·

Llama-3 is one of the models provided by AWS Bedrock which offers pay as you go pricing. I'm curious how it would break down on that. LLAMA 8B on Bedrock is $0

by throwaway2016a 2y ago

Llama-3 is one of the models provided by AWS Bedrock which offers pay as you go pricing. I'm curious how it would break down on that. LLAMA 8B on Bedrock is $0.40 per 1M input tokens and $0.60 per 1M output tokens which is a lot cheaper than OpenAI models. Edit: to add to that, as technical people we tend to discount the value of our own time. Bedrock and the OpenAI are both very easy to integrate with and get started. How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? That's got to be part of the break-even equation.

croddin 2y ago
Groq also has pay as you go pricing for llama3 8B for only $0.05/$0.08 that is very fast.
- sergiotapia 2y ago
  Groq is actually allowing you to pay now and get real service?
  - coder543 2y ago
    The option to pay is still listed as coming soon, but I also see pricing information in the settings page, so maybe it actually is coming somewhat sooner. I’m seeing $0.05/1M input and $0.10/1M output for llama3 8B, which is not exactly identical to what the previous person quoted. Either way, I wish Groq would offer a real service to people willing to pay.
    - croddin 2y ago
      I found the .05/.08 here: https://wow.groq.com/ https://wow.groq.com/
  - refulgentis 2y ago
    tl;dr: no-ish, it's getting better but still not there. I don't really get it, only thing I can surmise is it'd be such a no-brainer in various cases, that if they tried supporting it as a service, they'd have to cut users. I've seen multiple big media company employees begging for some sort of response on their discord.
- localfirst 2y ago
  didn't know they finally turned on pricing plans
VagabundoP 2y ago
Just to bounce off this a little. If you are looking to fine-tune using an on demand service it seems Amazon Sagemaker can do it at seemingly decent prices: https://aws.amazon.com/sagemaker/pricing/ https://aws.amazon.com/sagemaker/pricing/ I'd love to hear someones experience using this as I want to make an RPG rules bot tied to a specific ruleset as a project but I fear AWS as it might bankrupt me!
- zsyllepsis 2y ago
  In my experience SageMaker was relatively straightforward for fine-tuning models that could fit on a single instance, but distributed training still requires a good bit of detailed understanding of how things work under the covers. SageMaker Jumpstart includes some pretty easy out-of-the-box configurations for fine-tuning models that are a good starting point. They will incorporate some basic quantization and other cost-savings techniques to help reduce the total compute time. To help control costs, you can choose pretty conservative settings in terms of how long you want to let the model train for. Once that iteration is done and you have a model artifact saved, you can always pick back up and perform more rounds of training using the previous checkpoint as a starting point.
veryrealsid 2y ago
> How long did this server take to build? About 3 days [from 0 and iterating multiple times to the final solution] > How much time does it take to maintain and make sure all the security patches are applied each month? A lot > How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? All really good points, the exercise to self host is really just to see what is possible but completely agree that self hosting makes little to no sense unless you have a business case that can justify it. Not to mention if you sign customers with SLAs and then end up having downtime would put even more pressure on your self hosted hardware
johnklos 2y ago
> How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? That's got to be part of the break-even equation. All of these are the kinds of things that people say to non-technical people to try to sell cloud. It's all fluff. Do you really think that cloud computing doesn't have security issues, or crashes, or data loss, or that it doesn't involve lots of administration? Thinking that we don't know any better is both disingenuous and a bit insulting.
- websap 2y ago
  I've managed fleets on cloud providers with over 100k instances, even with all the excellent features through APIs, managing instances can quickly get tricky. Tbh, your comment is kind of insulting and belittles how far we've come ahead in infrastructure management. The cloud is probably more secure than a set of janky servers that you have running in your basement. You can totally automate away 0-days, cves and get access to better security primitives.
  - johnklos 2y ago
    If my comment is insulting, I apologize. That was not my intention. My intention was to say that writing sales speak in a technical discussion is insulting to those of us who know better. However, you've now gone out of your way to try to be insulting. You know nothing about me, yet you want to suggest that the cloud is more secure than my servers, and that my servers are "janky"? Please try a little harder to engage in reasonable discourse.
  - yjftsjthsd-h 2y ago
    > The cloud is probably more secure than a set of janky servers that you have running in your basement. Apples/Oranges. Your janky cloud[0] is less secure than the servers in my basement, because I'm a mostly competent sysadmin. Cloud lets you trade some operational concerns for higher costs, but not all of them. [0] If you can assume servers run by somebody who doesn't know how to do it properly, obviously I can assume the same about cloud configuration. Have fun with your leaked API keys.

croddin 2y ago

Groq also has pay as you go pricing for llama3 8B for only $0.05/$0.08 that is very fast.

sergiotapia 2y ago
Groq is actually allowing you to pay now and get real service?
- coder543 2y ago
  The option to pay is still listed as coming soon, but I also see pricing information in the settings page, so maybe it actually is coming somewhat sooner. I’m seeing $0.05/1M input and $0.10/1M output for llama3 8B, which is not exactly identical to what the previous person quoted. Either way, I wish Groq would offer a real service to people willing to pay.
  - croddin 2y ago
    I found the .05/.08 here: https://wow.groq.com/ https://wow.groq.com/
- refulgentis 2y ago
  tl;dr: no-ish, it's getting better but still not there. I don't really get it, only thing I can surmise is it'd be such a no-brainer in various cases, that if they tried supporting it as a service, they'd have to cut users. I've seen multiple big media company employees begging for some sort of response on their discord.
localfirst 2y ago
didn't know they finally turned on pricing plans

VagabundoP 2y ago

Just to bounce off this a little. If you are looking to fine-tune using an on demand service it seems Amazon Sagemaker can do it at seemingly decent prices: https://aws.amazon.com/sagemaker/pricing/ https://aws.amazon.com/sagemaker/pricing/ I'd love to hear someones experience using this as I want to make an RPG rules bot tied to a specific ruleset as a project but I fear AWS as it might bankrupt me!

zsyllepsis 2y ago
In my experience SageMaker was relatively straightforward for fine-tuning models that could fit on a single instance, but distributed training still requires a good bit of detailed understanding of how things work under the covers. SageMaker Jumpstart includes some pretty easy out-of-the-box configurations for fine-tuning models that are a good starting point. They will incorporate some basic quantization and other cost-savings techniques to help reduce the total compute time. To help control costs, you can choose pretty conservative settings in terms of how long you want to let the model train for. Once that iteration is done and you have a model artifact saved, you can always pick back up and perform more rounds of training using the previous checkpoint as a starting point.

veryrealsid 2y ago

> How long did this server take to build? About 3 days [from 0 and iterating multiple times to the final solution] > How much time does it take to maintain and make sure all the security patches are applied each month? A lot > How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? All really good points, the exercise to self host is really just to see what is possible but completely agree that self hosting makes little to no sense unless you have a business case that can justify it. Not to mention if you sign customers with SLAs and then end up having downtime would put even more pressure on your self hosted hardware

johnklos 2y ago

> How long did this server take to build? How much time does it take to maintain and make sure all the security patches are applied each month? How often does it crash and how much time will be needed to recover it? Do you keep spare parts on hand / how much is the cost of downtime if you have to wait to get a replacement part in the mail? That's got to be part of the break-even equation. All of these are the kinds of things that people say to non-technical people to try to sell cloud. It's all fluff. Do you really think that cloud computing doesn't have security issues, or crashes, or data loss, or that it doesn't involve lots of administration? Thinking that we don't know any better is both disingenuous and a bit insulting.

websap 2y ago
I've managed fleets on cloud providers with over 100k instances, even with all the excellent features through APIs, managing instances can quickly get tricky. Tbh, your comment is kind of insulting and belittles how far we've come ahead in infrastructure management. The cloud is probably more secure than a set of janky servers that you have running in your basement. You can totally automate away 0-days, cves and get access to better security primitives.
- johnklos 2y ago
  If my comment is insulting, I apologize. That was not my intention. My intention was to say that writing sales speak in a technical discussion is insulting to those of us who know better. However, you've now gone out of your way to try to be insulting. You know nothing about me, yet you want to suggest that the cloud is more secure than my servers, and that my servers are "janky"? Please try a little harder to engage in reasonable discourse.
- yjftsjthsd-h 2y ago
  > The cloud is probably more secure than a set of janky servers that you have running in your basement. Apples/Oranges. Your janky cloud[0] is less secure than the servers in my basement, because I'm a mostly competent sysadmin. Cloud lets you trade some operational concerns for higher costs, but not all of them. [0] If you can assume servers run by somebody who doesn't know how to do it properly, obviously I can assume the same about cloud configuration. Have fun with your leaked API keys.