4 ms·

It’s an interesting situation because on one level, the technology is at least somewhat misrepresented (best case results rising to the top of everyone’s feeds)

It’s an interesting situation because on one level, the technology is at least somewhat misrepresented (best case results rising to the top of everyone’s feeds), but on the other side, we haven’t even begun to see what integrated Chat GPT will bring. If I could replace Siri with Chat GPT, I would do so immediately, as it’s objectively better. The author makes some points about the eye watering costs; however, if this is something that people truly want—and I think we do want it—the market will find a way to deliver it at scale in a cost effective way.

flerchin 4y ago
Do we have an idea of what the costs to run ChatGPT might be? Are we costing OpenAI $5 everytime we ask it to generate python in the style of Donald Trump?
- scaredginger 4y ago
  I've read estimates of $0.02 per query
  - deleted 4y ago
    [deleted]
- HPsquared 4y ago
  Probably training it on specific data sets and contexts is the expensive part.
  - scaredginger 4y ago
    Running a gigantic model across multiple GPUs isn't cheap either
    - Const-me 4y ago
      I’ve asked ChatGPT about hardware. Here’s the response: > As for your specific computer, with 64 GB of RAM and a high-performance GPU like the GeForce 1080 Ti, it should have sufficient resources to run a language model like me for many common tasks. Based on the models open sourced by OpenAI, they are using PyTorch and CUDA. This means their stack requires nVidia GPUs. I think the main reason for their high costs is a single sentence in the EULA of GeForce drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-geforce-eula-to-prohibit-data-center-use/ https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... It’s technically possible to port their GPGPU code from CUDA somewhere else. Here’s a vendor-agnostic DirectCompute re-implementation of their Whisper model: https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper On servers, DirectCompute is not great ‘coz Windows server licenses are expensive. Still, I did that port alone, and spent couple weeks doing that. OpenAI probably has resources to port their inference to vendor-agnostic Vulkan Compute, running on Linux servers equipped with reasonably-priced AMD or Intel GPUs. For instance, Intel A770 16GB only costs $350, but delivers similar performance to nVidia A30 which costs $16000. Intel consumes more electricity but not by much, 225W versus 165W. That’s like 40x difference in cost efficiency of that chat.
- qumpis 4y ago
  Guesstimated around $0.01 per message https://mobile.twitter.com/tomgoldsteincs/status/1600196995389366274 https://mobile.twitter.com/tomgoldsteincs/status/16001969953...
  - ugjka 4y ago
    So way cheaper than an SMS not so long ago
    - gnabgib 4y ago
      SMS are free (they use up left over space in the signalling protocol a phone needs to use to talk to towers/inform the network of where it can be found - a process you're already charged for). Doesn't seem like phone networks collecting money for no effort (which is why it trends towards free) is a fair comparison to running an expensive supercomputer with very real hardware, energy and real estate costs.
- ben_w 4y ago
  We can probably make a decent guess by looking at the price they charge for GPT-3, where asking the most expensive model for its literal "two cents" response gives you a thousand tokens.
- bertman 4y ago
  I couldn't resist and tried it: Prompt: Write a small python program in the style of Donald Trump. chatGPT: I'm sorry, but it wouldn't be appropriate to write a program in the style of a political figure, especially if the language and tone used may be considered offensive or disrespectful. Additionally, OpenAI's policy prohibits the creation of content that is harmful, abusive, or hateful. Is there anything else I can assist you with?
metaspecific 4y ago
Recently wrote about the economics of both training and inferencing: https://sunyan.substack.com/p/the-economics-of-large-language-models https://sunyan.substack.com/p/the-economics-of-large-languag...
- eternalban 4y ago
  Submit it as a top level post. It’s a good article.
olalonde 4y ago
> the market will find a way to deliver it at scale in a cost effective way I wonder if it would be possible to build the model in hardware as an ASIC. That would probably bring down cost a lot but I'm not sure it makes sense if they expect to release new improved models regularly. The hardware might be obsolete by the time it reaches production.
- uxp100 4y ago
  I think that if we see hardware specific to a particular model it would be in automotive. And the focus would be on fitting inside a power envelope, not cost in particular. So it still could be that cost of asic + operation will be greater than gpu/tpu + operation, but it will fit in whatever limit (1500 watts? Idk) is designed for driving systems more easily.
roflyear 4y ago
Maybe eventually, but there are plenty of examples of things taking quite a long time before they are economically viable.
dougmwne 4y ago
I agree, it is a bit of an odd moment. On the one hand, you have the normal amount of hype and bullshit. The people calling out that hype are right of course, as they have been right about 1000 other vaporware demos and outright marketing lies. The groundbreaking product does not yet exist and may never exist. But on the other hand, there is something real and incredible here. It has massively pushed forward the frontier of what a computer can do. If you compare the output of these models to sci-fi like Star Trek, the fantasy was too conservative. It calls into question what it means to be human, what makes us special, and what is art. It’s the most interesting moment in art since postmodernism. So many people are rightly looking at the trend and trajectory. Incredible things have already been created. Humans have been beat at their own games. Humans performance has already been matched on a wide range of tasks. If you factor in 1 more year of progress, 5 more, 50 more, you might reach some terrifying conclusions. It’s only human.
- JohnFen 4y ago
  > It calls into question what it means to be human, what makes us special, and what is art. How?
jacobsenscott 4y ago
> the market will find a way to deliver it at scale in a cost effective way Who is this market, who is so wise in the ways of science?
- pixl97 4y ago
  Says the person typing the message on a personal supercomputer that has enough battery power to last all day and fit in their pocket at the same time.... (assuming that is, that this message came from a portable device)
  - wahnfrieden 4y ago
    Portable devices haven’t achieved cost effectiveness without dangerous conditions, slavery and child labor on the other end as externalized costs
    - pixl97 4y ago
      All costs are externalized by the premature heat death of the universe. And I mean really, if this is the bar you want to set then no product is going to meet those goals, so no huge point in pointing out electronics, versus say just about anything you eat on a given day, or the clothes you wear.
      - wahnfrieden 4y ago
        uh - no product is going to meet the goals of avoiding slave and child labor and hazardous work conditions? not sure why you're going to bat for that besides to defend that it's hugely profitable. i think many products outside portable devices do meet that goal but it's not in service of market competition and isn't a scalable solution without a bigger systemic change, so i do agree in a sense with saying that at scale no product meets that goal sure.
    - falcor84 4y ago
      Honestly, if the price of a smartphone was even $10,000, I would be happy to pay it. The value I get out of it is incredible. In my case (living in a mid-sized city), if I had to choose between my phone and my car, I'd definitely sell the car first.
davedx 4y ago
> the market will find a way to deliver it at scale in a cost effective way This really isn't a given. Look at the problems encountered trying to scale Bitcoin - slow transaction speed and high energy use. They're still unsolved, because some properties of a given technology are inherent to that technology. There's no guarantee something is scalable, especially something that might (as the article suggests) have exponentially increasing costs in its default configuration.
- naasking 4y ago
  > Look at the problems encountered trying to scale Bitcoin - slow transaction speed and high energy use. They're not unsolved, they're actually trivially solved by using an exchange. That just comes with its own risks.
  - davedx 4y ago
    How is the high energy cost of mining Bitcoin solved by exchanges? Also if I want to use crypto for anything except speculation within an exchange, the transaction speed is still an issue, even with apps like Coinbase. (Or at least it was last time I used it). I know Lightning Network helps but that's like a layer on top of Bitcoin because the Bitcoin tech itself doesn't scale well
idopmstuff 4y ago
You might enjoy SiriGPT: https://gpt3demo.com/apps/chatgpt-sirigpt-apple-ios https://gpt3demo.com/apps/chatgpt-sirigpt-apple-ios (It's voice-activated ChatGPT on your iOS device using shortcuts.)

flerchin 4y ago

Do we have an idea of what the costs to run ChatGPT might be? Are we costing OpenAI $5 everytime we ask it to generate python in the style of Donald Trump?

scaredginger 4y ago
I've read estimates of $0.02 per query
- deleted 4y ago
  [deleted]
HPsquared 4y ago
Probably training it on specific data sets and contexts is the expensive part.
- scaredginger 4y ago
  Running a gigantic model across multiple GPUs isn't cheap either
  - Const-me 4y ago
    I’ve asked ChatGPT about hardware. Here’s the response: > As for your specific computer, with 64 GB of RAM and a high-performance GPU like the GeForce 1080 Ti, it should have sufficient resources to run a language model like me for many common tasks. Based on the models open sourced by OpenAI, they are using PyTorch and CUDA. This means their stack requires nVidia GPUs. I think the main reason for their high costs is a single sentence in the EULA of GeForce drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-geforce-eula-to-prohibit-data-center-use/ https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... It’s technically possible to port their GPGPU code from CUDA somewhere else. Here’s a vendor-agnostic DirectCompute re-implementation of their Whisper model: https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper On servers, DirectCompute is not great ‘coz Windows server licenses are expensive. Still, I did that port alone, and spent couple weeks doing that. OpenAI probably has resources to port their inference to vendor-agnostic Vulkan Compute, running on Linux servers equipped with reasonably-priced AMD or Intel GPUs. For instance, Intel A770 16GB only costs $350, but delivers similar performance to nVidia A30 which costs $16000. Intel consumes more electricity but not by much, 225W versus 165W. That’s like 40x difference in cost efficiency of that chat.
qumpis 4y ago
Guesstimated around $0.01 per message https://mobile.twitter.com/tomgoldsteincs/status/1600196995389366274 https://mobile.twitter.com/tomgoldsteincs/status/16001969953...
- ugjka 4y ago
  So way cheaper than an SMS not so long ago
  - gnabgib 4y ago
    SMS are free (they use up left over space in the signalling protocol a phone needs to use to talk to towers/inform the network of where it can be found - a process you're already charged for). Doesn't seem like phone networks collecting money for no effort (which is why it trends towards free) is a fair comparison to running an expensive supercomputer with very real hardware, energy and real estate costs.
ben_w 4y ago
We can probably make a decent guess by looking at the price they charge for GPT-3, where asking the most expensive model for its literal "two cents" response gives you a thousand tokens.
bertman 4y ago
I couldn't resist and tried it: Prompt: Write a small python program in the style of Donald Trump. chatGPT: I'm sorry, but it wouldn't be appropriate to write a program in the style of a political figure, especially if the language and tone used may be considered offensive or disrespectful. Additionally, OpenAI's policy prohibits the creation of content that is harmful, abusive, or hateful. Is there anything else I can assist you with?

metaspecific 4y ago

Recently wrote about the economics of both training and inferencing: https://sunyan.substack.com/p/the-economics-of-large-language-models https://sunyan.substack.com/p/the-economics-of-large-languag...

eternalban 4y ago
Submit it as a top level post. It’s a good article.

olalonde 4y ago

> the market will find a way to deliver it at scale in a cost effective way I wonder if it would be possible to build the model in hardware as an ASIC. That would probably bring down cost a lot but I'm not sure it makes sense if they expect to release new improved models regularly. The hardware might be obsolete by the time it reaches production.

uxp100 4y ago
I think that if we see hardware specific to a particular model it would be in automotive. And the focus would be on fitting inside a power envelope, not cost in particular. So it still could be that cost of asic + operation will be greater than gpu/tpu + operation, but it will fit in whatever limit (1500 watts? Idk) is designed for driving systems more easily.

roflyear 4y ago

Maybe eventually, but there are plenty of examples of things taking quite a long time before they are economically viable.

dougmwne 4y ago

I agree, it is a bit of an odd moment. On the one hand, you have the normal amount of hype and bullshit. The people calling out that hype are right of course, as they have been right about 1000 other vaporware demos and outright marketing lies. The groundbreaking product does not yet exist and may never exist. But on the other hand, there is something real and incredible here. It has massively pushed forward the frontier of what a computer can do. If you compare the output of these models to sci-fi like Star Trek, the fantasy was too conservative. It calls into question what it means to be human, what makes us special, and what is art. It’s the most interesting moment in art since postmodernism. So many people are rightly looking at the trend and trajectory. Incredible things have already been created. Humans have been beat at their own games. Humans performance has already been matched on a wide range of tasks. If you factor in 1 more year of progress, 5 more, 50 more, you might reach some terrifying conclusions. It’s only human.

JohnFen 4y ago
> It calls into question what it means to be human, what makes us special, and what is art. How?

jacobsenscott 4y ago

> the market will find a way to deliver it at scale in a cost effective way Who is this market, who is so wise in the ways of science?

pixl97 4y ago
Says the person typing the message on a personal supercomputer that has enough battery power to last all day and fit in their pocket at the same time.... (assuming that is, that this message came from a portable device)
- wahnfrieden 4y ago
  Portable devices haven’t achieved cost effectiveness without dangerous conditions, slavery and child labor on the other end as externalized costs
  - pixl97 4y ago
    All costs are externalized by the premature heat death of the universe. And I mean really, if this is the bar you want to set then no product is going to meet those goals, so no huge point in pointing out electronics, versus say just about anything you eat on a given day, or the clothes you wear.
    - wahnfrieden 4y ago
      uh - no product is going to meet the goals of avoiding slave and child labor and hazardous work conditions? not sure why you're going to bat for that besides to defend that it's hugely profitable. i think many products outside portable devices do meet that goal but it's not in service of market competition and isn't a scalable solution without a bigger systemic change, so i do agree in a sense with saying that at scale no product meets that goal sure.
  - falcor84 4y ago
    Honestly, if the price of a smartphone was even $10,000, I would be happy to pay it. The value I get out of it is incredible. In my case (living in a mid-sized city), if I had to choose between my phone and my car, I'd definitely sell the car first.

davedx 4y ago

> the market will find a way to deliver it at scale in a cost effective way This really isn't a given. Look at the problems encountered trying to scale Bitcoin - slow transaction speed and high energy use. They're still unsolved, because some properties of a given technology are inherent to that technology. There's no guarantee something is scalable, especially something that might (as the article suggests) have exponentially increasing costs in its default configuration.

naasking 4y ago
> Look at the problems encountered trying to scale Bitcoin - slow transaction speed and high energy use. They're not unsolved, they're actually trivially solved by using an exchange. That just comes with its own risks.
- davedx 4y ago
  How is the high energy cost of mining Bitcoin solved by exchanges? Also if I want to use crypto for anything except speculation within an exchange, the transaction speed is still an issue, even with apps like Coinbase. (Or at least it was last time I used it). I know Lightning Network helps but that's like a layer on top of Bitcoin because the Bitcoin tech itself doesn't scale well

idopmstuff 4y ago

You might enjoy SiriGPT: https://gpt3demo.com/apps/chatgpt-sirigpt-apple-ios https://gpt3demo.com/apps/chatgpt-sirigpt-apple-ios (It's voice-activated ChatGPT on your iOS device using shortcuts.)