4 ms·

I’m confused by the evidence they use: > to justify those costs, the technology must be able to solve complex problems, which it isn't designed to do Planning

I’m confused by the evidence they use: > to justify those costs, the technology must be able to solve complex problems, which it isn't designed to do Planning and reasoning are the two greatest areas of research in AI right now, with an OOM more researchers devoted to it than there were to the first generation of generative AI architectures > In our experience, even basic summarization tasks often yield illegible and nonsensical results Summarization with current generation models is excellent. I can get a summarization of a several-hour-long-call with better recall than I could have had myself, for less than $2 in inference costs. > even if costs decline, they would have to do so dramatically to make automating tasks with AI affordable We’ve seen a literal 10x decrease in cost from gpt-4-32k to gpt-4o in a single year of AI development (3:1 cost blend). And that ignores that sonnet-3.5 is 50x cheaper than gpt-4-32k while getting better scores on pretty much all benchmarks? > the human brain is 10,000x more effective per unit of power in performing cognitive tasks vs. generative AI Patently false, we’re not untethered brains floating around and require shelter, food, and a ton of other energy intensive requirements to live, and an AI system can perform a task that it is designed to do easily 10-20x faster than a human could. If anything this makes me more bullish about AI systems having a positive ROI; the criticisms they have are based on extraordinarily (if not nefariously) dumb assumptions.

dist-epoch 2y ago
> the human brain is 10,000x more effective per unit of power Even if true, is completely irrelevant. What matters is effectiveness per dolar, not per watt. Does Goldman Sachs have an energy budget per employee, do they hire based on energy consumption? Do they prefer hiring someone living in a small house versus someone living in a mansion with a gigantic energy bill when both request the same salary?
- chaos_emergent 2y ago
  Yeah, an incredibly dumb analysis. I did a back of the hand estimate based off of the following assumptions: gpt-4o: 10 H100s = ~7000W power requirement, 30 seconds to generate a summary for an hour-long call human: ~10.7 kWh/year energy requirement (avg per American), 20 minutes to generate the same summary The numbers come out to 60 Wh for 4o and 400 Wh for a human
  - claytongulick 2y ago
    Except that the human will be using that energy regardless, it's sunk cost. The big question is whether the additional 60Wh are worth it.
    - chaos_emergent 2y ago
      I agree! I think it’s “worth it”. A parallel: it costs an OOM more to drive places than, say, bike places, but it allows us to do the things we did before an OOM faster and expands the types of things we can do. You can make the argument that it’s not “worth it” in terms of cost to our environment, but you have to acknowledge the upside of it simultaneously. And the dynamics are different in that we’ve reached an efficiency asymptote with cars and are still in the middle (or perhaps at the start) of the S curve with AI systems.
  - FireBeyond 2y ago
    Speaking of suboptimal analyses. The human isn't a single core, 100% pegged on that task for those 20 minutes. Even actively working at something that energy consumption is likely to be less than 10% of that. There's a whole lot of energy being burned by the cardiovascular system, respiratory, digestive and renal, endocrine, muscular system (all that typing, moving and focusing of the eyes), and we haven't even got to the energy of the "OS" execution in the brain to control all that. I don't think your numbers are the insight you think they are, nor accurate - not even particularly close.
    - chaos_emergent 2y ago
      We're neither disembodied brains nor bodies in isolation of basic needs like shelter or desires for energy-intensive conveniences, so I challenge that we should try to assess energy consumption in the way you're proposing. The outcome and the inputs required to get to it are the two relevant variables IMO. Being consistent, though, I should've included the amortized cost of training and the energy cost of the rest of the data center to the 4o analysis. I don't think that those costs would be more than a factor constant greater. When we can create disembodied brains that don't have needs for shelter and desires for energy-intensive conveniences, I buy that we can do 10% of the energy accounting.
llm_trw 2y ago
>Summarization with current generation models is excellent. I can get a summarization of a several-hour-long-call with better recall than I could have had myself, for less than $2 in inference costs. The company that I'm in has moved to all calls being video calls and recorded so we can get minutes out of them automatically. I had the bright idea of using magic phrases to add an item to a action list. Now everyone says nonsensical words so a computer can remember what we want to do for us.
- chaos_emergent 2y ago
  It sure does beat needing to have someone to do it for you :-)
  - llm_trw 2y ago
    The most ridiculous thing was that I got the inspiration from star trek. Added a bunch of "computer take note:" to an existing transcript and it wrote 10 out of 10 notes in a 'notes' section at the end of the minutes.
    - chaos_emergent 2y ago
      You should make it something random like “pineapple”
poikroequ 2y ago
I would love to know what you're doing right and I'm doing wrong, because my experience with AI has been mostly crap. I've tried using AI to summarize articles, but then the AI will say something interesting, so I actually look at the article and quickly realize the AI summary was wrong. You might want to double check that what the AI is telling you is actually accurate and not just blindly trust its output.
- chaos_emergent 2y ago
  I’d have to give away some of my secret system prompt sauce to tell you, but I’m sure that the summarizations I get are solid, we’ve done extensive evaluation of truthiness. I have tens of thousands of hours of calls summarized monthly from paying customers who rely on them to be accurate. We’ve only ever gotten hallucinations when we accidentally omitted the transcript, and even that was fixed by allowing the LLM to alert us when it didn’t see a transcript.
  - poikroequ 2y ago
    Bold claims with nothing to back it up. I'll take your "secret sauce" with a grain of salt.
    - chaos_emergent 2y ago
      You can try it for yourself, my company is in my profile.
  - FireBeyond 2y ago
    I use an AI noise canceling/transcription service to help with my extensive meeting agenda. I've lived in the US nearly 20 years, coming from Australia. Whether it's my accent or something else, most transcripts for me need probably in the order of 20% meeting duration for me to tidy and edit to be free of, in some cases, quite asinine, transcription errors. Unless your customers are doing that, I have a hard time, using the principle of garbage in garbage out, of believing "zero hallucinations" (and even then), but even then, the prompts would be something along the lines of "transcribe this, but just ... do better".
pants2 2y ago
In fact, if you can predict LLMs getting another 10X cheaper and 10X smarter over the next few years, it makes sense to start building LLM integrations today; even if they don't work well and cost too much today, every few months your feature will get better and cheaper, with very little work by you. Sounds like a fantastic ROI to me.
marcosdumay 2y ago
> Planning and reasoning are the two greatest areas of research in AI right now Yes, but it's not something LLMs do. At all. Each day I get more certain that AGI will surprise everybody and come from a nearly moneyless garage. And it will break every AI strategy people make. > I can get a summarization of a several-hour-long-call with better recall than I could have had myself The problem is, it works most of the time, when it doesn't it's dangerous. That "most of the time" part isn't large enough for people to ignore the failures. But this is one of the things where LLMs can shine. Your example looks more like it's solving a really bad problem that shouldn't exist, but it's possible that this adds lots of value somewhere.
- fauigerzigerk 2y ago
  >The problem is, it works most of the time, when it doesn't it's dangerous. That "most of the time" part isn't large enough for people to ignore the failures. I doubt that. A lot of failures will be ignored and only fixed if and when necessary because they are not usually very dangerous. A lot of information we produce is never acted on. And a lot of information we do act on is incorrect or unfounded anyway. Also, in places where hallucinations are unacceptable, double checking some facts is still far cheaper than manually producing the whole thing from scratch. I bet we will see an avalanche of incorrect and funny information coming from customer support systems. And yet very few people will want to pay for better fact checking.

dist-epoch 2y ago

> the human brain is 10,000x more effective per unit of power Even if true, is completely irrelevant. What matters is effectiveness per dolar, not per watt. Does Goldman Sachs have an energy budget per employee, do they hire based on energy consumption? Do they prefer hiring someone living in a small house versus someone living in a mansion with a gigantic energy bill when both request the same salary?

chaos_emergent 2y ago
Yeah, an incredibly dumb analysis. I did a back of the hand estimate based off of the following assumptions: gpt-4o: 10 H100s = ~7000W power requirement, 30 seconds to generate a summary for an hour-long call human: ~10.7 kWh/year energy requirement (avg per American), 20 minutes to generate the same summary The numbers come out to 60 Wh for 4o and 400 Wh for a human
- claytongulick 2y ago
  Except that the human will be using that energy regardless, it's sunk cost. The big question is whether the additional 60Wh are worth it.
  - chaos_emergent 2y ago
    I agree! I think it’s “worth it”. A parallel: it costs an OOM more to drive places than, say, bike places, but it allows us to do the things we did before an OOM faster and expands the types of things we can do. You can make the argument that it’s not “worth it” in terms of cost to our environment, but you have to acknowledge the upside of it simultaneously. And the dynamics are different in that we’ve reached an efficiency asymptote with cars and are still in the middle (or perhaps at the start) of the S curve with AI systems.
- FireBeyond 2y ago
  Speaking of suboptimal analyses. The human isn't a single core, 100% pegged on that task for those 20 minutes. Even actively working at something that energy consumption is likely to be less than 10% of that. There's a whole lot of energy being burned by the cardiovascular system, respiratory, digestive and renal, endocrine, muscular system (all that typing, moving and focusing of the eyes), and we haven't even got to the energy of the "OS" execution in the brain to control all that. I don't think your numbers are the insight you think they are, nor accurate - not even particularly close.
  - chaos_emergent 2y ago
    We're neither disembodied brains nor bodies in isolation of basic needs like shelter or desires for energy-intensive conveniences, so I challenge that we should try to assess energy consumption in the way you're proposing. The outcome and the inputs required to get to it are the two relevant variables IMO. Being consistent, though, I should've included the amortized cost of training and the energy cost of the rest of the data center to the 4o analysis. I don't think that those costs would be more than a factor constant greater. When we can create disembodied brains that don't have needs for shelter and desires for energy-intensive conveniences, I buy that we can do 10% of the energy accounting.

llm_trw 2y ago

>Summarization with current generation models is excellent. I can get a summarization of a several-hour-long-call with better recall than I could have had myself, for less than $2 in inference costs. The company that I'm in has moved to all calls being video calls and recorded so we can get minutes out of them automatically. I had the bright idea of using magic phrases to add an item to a action list. Now everyone says nonsensical words so a computer can remember what we want to do for us.

chaos_emergent 2y ago
It sure does beat needing to have someone to do it for you :-)
- llm_trw 2y ago
  The most ridiculous thing was that I got the inspiration from star trek. Added a bunch of "computer take note:" to an existing transcript and it wrote 10 out of 10 notes in a 'notes' section at the end of the minutes.
  - chaos_emergent 2y ago
    You should make it something random like “pineapple”

poikroequ 2y ago

I would love to know what you're doing right and I'm doing wrong, because my experience with AI has been mostly crap. I've tried using AI to summarize articles, but then the AI will say something interesting, so I actually look at the article and quickly realize the AI summary was wrong. You might want to double check that what the AI is telling you is actually accurate and not just blindly trust its output.

chaos_emergent 2y ago
I’d have to give away some of my secret system prompt sauce to tell you, but I’m sure that the summarizations I get are solid, we’ve done extensive evaluation of truthiness. I have tens of thousands of hours of calls summarized monthly from paying customers who rely on them to be accurate. We’ve only ever gotten hallucinations when we accidentally omitted the transcript, and even that was fixed by allowing the LLM to alert us when it didn’t see a transcript.
- poikroequ 2y ago
  Bold claims with nothing to back it up. I'll take your "secret sauce" with a grain of salt.
  - chaos_emergent 2y ago
    You can try it for yourself, my company is in my profile.
- FireBeyond 2y ago
  I use an AI noise canceling/transcription service to help with my extensive meeting agenda. I've lived in the US nearly 20 years, coming from Australia. Whether it's my accent or something else, most transcripts for me need probably in the order of 20% meeting duration for me to tidy and edit to be free of, in some cases, quite asinine, transcription errors. Unless your customers are doing that, I have a hard time, using the principle of garbage in garbage out, of believing "zero hallucinations" (and even then), but even then, the prompts would be something along the lines of "transcribe this, but just ... do better".

pants2 2y ago

In fact, if you can predict LLMs getting another 10X cheaper and 10X smarter over the next few years, it makes sense to start building LLM integrations today; even if they don't work well and cost too much today, every few months your feature will get better and cheaper, with very little work by you. Sounds like a fantastic ROI to me.

marcosdumay 2y ago

> Planning and reasoning are the two greatest areas of research in AI right now Yes, but it's not something LLMs do. At all. Each day I get more certain that AGI will surprise everybody and come from a nearly moneyless garage. And it will break every AI strategy people make. > I can get a summarization of a several-hour-long-call with better recall than I could have had myself The problem is, it works most of the time, when it doesn't it's dangerous. That "most of the time" part isn't large enough for people to ignore the failures. But this is one of the things where LLMs can shine. Your example looks more like it's solving a really bad problem that shouldn't exist, but it's possible that this adds lots of value somewhere.

fauigerzigerk 2y ago
>The problem is, it works most of the time, when it doesn't it's dangerous. That "most of the time" part isn't large enough for people to ignore the failures. I doubt that. A lot of failures will be ignored and only fixed if and when necessary because they are not usually very dangerous. A lot of information we produce is never acted on. And a lot of information we do act on is incorrect or unfounded anyway. Also, in places where hallucinations are unacceptable, double checking some facts is still far cheaper than manually producing the whole thing from scratch. I bet we will see an avalanche of incorrect and funny information coming from customer support systems. And yet very few people will want to pay for better fact checking.