3 ms·

I am... unsure why anyone would think LLMs would be able to do this. They are not magic oracles. Like I think even most humans would be extremely bad at this.

I am... unsure why anyone would think LLMs would be able to do this. They are not magic oracles. Like I think even most humans would be extremely bad at this. Like, are people actually using LLMs for this? Please do not, it won't work.

Nicook 6mo ago
You are severely overestimating the average, or even above average understanding of LLMs.
- bluefirebrand 6mo ago
  Not to mention the fact that LLM marketing is trying to convince us that they can do anything
deleted 6mo ago
[deleted]
tarkin2 6mo ago
OpenAI etc, are, however advertising them like they are magical oracles, on the verge of lifting humanity to next phrase of civilisation. The idea the majority of users know what nondeterministic even means it's a massive, massive ask
PUSH_AX 6mo ago
It’s worse, I bet there are apps in the App Store that do this, the users just have no idea on the accuracy
- vector_spaces 6mo ago
  There is a very popular app for macro counting called Cal AI that was reported to have been written by a high school student with over $1M in revenue. Looks like it was just acquired by MyFitnessPal
  - lordgrenville 6mo ago
    Wow, yeah. "The result is an app that the creators say is 90% accurate". https://techcrunch.com/2025/03/16/photo-calorie-app-cal-ai-downloaded-over-a-million-times-was-built-by-two-teenagers/ https://techcrunch.com/2025/03/16/photo-calorie-app-cal-ai-d...
Jtarii 6mo ago
>I am... unsure why anyone would think LLMs would be able to do this. Well firstly the average IQ is 100. And also because people market products to consumers that claim to be able to count carbs from images. If you don't know the limitations of LLMs then there would be little reason to doubt it for an uniformed or below average intelligence person, of which there are hundreds of millions.
kioleanu 6mo ago
Yes, people are using LLMs for this because that is how they've been marketed, like being able to solve every day tasks like a personal assistant on one hand, but also like researchers being able to solve old problems that humans couldn't crack. Does the model say it can't do that when asked? No, it answers confidentely. Also it's easy to trust it if you don't know how it works
- drtz 6mo ago
  Would people really trust their personal assistant to tell them how many calories are in a sandwich just by glancing at it on a plate? I'm doubtful, and I would also expect a diabetic to be even more skeptical.
  - TychoCelchuuu 6mo ago
    [dead]
  - tempaccount5050 6mo ago
    Yes.
sjsdaiuasgdia 6mo ago
Some people are asking LLMs what's on the menu of restaurants they are actively sitting in, possibly with a menu on the table in front of them. Some people have a very poor understanding of what LLMs are good for. Some people do see them as magic oracles.
throwaway260124 6mo ago
But nothing prevents llms from being RLed to do this right? But does training llms to be better at this, improves their world model or does it only make changes at the surface?
- vidarh 6mo ago
  Yes, something prevents llms from being RLed to do this: You can't see through something opaque to determine whether there's something high calorie or low calorie out of sight. The problem itself is unsolvable given the data provided. You could conceivable make it better at making guesses, but they will inherently always be guesses that will sometimes be wildly off.
  - pjc50 6mo ago
    > You can't see through something opaque to determine whether there's something high calorie or low calorie out of sight https://www-users.york.ac.uk/~ss44/joke/3.htm https://www-users.york.ac.uk/~ss44/joke/3.htm "There is at least one field, containing at least one sheep, of which at least one side is black."
- ben_w 6mo ago
  Estimate the calorie count of this door handle: https://m.youtube.com/watch?v=VDSzY52Mkrw&pp=0gcJCVACo7VqN5tD&ra=m https://m.youtube.com/watch?v=VDSzY52Mkrw&pp=0gcJCVACo7VqN5t... Extreme example perhaps, but no, you can't just turn pixels into calories. Right now I'd be impressed if we could reliably estimate volume to within 30% from a photo, but even with that correct the contents of the food can easily be way off without visible sign.
- rsynnott 6mo ago
  Okay, so take the sandwich. There is no way to know what is in it by looking at it. No amount of optimisation will fix this. I'm sure one could produce a CV model that was a lot better at guessing here than these LLMs are, but fundamentally it is still guessing.
AndrewKemendo 6mo ago
The vast majority of people using LLMs in my experience use them as though they are Oracles They are surprised and upset when the Oracle is not perfect Go ahead and search around on hacker news you’ll see precisely the same pattern with people who are ostensibly engineers and hackers It’s actually pretty mind boggling but then again humans never fail to surprise and disappoint
hansmayer 6mo ago
I mean people will shamelessly paste you a wall of text from LLM while chatting with you to prove a point, probably thinking how they outsmarted you now...
faangguyindia 6mo ago
It’s because AI can debug a programme and people start thinking it can do fitness and health stuff too, but the thing is, there is no “instant-reacting compiler" for health or fitness. Things change over a long time, till then AI would have run out of context or lost the data from its cache, or the user may have got bored and deleted their account.
acchow 6mo ago
Most people are convinced LLMs can do this. Cal AI, which claims to generate a nutritional breakdown based off a photo, has $30 million in annual recurring revenue.
- rsynnott 6mo ago
  ... Bloody hell. I mean that's basically fraud, surely. It is _not possible to do this even vaguely accurately_.
heysoup 6mo ago
They sold the idea that LLMs "have" information. That the LLM "is" intelligent. Truth is the LLM is good at making intelligent decisions. But in order to make intelligent decision, you need context. If you give proper context -> ask the LLM -> get almost perfect result every time. Anything else is rolling dice, a very special type of dice, but dice anyhow. Not magic.
jeroenhd 6mo ago
https://xkcd.com/1425/ https://xkcd.com/1425/ strikes again. As far as consumers know, LLMs can identify the towns pictures were taken (without metadata), can summarize entire movies, generate clips of your kid flying a rocket to the moon, can translate images from any language imaginable, but somehow they cannot estimate the calories in a cheese sandwich. The supposed professional posting about an LLM deleting their prod database for their non-existent company asked the AI to explain itself. That's the level of LLM knowledge you should expect from most people that actually work with these tools.
kdheiwns 6mo ago
They're marketed as AI. AI has a long standing image built up by movies and other media of being some omniscient computer capable of analyzing the world. These AI companies are very aware of this and leverage it. And a person with sufficient knowledge could easily give a rough estimate of the calories. A slice of store bought sandwich bread of a given thickness generally has calories within a certain range. So do cheese slices. It's elementary school health class material. We all learn how to calculate calories in a meal. Packaging on food also always has calories, so clearly people know how to estimate it fairly accurately. If a fifth grader can calculate it but an AI can't, that says a lot about how bad these AIs are. We'll get another series of paid and bought articles saying "AI analyzed IMPOSSIBLE math problem beyond human comprehension and solved it with FACTS and LOGIC", while at the same time being told "bro no you can't expect an ai to calculate calories in a sandwich bro that's impossible bro if you even try that then you're insane for even thinking ai should be used that way bro". These companies need to decide: is AI smart enough to solve hard questions, or is it too useless to calculate something any kid could do by googling calories in a slice of bread and doing some basic arithmetic?
- rsynnott 6mo ago
  > Packaging on food also always has calories, so clearly people know how to estimate it fairly accurately. That's not done by looking at it and guessing (or at least it _shouldn't_ be; manufacturers have been known to do this but it's bad practice and may cause them regulatory problems). In an ideal world it's done with one of these: https://en.wikipedia.org/wiki/Calorimeter https://en.wikipedia.org/wiki/Calorimeter ; less ideally it can be estimated based on the ingredients.
jihadjihad 6mo ago
> They are not magic oracles. I came across a LinkedIn post a couple days ago where someone had asked ChatGPT, "What are the top things you get asked about $NICHE_INDUSTRY_THING_I_AM_SELLING?" As if there is introspection like that at the meta level, where ChatGPT could actually provide hard numbers around its own usage and request patterns. The fact that these products work with natural language beguiles people into thinking they are, indeed, magic oracles.
- Ekaros 6mo ago
  This is the weird intersection where I think that data might exist and LLM might be able to query it. But any company would never give it out. So the bot would not have access to it.
pjc50 6mo ago
> They are not magic oracles. Anthropic's trillion dollar valuation hinges on the idea that it is just that, a magic oracle that can replace any worker for any type of task. Any programmer, any author, any musician, any kind of clerical work. All we've asked here is "sudo evaluate me a sandwich", the sort of estimation task that humans with internet resources might reasonably be expected to do, and it's given up? (It would be fun to compare this to sending the picture out on Mechanical Turk and asking humans to eyeball the calorie count of said sandwich...)
ambicapter 6mo ago
If the LLM can correctly identify a food item some high percentage of the time, why would it be magic for it to guess the amount of calories in an object? It's perhaps a lookup and some simple math as an extra step.

Nicook 6mo ago

You are severely overestimating the average, or even above average understanding of LLMs.

bluefirebrand 6mo ago
Not to mention the fact that LLM marketing is trying to convince us that they can do anything

deleted 6mo ago

[deleted]

tarkin2 6mo ago

OpenAI etc, are, however advertising them like they are magical oracles, on the verge of lifting humanity to next phrase of civilisation. The idea the majority of users know what nondeterministic even means it's a massive, massive ask

PUSH_AX 6mo ago

It’s worse, I bet there are apps in the App Store that do this, the users just have no idea on the accuracy

vector_spaces 6mo ago
There is a very popular app for macro counting called Cal AI that was reported to have been written by a high school student with over $1M in revenue. Looks like it was just acquired by MyFitnessPal
- lordgrenville 6mo ago
  Wow, yeah. "The result is an app that the creators say is 90% accurate". https://techcrunch.com/2025/03/16/photo-calorie-app-cal-ai-downloaded-over-a-million-times-was-built-by-two-teenagers/ https://techcrunch.com/2025/03/16/photo-calorie-app-cal-ai-d...

Jtarii 6mo ago

>I am... unsure why anyone would think LLMs would be able to do this. Well firstly the average IQ is 100. And also because people market products to consumers that claim to be able to count carbs from images. If you don't know the limitations of LLMs then there would be little reason to doubt it for an uniformed or below average intelligence person, of which there are hundreds of millions.

kioleanu 6mo ago

Yes, people are using LLMs for this because that is how they've been marketed, like being able to solve every day tasks like a personal assistant on one hand, but also like researchers being able to solve old problems that humans couldn't crack. Does the model say it can't do that when asked? No, it answers confidentely. Also it's easy to trust it if you don't know how it works

drtz 6mo ago
Would people really trust their personal assistant to tell them how many calories are in a sandwich just by glancing at it on a plate? I'm doubtful, and I would also expect a diabetic to be even more skeptical.
- TychoCelchuuu 6mo ago
  [dead]
- tempaccount5050 6mo ago
  Yes.

sjsdaiuasgdia 6mo ago

Some people are asking LLMs what's on the menu of restaurants they are actively sitting in, possibly with a menu on the table in front of them. Some people have a very poor understanding of what LLMs are good for. Some people do see them as magic oracles.

throwaway260124 6mo ago

But nothing prevents llms from being RLed to do this right? But does training llms to be better at this, improves their world model or does it only make changes at the surface?

vidarh 6mo ago
Yes, something prevents llms from being RLed to do this: You can't see through something opaque to determine whether there's something high calorie or low calorie out of sight. The problem itself is unsolvable given the data provided. You could conceivable make it better at making guesses, but they will inherently always be guesses that will sometimes be wildly off.
- pjc50 6mo ago
  > You can't see through something opaque to determine whether there's something high calorie or low calorie out of sight https://www-users.york.ac.uk/~ss44/joke/3.htm https://www-users.york.ac.uk/~ss44/joke/3.htm "There is at least one field, containing at least one sheep, of which at least one side is black."
ben_w 6mo ago
Estimate the calorie count of this door handle: https://m.youtube.com/watch?v=VDSzY52Mkrw&pp=0gcJCVACo7VqN5tD&ra=m https://m.youtube.com/watch?v=VDSzY52Mkrw&pp=0gcJCVACo7VqN5t... Extreme example perhaps, but no, you can't just turn pixels into calories. Right now I'd be impressed if we could reliably estimate volume to within 30% from a photo, but even with that correct the contents of the food can easily be way off without visible sign.
rsynnott 6mo ago
Okay, so take the sandwich. There is no way to know what is in it by looking at it. No amount of optimisation will fix this. I'm sure one could produce a CV model that was a lot better at guessing here than these LLMs are, but fundamentally it is still guessing.

AndrewKemendo 6mo ago

The vast majority of people using LLMs in my experience use them as though they are Oracles They are surprised and upset when the Oracle is not perfect Go ahead and search around on hacker news you’ll see precisely the same pattern with people who are ostensibly engineers and hackers It’s actually pretty mind boggling but then again humans never fail to surprise and disappoint

hansmayer 6mo ago

I mean people will shamelessly paste you a wall of text from LLM while chatting with you to prove a point, probably thinking how they outsmarted you now...

faangguyindia 6mo ago

It’s because AI can debug a programme and people start thinking it can do fitness and health stuff too, but the thing is, there is no “instant-reacting compiler" for health or fitness. Things change over a long time, till then AI would have run out of context or lost the data from its cache, or the user may have got bored and deleted their account.

acchow 6mo ago

Most people are convinced LLMs can do this. Cal AI, which claims to generate a nutritional breakdown based off a photo, has $30 million in annual recurring revenue.

rsynnott 6mo ago
... Bloody hell. I mean that's basically fraud, surely. It is _not possible to do this even vaguely accurately_.

heysoup 6mo ago

They sold the idea that LLMs "have" information. That the LLM "is" intelligent. Truth is the LLM is good at making intelligent decisions. But in order to make intelligent decision, you need context. If you give proper context -> ask the LLM -> get almost perfect result every time. Anything else is rolling dice, a very special type of dice, but dice anyhow. Not magic.

jeroenhd 6mo ago

https://xkcd.com/1425/ https://xkcd.com/1425/ strikes again. As far as consumers know, LLMs can identify the towns pictures were taken (without metadata), can summarize entire movies, generate clips of your kid flying a rocket to the moon, can translate images from any language imaginable, but somehow they cannot estimate the calories in a cheese sandwich. The supposed professional posting about an LLM deleting their prod database for their non-existent company asked the AI to explain itself. That's the level of LLM knowledge you should expect from most people that actually work with these tools.

kdheiwns 6mo ago

They're marketed as AI. AI has a long standing image built up by movies and other media of being some omniscient computer capable of analyzing the world. These AI companies are very aware of this and leverage it. And a person with sufficient knowledge could easily give a rough estimate of the calories. A slice of store bought sandwich bread of a given thickness generally has calories within a certain range. So do cheese slices. It's elementary school health class material. We all learn how to calculate calories in a meal. Packaging on food also always has calories, so clearly people know how to estimate it fairly accurately. If a fifth grader can calculate it but an AI can't, that says a lot about how bad these AIs are. We'll get another series of paid and bought articles saying "AI analyzed IMPOSSIBLE math problem beyond human comprehension and solved it with FACTS and LOGIC", while at the same time being told "bro no you can't expect an ai to calculate calories in a sandwich bro that's impossible bro if you even try that then you're insane for even thinking ai should be used that way bro". These companies need to decide: is AI smart enough to solve hard questions, or is it too useless to calculate something any kid could do by googling calories in a slice of bread and doing some basic arithmetic?

rsynnott 6mo ago
> Packaging on food also always has calories, so clearly people know how to estimate it fairly accurately. That's not done by looking at it and guessing (or at least it _shouldn't_ be; manufacturers have been known to do this but it's bad practice and may cause them regulatory problems). In an ideal world it's done with one of these: https://en.wikipedia.org/wiki/Calorimeter https://en.wikipedia.org/wiki/Calorimeter ; less ideally it can be estimated based on the ingredients.

jihadjihad 6mo ago

> They are not magic oracles. I came across a LinkedIn post a couple days ago where someone had asked ChatGPT, "What are the top things you get asked about $NICHE_INDUSTRY_THING_I_AM_SELLING?" As if there is introspection like that at the meta level, where ChatGPT could actually provide hard numbers around its own usage and request patterns. The fact that these products work with natural language beguiles people into thinking they are, indeed, magic oracles.

Ekaros 6mo ago
This is the weird intersection where I think that data might exist and LLM might be able to query it. But any company would never give it out. So the bot would not have access to it.

pjc50 6mo ago

> They are not magic oracles. Anthropic's trillion dollar valuation hinges on the idea that it is just that, a magic oracle that can replace any worker for any type of task. Any programmer, any author, any musician, any kind of clerical work. All we've asked here is "sudo evaluate me a sandwich", the sort of estimation task that humans with internet resources might reasonably be expected to do, and it's given up? (It would be fun to compare this to sending the picture out on Mechanical Turk and asking humans to eyeball the calorie count of said sandwich...)

ambicapter 6mo ago

If the LLM can correctly identify a food item some high percentage of the time, why would it be magic for it to guess the amount of calories in an object? It's perhaps a lookup and some simple math as an extra step.