5 ms·

I got it all set up and tested out Gemma3 1B on a Pixel 8a. That only took a few minutes, which was nice. But it was garbage. It barely parsed the question, di

by smeej 1y ago

I got it all set up and tested out Gemma3 1B on a Pixel 8a. That only took a few minutes, which was nice. But it was garbage. It barely parsed the question, didn't even attempt to answer it, and replied in what was barely serviceable English. All I asked was how it was small enough to run locally on my phone. It was bad enough for me to abandon the model entirely, which is saying a lot, because I feel like I have pretty low expectations for AI work in the first place.

throwaway314155 1y ago
> All I asked was how it was small enough to run locally on my phone Bit off-topic, but did you expect to see a real or honest answer about itself? I see many people under the impression that models know information about themselves that isn't in the system prompt. Couldn't be further from the truth. In face, those questions specifically lead to hallucinations more often resulting in an overconfident assertion with a "reasonable" answer. The information the model knows (offline - no tools allowed) stops weeks if not months if not years prior to when the model is done training. There is _zero_ information about its inception, how it works, or anything similar in its weights. Sorry, this is mostly directed at the masses - not you.
- smeej 1y ago
  Not really, but I did expect an answer, or at least a non-answer, that showed it understood the question, and that an answer was expected.
DrSiemer 1y ago
Why would you ask a 1B model anything? Those are only useful for rephrasing output at best.
- smeej 1y ago
  ...because they advertise its potential for on-device usage for basic functionality in this very product and I wanted to see how it worked? I'm not sure what you want MW to say. I tested the product because I wanted to know if the product worked. That was the extent of it.

5 ms·

I got it all set up and tested out Gemma3 1B on a Pixel 8a. That only took a few minutes, which was nice. But it was garbage. It barely parsed the question, di

by smeej 1y ago

throwaway314155 1y ago
> All I asked was how it was small enough to run locally on my phone Bit off-topic, but did you expect to see a real or honest answer about itself? I see many people under the impression that models know information about themselves that isn't in the system prompt. Couldn't be further from the truth. In face, those questions specifically lead to hallucinations more often resulting in an overconfident assertion with a "reasonable" answer. The information the model knows (offline - no tools allowed) stops weeks if not months if not years prior to when the model is done training. There is _zero_ information about its inception, how it works, or anything similar in its weights. Sorry, this is mostly directed at the masses - not you.
- smeej 1y ago
  Not really, but I did expect an answer, or at least a non-answer, that showed it understood the question, and that an answer was expected.
DrSiemer 1y ago
Why would you ask a 1B model anything? Those are only useful for rephrasing output at best.
- smeej 1y ago
  ...because they advertise its potential for on-device usage for basic functionality in this very product and I wanted to see how it worked? I'm not sure what you want MW to say. I tested the product because I wanted to know if the product worked. That was the extent of it.