4 ms·

I love the idea, that's the future. However you should be aware that the explanation of second law of thermodynamics generated by the LLM you used in your app s

by madlag 3y ago

I love the idea, that's the future. However you should be aware that the explanation of second law of thermodynamics generated by the LLM you used in your app store screenshot is wrong: the LLM has it backwards. Energy transfers to less stable states from more stable states, and not the reverse. (I use LLMs for science education apps like https://apps.apple.com/fr/app/explayn-learn-chemistry/id6448284993 https://apps.apple.com/fr/app/explayn-learn-chemistry/id6448..., so I am quite used to spot that kind of errors in LLM outputs...)

Horffupolde 3y ago
How do you define stability in that context?
- madlag 3y ago
  Stability is actually defined by having a lower energy level. That explains why energy can only flow from a less stable system to a more stable system : the more stable system does not have available energy to give.
wannabag 3y ago
Oh, that's an interesting app and in French too... is that something you plan to have on Android as well?
- madlag 3y ago
  Yes, it's Unity based, so quite easy. There is another version on Quest too, so running on Android : https://www.meta.com/fr-fr/experiences/6113695908674751/ https://www.meta.com/fr-fr/experiences/6113695908674751/ .
Const-me 3y ago
Is that explanation better? https://github.com/Const-me/Cgml/blob/master/Mistral/MistralChat/screenshot.png https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... Same Mistral Instruct 0.2 model, different implementation.
kkielhofner 3y ago
Strongly agree. Local, app embedded, and purpose-built targeted experts is clearly the future in my mind for a variety of reasons. Looking at TPUs in Android devices and neural engine in Apple hardware it's pretty clear. Xcode already has an ML studio, for example, that can not only embed and integrate models in apps but also finetune, etc. It's obvious to me that at some point most apps will have embedded models in the app (or device) for specific purposes. No AI can compare to humans and even we specialize. You wouldn't hire a plumber to perform brain surgery and you wouldn't hire a neurosurgeon to fix your toilet. Mixture of experts with AI models is a thing of course but when we look at how we primarily interact with technology and the functionality it provides it's generally pretty well siloed to specific purposes. A purposed domain and context trained/tuned small model doing stuff on your on-device data would likely do nearly as well if not better for some applications than even ChatGPT. Think of the next version of device keyboards doing RAG+LLM through your text messages to generate replies. Stack it up with speech to text, vision, multimodal models, and who knows what and yeah, interesting. Throw in the automatic scaling, latency, and privacy and the wins really stack up. Some random app developer can integrate a model in their application and scale higher with better performance than ChatGPT without setting money on fire.
- jorvi 3y ago
  > Local, app embedded, and purpose-built targeted experts is clearly the future in my mind for a variety of reasons. Looking at TPUs in Android devices and neural engine in Apple hardware it's pretty clear. I think that’s only true for delay-intolerant or privacy-focused features. For most situations, a remote model running on an external server will outperform a local model. There is no thermal, battery or memory headroom for the local model to ever do better. The cost being a mere hundred milliseconds delay at most. I expect most models triggered on consumer devices to run remotely, with a degraded local service option in case of connection problems.
  - kkielhofner 3y ago
    Snapchat filters, iPhone photo processing/speech to text/always-on Hey Siri/OCR/object detection and segmentation - there are countless applications and functionality doing this on device today (and for years). For something like the RAG approach I mentioned the sync and coordination of your local content to a remote API would be more taxing on the battery just in terms of the radio than what we already see from on device neural engines and TPUs as leveraged by the functionality I described. These applications would also likely be very upload heavy (photo/video inference - massive upload, tiny JSON response) which could very likely end up taxing cell networks further. Even RAG is thousands of tokens in and a few hundred out (in most cases). There's also the issue of Nvidia GPUs having > 1 yr lead times and the exhaustion of GPUs available from various cloud providers. LLMs especially use tremendous resources for training and this increase is leading to more and more contention for available GPU resources. People are going to be looking more and more to save the clouds and big GPUs for what you really need to do there - big training. Plus, not everyone can burn $1m/day like ChatGPT. If AI keeps expanding and eating more and more functionality the remote-first approach just isn't sustainable. There will likely always be some sort of blend (with serious heavy lifting being cloud, of course) but it's going to shift more and more to local and on-device. There's just no other way.
reexpressionist 3y ago
[dead]

by madlag 3y ago

Horffupolde 3y ago

How do you define stability in that context?

madlag 3y ago
Stability is actually defined by having a lower energy level. That explains why energy can only flow from a less stable system to a more stable system : the more stable system does not have available energy to give.

wannabag 3y ago

Oh, that's an interesting app and in French too... is that something you plan to have on Android as well?

madlag 3y ago
Yes, it's Unity based, so quite easy. There is another version on Quest too, so running on Android : https://www.meta.com/fr-fr/experiences/6113695908674751/ https://www.meta.com/fr-fr/experiences/6113695908674751/ .

Const-me 3y ago

Is that explanation better? https://github.com/Const-me/Cgml/blob/master/Mistral/MistralChat/screenshot.png https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... Same Mistral Instruct 0.2 model, different implementation.

kkielhofner 3y ago

Strongly agree. Local, app embedded, and purpose-built targeted experts is clearly the future in my mind for a variety of reasons. Looking at TPUs in Android devices and neural engine in Apple hardware it's pretty clear. Xcode already has an ML studio, for example, that can not only embed and integrate models in apps but also finetune, etc. It's obvious to me that at some point most apps will have embedded models in the app (or device) for specific purposes. No AI can compare to humans and even we specialize. You wouldn't hire a plumber to perform brain surgery and you wouldn't hire a neurosurgeon to fix your toilet. Mixture of experts with AI models is a thing of course but when we look at how we primarily interact with technology and the functionality it provides it's generally pretty well siloed to specific purposes. A purposed domain and context trained/tuned small model doing stuff on your on-device data would likely do nearly as well if not better for some applications than even ChatGPT. Think of the next version of device keyboards doing RAG+LLM through your text messages to generate replies. Stack it up with speech to text, vision, multimodal models, and who knows what and yeah, interesting. Throw in the automatic scaling, latency, and privacy and the wins really stack up. Some random app developer can integrate a model in their application and scale higher with better performance than ChatGPT without setting money on fire.

jorvi 3y ago
> Local, app embedded, and purpose-built targeted experts is clearly the future in my mind for a variety of reasons. Looking at TPUs in Android devices and neural engine in Apple hardware it's pretty clear. I think that’s only true for delay-intolerant or privacy-focused features. For most situations, a remote model running on an external server will outperform a local model. There is no thermal, battery or memory headroom for the local model to ever do better. The cost being a mere hundred milliseconds delay at most. I expect most models triggered on consumer devices to run remotely, with a degraded local service option in case of connection problems.
- kkielhofner 3y ago
  Snapchat filters, iPhone photo processing/speech to text/always-on Hey Siri/OCR/object detection and segmentation - there are countless applications and functionality doing this on device today (and for years). For something like the RAG approach I mentioned the sync and coordination of your local content to a remote API would be more taxing on the battery just in terms of the radio than what we already see from on device neural engines and TPUs as leveraged by the functionality I described. These applications would also likely be very upload heavy (photo/video inference - massive upload, tiny JSON response) which could very likely end up taxing cell networks further. Even RAG is thousands of tokens in and a few hundred out (in most cases). There's also the issue of Nvidia GPUs having > 1 yr lead times and the exhaustion of GPUs available from various cloud providers. LLMs especially use tremendous resources for training and this increase is leading to more and more contention for available GPU resources. People are going to be looking more and more to save the clouds and big GPUs for what you really need to do there - big training. Plus, not everyone can burn $1m/day like ChatGPT. If AI keeps expanding and eating more and more functionality the remote-first approach just isn't sustainable. There will likely always be some sort of blend (with serious heavy lifting being cloud, of course) but it's going to shift more and more to local and on-device. There's just no other way.

reexpressionist 3y ago

[dead]