5 ms·

The bits of this controversy about "where's the LAM?" are strange to me. My impression from the beginning was that the "Large Action Model" was a server-side th

by kemayo 2y ago

The bits of this controversy about "where's the LAM?" are strange to me. My impression from the beginning was that the "Large Action Model" was a server-side thing, and the hardware was going to be almost-entirely a way to talk to their services. Apparently this impression was far from universal.

nkozyra 2y ago
I recall the LAM being a/the (stated) reason why it couldn't "just be an app." If it's vaporware on-device it's more evidence that it could have just been an app. Perhaps there was special on-device sauce coming in a future update, but as of now it's a client making http calls.
- suyula 2y ago
  You probably wouldn't want an app to be able to press buttons on another app so if that's how it works (or worked) then that seems fair enough to me.
  - 1shooner 2y ago
    You probably really wouldn't want it to be doing that on someone else's server, using your authentications to all your apps. Which is what it actually does (or claims to, anyway).
- bippihippi1 2y ago
  it's an app with extra deep data collection, would be my guess. Also it's a nice construction to get attention.
- beefnugs 2y ago
  Yeah the whole point was supposedly you can "train it" by operating a generic "app" and it would learn so in the future other people could operate these apps via voice only. A clearly as yet non-existant feature. But would definitely require super obervation, recording, and control above standard android. Also insecure as hell. It would be super dumbo-bonkers insecure if you werent actually running these "Generic app training" on your own device, but in some cloud emulation. yikes This is as dumb as microsoft selling their next big feature of ai knowing how to control generic windows applications... that means they are going to record EVERYONES screens and actions without consent, we all know how microsoft respects users consent
  - ugh123 2y ago
    Why couldn't the AI be done on-device rather than server side? Isn't that where things are going with mobile hardware?
    - unsatchmo 2y ago
      Mobile devices don’t have the memory or the memory bandwidth to run an LLM that’s big enough to be good at much. Plus the fixed battery and thermal constraints.
      - bendews 2y ago
        my M1 Macbook air can run LLM's pretty well.. worse specs than the latest iPad Pro (and iPhone pro wouldn't be too far behind).
        nkozyra 2y ago
        Running them is a whole lot less resource intensive than training them. Unless the plan was just to build a RAG source from your personal data, in which case it would be yet another underwhelming feature.
      - ugh123 2y ago
        I guess that will never change huh
- lawgimenez 2y ago
  If this were an app, my guess is that this will never pass Google’s test with all the data passed to their servers, with all the privacy requirements.
- deleted 2y ago
  [deleted]
shmatt 2y ago
As someone whose job is to inference on the edge. No one is getting a mass product consistently inferencing on the edge for $199 not today and not next year (and I’d bet not in 2026) Your impression was spot on
- imranq 2y ago
  What do you mean by edge? There are already many apps that use LLMs and diffusion models locally on even a mac M1, which means iPhones and tensor chips aren't far behind
  - maronato 2y ago
    Macs and iPhones don’t cost $200
    - imranq 2y ago
      Take a look at an app like this: https://apps.apple.com/us/app/offline-chat-private-ai/id6474077941?platform=iphone https://apps.apple.com/us/app/offline-chat-private-ai/id6474... This will perform inference on Phi-2 or Mistral on almost any iPhone device. The first neural engine came with the A11 bionic chip which is in the iPhone 8. People can buy that phone for $150 used on Amazon. I think that's evidence that inference on the edge is already here
      - KomoD 2y ago
        > or Mistral on almost any iPhone device > "Small" requires iPhone 14 and 15 Pros > "Small" and "Medium" models are based on Mistral 7B 0.2 So no, also that app wouldn't work on an iPhone 8 either since it requires iOS 17 or later
        imranq 2y ago
        You could even go up a few iPhone versions and still be <$199. iPhone 12 used ~$190 supports iOS 17 and can run the Tiny models (phi-2). For small models, today a used iphone 14 costs ~$300. Which means next year it will by down to ~$200. I'm pretty sure by 2026, we will be able to inference "medium" models like Mistral/Llama3 8B on a <$199 device. That's not even factoring in how much better models will get by that time
theolivenbaum 2y ago
What I find interesting from the analysis is that the hard-coded, per-app event handlers suggests their integrations are nothing more than traditional wrappers around the services they support, instead of some magic, AI powered LAM living on the cloud that can interact with anything: UberEventHandler SpotifyEventHandler FoodDeliveryEventHandler MidjourneyEventHandler TranslatorEventHandler SearchWeatherInfo SearchStockInfo

by kemayo 2y ago

nkozyra 2y ago

I recall the LAM being a/the (stated) reason why it couldn't "just be an app." If it's vaporware on-device it's more evidence that it could have just been an app. Perhaps there was special on-device sauce coming in a future update, but as of now it's a client making http calls.

suyula 2y ago
You probably wouldn't want an app to be able to press buttons on another app so if that's how it works (or worked) then that seems fair enough to me.
- 1shooner 2y ago
  You probably really wouldn't want it to be doing that on someone else's server, using your authentications to all your apps. Which is what it actually does (or claims to, anyway).
bippihippi1 2y ago
it's an app with extra deep data collection, would be my guess. Also it's a nice construction to get attention.
beefnugs 2y ago
Yeah the whole point was supposedly you can "train it" by operating a generic "app" and it would learn so in the future other people could operate these apps via voice only. A clearly as yet non-existant feature. But would definitely require super obervation, recording, and control above standard android. Also insecure as hell. It would be super dumbo-bonkers insecure if you werent actually running these "Generic app training" on your own device, but in some cloud emulation. yikes This is as dumb as microsoft selling their next big feature of ai knowing how to control generic windows applications... that means they are going to record EVERYONES screens and actions without consent, we all know how microsoft respects users consent
- ugh123 2y ago
  Why couldn't the AI be done on-device rather than server side? Isn't that where things are going with mobile hardware?
  - unsatchmo 2y ago
    Mobile devices don’t have the memory or the memory bandwidth to run an LLM that’s big enough to be good at much. Plus the fixed battery and thermal constraints.
    - bendews 2y ago
      my M1 Macbook air can run LLM's pretty well.. worse specs than the latest iPad Pro (and iPhone pro wouldn't be too far behind).
      - nkozyra 2y ago
        Running them is a whole lot less resource intensive than training them. Unless the plan was just to build a RAG source from your personal data, in which case it would be yet another underwhelming feature.
    - ugh123 2y ago
      I guess that will never change huh
lawgimenez 2y ago
If this were an app, my guess is that this will never pass Google’s test with all the data passed to their servers, with all the privacy requirements.
deleted 2y ago
[deleted]

shmatt 2y ago

As someone whose job is to inference on the edge. No one is getting a mass product consistently inferencing on the edge for $199 not today and not next year (and I’d bet not in 2026) Your impression was spot on

imranq 2y ago
What do you mean by edge? There are already many apps that use LLMs and diffusion models locally on even a mac M1, which means iPhones and tensor chips aren't far behind
- maronato 2y ago
  Macs and iPhones don’t cost $200
  - imranq 2y ago
    Take a look at an app like this: https://apps.apple.com/us/app/offline-chat-private-ai/id6474077941?platform=iphone https://apps.apple.com/us/app/offline-chat-private-ai/id6474... This will perform inference on Phi-2 or Mistral on almost any iPhone device. The first neural engine came with the A11 bionic chip which is in the iPhone 8. People can buy that phone for $150 used on Amazon. I think that's evidence that inference on the edge is already here
    - KomoD 2y ago
      > or Mistral on almost any iPhone device > "Small" requires iPhone 14 and 15 Pros > "Small" and "Medium" models are based on Mistral 7B 0.2 So no, also that app wouldn't work on an iPhone 8 either since it requires iOS 17 or later
      - imranq 2y ago
        You could even go up a few iPhone versions and still be <$199. iPhone 12 used ~$190 supports iOS 17 and can run the Tiny models (phi-2). For small models, today a used iphone 14 costs ~$300. Which means next year it will by down to ~$200. I'm pretty sure by 2026, we will be able to inference "medium" models like Mistral/Llama3 8B on a <$199 device. That's not even factoring in how much better models will get by that time

theolivenbaum 2y ago

What I find interesting from the analysis is that the hard-coded, per-app event handlers suggests their integrations are nothing more than traditional wrappers around the services they support, instead of some magic, AI powered LAM living on the cloud that can interact with anything: UberEventHandler SpotifyEventHandler FoodDeliveryEventHandler MidjourneyEventHandler TranslatorEventHandler SearchWeatherInfo SearchStockInfo