4 ms·

2GB for 0.5B smallest model. it does not make sense for each app to download this. apple must have plans to pre-load these models on os level and expose SDK for

2GB for 0.5B smallest model. it does not make sense for each app to download this. apple must have plans to pre-load these models on os level and expose SDK for all apps to call these models locally. exciting times! opened issue for them to confirm this: https://github.com/apple/ml-fastvlm/issues/7 https://github.com/apple/ml-fastvlm/issues/7

cube2222 1y ago
That’s what they suggested about LLMs at last year’s WWDC iirc. The core models are provided by the OS, while apps bring LORAs to fine-tune them / bring custom heads for them.
babl-yc 1y ago
You could probably get away with f16 or even quantize to int8 and have a much smaller model, but your point stands. Users won't be thrilled to download a 500MB model for an app either.
- nikolayasdf123 1y ago
  haha latest Uber build for iOS 18 is 500MB... without LLM models <face-palm/>
  - ukuina 1y ago
    What are they doing in there? Is it mostly visual assets?
    - bastawhiz 1y ago
      If I was going to guess, I'd get there's a ton of third party code for things like payment method SDKs. Every local payment method around the world is going to have its own package that you need to import, and you can't just load in new executable code on the fly after the app is installed.
      - victorbjorklund 1y ago
        You can actually do over the air updates of apps (how easy it is depends on what you wrote your app in) and not adding a new feature (like just adding an additional payment provider) would not require an update on the App Store.
        bastawhiz 1y ago
        You can't download only part of your app and lazy load functionality that your user probably won't use.
        victorbjorklund 1y ago
        You could but it probably would either require a server solution like Liveview Native (you only get the code when you use it) or a pretty bad UX where you have to wait to download something first
      - nikolayasdf123 1y ago
        wouldn't you want to create payment gateway and abstract away logic such that client is agnostic of payment processes and backend confirms internal payment process into external specific ones? (in worst case redirect to other apps with universal links or webview)
        bastawhiz 1y ago
        You're not doing it in a browser window. You're integrating against device APIs like NFC, you've got custom UI (which you probably want to be native), you've got stuff like camera access to read QR codes and OCR a credit card. Want to pay by topping up a wallet at 7-Eleven? Now you need a map to show where the nearest one is.
    - philipkglass 1y ago
      A lot of geography-specific scenarios are compiled into the app, including regional payment SDKs. There's a great comment from a former Uber engineer explaining it here: https://news.ycombinator.com/item?id=25376346 https://news.ycombinator.com/item?id=25376346
    - nikolayasdf123 1y ago
      I think they using vector graphics and vector animations (say Rive). Rive takes order of 10s of KBs. Lottie is much larger to 100s of KBs. Even then you would need 5000 animations to reach 500MB, unlikely! raster graphics and videos are likely not included in build probably some unused code (libraries) got into it, it can grow quite large or maybe some ML models?
HanClinto 1y ago
I think that there is fantastic potential in having open-weight, OS-standard foundation models. Especially if the API gives opportunity for app developers to load their custom LoRa fine-tunings onto OS-standard foundation models at runtime, then you can (ideally) have the best of both worlds -- fine-tuned app-specific models with reasonable app sizes.
- HappMacDonald 1y ago
  I haven't seen much done with loras for LLMs though, only for diffusion image gen models. From what I've heard it sounds like a difference in benefit due to architecture.
gessha 1y ago
My guess is that they won’t confirm it unless it’s a big presentation. WWDC maybe?

cube2222 1y ago

That’s what they suggested about LLMs at last year’s WWDC iirc. The core models are provided by the OS, while apps bring LORAs to fine-tune them / bring custom heads for them.

babl-yc 1y ago

You could probably get away with f16 or even quantize to int8 and have a much smaller model, but your point stands. Users won't be thrilled to download a 500MB model for an app either.

nikolayasdf123 1y ago
haha latest Uber build for iOS 18 is 500MB... without LLM models <face-palm/>
- ukuina 1y ago
  What are they doing in there? Is it mostly visual assets?
  - bastawhiz 1y ago
    If I was going to guess, I'd get there's a ton of third party code for things like payment method SDKs. Every local payment method around the world is going to have its own package that you need to import, and you can't just load in new executable code on the fly after the app is installed.
    - victorbjorklund 1y ago
      You can actually do over the air updates of apps (how easy it is depends on what you wrote your app in) and not adding a new feature (like just adding an additional payment provider) would not require an update on the App Store.
      - bastawhiz 1y ago
        You can't download only part of your app and lazy load functionality that your user probably won't use.
        victorbjorklund 1y ago
        You could but it probably would either require a server solution like Liveview Native (you only get the code when you use it) or a pretty bad UX where you have to wait to download something first
    - nikolayasdf123 1y ago
      wouldn't you want to create payment gateway and abstract away logic such that client is agnostic of payment processes and backend confirms internal payment process into external specific ones? (in worst case redirect to other apps with universal links or webview)
      - bastawhiz 1y ago
        You're not doing it in a browser window. You're integrating against device APIs like NFC, you've got custom UI (which you probably want to be native), you've got stuff like camera access to read QR codes and OCR a credit card. Want to pay by topping up a wallet at 7-Eleven? Now you need a map to show where the nearest one is.
  - philipkglass 1y ago
    A lot of geography-specific scenarios are compiled into the app, including regional payment SDKs. There's a great comment from a former Uber engineer explaining it here: https://news.ycombinator.com/item?id=25376346 https://news.ycombinator.com/item?id=25376346
  - nikolayasdf123 1y ago
    I think they using vector graphics and vector animations (say Rive). Rive takes order of 10s of KBs. Lottie is much larger to 100s of KBs. Even then you would need 5000 animations to reach 500MB, unlikely! raster graphics and videos are likely not included in build probably some unused code (libraries) got into it, it can grow quite large or maybe some ML models?

HanClinto 1y ago

I think that there is fantastic potential in having open-weight, OS-standard foundation models. Especially if the API gives opportunity for app developers to load their custom LoRa fine-tunings onto OS-standard foundation models at runtime, then you can (ideally) have the best of both worlds -- fine-tuned app-specific models with reasonable app sizes.

HappMacDonald 1y ago
I haven't seen much done with loras for LLMs though, only for diffusion image gen models. From what I've heard it sounds like a difference in benefit due to architecture.

gessha 1y ago

My guess is that they won’t confirm it unless it’s a big presentation. WWDC maybe?