10 ms·

Firstly compliments to Apple for all these incredible accessibility features. I think there is an important little nod to the future in this announcement. "Per

Firstly compliments to Apple for all these incredible accessibility features. I think there is an important little nod to the future in this announcement. "Personal Voice" is training (likely fine tuning) and then running a local AI model to generate the user's voice. This is a sneak peak of the future of Apple. They are in the unique position to enable local AI tools, such as assistance or text generation, without the difficulties and privacy concerns with the cloud. Apple silicone is primed for local AI models with its GPU, Neural Cores and the unified memory architecture. I suspect that Apple is about to surprise everyone with what they do next. I'm very excited to see where they go with the M3, and what they release for both users and developers looking to harness the progress made in AI but locally.

detourdog 3y ago
Just yesterday I started using a new maxed out Mac mini and everything about it is snappy. I have no doubt that it is ready for enormous amount of background processing. Heavy background work is the only way to use the processing power in that little computer.
- samwillis 3y ago
  Think Siri+ChatGPT trained on all your email, documents, browser history, messages, movements, everything. All local, no cloud, complete privacy. "Hey Siri, I had a meeting last summer in New York about project X, could you bring up all relevant documents and give me a brief summary of what we discussed and decisions we made. Oh and while you're at it, we ate at an awesome restaurant that evening, can you book a table for me for our meeting next week."
  - _xnmw 3y ago
    This is already available with Rewind.ai + their GPT plugin and I've already asked such questions and got good answers.
    - edgyquant 3y ago
      Thats not what’s being discussed though, which is Siri having these abilities built in
    - samwillis 3y ago
      Rewind.ai looks awesome, but only like 10% of what I'm thinking Apple could do.
  - lotsofpulp 3y ago
    All of my current experience with Siri tells me there is a 50-50 chance of the result coming back as “Sorry, I have having trouble connecting to the network” or playing a random song from Apple Music. Just last night, we were entertaining our toddler with animal sounds. It worked with “Hey Siri, what does a goat sound like?”, then we were able to do horse, cow, sheep, boar, and it somehow got tripped up on pig, for which it responded with the Wikipedia entry and told us to look at the phone for more info.
    - iknowstuff 3y ago
      [flagged]
      - lotsofpulp 3y ago
        I thought the lack of ability to execute on current “easy” queries would indicate something about ability to execute something as complicated as figuring out the restaurant you ate at and making a reservation. At least anytime in the next few years.
        mason55 3y ago
        Siri today is built on what’s essentially completely different concepts from something like ChatGPT. There are demos of using ChatGPT to turn normal English into Alexa commands and it’s pretty flawless. If you assume Apple can pretty easily leverage LLM tech on Siri and do it locally via silicon in the M3 or M4, it’s only a matter of chip lead time before Siri has multiple orders of magnitude improvement.
        lotsofpulp 3y ago
        That would be a pleasant surprise!
        rschoultz 3y ago
        That experience likely isn’t transferable to Siri, that has deeper problems. People, me included, are reporting their problems with Siri, e.g. setting it to transcribing what they and Siri says as text on the screen, and then are able to show that given input as “Please add milk to the shopping list” results in Siri responding “I do not understand what speaker you refer to.”, in writing. Likely problems like these could be overcome, but preparing better input would probably not address the root cause of the problems with Siri.
  - blitzar 3y ago
    "Playing Our Last Summer by ABBA on Apple Music"
  - addandsubtract 3y ago
    Fat chance Apple will alow us to do this locally. More like, upgrade to Apple Cloud Plus to get these features. But yeah, I've also dreamt of what my Apple hardware could do.
    - fastball 3y ago
      Apple already does all existing AI stuff locally. The main one being categorizing images in your Photos library.
    - microtonal 3y ago
      Why not? They do on-device training of face recognition: https://machinelearning.apple.com/research/recognizing-people-photos https://machinelearning.apple.com/research/recognizing-peopl...
    - insane_dreamer 3y ago
      Actually Apple is much more device-first than cloud-first
  - throw384624 3y ago
    I would love if Apple created a personal iCloud device too. Let me sync my information to something local. I know there's Nextcloud, but it's not as seamless as iCloud.
    - tough 3y ago
      We should get the European Union to force Apple to offer an Icloud Local server thingy
      - scarface74 3y ago
        Yes and we should also have EU regulators at every design meeting for every company. They did such a good job with the GDPR making the user experience better on the web
        another2another 3y ago
        Yes, alas they didn't leave room for a 'cookie preferences' cookie, so that whenever I choose the option 'reject all', it's of course going to ask me again, every time I visit the website. saying that, their intentions were good, I'm always horrifically amazed at the number of cookies used whenever I see the preferences popup. I honestly had no idea how many tracking cookies were used by the average website.
  - nicbou 3y ago
    I love Steve Jobs' "bicycle for the mind" metaphor, and what you describe is the best possible example of this concept. A computer that does that would enable us to do so much more. This is the sort of AI I want; a true personal assistant, not a bullshit generator.
    - y04nn 3y ago
      It appears that we are tantalizingly close to have the perfect voice assistant. But for some inexplicable reason, it does not exist yet. Siri was introduced over a decade ago, and it seems that its development has not progressed as anticipated. Meanwhile, language models have made significant advancements. I am uncertain as to what is preventing Apple, a company with boundless resources, from enhancing Siri. Perhaps it is the absence of competition and the duopoly maintained by Apple and Google, both of whom seem reluctant to engage in a competitive battle within this domain.
      - lamontcg 3y ago
        It is probably a people problem. The people who really understood Siri have probably left, the managers left running it are scored primarily on not making any mistakes and staying off the headlines. Any engineers who understand what it would take to upgrade it aren't given the resources and spend their days on maintenance tasks that nobody really sees.
        slowmovintarget 3y ago
        It's more likely a perverse incentive problem. Voice activated "assistants" weren't viewed as assistance for end users. They were universally viewed as one of two things: A way of treating the consumer as a product, or a feature check-box. That Siri went from useful to far less useful had more to do with the aim to push products at you rather than actually accomplishing the task you set for Siri. If Apple actually delivers an assistant that works locally, doesn't make me the product, and generally makes it easier to accomplish my tasks, then that's a product worth paying for. When anyone asks "who benefits from 'AI'?" the answer is almost invariably "the people running the AI." Microsoft and OpenAI get more user data, and subscriptions. Google gets another vehicle for attention-injection. But if I run Vicuna or Alpaca (or some eventual equivalent) on my hardware, I can ensure I get what I need, and that there's much less hijacking of my intentions. So Microsoft, if you're listening: I don't want Bing Chat search, I want Cortana Local.
  - bredren 3y ago
    > All local, no cloud, complete privacy. This is already felt in use of Stable Diffusion, where M2 is fully capable offline. Anything that can be done to reduce the need to “dial out” for processing protects the individual. It erodes the ability of business and governmental organizations to use knowledge of otherwise private matters to target and influence. The potential of moving a HQ LLM like GPT to the edge to answer everyday questions reminds me of my move from Google to DDG as my default search engine. Except it’s even a bigger deal than that. It reduces private data exhaust from search to zero, making going to the net a backup plan instead of a necessity. Apple delivering this on device is a major threat to OpenAI, which will have to provide some LLM model with training that Apple can’t or won’t. Savvy users will begin to leer at having to produce queries over the wire, feeding valuable data (proven by ShareGPT) Even then, Apple will likely chose to or be forced to open up on device AI to allow user contributed apps like LORAs which would ask the question why does OpenAI need to exist? Also fascinating the potential to do this at the Server level for enterprise. If Apple produced a stack for enterprise training it could replace generalized data compute needs, shifting IT back to local or intranet.
  - mattl 3y ago
    Sounds like Knowledge Navigator https://youtu.be/umJsITGzXd0 https://youtu.be/umJsITGzXd0
    - detourdog 3y ago
      My senior thesis was influenced by that.
  - sroussey 3y ago
    I’m working on something very much like this!
  - tomcam 3y ago
    Apparently, you are not an actual user of Siri, because I get jack shit out of her. speech to text is infinitely worse than the first week Siri was released.
    - ceejayoz 3y ago
      Siri has felt like abandonware for years. My hope is something like a ChatGPT model is its replacement soon.
  - precompute 3y ago
    >Think Siri+ChatGPT trained on all your email, documents, browser history, messages, movements, everything. All local, no cloud, complete privacy. That sounds absolutely horrifying if you remove the "all local" part. And that part's a pipe dream anyway. Plus, when using a model you'd basically become subservient / limited to the type of data in the model, which would necessarily abide by Apple's TOS, so a couple of hundred million people would be the Apple TOS but in human form. I don't understand why apple fanboys don't get this. Apple is pretty shoddy when privacy is concerned. Are these apple employees making these posts?
- hammock 3y ago
  Do you think it would be good/capable for audio or video editing?
  - detourdog 3y ago
    I think so. It is stunningly fast. It may be the biggest boost I have seen. The last big boost I saw was the move to SSD.
- Maursault 3y ago
  > Just yesterday I started using a new maxed out Mac mini and everything about it is snappy. Really?! I didn't think anyone here would fall for that. Mac Mini 12-core M2, 19-core GPU, 32GB, 10Gbit, 8TB storage? $4500 Mac Studio 20-core M1, 48-core GPU, 64GB, 10Gbit, 1TB storage is $4000. 128GB of RAM is $800 more but either Studio RAM configuration obviously spanks the M2 mini. It's sacrificing Apple's expensive storage, but with Thunderbolt 3 it's pretty academic to find 8TB or more of NVMe storage, probably 32GB of NVMe RAID[1], for less than Apple's charge of $2200 above cost of 1TB. [1] https://eshop.macsales.com/shop/express-4m2 https://eshop.macsales.com/shop/express-4m2
  - detourdog 3y ago
    I specced the smallest SSD. I use netwomr homes. The mini is a stop gap waiting for the pro. Drive size Indont really consider a performance item anymore. I spent just over $2,000. Mac mini With the following configuration: Apple M2 Pro with 12‑core CPU, 19-core GPU, 16‑core Neural Engine 32GB unified memory 512GB SSD storage Four Thunderbolt 4 ports, HDMI port, two USB‑A ports, headphone jack 10 Gigabit Ethernet Im satauisfied.
    - Maursault 3y ago
      Not awful, but for $2K you could have had 16-core CPU, 20-core GPU, 32-core Neural Engine, 48GB unified memory, 512K SSD storage, Four Thunderbolt 4 ports, two HDMI ports, four USB-A ports, two headphone jacks, two Gigabit Ethernet.
      - detourdog 3y ago
        Yes. I wanted the 10Gbt Ethernet. My purchasing question is when is the right time to buy a great monitor. In the CRT days the monitor lasted the longest and buying the best one could afford worked for me.
        detourdog 3y ago
        I just went back to compare the Mini with the Studio again. Despite your advice I would buy the Mini again for these reasons: I'm on a newer generation chip that has a lower power draw. Meets my network speed minimum. All for the price of the entry level Studio. This box is basically an experiment to see how much processing power I need. I have a very specific project that will require the benchmarking of Apple's machine learning frameworks. I want to see how much of a machine learning load this Mini can handle. Once I have benchmarks maybe the Pro will exist and I will be in good shape to shop and understand what I'm buying. I think a Mini of any spec is a great value. The studio has a place but I'm hoping the Pro ends up being like an old Sun E450. This Mini experiment is to help me frame the hardware power vs. the software loads.
brookst 3y ago
UMA may turn out to be visionary. I really wonder if they saw the AI/ML trend or just lucked out. Either way, the apple silicon arch is looking very strong for local AI. It’s a lot easier to beef up the NPU than to redo memory arch.
- smoldesu 3y ago
  I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. All of them are pretty weak for local training. But having reasonably powerful inferencing hardware isn't very hard at all, UMA doesn't seem very visionary to me in an era of MMAPed AI models.
  - microtonal 3y ago
    I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. Apple Silicon AMX units provide the matrix multiplication performance of many core CPUs or faster at a fraction of the wattage. See eg. https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
    - smoldesu 3y ago
      Yes, and generic multicore ARM CPUs can run ARM's standard compute library regardless of their hardware: https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Plus, the benchmark you've linked to is comparing hardware accelerated inferencing to the notoriously crippled MKL execution. A more appropriate comparison would test Apple's AMX units against the Ryzen's AVX-optimized inferencing.
      - microtonal 3y ago
        Nope, the benchmarks are done by disabling the MKL AMD cripple (I did these benchmarks). It’s not faster with eg. AMD BLIS.
ec109685 3y ago
The question is whether there will be models that can’t fit into an iPhone that apple will miss out on because they find cloud based personalization so abhorrent. Agree these are tremendously good features and having them run locally will provide the best possible experience.
- samwillis 3y ago
  My hope for the surprise is an almost order of magnitude increase in memory on these chips. That wold be transformative for local AI.
  - fastball 3y ago
    Order of magnitude increase in memory without a corresponding increase in size is a pipe dream if I've ever heard one.
    - samwillis 3y ago
      We're all aloud our pipe dreams, the future is built on them.
  - astrange 3y ago
    Have you tried using less memory?
- reaperducer 3y ago
  The question is whether there will be models that can’t fit into an iPhone that apple will miss out on Fortunately, Apple isn't an adolescent, so it doesn't suffer from FOMO. because they find cloud based personalization so abhorrent. I'm not even sure what "cloud based personalization" means to the user, other than "Hoover up all of your personal information."
  - whimsicalism 3y ago
    > I'm not even sure what "cloud based personalization" means to the user, other than "Hoover up all of your personal information." It means having actually good ML. I see so many posts around here saying Apple is absolutely well positioned to dominate in ML. It's just not true. Nobody who is a top AI player wants to work at Apple where they have few if any AI products, no data, don't pay particularly well, not a big research culture, etc. etc. The only thing they have going for them in this space is a good ARM architecture for low power matrix multiplication.
huijzer 3y ago
Sounds plausible. Also due to the news yesterday that Apple uses 90% of TSMC‘s 3nm space in 2023 [1]. Whereas everyone is talking about a recession, Apple seems to see opportunities. Or maybe they just had too much cash on hand. Also possible. [1]: https://news.ycombinator.com/item?id=35947339 https://news.ycombinator.com/item?id=35947339
- smoldesu 3y ago
  Density doesn't always matter. I'm reminded of Apple's 5nm M1 Ultra struggling to keep up with Nvidia's 10nm RTX 3080 in standard use. Having such a minor node advantage won't necessarily save them here, especially since Nvidia's currently occupying the TSMC 4nm supply.
  - deleted 3y ago
    [deleted]
  - valine 3y ago
    Nvidia is somewhat encumbered by their need to optimize for raster performance. Ideally, all those transistors should be going toward tensor cores. Apple has never really taken the gaming market seriously. If they wanted to, they could ship their next M3 chip with identical GPU performance and use all that new 3nm die space for AI accelerators.
  - Someone 3y ago
    > Having such a minor node advantage Is that a minor advantage? I would think that, the smaller the nodes get, the larger the impact of a 1nm difference. Because transistors have area, I think the math, in ≈transistor count would be 3nm:4nm = ⅓²:¼², and that’s 1,777… so a 3nm node could have 75% more transistors on a given die area than a 4nm one (roughly).
    - Kirby64 3y ago
      4nm -> 3nm no longer means size goes down as a result directly ratiometricly. You have to look at what TSMC is claiming for their improvements. They're claiming 5nm -> 3nm is a 70% density improvement (I can't find any 4nm -> 3nm claims)... so 4->3 must be much less. Also, most folks seem to have gone directly from 5nm to 3nm, and skipped 4nm altogether.
mupuff1234 3y ago
Do users actually care whether something is local or not? (Edit: In terms of privacy as there are benefits in terms of speed and offline work) It's not like we're not already storing all of our media on the cloud (including voice), passwords and other sensitive data.
- heyjamesknight 3y ago
  I do, when I’m without a quality internet connection and a basic request to Siri to turn on a timer fails.
- criddell 3y ago
  A lot of end users do not and they have no interest in spending the time figuring it out. That's why it's very important that the companies behind the technology we use make ethical choices that are good for their users and when that doesn't happen, legislators need to step in. Apple has been on both sides of that coin and what is ethical isn't always clear.
  - fastball 3y ago
    > companies need to be ethical > what is ethical isn't always clear So...?
    - criddell 3y ago
      So communicate. Have guiding principles for the company. Be open about the choices you make and why you make them. Listen and respond to criticism.
      - otterley 3y ago
        Apple has one: "Privacy is a fundamental human right."
        LocalH 3y ago
        Not for their employees, otherwise they'd develop a setup to keep personal and company information separate internally. https://www.theverge.com/22648265/apple-employee-privacy-icloud-id https://www.theverge.com/22648265/apple-employee-privacy-icl...
jayd16 3y ago
Is it really that unique a position? Doesn't Google's pixel phones have a neural core and similar architecture as well?
- Grustaf 3y ago
  The uniqueness is not in hardware it’s in trust. And the ability to make software for normal human beings.
  - JustSomeNobody 3y ago
    Trust. That's an interesting word. Personally, I don't trust corporations. Their motive is always money.
    - reaperducer 3y ago
      Personally, I don't trust corporations. How do you eat?
      - deleted 3y ago
        [deleted]
      - nawgz 3y ago
        Because a food corporation that sells food that kills people is unlikely to make piles of money.
        paganel 3y ago
        Not for lack of trying. If anyone were to write a chronological history of regulations imposed by different authorities throughoug history I think that it is a fair assumption to make that regulations related to making bread would already show up in the first chapters of the book.
      - JustSomeNobody 3y ago
        Slowly, chewing my food completely.
uncletammy 3y ago
> They are in the unique position to enable local AI tools, such as assistance or text generation, without the difficulties and privacy concerns with the cloud. I don't see why client-side processing mitigates the privacy concerns. That doesn't stop Apple from "locally" spying on you then later sending that data to their servers.
- marricks 3y ago
  Ok, sure, but surely you see how it is that much harder to do? Also since Apple is built around selling expensive devices and services you could also see why they’d have much less incentive to spy and collect data than, say, Google or Facebook? The cynicism of “everything is equally bad so why care” is destructive.
  - SoftTalker 3y ago
    Now. It was just two decades ago that Apple was on life support. That could happen again. And the temptation would be much stronger to start monitizing their user's data.
    - marricks 3y ago
      Very good point: if things changed drastically they could change drastically.
- aardvarkr 3y ago
  Nope but that is something that can be monitored and investigated, pretty easily I might add.
yyyk 3y ago
>They are in the unique position to enable local AI tools The only unique Apple thing here is how bad their AI products here and how behind they are in AI. This is the only thing that matters here - performance is adequate or better for the other processors out there, but you can't get anywhere without the appropriate software. Maybe they'll get smart enough to buy some AI startups/companies to get the missing talent.
- andy_ppp 3y ago
  Which AI products that they have actually implemented are bad? I think Siri is pretty poor to be fair and improves at a glacial pace. Pretty much everything else I'd say is state of the art from things like text selection from images, cutting out of image subject, their computational photography, even Map directions have come a long way.
  - yyyk 3y ago
    When people talk about AI they mean the new tech like LLM or Diffusion, and the only relevant Apple offering (Siri) is way behind and there's no evidence they have anything to replace it. (Aside, their image manipulation and Map is worse - though with Maps I dunno what's the underlying issue, and OCR was already mostly solved. I'm far from a photography expert so can't compare there).
    - brookst 3y ago
      With Apple it is always good to draw the distinction (as you did) between them being behind and there being no evidence of them not being behind.
      - yyyk 3y ago
        True that. But I won't give Apple credit for products we can't see or assume good performance without proof. As they say in the movies: "Show me the money". I'll say though that no multibillion company is under existential threat. Not Apple, not Google and not even Intel. At worst they will lose a couple tens of billions and some marketshare. Even IBM still exists and took a long long time to fall to where it is still today.
cubefox 3y ago
> Apple silicone is primed for local AI models with its GPU, Neural Cores and the unified memory architecture. Is there any information on how they are doing with respect to local AI compared to other Smartphone SoCs? (Snapdragon, Tensor, Kirin, Helio etc)
xnx 3y ago
> They are in the unique position to enable local AI tools Nothing special about Apple with regards to AI. M2 beats x86 in power efficiency, but not significantly better than other ARM processors.
- shepherdjerred 3y ago
  I think the parent is referring to the Apple Neural Engine in Apple Silicon, which aren't widely used today (as far as I know)
- gcr 3y ago
  Some workloads on M1 absolutely smash other ARM processors in part because of M1's special-purpose hardware. In particular, the undocumented AMX chip is really nice for distance matrix calculations, vector search, embeddings, etc. Non-scientific example: for inference, whisper.cpp links with Accelerate.framework to do fast matrix multiplies. On M1, one configuration gets ~6x realtime speed, but on a very beefy AWS Gravatron processor, the same configuration only achieves 0.5x realtime, even after choosing an optimal threadcount, even linking with NEON-optimized BLAS. (Maybe I'm doing something wrong though).
rodgerd 3y ago
> I think there is an important little nod to the future in this announcement. "Personal Voice" is training (likely fine tuning) and then running a local AI model to generate the user's voice. My Kiwi accent needs this so bad.
archon810 3y ago
Sneak peek*

detourdog 3y ago

Just yesterday I started using a new maxed out Mac mini and everything about it is snappy. I have no doubt that it is ready for enormous amount of background processing. Heavy background work is the only way to use the processing power in that little computer.

samwillis 3y ago
Think Siri+ChatGPT trained on all your email, documents, browser history, messages, movements, everything. All local, no cloud, complete privacy. "Hey Siri, I had a meeting last summer in New York about project X, could you bring up all relevant documents and give me a brief summary of what we discussed and decisions we made. Oh and while you're at it, we ate at an awesome restaurant that evening, can you book a table for me for our meeting next week."
- _xnmw 3y ago
  This is already available with Rewind.ai + their GPT plugin and I've already asked such questions and got good answers.
  - edgyquant 3y ago
    Thats not what’s being discussed though, which is Siri having these abilities built in
  - samwillis 3y ago
    Rewind.ai looks awesome, but only like 10% of what I'm thinking Apple could do.
- lotsofpulp 3y ago
  All of my current experience with Siri tells me there is a 50-50 chance of the result coming back as “Sorry, I have having trouble connecting to the network” or playing a random song from Apple Music. Just last night, we were entertaining our toddler with animal sounds. It worked with “Hey Siri, what does a goat sound like?”, then we were able to do horse, cow, sheep, boar, and it somehow got tripped up on pig, for which it responded with the Wikipedia entry and told us to look at the phone for more info.
  - iknowstuff 3y ago
    [flagged]
    - lotsofpulp 3y ago
      I thought the lack of ability to execute on current “easy” queries would indicate something about ability to execute something as complicated as figuring out the restaurant you ate at and making a reservation. At least anytime in the next few years.
      - mason55 3y ago
        Siri today is built on what’s essentially completely different concepts from something like ChatGPT. There are demos of using ChatGPT to turn normal English into Alexa commands and it’s pretty flawless. If you assume Apple can pretty easily leverage LLM tech on Siri and do it locally via silicon in the M3 or M4, it’s only a matter of chip lead time before Siri has multiple orders of magnitude improvement.
        lotsofpulp 3y ago
        That would be a pleasant surprise!
        rschoultz 3y ago
        That experience likely isn’t transferable to Siri, that has deeper problems. People, me included, are reporting their problems with Siri, e.g. setting it to transcribing what they and Siri says as text on the screen, and then are able to show that given input as “Please add milk to the shopping list” results in Siri responding “I do not understand what speaker you refer to.”, in writing. Likely problems like these could be overcome, but preparing better input would probably not address the root cause of the problems with Siri.
- blitzar 3y ago
  "Playing Our Last Summer by ABBA on Apple Music"
- addandsubtract 3y ago
  Fat chance Apple will alow us to do this locally. More like, upgrade to Apple Cloud Plus to get these features. But yeah, I've also dreamt of what my Apple hardware could do.
  - fastball 3y ago
    Apple already does all existing AI stuff locally. The main one being categorizing images in your Photos library.
  - microtonal 3y ago
    Why not? They do on-device training of face recognition: https://machinelearning.apple.com/research/recognizing-people-photos https://machinelearning.apple.com/research/recognizing-peopl...
  - insane_dreamer 3y ago
    Actually Apple is much more device-first than cloud-first
- throw384624 3y ago
  I would love if Apple created a personal iCloud device too. Let me sync my information to something local. I know there's Nextcloud, but it's not as seamless as iCloud.
  - tough 3y ago
    We should get the European Union to force Apple to offer an Icloud Local server thingy
    - scarface74 3y ago
      Yes and we should also have EU regulators at every design meeting for every company. They did such a good job with the GDPR making the user experience better on the web
      - another2another 3y ago
        Yes, alas they didn't leave room for a 'cookie preferences' cookie, so that whenever I choose the option 'reject all', it's of course going to ask me again, every time I visit the website. saying that, their intentions were good, I'm always horrifically amazed at the number of cookies used whenever I see the preferences popup. I honestly had no idea how many tracking cookies were used by the average website.
- nicbou 3y ago
  I love Steve Jobs' "bicycle for the mind" metaphor, and what you describe is the best possible example of this concept. A computer that does that would enable us to do so much more. This is the sort of AI I want; a true personal assistant, not a bullshit generator.
  - y04nn 3y ago
    It appears that we are tantalizingly close to have the perfect voice assistant. But for some inexplicable reason, it does not exist yet. Siri was introduced over a decade ago, and it seems that its development has not progressed as anticipated. Meanwhile, language models have made significant advancements. I am uncertain as to what is preventing Apple, a company with boundless resources, from enhancing Siri. Perhaps it is the absence of competition and the duopoly maintained by Apple and Google, both of whom seem reluctant to engage in a competitive battle within this domain.
    - lamontcg 3y ago
      It is probably a people problem. The people who really understood Siri have probably left, the managers left running it are scored primarily on not making any mistakes and staying off the headlines. Any engineers who understand what it would take to upgrade it aren't given the resources and spend their days on maintenance tasks that nobody really sees.
      - slowmovintarget 3y ago
        It's more likely a perverse incentive problem. Voice activated "assistants" weren't viewed as assistance for end users. They were universally viewed as one of two things: A way of treating the consumer as a product, or a feature check-box. That Siri went from useful to far less useful had more to do with the aim to push products at you rather than actually accomplishing the task you set for Siri. If Apple actually delivers an assistant that works locally, doesn't make me the product, and generally makes it easier to accomplish my tasks, then that's a product worth paying for. When anyone asks "who benefits from 'AI'?" the answer is almost invariably "the people running the AI." Microsoft and OpenAI get more user data, and subscriptions. Google gets another vehicle for attention-injection. But if I run Vicuna or Alpaca (or some eventual equivalent) on my hardware, I can ensure I get what I need, and that there's much less hijacking of my intentions. So Microsoft, if you're listening: I don't want Bing Chat search, I want Cortana Local.
- bredren 3y ago
  > All local, no cloud, complete privacy. This is already felt in use of Stable Diffusion, where M2 is fully capable offline. Anything that can be done to reduce the need to “dial out” for processing protects the individual. It erodes the ability of business and governmental organizations to use knowledge of otherwise private matters to target and influence. The potential of moving a HQ LLM like GPT to the edge to answer everyday questions reminds me of my move from Google to DDG as my default search engine. Except it’s even a bigger deal than that. It reduces private data exhaust from search to zero, making going to the net a backup plan instead of a necessity. Apple delivering this on device is a major threat to OpenAI, which will have to provide some LLM model with training that Apple can’t or won’t. Savvy users will begin to leer at having to produce queries over the wire, feeding valuable data (proven by ShareGPT) Even then, Apple will likely chose to or be forced to open up on device AI to allow user contributed apps like LORAs which would ask the question why does OpenAI need to exist? Also fascinating the potential to do this at the Server level for enterprise. If Apple produced a stack for enterprise training it could replace generalized data compute needs, shifting IT back to local or intranet.
- mattl 3y ago
  Sounds like Knowledge Navigator https://youtu.be/umJsITGzXd0 https://youtu.be/umJsITGzXd0
  - detourdog 3y ago
    My senior thesis was influenced by that.
- sroussey 3y ago
  I’m working on something very much like this!
- tomcam 3y ago
  Apparently, you are not an actual user of Siri, because I get jack shit out of her. speech to text is infinitely worse than the first week Siri was released.
  - ceejayoz 3y ago
    Siri has felt like abandonware for years. My hope is something like a ChatGPT model is its replacement soon.
- precompute 3y ago
  >Think Siri+ChatGPT trained on all your email, documents, browser history, messages, movements, everything. All local, no cloud, complete privacy. That sounds absolutely horrifying if you remove the "all local" part. And that part's a pipe dream anyway. Plus, when using a model you'd basically become subservient / limited to the type of data in the model, which would necessarily abide by Apple's TOS, so a couple of hundred million people would be the Apple TOS but in human form. I don't understand why apple fanboys don't get this. Apple is pretty shoddy when privacy is concerned. Are these apple employees making these posts?
hammock 3y ago
Do you think it would be good/capable for audio or video editing?
- detourdog 3y ago
  I think so. It is stunningly fast. It may be the biggest boost I have seen. The last big boost I saw was the move to SSD.
Maursault 3y ago
> Just yesterday I started using a new maxed out Mac mini and everything about it is snappy. Really?! I didn't think anyone here would fall for that. Mac Mini 12-core M2, 19-core GPU, 32GB, 10Gbit, 8TB storage? $4500 Mac Studio 20-core M1, 48-core GPU, 64GB, 10Gbit, 1TB storage is $4000. 128GB of RAM is $800 more but either Studio RAM configuration obviously spanks the M2 mini. It's sacrificing Apple's expensive storage, but with Thunderbolt 3 it's pretty academic to find 8TB or more of NVMe storage, probably 32GB of NVMe RAID[1], for less than Apple's charge of $2200 above cost of 1TB. [1] https://eshop.macsales.com/shop/express-4m2 https://eshop.macsales.com/shop/express-4m2
- detourdog 3y ago
  I specced the smallest SSD. I use netwomr homes. The mini is a stop gap waiting for the pro. Drive size Indont really consider a performance item anymore. I spent just over $2,000. Mac mini With the following configuration: Apple M2 Pro with 12‑core CPU, 19-core GPU, 16‑core Neural Engine 32GB unified memory 512GB SSD storage Four Thunderbolt 4 ports, HDMI port, two USB‑A ports, headphone jack 10 Gigabit Ethernet Im satauisfied.
  - Maursault 3y ago
    Not awful, but for $2K you could have had 16-core CPU, 20-core GPU, 32-core Neural Engine, 48GB unified memory, 512K SSD storage, Four Thunderbolt 4 ports, two HDMI ports, four USB-A ports, two headphone jacks, two Gigabit Ethernet.
    - detourdog 3y ago
      Yes. I wanted the 10Gbt Ethernet. My purchasing question is when is the right time to buy a great monitor. In the CRT days the monitor lasted the longest and buying the best one could afford worked for me.
      - detourdog 3y ago
        I just went back to compare the Mini with the Studio again. Despite your advice I would buy the Mini again for these reasons: I'm on a newer generation chip that has a lower power draw. Meets my network speed minimum. All for the price of the entry level Studio. This box is basically an experiment to see how much processing power I need. I have a very specific project that will require the benchmarking of Apple's machine learning frameworks. I want to see how much of a machine learning load this Mini can handle. Once I have benchmarks maybe the Pro will exist and I will be in good shape to shop and understand what I'm buying. I think a Mini of any spec is a great value. The studio has a place but I'm hoping the Pro ends up being like an old Sun E450. This Mini experiment is to help me frame the hardware power vs. the software loads.

brookst 3y ago

UMA may turn out to be visionary. I really wonder if they saw the AI/ML trend or just lucked out. Either way, the apple silicon arch is looking very strong for local AI. It’s a lot easier to beef up the NPU than to redo memory arch.

smoldesu 3y ago
I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. All of them are pretty weak for local training. But having reasonably powerful inferencing hardware isn't very hard at all, UMA doesn't seem very visionary to me in an era of MMAPed AI models.
- microtonal 3y ago
  I think pretty much any multicore ARM CPU with a post ARMv8 ISA is looking pretty strong for local AI right now. Same goes for x86 chips with AVX2 support. Apple Silicon AMX units provide the matrix multiplication performance of many core CPUs or faster at a fraction of the wattage. See eg. https://explosion.ai/blog/metal-performance-shaders https://explosion.ai/blog/metal-performance-shaders https://github.com/danieldk/gemm-benchmark#1-to-16-threads https://github.com/danieldk/gemm-benchmark#1-to-16-threads
  - smoldesu 3y ago
    Yes, and generic multicore ARM CPUs can run ARM's standard compute library regardless of their hardware: https://github.com/ARM-software/armnn https://github.com/ARM-software/armnn Plus, the benchmark you've linked to is comparing hardware accelerated inferencing to the notoriously crippled MKL execution. A more appropriate comparison would test Apple's AMX units against the Ryzen's AVX-optimized inferencing.
    - microtonal 3y ago
      Nope, the benchmarks are done by disabling the MKL AMD cripple (I did these benchmarks). It’s not faster with eg. AMD BLIS.

ec109685 3y ago

The question is whether there will be models that can’t fit into an iPhone that apple will miss out on because they find cloud based personalization so abhorrent. Agree these are tremendously good features and having them run locally will provide the best possible experience.

samwillis 3y ago
My hope for the surprise is an almost order of magnitude increase in memory on these chips. That wold be transformative for local AI.
- fastball 3y ago
  Order of magnitude increase in memory without a corresponding increase in size is a pipe dream if I've ever heard one.
  - samwillis 3y ago
    We're all aloud our pipe dreams, the future is built on them.
- astrange 3y ago
  Have you tried using less memory?
reaperducer 3y ago
The question is whether there will be models that can’t fit into an iPhone that apple will miss out on Fortunately, Apple isn't an adolescent, so it doesn't suffer from FOMO. because they find cloud based personalization so abhorrent. I'm not even sure what "cloud based personalization" means to the user, other than "Hoover up all of your personal information."
- whimsicalism 3y ago
  > I'm not even sure what "cloud based personalization" means to the user, other than "Hoover up all of your personal information." It means having actually good ML. I see so many posts around here saying Apple is absolutely well positioned to dominate in ML. It's just not true. Nobody who is a top AI player wants to work at Apple where they have few if any AI products, no data, don't pay particularly well, not a big research culture, etc. etc. The only thing they have going for them in this space is a good ARM architecture for low power matrix multiplication.

huijzer 3y ago

Sounds plausible. Also due to the news yesterday that Apple uses 90% of TSMC‘s 3nm space in 2023 [1]. Whereas everyone is talking about a recession, Apple seems to see opportunities. Or maybe they just had too much cash on hand. Also possible. [1]: https://news.ycombinator.com/item?id=35947339 https://news.ycombinator.com/item?id=35947339

smoldesu 3y ago
Density doesn't always matter. I'm reminded of Apple's 5nm M1 Ultra struggling to keep up with Nvidia's 10nm RTX 3080 in standard use. Having such a minor node advantage won't necessarily save them here, especially since Nvidia's currently occupying the TSMC 4nm supply.
- deleted 3y ago
  [deleted]
- valine 3y ago
  Nvidia is somewhat encumbered by their need to optimize for raster performance. Ideally, all those transistors should be going toward tensor cores. Apple has never really taken the gaming market seriously. If they wanted to, they could ship their next M3 chip with identical GPU performance and use all that new 3nm die space for AI accelerators.
- Someone 3y ago
  > Having such a minor node advantage Is that a minor advantage? I would think that, the smaller the nodes get, the larger the impact of a 1nm difference. Because transistors have area, I think the math, in ≈transistor count would be 3nm:4nm = ⅓²:¼², and that’s 1,777… so a 3nm node could have 75% more transistors on a given die area than a 4nm one (roughly).
  - Kirby64 3y ago
    4nm -> 3nm no longer means size goes down as a result directly ratiometricly. You have to look at what TSMC is claiming for their improvements. They're claiming 5nm -> 3nm is a 70% density improvement (I can't find any 4nm -> 3nm claims)... so 4->3 must be much less. Also, most folks seem to have gone directly from 5nm to 3nm, and skipped 4nm altogether.

mupuff1234 3y ago

Do users actually care whether something is local or not? (Edit: In terms of privacy as there are benefits in terms of speed and offline work) It's not like we're not already storing all of our media on the cloud (including voice), passwords and other sensitive data.

heyjamesknight 3y ago
I do, when I’m without a quality internet connection and a basic request to Siri to turn on a timer fails.
criddell 3y ago
A lot of end users do not and they have no interest in spending the time figuring it out. That's why it's very important that the companies behind the technology we use make ethical choices that are good for their users and when that doesn't happen, legislators need to step in. Apple has been on both sides of that coin and what is ethical isn't always clear.
- fastball 3y ago
  > companies need to be ethical > what is ethical isn't always clear So...?
  - criddell 3y ago
    So communicate. Have guiding principles for the company. Be open about the choices you make and why you make them. Listen and respond to criticism.
    - otterley 3y ago
      Apple has one: "Privacy is a fundamental human right."
      - LocalH 3y ago
        Not for their employees, otherwise they'd develop a setup to keep personal and company information separate internally. https://www.theverge.com/22648265/apple-employee-privacy-icloud-id https://www.theverge.com/22648265/apple-employee-privacy-icl...

jayd16 3y ago

Is it really that unique a position? Doesn't Google's pixel phones have a neural core and similar architecture as well?

Grustaf 3y ago
The uniqueness is not in hardware it’s in trust. And the ability to make software for normal human beings.
- JustSomeNobody 3y ago
  Trust. That's an interesting word. Personally, I don't trust corporations. Their motive is always money.
  - reaperducer 3y ago
    Personally, I don't trust corporations. How do you eat?
    - deleted 3y ago
      [deleted]
    - nawgz 3y ago
      Because a food corporation that sells food that kills people is unlikely to make piles of money.
      - paganel 3y ago
        Not for lack of trying. If anyone were to write a chronological history of regulations imposed by different authorities throughoug history I think that it is a fair assumption to make that regulations related to making bread would already show up in the first chapters of the book.
    - JustSomeNobody 3y ago
      Slowly, chewing my food completely.

uncletammy 3y ago

> They are in the unique position to enable local AI tools, such as assistance or text generation, without the difficulties and privacy concerns with the cloud. I don't see why client-side processing mitigates the privacy concerns. That doesn't stop Apple from "locally" spying on you then later sending that data to their servers.

marricks 3y ago
Ok, sure, but surely you see how it is that much harder to do? Also since Apple is built around selling expensive devices and services you could also see why they’d have much less incentive to spy and collect data than, say, Google or Facebook? The cynicism of “everything is equally bad so why care” is destructive.
- SoftTalker 3y ago
  Now. It was just two decades ago that Apple was on life support. That could happen again. And the temptation would be much stronger to start monitizing their user's data.
  - marricks 3y ago
    Very good point: if things changed drastically they could change drastically.
aardvarkr 3y ago
Nope but that is something that can be monitored and investigated, pretty easily I might add.

yyyk 3y ago

>They are in the unique position to enable local AI tools The only unique Apple thing here is how bad their AI products here and how behind they are in AI. This is the only thing that matters here - performance is adequate or better for the other processors out there, but you can't get anywhere without the appropriate software. Maybe they'll get smart enough to buy some AI startups/companies to get the missing talent.

andy_ppp 3y ago
Which AI products that they have actually implemented are bad? I think Siri is pretty poor to be fair and improves at a glacial pace. Pretty much everything else I'd say is state of the art from things like text selection from images, cutting out of image subject, their computational photography, even Map directions have come a long way.
- yyyk 3y ago
  When people talk about AI they mean the new tech like LLM or Diffusion, and the only relevant Apple offering (Siri) is way behind and there's no evidence they have anything to replace it. (Aside, their image manipulation and Map is worse - though with Maps I dunno what's the underlying issue, and OCR was already mostly solved. I'm far from a photography expert so can't compare there).
  - brookst 3y ago
    With Apple it is always good to draw the distinction (as you did) between them being behind and there being no evidence of them not being behind.
    - yyyk 3y ago
      True that. But I won't give Apple credit for products we can't see or assume good performance without proof. As they say in the movies: "Show me the money". I'll say though that no multibillion company is under existential threat. Not Apple, not Google and not even Intel. At worst they will lose a couple tens of billions and some marketshare. Even IBM still exists and took a long long time to fall to where it is still today.

cubefox 3y ago

> Apple silicone is primed for local AI models with its GPU, Neural Cores and the unified memory architecture. Is there any information on how they are doing with respect to local AI compared to other Smartphone SoCs? (Snapdragon, Tensor, Kirin, Helio etc)

xnx 3y ago

> They are in the unique position to enable local AI tools Nothing special about Apple with regards to AI. M2 beats x86 in power efficiency, but not significantly better than other ARM processors.

shepherdjerred 3y ago
I think the parent is referring to the Apple Neural Engine in Apple Silicon, which aren't widely used today (as far as I know)
gcr 3y ago
Some workloads on M1 absolutely smash other ARM processors in part because of M1's special-purpose hardware. In particular, the undocumented AMX chip is really nice for distance matrix calculations, vector search, embeddings, etc. Non-scientific example: for inference, whisper.cpp links with Accelerate.framework to do fast matrix multiplies. On M1, one configuration gets ~6x realtime speed, but on a very beefy AWS Gravatron processor, the same configuration only achieves 0.5x realtime, even after choosing an optimal threadcount, even linking with NEON-optimized BLAS. (Maybe I'm doing something wrong though).

rodgerd 3y ago

> I think there is an important little nod to the future in this announcement. "Personal Voice" is training (likely fine tuning) and then running a local AI model to generate the user's voice. My Kiwi accent needs this so bad.

archon810 3y ago

Sneak peek*