35 ms·

Together with next-generation ML accelerators in the CPU, the high-performance GPU, and higher-bandwidth unified memory, the Neural Engine makes M4 an outrageou

• krunck 2y ago
  Yes, that would be great. But without the ability for us to verify this who's to say they won't use the edge resources(your computer and electricity) to process data(your data) and then send the results to their data center? It would certainly save them a lot of money.
  • astrange 2y ago
    You seem to be describing face recognition in Photos like it's a conspiracy against you. You'd prefer the data center servers looking at your data?
  • IggleSniggle 2y ago
    When you can do all inference at the edge, you can keep it disconnected from the network if you don't trust the data handling. I happen to think they wouldn't, simply because sending this data back to Apple in any form that they could digest it is not aligned with their current privacy-first strategies. But if they make a device that still works if it stays disconnected, the neat thing is that you can just...keep it disconnected. You don't have to trust them.
    • chem83 2y ago
      Except that's an unreasonable scenario for a smart phone. It doesn't prove that the minute the user goes online it won't be egressing data willingly or not.
      • IggleSniggle 2y ago
        I don't disagree, although when I composed my comment I had desktop/laptop in mind, as I think genuinely useful on-device smartphone-AI is a ways of yet, and who knows what company Apple will be by then.
    • bee_rider 2y ago
      To use a proprietary system and not trust the vendor, you have to never connect it. That’s possible of course, but it seems pretty limiting, right?
  • chem83 2y ago
    +1 The idea that it's on device, hence it's privacy-preserving is Apple's marketing machine speaking and that doesn't fly anymore. They have to do better to convince any security and privacy expert worth their salt that their claims and guarantees can be independently verified on behalf of iOS users. Google did some of that on Android, which means open-sourcing their on-device TEE implementation, publishing a paper about it etc.
  • robbomacrae 2y ago
    They already do this. It's called federated learning and its a way for them to use your data to help personalize the model for you and also (to a much lesser extent) the global model for everyone whilst still respecting your data privacy. It's not to save money, it's so they can keep your data private on device and still use ML. https://www.technologyreview.com/2019/12/11/131629/apple-ai-personalizes-siri-federated-learning/ https://www.technologyreview.com/2019/12/11/131629/apple-ai-...
  • victorbjorklund 2y ago
    If you trust that Apple doesn't film you with the camera when you use the phone while sitting on the toilet. Why wouldn't you trust Apple now? It would have to be a huge conspiracy with all Apples employees. And you can easily just listen to the network and see if they do it or not.
    • xanderlewis 2y ago
      I find it somewhat hard to believe that wouldn’t be in contravention of some law or other. Or am I wrong? Of course we can then worry that companies are breaking the law, but you have to draw the line somewhere… and what have they to gain anyway?
• joelthelion 2y ago
  >n case it is not abundantly clear by now: Apple's AI strategy is to put inference (and longer term even learning) I'm curious: is anyone seriously using apple hardware to train Ai models at the moment? Obviously not the big players, but I imagine it might be a viable option for Ai engineers in smaller, less ambitious companies.
  • andrewmcwatters 2y ago
    Yes, it can be more cost effective for smaller businesses to do all their work on Mac Studios, versus having a dedicated Nvidia rig plus Apple or Linux hardware for your workstation. Honestly, you can train basic models just fine on M-Series Max MacBook Pros.
    • nightski 2y ago
      A decked out Mac Studio is like $7k for far less GPU power. I find that highly unlikely.
      • inciampati 2y ago
        But you get access to a very large amount of RAM for that price.
      • softfalcon 2y ago
        Don't attack me, I'm not disagreeing with you that an nVidia GPU is far superior at that price point. I simply want to point out that these folks don't really care about that. They want a Mac for more reasons than "performance per watt/dollar" and if it's "good enough", they'll pay that Apple tax. Yes, yes, I know, it's frustrating and they could get better Linux + GPU goodness with an nVidia PC running Ubuntu/Arch/Debian, but macOS is painless for the average science AI/ML training person to set up and work with. There are also known enterprise OS management solutions that business folks will happily sign off on. Also, $7000 is chump change in the land of "can I get this AI/ML dev to just get to work on my GPT model I'm using to convince some VC's to give me $25-500 million?" tldr; they're gonna buy a Mac cause it's a Mac and they want a Mac and their business uses Mac's. No amount of "but my nVidia GPU = better" is ever going to convince them otherwise as long as there is a "sort of" reasonable price point inside Apple's ecosystem.
• legitster 2y ago
  > This is completely coherent with their privacy-first strategy (which would be at odds with sending data up to the cloud for processing). I feel like people are being a bit naïve here. Apple's "Privacy First" strategy was a marketing spin developed in response to being dead-last in web-development/cloud computing/smart features. Apple has had no problem changing their standards by 180 degrees and being blatantly anti-consumer whenever they have a competitive advantage to do so.
  • seec 2y ago
    Don't bother the fanboys have an Apple can't do anything wrong/malicious. At this point it's closer to a religion than ever. You would be amazed at the response of some of them when I point out some shit Apple does that make their products clearly lacking for the price, the cognitive dissonance is so strong they don't know how to react in any other way than lying or pretending it doesn't matter.
    • acdha 2y ago
      If you’re annoyed about quasi-religious behavior, consider that your comment has nothing quantifiable and contributed nothing to this thread other than letting us know that you don’t like Apple products for non-specific reasons. Maybe you could try to model the better behavior you want to see?
      • seec 2y ago
        How do you even come to the conclusion that I don't like Apple products? I have a phone, watch and computer from them. It's not like I hate the products. I have very specific reasons to be annoyed but they are far too many to list them all in a simple post. I was working with and buying Apple stuff before the turn of the millenium and I have worked as an Apple technician and helped way more Apple users than I could care to list. This is my experience and your comment precisely illustrates that. My comment was about the delusion of privacy first marketing bullshit that they came up with to excuse the limitations of some of their stuff. But since Apple can't do anything wrong, we are not going anywhere. Whatever, keep on believing.
• croes 2y ago
  It isn't privacy if Apple knows. They are the gatekeeper of your data for their benefit not yours.
  • jajko 2y ago
    Yes at the end its just some data representing user's trained model. Is there a contractual agreement with users that apple will never ever transfer a single byte of those, otherwise huge penalties will happen? If not, its pinky PR promise that sounds nice.
    • threeseed 2y ago
      Apple publicly documents their privacy and security practices. At minimum, laws around the world prevent companies from knowingly communicating false information to consumers. And in many countries the rules around privacy are much more stringent.
      • croes 2y ago
        I bet Boeing also has documentation about their security practices. Talk is cheap and in Apple's case it's part of their PR.
        • bamboozled 2y ago
          What is wrong with Boeing's security?
        • dudeinjapan 2y ago
          > What is wrong with Boeing's security? Too many holes.
        • threeseed 2y ago
          But what does that have to do with the price of milk in Turkmenistan. Because Boeing's issues have nothing to do with privacy or security and since they are not consumer facing have no relevance to what we are talking about.
• dheera 2y ago
  > This is completely coherent with their privacy-first strategy Apple has never been privacy-first in practice. They give you the illusion of privacy but in reality it's a closed-source system and you are forced to trust Apple with your data. They also make it a LOT harder than Android to execute your own MITM proxies to inspect what exact data is being sent about you by all of your apps including the OS itself.
  • deadmutex 2y ago
    Yeah, given that they resisted putting RCS in iMessage so long, I am a bit skeptical about the whole privacy narrative. Especially when Apple's profit is at odds with user privacy.
    • notaustinpowers 2y ago
      From my understanding, the reason RCS was delayed is because Google's RCS was E2EE only in certain cases (both users using RCS). But also because Google's RCS runs through Google servers. If Apple enabled RCS in messages back then, but the recipient was not using RCS, then Google now has the decrypted text message, even when RCS advertises itself as E2EE. With iMessage, at least I know all of my messages are E2EE when I see a blue bubble. Even now, RCS is available on Android if using Google Messages. Yes, it's pre-installed on all phones, but OEMs aren't required to use it as the default. It opens up more privacy concerns because now I don't know if my messages are secure. At least with the green bubbles, I can assume that anything I send is not encrypted. With RCS, I can't be certain unless I verify the messaging app the recipient is using and hope they don't replace it with something else that doesn't support RCS.
      • vel0city 2y ago
        You know what would really help Apple customers increase their privacy when communicating with non-Apple devices? Having iMessage available to everyone regardless of their mobile OS.
        • 2y ago
• sergiotapia 2y ago
  > privacy-first strategy That's just their way of walled gardening apple customers. Then they can extort devs and other companies dry without any middle-men.
• MyFirstSass 2y ago
  I've been saying the same thing since ANE and the incredible new chips with shared ram, suddenly everyone could run capable local models - but then Apple decided to be catastrophically stingy once again putting ridiculous 8gb's of ram in these new iPads' and their new macbook air's destroying having a widespread "intelligent local siri" because now half the new generation can't run anything. Apple is an amazing powerhouse but also disgustingly elitist and wasteful if not straight up vulgar in its profit motives. There's really zero idealism there despite their romantic and creative legacy. There's always some straight idiotic limitations in their otherwise incredible machines, with no other purpose than to create planned obsolescence, "PRO" exclusivity and piles e-waste.
• KaiserPro 2y ago
  > This is completely coherent with their privacy-first strategy (which would be at odds with sending data up to the cloud for processing). I mean yeah, that makes good marketing copy, but its more due to reducing latency and keeping running costs down. but as this is mostly marketing fluff we'll need to actually see how it performs before casting judgment on how "revolutionary" it is.
• lunfard000 2y ago
  Prob beacuse they are like super-behind in the cloud space, it is not like they wouldn't like to sell the service. They ignored photos privacy quite a few times in the icloud.
  • dylan604 2y ago
    is it surprising since they effectively given the finger to data center hardware designs?
• jablongo 2y ago
  So for hardware accelerated training with something like PyTorch, does anyone have a good comparison between Metal vs Cuda, both in terms of performance and capabilities?
• s1k3s 2y ago
  For everyone else who doesn't understand what this means, he's saying Apple wants you to be able to run models on their devices, just like you've been doing on nvidia cards for a while.
  • nomel 2y ago
    I think he's saying they want to make local AI a first class, default, capability, which is very unlike buying a $1k peripheral to enable it. At this point (though everyone seems to be working on it), other companies need to include a gaming GPU in every laptop, and tablet now (lol), to enable this.
  • Petersipoi 2y ago
    Awesome. I'm going to go tell my mom she can just pull her Nvidia card out of her pocket at the train station to run some models. On second thought.. maybe it isn't "just like you've been doing on Nvidia cards for a while"
• andsoitis 2y ago
  > In case it is not abundantly clear by now: Apple's AI strategy is to put inference (and longer term even learning) on edge devices. This is completely coherent with their privacy-first strategy (which would be at odds with sending data up to the cloud for processing). Their primary business goal is to sell hardware. Yes, they’ve diversified into services and being a shopping mall for all, but it is about selling luxury hardware. The promise of privacy is one way in which they position themselves, but I would not bet the bank on that being true forever.
  • ActorNightly 2y ago
    >The promise of privacy is one way in which they position themselves, but I would not bet the bank on that being true forever. Everyone seems to have forgotten about the Celebrity iCloud photo leak.
  • serial_dev 2y ago
    It doesn't need to stay true forever. The alternative is Google / Android devices and OpenAI wrapper apps, both of which usually offer a half baked UI, poor privacy practices, and a completely broken UX when the internet connection isn't perfect. Pair this with the completely subpar Android apps, Google dropping support for an app about once a month, and suddenly I'm okay with the lesser of two evils. I know they aren't running a charity, I even hypothesized that Apple just can't build good services so they pivoted to focusing on this fake "privacy" angle. In the end, iPhones are likely going to be better for edge AI than whatever is out there, so I'm looking forward to this.
    • jocaal 2y ago
      > better for edge AI than whatever is out there, so I'm looking forward to this What exactly are you expecting? The current hype for AI is large language models. The word 'large' has a certain meaning in that context. Much larger that can fit on your phone. Everyone is going crazy about edge AI, what am I missing?
      • jchanimal 2y ago
• SpaceManNabs 2y ago
  This comment is odd. I wouldn't say it is misleading, but it is odd because it borders on such definition. > Apple's AI strategy is to put inference (and longer term even learning) on edge devices This is pretty much everyone's strategy. Model distillation is huge because of this. This goes in line with federated learning. This goes in line with model pruning too. And parameter efficient tuning and fine tuning and prompt learning etc. > This is completely coherent with their privacy-first strategy Apple's marketing for their current approach is privacy-first. They are not privacy first. If they were privacy first, you would not be able to use app tracking data on their first party ad platform. They shut it off for everyone else but themselves. Apple's approach is walled garden first. > Processing data at the edge also makes for the best possible user experience because of the complete independence of network connectivity as long as you don't depend on graph centric problems where keeping a local copy of that graph is prohibitive. Graph problems will become more common. Not sure if this is a problem for apple though. I am just commenting in general. > If (and that's a big if) they keep their APIs open to run any kind of AI workload on their chips Apple does not have a good track record of this; they are quite antagonistic when it comes to this topic. Gaming on apple was dead for nearly a decade (and pretty much still is) because steve jobs did not want people gaming on macs. Apple has eased up on this, but it very much seems that if they want you to use their devices (not yours) in a certain way, then they make it expensive to do anything else. Tbf, I don't blame apple for any of this. It is their strategy. Whether it works or not, it doesn't matter. I just found this comment really odd since it almost seemed like evangelism. edit: weird to praise apple for on device training when it is not publicly known if they have trained any substantial model even on cloud.
  • nomel 2y ago
    > This is pretty much everyone's strategy. I think this is being too charitable on the state of "everyone". It's everyone's goal. Apple is actively achieving that goal, with their many year strategy of in house silicon/features.
    • SpaceManNabs 2y ago
      > Apple is actively achieving that goal, with their many year strategy of in house silicon/features So are other companies, with their many year strategy of actually building models that accessible to the public. yet Apple is "actively" achieving the goal without any distinct models.
• strangescript 2y ago
  The fundamental problem with this strategy is model size. I want all my apps to be privacy first with local models, but there is no way they can share models in any kind of coherent way. Especially when good apps are going to fine tune their models. Every app is going to be 3GB+
  • tyho 2y ago
    Foundation models will be the new .so files.
    • flawsofar 2y ago
      And fine tuning datasets will be compressed and sent rather than the whole model
    • strangescript 2y ago
      This would be interesting but also feels a little restrictive. Maybe something like LoRa could bridge the capability gap but if a competitor then drops a much more capable model then you either have to ignore it or bring it into your app. (assuming companies won't easily share all their models for this kind of effort)
  • SpaceManNabs 2y ago
    I don't think HN understands how important model distillation still is for federated learning. Hype >> substance ITT
  • Havoc 2y ago
    You could always mix and match. Do lighter task on device and outsource to cloud if needed
  • mr_toad 2y ago
    Gemini nano is 1.8B 4 bit parameters, so a little under a GB. And hopefully each app won’t include a full copy of their models.
  • int_19h 2y ago
• kernal 2y ago
  > This is completely coherent with their privacy-first strategy How can they have a privacy first strategy when they operate an Ad network and have their Chinese data centers run by state controlled companies?
  • KerrAvon 2y ago
    How can I have mint choc and pineapple swirl ice cream when there are children starving in Africa?
    • edm0nd 2y ago
      Fuck them kids. It's delicious af.
  • n9 2y ago
    ... I think that the more correct assertion would be that Apple is a sector leader in privacy. If only because their competitors make no bones about violating the privacy of their customers as it is the basis of thier business model. So it's not that Apple is A+ so much as the other students are getting Ds and Fs.
• macns 2y ago
  > This is completely coherent with their privacy-first strategy You mean .. with their said privacy-first strategy
• ptman 2y ago
  Apple privacy is marketing https://www.eurekalert.org/news-releases/1039938 https://www.eurekalert.org/news-releases/1039938
• xipix 2y ago
  How is local more private? Whether AI runs on my phone or in a data center I still have to trust third parties to respect my data. That leaves only latency and connectivity as possible reasons to wish for endpoint AI.
  • chatmasta 2y ago
    If you can run AI in airplane mode, you are not trusting any third party, at least until you reconnect to the Internet. Even if the model was malware, it wouldn’t be able to exfiltrate any data prior to reconnecting. You’re trusting the third party at training time, to build the model. But you’re not trusting it at inference time (or at least, you don’t have to, since you can airgap inference).
• thefourthchime 2y ago
  Yes is it completely clear. My guess is they do something like "Siri-powered shortcuts". Where you can ask it to do a couple things and it'll dynamically create a script and execute it. I can see a smaller model trained to do that may work well enough, however, I've never seen any real working examples of this work, that rabit device is heading in that direction, but it's mostly vaporware now.
  • _boffin_ 2y ago
    Pretty much my thoughts too. Going to have a model that’s smaller than 3B built in. The’ll have tokens that directly represent functions / shortcuts.
• deleted 2y ago
  [deleted]
• LtWorf 2y ago
  > This is completely coherent with their privacy-first strategy Is this the same apple whose devices do not work at all unless you register an apple account?
  • yazzku 2y ago
    Some people really seem to be truly delusional. It's obvious that the company's "privacy" is a marketing gimmick when you consider the facts. Do people not consider the facts anymore? How does somebody appeal to the company's "privacy-first strategy" with a straight face in light of the facts? I suppose they are not aware of the advertising ID that is embedded in all Apple operating systems. That one doesn't even require login.
    • LtWorf 2y ago
      Considering the facts is much harder when admitting a mistake is involved.
      • yazzku 2y ago
        A "mistake" seems to be putting it lightly when the thing has been reiterated multiple times throughout the years, but yeah. Seems more like blind dogma. Obviously people don't like the facts pointed out to them either as you can tell by the down votes on my comment. If I am wrong, please tell me how in a reply.
    • kalleboo 2y ago
      What are the facts that people are not considering? The advertising ID is useless as of App Tracking Transparency.
  • kalleboo 2y ago
    That's not true though? I reset and set up devices for testing all the time, and you can skip logging into an Apple ID.
    • LtWorf 2y ago
• choppaface 2y ago
  On “privacy”: If Apple owned the Search app versus paying Google, and used their own ad network (which they have for App Store today), Apple will absolutely use your data and location etc to target you with ads. It can even be third party services sending ad candidates directly to your phone and then the on-device AI chooses which is relevant. Privacy is a contract not the absence of a clear business opportunity. Just look at how Apple does testing internally today. They have no more respect for human privacy than any of their competitors. They just differentiate through marketing and design.
• aiauthoritydev 2y ago
  Edge inference and cloud inference are not mutually exclusive and chances are any serious player would be dipping their toes in both.
  • MBCook 2y ago
    Right. The difference is that Apple has a ton of edge capacity, they’ve been building it for a long time. Google and Samsung have been building it too, at different speeds. Intel and AMD seem further behind (at the moment) unless the user has a strong GPU, which is especially uncommon on the most popular kind of computer: laptops. And if you’re not one of those four companies… you probably don’t have much capable consumer edge hardware.
• 7speter 2y ago
  >I personally really really welcome as I don't want the AI future to be centralised in the hands of a few powerful cloud providers. Watch out for being able to using ai on your local machine and those ai services using telemetry to send your data (recorded conversations, for instance) to their motherships.
  • happyopossum 2y ago
    > and those ai services using telemetry to send your data (recorded conversations, for instance) to their motherships This doesn’t require Ai and I am not aware of any instances of this happening today, so what exactly are we watching out for?
  • shinycode 2y ago
    I agree but for a different reason. Now the subscription is 20$ a month and the API price is accessible. What will happen when they all decide to x100 or x1000 the price of their API ? All the companies that got rid of people in favor of AI, might have lost the knowledge as well. This is dangerous and might kill a lot of companies no ?
• Powdering7082 2y ago
  Also they don't have to pay either the capex or opex costs for training a model if they get user's devices to train the models
• aiauthoritydev 2y ago
  I think these days everyone links their products with AI. Today even BP CEO linked his business with AI. Edge inference and cloud inference are not mutually exclusive choices. Any serious provider will provide both and the improvement in quality of services come from you giving more of your data to the service provider. Most people are totally fine with that and that will not change any time sooner. Privacy paranoia is mostly a fringe thing in consumer tech.
  • twototango 2y ago
    Hey could i get a source on the BP stuff please. Just curious as I couldn't find anything on the interwebs
    • aiauthoritydev 2y ago
      https://x.com/zerohedge/status/1787822478686851122 https://x.com/zerohedge/status/1787822478686851122
      • aiauthoritydev 2y ago
        https://www.reddit.com/r/stocks/comments/1cmi5gj/bp_earnings_call_we_need_70_less_coders_from/ https://www.reddit.com/r/stocks/comments/1cmi5gj/bp_earnings...
  • MBCook 2y ago
    I agree. Apple has been on this path for a while, the first processor with a Neural Engine was the A11 in 2017 or so. The path didn’t appear to change at all. The big differences today that stood out to me were adopting AI as a term (they used machine learning before) and repeating the term AI everywhere they could shove it in since that’s obviously what the street wants to hear. That’s all that was different. And I’m not surprised they emphasized it given all the weird “Apple is behind on AI“ articles that have been going around.
• gtirloni 2y ago
  > complete independence of network connectivity and hence minimal latency. Does it matter that each token takes additional milliseconds on the network if the local inference isn't fast? I don't think it does. The privacy argument makes some sense, if there's no telemetry leaking data.
• kshahkshah 2y ago
  Why is Siri still so terrible though?
• w1nst0nsm1th 2y ago
  > to put inference on edge devices... It will take a long time before you can put performant inference on edge device. Just download one of the various open source large(st) langage model and test it on your desktop... Compute power and memory and storage requirements are insane if you want decent result... I mean not just Llama gibberish. Until such requirement are satisfied, distant model are the way to go, at least for conversational model. Aside llm, AlphaGo would not run on any end user device, by a long shot, even if it is an already 'old' technology. I think 'neural engine' on end user device is just marketing nonsense at this current state of the art.
• tweetle_beetle 2y ago
  I wonder if BYOE (bring your own electricity) also plays a part in their long term vision? Data centres are expensive in terms of hardware, staffing and energy. Externalising this cost to customers saves money, but also helps to paint a green(washing) narrative. It's more meaningful to more people to say they've cut their energy consumption by x than to say they have a better server obselesence strategy, for example.
  • dyauspitr 2y ago
    Why would it be green washing? Aren’t their data centers and commercial operations run completely on renewable energy?
    • ironmagma 2y ago
      If you offload data processing to the end user, then your data center uses less energy on paper. The washing part is that work is still being done and spending energy, just outside of the data center.
      • dyauspitr 2y ago
        Which honestly is still good for the environment to have the work distributed across the entire electricity grid. That work needs to be done anyways and Apple is doing it in the cleanest way possible. What’s an alternative in your mind, just don’t do the processing? That sounds like making progress towards being green. If you’re making claims of green washing you need to be able to back it up with what alternative would actually be “green”.
        • ironmagma 2y ago
          I didn't make any claims, I just explained what the parent was saying. There could be multiple ways to make it more green: one being not doing the processing, or another perhaps just optimizing the work being done. But actually, no, you don't need a viable way to be green in order to call greenwashing "greenwashing." It can just be greenwashing, with no alternative that is actually green.
• aborsy 2y ago
  What are the example of the edge devices made by Apple?
  • floam 2y ago
    MacBook, iPhone, iPad?
    • bingbingbing777 2y ago
      So laptops are now edge devices?
      • MBCook 2y ago
        Doesn’t it just refer to the end user device as opposed to a server somewhere or a middle box?
        • Dylan16807 2y ago
          Last time I looked for the definition, nobody can agree on whether client devices count as edge or not.
      • aborsy 2y ago
        These don’t sit on the edge of the internet , and typically are not called edge devices. It’s usually a more powerful device such as a router or mini server between LAN and internet.
• dancemethis 2y ago
  Apple is privacy last, if anything. Forgotten PRISM already?
• royaltjames 2y ago
  Yes this began with the acquisition of xnor.ai. Absolutely amazing what will be done (and is being done) with edge computing.
• Pesthuf 2y ago
  Honestly, if they manage this, they have my money. But to get actually powerful models running, they need to supply the devices with enough RAM - and that's definitely not what Apple like to do.
• jonplackett 2y ago
  I think it will be a winning strategy. Lag is a real killer for LLMs. I think they'll have another LLM on a server (maybe a deal for openai/gemini) that the one on the device can use like ChatGPT uses plugins. But on device Apple have a gigantic advantage. Rabbit and Humane are good ideas humbled by shitty hardware that runs out of battery, gets too hot, has to connect to the internet to do literally anything. Apple is in a brilliant position to solve all those things. I hope they announce something good at WWDC
  • threeseed 2y ago
    There really isn't enough emphasis on the downsides of server side platforms. So many of these are only deployed in US and so if you're say in country Australia not only do you have all your traffic going to the US but it will be via slow and intermittent cellular connections. It makes using services like LLMs unusably slow. I miss the 90s and having applications and data reside locally.
    • whimsicalism 2y ago
      Unusably slow? It's like 0.3 seconds to first token and then pretty much all of the tokens can follow within a second. I find it hard to understand the edge usecase for text-based models.
      • deleted 2y ago
        [deleted]
    • EarthMephit 2y ago
      Even in Australia is the LLM lag to a server noticable? Generally an LLM seems to take about 3s or more to respond, and the network delay to the US is a couple of hundred milliseconds. The network delay seems minimal compared to the actual delay of the LLM.
  • whimsicalism 2y ago
    Having used groq and other fast LLM services a fair bit, lag seems negligible. You're literally just passing text at close to the speed of light.
• throwaway48476 2y ago
  >Apple's AI strategy is to put inference (and longer term even learning) on edge devices Ironic, given that AI requires lots of VRAM.
• lagt_t 2y ago
  Has nothing to do with privacy, google is also pushing gemini nano to the device. The sector is discovering the diminishing returns of LLMs. With the ai cores on phones they can cover your average user use cases with a light model without the server expense.
  • SpaceManNabs 2y ago
    don't bother. apple's marketing seems to have won on here. i made a similar point only for people to tell me that apple is the only org seriously pushing federated learning.
• zachbee 2y ago
  If they're doing inference on edge devices, one challenge I see is protecting model weights. If you want to deploy a proprietary model on an edge AI chip, the weights can get stolen via side-channel attacks [1]. Obviously this isn't a concern for open models, but I doubt Apple would go the open models route. [1] https://spectrum.ieee.org/how-prevent-ai-power-usage-secrets https://spectrum.ieee.org/how-prevent-ai-power-usage-secrets
  • Havoc 2y ago
    Nobody is taking particular care protecting weights for edge class models
• bmitc 2y ago
  > [Apple's] privacy-first strategy That is a marketing and advertising strategy.
  • moosemess 2y ago
    - dozens of horrific 0days cves every year because not enough is invested in security, making private virtually impossible - credit card required to install free apps such as the "private" Signal messenger - location required just to show me the weather in a static place, lol - claims to be e2e but apple controls all keys and identities - basically sells out all users' icloud data in china, and totally doesn't do the same in the US, because tim pinky swears - everything is closed source
  • jascination 2y ago
    I'm a cross-platform app developer and can assure you iOS is much stronger in terms of what data you can/can't get from a user
    • bmitc 2y ago
      That says nothing about what Apple does with data.
• cyanydeez 2y ago
  their privacy strategy is to make you feel comfortable with their tech so you don't mind when they shop it around to the highest bidder. Make no mistake, they're just waiting for the right MBA to walk through the door, see the sky high value of their users and start chop shopping that. Enshitiffication is always available to the next CEO, and this is just going to be more and more tempting as the value of the walled garden increases.
  • acdha 2y ago
    Yes, it’s possible that they’ll change in the future but that doesn’t make it inevitable. Everything you describe could have happened at any point in the last decade or two but didn’t, which suggests that it’s not “waiting for the right MBA” but an active effort to keep the abusive ones out. One thing to remember is that they understand the value of long-term investments. They aren’t going to beat Google and Facebook at advertising and have invested billions in a different model those companies can’t easily adopt, and I’m sure someone has done the math on how expensive it would be to switch.
• felixding 2y ago
  > This is completely coherent with their privacy-first strategy Do not believe what they say, watch what they do.
• SergeAx 2y ago
  Then putting only 256gb into their cheaper devices is really bad move. Even simple models like Whisper require hundreds of megabytes of storage.
• the_king 2y ago
  I'm all for running as much on the edge as possible, but we're not even close to being able to do real-time inference on Frontier models on Macs or iPads, and that's just for vanilla LLM chatbots. Low-precision Llama 3-8b is awesome, but it isn't a Claude 3 replacer, totally drains my battery, and is slow (M1 Max). Multimodal agent setups are going to be data center/home-lab only for at least the next five years. Apple isn't about to put 80GB on VRAM in an iPad for about 15 reasons.
• ChuckMcM 2y ago
  Something they should be able to do now, but do not seem to, is to allow you to train Siri to recognize exactly your voice and accent. Which is to say, to take the speech-to-text model that is listening and putting it into the Siri integration API, to both be 99.99% accurate for your speech and to recognize you and only you when it comes to invoking voice commands. It could, if it chose to, continue to recognize all voices but at the same time limit the things the non-owner could ask for based on owner preferences.
  • jilijeanlouis 2y ago
    This is really easy to do: it's just an embedding of your voice. So typically like 10/30 sec max of your voice to configure this. You already do a similar setup for faceId. I agree with you, I don't understand why they don't do it.
• eru 2y ago
  > Processing data at the edge also makes for the best possible user experience because of the complete independence of network connectivity and hence minimal latency. That's one particular set of trade-offs, but not necessarily the best. Eg if your network connection and server processing speed is sufficiently faster than your local processing speed, the latency would be higher for doing it locally. Local inference can also use more battery power. And you need a more beefy device, all else being equal.
• chipdart 2y ago
  > This is completely coherent with their privacy-first strategy (...) I think you're trying too hard to rationalize this move as pro-privacy and pro-consumer. Apple is charging a premium for hardware based on performance claims, which they need to create relevance and demand for it. There is zero demand for the capacity for running computationally demanding workloads beyond very niche applications, for what classifies as demanding for the consumer-grade hardware being sold for the past two decades. If Apple offloads these workloads to the customer's own hardware, they don't have to provide this computing capacity themselves. This means no global network of data centers, no infrastructure, no staff, no customer support, no lawyer, nothing. More importantly, Apple claims to be pro privacy but their business moves are in reality in the direction of ensuring that they are in sole control of their users' data. Call it what you want but leveraging their position to ensure they hold a monopoly over a market created over their userbase is not a pro privacy move, just like Apple's abuse of their control over the app store is not a security move.
  • deleted 2y ago
    [deleted]
• carabiner 2y ago
  iPhone Photos app already does incredible image subject search via ML locally. Versus Android which does it via cloud.
• arvinsim 2y ago
  If inference on edge devices is their goal, then they would have to rethink their pricing on storage and RAM.
• fnord77 2y ago
  They're a hardware company, so yes, they want to sell thick clients.
• ActorNightly 2y ago
  >Apple's AI strategy is to put inference (and longer term even learning) on edge devices. Apple's AI strategy is to put inference (and longer term even learning) on edge devices...only for Apple stuff. There is a big difference. ANE right now is next to useless for anything not Apple.
  • davedx 2y ago
    I think that’s going to change with WWDC
    • ActorNightly 2y ago
      Nah. Apple doesn't have incentive to provide any more dev power. It will keep things locked down and charge people for Apple branded software products. That has been their business for the past decade.
      • davedx 2y ago
        I think there's always been a tension at Apple between keeping everything as locked down as possible and opening up parts because they need the developer driven app ecosystem. My prediction is Neural Engine is going to become more useful to third party developers. I could be wrong
• edanm 2y ago
  And yet Siri is super slow because it does the processing off-device, and is far less useful than it could be because it is cobbled with restrictions. I can't even find a way to resume playing whatever Audible book I was last playing. "Siri play audible" or something. As far as I know, this is impossible to do.
• blegr 2y ago
  I hope this means AI-accelerated frameworks get better support on Mx. Unified memory and Metal are a pretty good alternative for local deep learning development.
• unusualmonkey 2y ago
  There is no guarantee that local processing is going to have lower latency than remote processing. Given the huge compute needs of some AI models (e.g. chat gpt) the time saved by using larger compute likely dwarfs the relatively small time need to transmit a request.
• wilde 2y ago
  Ehhh at this point Apple’s privacy strategy is little more than marketing. Sure they’ll push stuff to the edge to save themselves money and book the win, but they also are addicted to the billions they make selling your searches to Google. Agreed on the UX improvements though.
• neilsimp1 2y ago
  The entire software stack is non-free and closed-source. This means you'd be taking Apple at their word on "privacy". Do you trust Apple? I wouldn't, given their track record.
  • madeofpalk 2y ago
    What track record?
    • Gh0stRAT 2y ago
      They fought the FBI over unlocking iPhones when they could have just quietly complied with the request. I'd say they have a decent track record.
      • MaxBarraclough 2y ago
        They might have been thinking of the recently discovered hardware backdoor issue, CVE-2023-38606 (see also Operation Triangulation). There was surprisingly little reporting on it. Discussion: https://news.ycombinator.com/item?id=38783112 https://news.ycombinator.com/item?id=38783112 Transcript of Security Now podcast episode discussing the issue: https://www.grc.com/sn/sn-955.htm https://www.grc.com/sn/sn-955.htm
    • m463 2y ago
      Just install little snitch and you'll see how MUCH gets sent back to the mothership. and that is just macos.
• garydgregory 2y ago
  My cynical view is that doing AI on the client is the only way they can try to keep selling luxury items (jewelry really) and increasing prices for what are essentially and functionally commodity devices.
• marticode 2y ago
  Every chip coming this year (Intel, AMD, Qualcomm) has an AI processor. I am not sure Apple is doing anything special here.
• unboxingelf 2y ago
  Apple is UX-first, not privacy-first.
• ethagnawl 2y ago
  > Processing data at the edge also makes for the best possible user experience because of the complete independence of network connectivity and hence minimal latency. I know a shop who's doing this and it's a very promising approach. The ability to offload the costs of cloud GPU time is a tremendous advantage. That's to say nothing of the decreased latency, increased privacy, etc. The glaring downside is that you are dependent upon your users to be willing and able to run native apps (or possibly WASM, I'm not sure) on bleeding edge hardware. However, for some target markets (e.g. video production, photography, designers, etc.) it's a "safe" assumption that they will be using the latest and greatest Macs. I've also been hearing people talk somewhat seriously about setting up their own training/inference farms using Macs because, at least for now, they're more readily available and cheaper to buy/run than big GPUs. That comes with a host of ops problems but it still may prove worthwhile for some use cases and addresses some of the same privacy concerns as edge computing if you're able to keep data/computation in-house.
• beestripes 2y ago
  Privacy can actually be reduced with on-device ai too. Now, without actually sending any data to iCloud apple can still have a general idea of what you’re doing. Imagine a state has a law that makes certain subjects illegal to discuss. They could compel apple to have their local AI detect that content and then to broadcast a ping in the AirTags network about the user and their location. No internet connection required on the target.
• amelius 2y ago
  In case it is not abundantly clear by now: Apple's strategy is to turn developers into slaves. Run away from these temptations. You will never truly own the hardware anyway.
• demondemidi 2y ago
  Every embedded company is pushing ML at the edge with inference engines. Check out MLPerfTiny. They’ve been benchmarking all sorts of edge AI since 2019.
• msla 2y ago
  Privacy from everyone but Apple, certainly.
• msla 2y ago
  Have you seen Apple's latest ad? It's a press crushing the tools and results of human creativity: https://nitter.esmailelbob.xyz/tim_cook/status/1787864325258162239 https://nitter.esmailelbob.xyz/tim_cook/status/1787864325258...
• Pr0ject217 2y ago
  "Which would be at odds with sending data up to the cloud for processing" I don't think there is enough information here for that to be a true claim - it is possible to send the input used for inference, to the cloud, while computing the result locally. It is also possible to store the input used for inference while offline, and send that later when the device is online.