8 ms·

Apple releases MGIE, an AI-based image editing model

118 points by gnicholas 3y ago

gnicholas 3y ago
The Github repo is here: https://github.com/apple/ml-mgie https://github.com/apple/ml-mgie
- MrThoughtful 3y ago
  I have an AWS account. Is there a type of EC2 instance I can fire up and run this thing on?
  - deleted 3y ago
    [deleted]
  - cheald 3y ago
    Inference should ideally be done on an instance with a GPU, and it should have enough GPU VRAM to hold the entire model. This one is only a 7b model, so it should run pretty easily in a modest amount of vram. You can likely run it locally in CPU-only mode, too, though it's likely to be rather slow. https://instances.vantage.sh/?min_gpus=1 https://instances.vantage.sh/?min_gpus=1 (wait for it to finish loading to filter) You could try running these in Google Collab (though tweaks would have to be made to load the files), or you might try something like runpod.io, as well, which gets you GPU instances for a lot less than you'd pay with AWS - for example, a Tesla V100 (16GB) community cloud instance runs you $0.24/hr vs the g5g.xlarge which runs you $0.42/hr.
deleted 3y ago
[deleted]
ericmcer 3y ago
It feels like Apple has been very quiet about their approach in some of the future spaces (self-driving, VR/AR, AI) but then they suddenly burst out with something like Vision Pro. It seems like they are biding their time and avoiding the hype knowing that they don't need to get their first. If they can produce a superior product their brand will let them dominate those markets whenever they decide to release.
- RyanHamilton 3y ago
  What makes you think vision pro is dominating? Unlike iPhone etc it has no distinguishing feature I know of.
  - ToucanLoucan 3y ago
    Speaking as a VR enthusiast who has virtually no interest in it: The Vision Pro is an entirely new product relative to any other VR headset I know of. It offers a comprehensive set of computing features (discounting an App Store which has only really just started) that puts it roughly on par with something like iPad OS. I would say it dominates currently for VR headsets if that is what you're after, because it's basically your only option. Desktop computing in VR is possible with things like a Vive or an Index but the experience is entirely different, worse on every front, and is basically done as a workaround in case you need to operate your PC while in VR without taking off the headset. And of course, all of those headsets are (quite intelligent, but remain merely) displays: they require a fairly powerful computer to attach to. There are headsets that operate independently too, like Meta's offerings, but those are much, much, much more restrictive in terms of capability than the Vision Pro. My interest in VR is basically solely for gaming, so the Vision Pro for me is a complete miss and I have no plans to purchase one. That being said, in the realm of productivity VR, I'd say they are dominating in that they're the sole entrant to the market that I'm aware of and their offering is extremely compelling.
    - romeros 3y ago
      To paraphrase YC/Sama etc. If there is a small grouop of initial users who are obsessively using it and can't get enough of it. Then VP will work and be here to stay. If not, then it is not their time. So, the question is this. If you know anyone who actually got VP. Are they using it obsessively? I got occulus rift a while ago and so did many of my friends. We bought it but never really used it. Like probably a few times in a year to show friends.
      - ToucanLoucan 3y ago
nickthegreek 3y ago
I was not expecting for Apple to be releasing any open source generative ai product. So the big players in the download it yourself gang are Meta & Apple? I don't think anyone would have taken that bet when OpenAI blew up.
- jedberg 3y ago
  Apple makes a ton of sense. They sell hardware (and services for that hardware). It would make sense that they want to commoditize foundation models while at the same time creating yet another reason to buy their most powerful hardware. Meta is the real wildcard here. They're betting on "commoditize your compliment" and hoping by making GenAI training nearly free (by doing it for you) that none of their competitors can use GenAI models to gain an advantage over them.
  - deleted 3y ago
    [deleted]
  - keepper 3y ago
    Why is Meta the wildcard? Meta has the largest track record of open sourcing everything. From hardware (OpenCompute suite of releases ) to pretty much all software ( HHVM, Hack, Buck, etc etc ) that would benefit wider use. Even Meta's ML research has a history of open sourcing everything, hello PyTorch. So Llama is just a continuation
    - jedberg 3y ago
      Only because all their open source didn't cost them billions to produce. Making a foundation model is at least a billion dollars in just hardware and power. But you're right they do have a good history of open source, and they should get credit for that.
  - yunwal 3y ago
    Historically, Apple doesn't sell hardware, they sell a combination of software and hardware that are supposedly more tightly integrated than is possible when those things are developed separately. It's interesting and surprising that they would develop and release an open source model.
gitfan86 3y ago
No one is talking about how Joi is the killer app for vision pro
- dvngnt_ 3y ago
  wha
achow 3y ago
Demo: https://huggingface.co/spaces/tsujuifu/ml-mgie https://huggingface.co/spaces/tsujuifu/ml-mgie (Long queue, I could not test it)
dosinga 3y ago
Interesting read, but then suddenly there's this passive aggressive paragraph: Apple stock has taken a beating as of late, in part because analysts have loudly proclaimed that the company is behind Meta, Google, and Microsoft in generative AI implementation. It's not clear why this wasn't a problem when it wasn't first to a mobile phone, a tablet, a smartwatch, or a VR headset, but is with generative AI. somebody is hurting
- system16 3y ago
  I also found that an odd statement. If you need a “why” analysts may not feel confident about Apple’s ability to deliver when it comes to AI, look no further than Siri and Apple Maps.
  - hospitalJail 3y ago
    I find it interesting 'analysts' are looking at Apple's product rather than user sentiment. If you are only looking at objective metrics, Apple is always going to perform at B and C levels. If you are looking at fanaticism, they are A++.
  - tiltowait 3y ago
    I'll grant you Siri, of course, but why Apple Maps? From my use, it should be held as an example of how Apple can convert a bad launch into a good product.
- pksebben 3y ago
  I dunno, the tone may be what it is but I kinda see the message re: analysts and their forecasts. Beat the managers, buy the index, yaknow.
jahabrewer 3y ago
Important clarification from reading tfa for a second: image _editing_. I can't believe Apple would sully their image with actual generation.
gcr 3y ago
The significant part of this, for me, is not the image model. It's that Apple rarely publishes in the vision field. Back in Neurips (2016 I want to say?) Apple had a workshop where they said "we're going to engage more with the academic community, we're going to publish more," but since then only a small number of CV projects have seen the academic light of day. I remember one on detection of flashing lights, a whitepaper/tech report on Hey Siri wakeword detection, some interesting synthetic image generation for iris recognition, now this. If I had to guess, I think Apple Research may have some institutional obstacles to getting research work out there. Perhaps this could indicate those walls are continuing to dissolve bit by bit?
- imjonse 3y ago
  They have at least two recent papers (MobileOne and FastViT) with code and weights for low-latency (as measured on iPhone) vision backbones. Also AIM (Autoregressive Image Models) released last month.
- Someone 3y ago
  I wouldn’t know how good they are, but https://machinelearning.apple.com/research?page=1&domain=Computer+Vision&type=article&type=paper https://machinelearning.apple.com/research?page=1&domain=Com... has about 50 papers in 2023. Aren’t those in computer vision?
  - gcr 3y ago
    Thanks! I legit had no idea. I retract my earlier comment.
barrkel 3y ago
The code repo this article is based on - https://github.com/apple/ml-mgie https://github.com/apple/ml-mgie - is based around InstructPix2Pix (which is based on Stable Diffusion) and LLaVA (multi-modal LLM that supports vision input). It's research code, not a product. I don't think Apple would ever productionize anything with data lineage from Stable Diffusion.
devinprater 3y ago
This is going to be really cool. Being able to have pictures described, with AI, and then edit them using text, like underline this part of the screenshot, or something like that. Especially with Llava 1.6, and Apple's other model where it understands spatial parts of images, this should be pretty possible for me, as a blind person, to do. So yeah I think I'll wait to get this September's iPhone.
_lvbh 3y ago
I’m curious why so many AI papers consist mostly of people with Chinese-like names. I’m surprised there aren’t more witch hunts from the far right & skepticism that they can embed CCP propaganda into the models.
- Am4TIfIsER0ppos 3y ago
  Why would we, the right-wing conspiracy nuts, need chinamen to do that when there are so many communists in america and europe who do it anyway.
  - atlas_hugged 3y ago
    Y’all just say the quiet part out loud now?

118 points by gnicholas 3y ago

gnicholas 3y ago

The Github repo is here: https://github.com/apple/ml-mgie https://github.com/apple/ml-mgie

MrThoughtful 3y ago
I have an AWS account. Is there a type of EC2 instance I can fire up and run this thing on?
- deleted 3y ago
  [deleted]
- cheald 3y ago
  Inference should ideally be done on an instance with a GPU, and it should have enough GPU VRAM to hold the entire model. This one is only a 7b model, so it should run pretty easily in a modest amount of vram. You can likely run it locally in CPU-only mode, too, though it's likely to be rather slow. https://instances.vantage.sh/?min_gpus=1 https://instances.vantage.sh/?min_gpus=1 (wait for it to finish loading to filter) You could try running these in Google Collab (though tweaks would have to be made to load the files), or you might try something like runpod.io, as well, which gets you GPU instances for a lot less than you'd pay with AWS - for example, a Tesla V100 (16GB) community cloud instance runs you $0.24/hr vs the g5g.xlarge which runs you $0.42/hr.

deleted 3y ago

[deleted]

ericmcer 3y ago

It feels like Apple has been very quiet about their approach in some of the future spaces (self-driving, VR/AR, AI) but then they suddenly burst out with something like Vision Pro. It seems like they are biding their time and avoiding the hype knowing that they don't need to get their first. If they can produce a superior product their brand will let them dominate those markets whenever they decide to release.

RyanHamilton 3y ago
What makes you think vision pro is dominating? Unlike iPhone etc it has no distinguishing feature I know of.
- ToucanLoucan 3y ago
  Speaking as a VR enthusiast who has virtually no interest in it: The Vision Pro is an entirely new product relative to any other VR headset I know of. It offers a comprehensive set of computing features (discounting an App Store which has only really just started) that puts it roughly on par with something like iPad OS. I would say it dominates currently for VR headsets if that is what you're after, because it's basically your only option. Desktop computing in VR is possible with things like a Vive or an Index but the experience is entirely different, worse on every front, and is basically done as a workaround in case you need to operate your PC while in VR without taking off the headset. And of course, all of those headsets are (quite intelligent, but remain merely) displays: they require a fairly powerful computer to attach to. There are headsets that operate independently too, like Meta's offerings, but those are much, much, much more restrictive in terms of capability than the Vision Pro. My interest in VR is basically solely for gaming, so the Vision Pro for me is a complete miss and I have no plans to purchase one. That being said, in the realm of productivity VR, I'd say they are dominating in that they're the sole entrant to the market that I'm aware of and their offering is extremely compelling.
  - romeros 3y ago
    To paraphrase YC/Sama etc. If there is a small grouop of initial users who are obsessively using it and can't get enough of it. Then VP will work and be here to stay. If not, then it is not their time. So, the question is this. If you know anyone who actually got VP. Are they using it obsessively? I got occulus rift a while ago and so did many of my friends. We bought it but never really used it. Like probably a few times in a year to show friends.
    - ToucanLoucan 3y ago

nickthegreek 3y ago

I was not expecting for Apple to be releasing any open source generative ai product. So the big players in the download it yourself gang are Meta & Apple? I don't think anyone would have taken that bet when OpenAI blew up.

jedberg 3y ago
Apple makes a ton of sense. They sell hardware (and services for that hardware). It would make sense that they want to commoditize foundation models while at the same time creating yet another reason to buy their most powerful hardware. Meta is the real wildcard here. They're betting on "commoditize your compliment" and hoping by making GenAI training nearly free (by doing it for you) that none of their competitors can use GenAI models to gain an advantage over them.
- deleted 3y ago
  [deleted]
- keepper 3y ago
  Why is Meta the wildcard? Meta has the largest track record of open sourcing everything. From hardware (OpenCompute suite of releases ) to pretty much all software ( HHVM, Hack, Buck, etc etc ) that would benefit wider use. Even Meta's ML research has a history of open sourcing everything, hello PyTorch. So Llama is just a continuation
  - jedberg 3y ago
    Only because all their open source didn't cost them billions to produce. Making a foundation model is at least a billion dollars in just hardware and power. But you're right they do have a good history of open source, and they should get credit for that.
- yunwal 3y ago
  Historically, Apple doesn't sell hardware, they sell a combination of software and hardware that are supposedly more tightly integrated than is possible when those things are developed separately. It's interesting and surprising that they would develop and release an open source model.

gitfan86 3y ago

No one is talking about how Joi is the killer app for vision pro

dvngnt_ 3y ago
wha

achow 3y ago

Demo: https://huggingface.co/spaces/tsujuifu/ml-mgie https://huggingface.co/spaces/tsujuifu/ml-mgie (Long queue, I could not test it)

dosinga 3y ago

Interesting read, but then suddenly there's this passive aggressive paragraph: Apple stock has taken a beating as of late, in part because analysts have loudly proclaimed that the company is behind Meta, Google, and Microsoft in generative AI implementation. It's not clear why this wasn't a problem when it wasn't first to a mobile phone, a tablet, a smartwatch, or a VR headset, but is with generative AI. somebody is hurting

system16 3y ago
I also found that an odd statement. If you need a “why” analysts may not feel confident about Apple’s ability to deliver when it comes to AI, look no further than Siri and Apple Maps.
- hospitalJail 3y ago
  I find it interesting 'analysts' are looking at Apple's product rather than user sentiment. If you are only looking at objective metrics, Apple is always going to perform at B and C levels. If you are looking at fanaticism, they are A++.
- tiltowait 3y ago
  I'll grant you Siri, of course, but why Apple Maps? From my use, it should be held as an example of how Apple can convert a bad launch into a good product.
pksebben 3y ago
I dunno, the tone may be what it is but I kinda see the message re: analysts and their forecasts. Beat the managers, buy the index, yaknow.

jahabrewer 3y ago

Important clarification from reading tfa for a second: image _editing_. I can't believe Apple would sully their image with actual generation.

gcr 3y ago

The significant part of this, for me, is not the image model. It's that Apple rarely publishes in the vision field. Back in Neurips (2016 I want to say?) Apple had a workshop where they said "we're going to engage more with the academic community, we're going to publish more," but since then only a small number of CV projects have seen the academic light of day. I remember one on detection of flashing lights, a whitepaper/tech report on Hey Siri wakeword detection, some interesting synthetic image generation for iris recognition, now this. If I had to guess, I think Apple Research may have some institutional obstacles to getting research work out there. Perhaps this could indicate those walls are continuing to dissolve bit by bit?

imjonse 3y ago
They have at least two recent papers (MobileOne and FastViT) with code and weights for low-latency (as measured on iPhone) vision backbones. Also AIM (Autoregressive Image Models) released last month.
Someone 3y ago
I wouldn’t know how good they are, but https://machinelearning.apple.com/research?page=1&domain=Computer+Vision&type=article&type=paper https://machinelearning.apple.com/research?page=1&domain=Com... has about 50 papers in 2023. Aren’t those in computer vision?
- gcr 3y ago
  Thanks! I legit had no idea. I retract my earlier comment.

barrkel 3y ago

The code repo this article is based on - https://github.com/apple/ml-mgie https://github.com/apple/ml-mgie - is based around InstructPix2Pix (which is based on Stable Diffusion) and LLaVA (multi-modal LLM that supports vision input). It's research code, not a product. I don't think Apple would ever productionize anything with data lineage from Stable Diffusion.

devinprater 3y ago

This is going to be really cool. Being able to have pictures described, with AI, and then edit them using text, like underline this part of the screenshot, or something like that. Especially with Llava 1.6, and Apple's other model where it understands spatial parts of images, this should be pretty possible for me, as a blind person, to do. So yeah I think I'll wait to get this September's iPhone.

_lvbh 3y ago

I’m curious why so many AI papers consist mostly of people with Chinese-like names. I’m surprised there aren’t more witch hunts from the far right & skepticism that they can embed CCP propaganda into the models.

Am4TIfIsER0ppos 3y ago
Why would we, the right-wing conspiracy nuts, need chinamen to do that when there are so many communists in america and europe who do it anyway.
- atlas_hugged 3y ago
  Y’all just say the quiet part out loud now?