9 ms·
Apple Releases Open Weights Video Model
- coolspot 11mo ago> STARFlow-V is trained on 96 H100 GPUs using approximately 20 million videos. They don’t say for how long.
- moondev 11mo agoApple intelligence: trained by Nvidia GPUs on Linux. Do the examples in the repo run inference on Mac?
- satvikpendem 11mo agoLooks good. I wonder what use case Apple has in mind though, or I suppose this is just what the researchers themselves were interested in, perhaps due to the current zeitgeist. I'm not really sure how it works at big tech companies with regards to research, are there top down mandates?
- ivape 11mo agoTo add things to videos you create with your phone. TikTok and Insta will probably add this soon, but I suppose Apple is trying to provide this feature on “some level”. That means you don’t have to send your video through a social media platform first to creatively edit it (the platforms being the few tools that let you do generative video). They should really buy Snapchat.
- ozim 11mo agoI guess Apple is big in video production and animation with some ties via Pixar and Disney. Since Jobs started Pixar and it all got tied up in myriad of different ways.
- nothrowaways 11mo agoWhere do they get the video training data?
- postalcoder 11mo agoFrom the paper: > Datasets. We construct a diverse and high-quality collection of video datasets to train STARFlow-V. Specifically, we leverage the high-quality subset of Panda (Chen et al., 2024b) mixed with an in-house stock video dataset, with a total number of 70M text-video pairs.
- justinclift 11mo ago> in-house stock video dataset Wonder if "iCloud backups" would be counted as "stock video" there? ;)
- fragmede 11mo agoTurn on advanced data protection so they don't train on yours.
- givinguflac 11mo agoThat has nothing to do with it, and Apple wouldn’t train on user content, they’re not Google. If they ever did there would be opt in at best. There’s a reason they’re walking and observing, not running and trying to be the forefront cloud AI leader, like some others.
- gaigalas 11mo agoWhy should I buy this "ethical Apple" argument? They shared audio Siri recordings with contractors in 2019. It became opt-in only after backlash, similar to other privacy controversies. This shows that they clearly prioritize not being sued or caught, which is slightly different from prioritizing user choices.
- anon7000 11mo agoI have to delete as many videos as humanly possible before backing up to avoid blowing through my iCloud storage quota so I guess I’m safe
- devinprater 11mo agoApple has a video understanding model too. I can't wait to find out what accessibility stuff they'll do with the models. As a blind person, AI has changed my life.
- densh 11mo ago> As a blind person, AI has changed my life. Something one doesn't see in news headlines. Happy to see this comment.
- badmonster 11mo agoWhat other accessibility features do you wish existed in video AI models? Real-time vs post-processing?
- devinprater 11mo agoMainly realtime processing. I play video games, and would love to play something like Legend of Zelda and just have the AI going, then ask it "read the menu options as I move between them," and it would speak each menu option as the cursor moves to it. Or when navigating a 3D environment, ask it to describe the surroundings, then ask it to tell me how to get to a place or object, then it guide me to it. That could be useful in real-world scenarios too.
- kulahan 11mo agoWeird question, but have you ever tried text adventures? It seems like it's inherently the ideal option, if you can get your screen reader going.
- devinprater 11mo agoYep, they're nice. There are even online versions.
- tippa123 11mo ago
- camillomiller 11mo agoHopefully this will make into some useful feature in the ecosystem and not contribute to having just more terrible slop. Apple has saved itself from the destruction of quality and taste that these model enabled, I hope it stays that way.
- yegle 11mo agoLooking at text to video examples (https://starflow-v.github.io/#text-to-video https://starflow-v.github.io/#text-to-video) I'm not impressed. Those gave me the feeling of the early Will Smith noodles videos. Did I miss anything?
- M4v3R 11mo agoThese are ~2 years behind state of the art from the looks of it. Still cool that they're releasing anything that's open for researchers to play with, but it's nothing groundbreaking.
- Mashimo 11mo agoBut 7b is rather small no? Are other open weight video models also this small? Can this run on a single consumer card?
- Maxious 11mo agoWan 2.2: "This generation was run on an RTX 3060 (12 GB VRAM) and took 900 seconds to complete at 840 × 420 resolution, producing 81 frames." https://www.nextdiffusion.ai/tutorials/how-to-run-wan22-image-to-video-gguf-models-in-comfyui-low-vram https://www.nextdiffusion.ai/tutorials/how-to-run-wan22-imag...
- dragonwriter 11mo ago> But 7b is rather small no? Sure, its smallish. > Are other open weight video models also this small? Apples models are weights-available not open weights, and yes, WAN 2.1, as well as the 14B models, also has 1.3B models; WAN 2.2, as well as the 14B models, also has a 5B model (the WAN 2.2 VAE used by Starflow-V is specifically the one used with the 5B model.) and because the WAN models are largely actually open weights models (Apache 2.0 licensed) there are lots of downstream open-licensed derivatives. > Can this run on a single consumer card? Modern model runtimes like ComfyUI can run models that do not fit in VRAM on a single consumer card by swapping model layers between RAM and VRAM as needed; models bigger than this can run on single consumer cards.
- ai_updates 11mo ago[flagged]
- MallocVoidstar 11mo agoyou don't "appreciate" anything, you're just posting LLM comments
- RobotToaster 11mo agoThe license[0] seems quite restrictive, limiting it's use to non commercial research. It doesn't meet the open source definition so it's more appropriate to call it weights available. [0]https://github.com/apple/ml-starflow/blob/main/LICENSE_MODEL https://github.com/apple/ml-starflow/blob/main/LICENSE_MODEL
- deleted 11mo ago[deleted]
- limagnolia 11mo agoThey haven't even released the weights yet... As for the license, happily, Model Weights are the product of machine output and not creative works, so not copyrightable under US law. Might depend on where you are from, but I would have no problem using Model Weights however I want to and ignoring pointless licenses.
- mdrzn 11mo ago"VAE: WAN2.2-VAE" so it's just a Wan2.2 edit, compressed to 7B.
- BoredPositron 11mo agoThey used the VAE of WAN like many other models do. For image models you see a lot of them using the flux VAE. Which is perfectly fine, they are released as apache2 and save you time to focus on your transformers architecture...
- kouteiheika 11mo agoThis doesn't necessarily mean that it's Wan2.2. People often don't train their own VAEs and just reuse an existing one, because a VAE isn't really what's doing the image generation part. A little bit more background for those who don't know what a VAE is (I'm simplifying here, so bear with me): it's essentially a model which turns raw RGB images into a something called a "latent space". You can think of it as a fancy "color" space, but on steroids. There are two main reasons for this: one is to make the model which does the actual useful work more computationally efficient. VAEs usually downscale the spatial dimensions of the images they ingest, so your model now instead of having to process a 1024x1024 image needs to work on only a 256x256 image. (However they often do increase the number of channels to compensate, but I digress.) The other reason is that, unlike raw RGB space, the latent space is actually a higher level representation of the image. Training a VAE isn't the most interesting part of image models, and while it is tricky, it's done entirely in an unsupervised manner. You give the VAE an RGB image, have it convert it to latent space, then have it convert it back to RGB, you take a diff between the input RGB image and the output RGB image, and that's the signal you use when training them (in reality it's a little more complex, but, again, I'm simplifying here to make the explanation more clear). So it makes sense to reuse them, and concentrate on the actually interesting parts of an image generation model.
- pulse7 11mo ago<joke> GGUF when? </joke>
- LoganDark 11mo ago> Model Release Timeline: Pretrained checkpoints will be released soon. Please check back or watch this repository for updates. > The checkpoint files are not included in this repository due to size constraints. So it's not actually open weights yet. Maybe eventually once they actually release the weights it will be. "Soon"
- vessenes 11mo agoFrom the paper, this is a research model aimed at dealing with the runaway error common in diffusion video models - the latent space is (proposed to be) causal and therefore it should have better coherence. For a 7b model the results look pretty good! If Apple gets a model out here that is competitive with wan or even veo I believe in my heart it will have been trained with images of the finest taste.
- dymk 11mo agoTitle is wrong, model isn’t released yet. Title also doesn’t appear in the link - why the editorializing?
- deleted 11mo ago[deleted]
- giancarlostoro 11mo agoI was upset the page didnt have videos immediately available, then I realized I have to click on some of the tabs. One red flag on their github is the license looks to be their own flavor of MIT (though much closer to MS-PL).
- Barry-Perkins 11mo ago[dead]
- andersa 11mo agoThe number of video models that are worse than Wan 2.2 and can safely be ignored has increased by 1.
- embedding-shape 11mo agoTo be fair, the sizes aren't comparable, and for the variant that is comparable, the results aren't that much worse.
- dragonwriter 11mo agoThe samples (and this may or may not be completely fair, either set could be more cherry picked than the other, It would be interesting to see a side-by-side comparison with comparable prompts) seem significantly worse than what I’ve seen from WAN 2.1 1.3B, which is both fron the previous WAN version and is smaller, proportionally, compared to Apple’s 7B than that model itself is compared to the 28B combination of the high and low noise 14B WAN 2.2 models that are typically used together. But also, Starflow-V is a research model with a substandard text encoder, it doesn't have to be competitive as-is to be an interesting spur for further research on the new architecture it presents. (Though it would be nice if it had some aspect where it offered a clear improvement.)
- wolttam 11mo agoThis doesn’t look like it was intended to compete. The research appears interesting
- gorgoiler 11mo agoIt’s not really relevant to this release specifically but it irks me that, in general, an “open weights model” is like an “open source machine code” version of Microsoft Windows. Yes, I guess I have open access to view the thing I am about to execute! This Apple license is click wrap MIT with the rights, at least, to modify and redistribute the model itself. I suppose I should be grateful for that much openness, at least.
- advisedwang 11mo agoGreat analogy. To extend the analogy, "closed source machine code" would be like conventional SaaS. There's an argument that shipping me a binary I can freely use is at least better than only providing SaaS.
- limagnolia 11mo agoI think you are looking at the code license, not the model license.
- Aloisius 11mo agoNo, it's the model license. There's a second license for the code. Of course, model weights almost certainly are not copyrightable so the license isn't enforceable anyway, at least in the US. The EU and the UK are a different matter since they have sui generis database rights which seemingly allows individuals to own /dev/random.
- pabs3 11mo agoThe output of a compiler is copyrightable, why aren't models similarly copyrightable?
- limagnolia 11mo agoThe output of a compiler is directly based on what you put in, the source code. That makes it a derivative work of the copyrightable source code, and thus copyright to the copyright holder of the source code, not the person who runs the compiler. One might argue that model weights are derivative of the training material and copyright held be the copyright holder of the training material. The counter argument would be that the weights are significantly transformational.
- cubefox 11mo agoInteresting that this is an autoregressive ("causal") model rather than a diffusion model.
- Invictus0 11mo agoApple's got to stop running their AI group like a university lab. Get some actual products going that we can all use--you know, with a proper fucking web UI and a backend.
- Jtsummers 11mo agoPersonally, I'm happy that Apple is spending the time and money on research. We have products that already do what this model does, the next step is to make it either more efficient or better (closer to the prompt, more realistic, higher quality output). That requires research, not more products.
- Invictus0 11mo agoNot complaining about the research. I'm complaining that I have to do work to even try this, instead of Apple offering a demo or portal of some kind.
- summerlight 11mo agoThis looks interesting. This project has some novelty as a research and actually delivered a promising PoC but as a product it implies that its training was severely constrained by computing resources, which correlates well with the report that their CFO overruled CEO's decision on ML infra investment. JG's recent departure and follow up massive reorg to get rid of AI, rumors on Tim's upcoming step down in early 2026... All of these signals indicate that those non-ML folks have won corporate politics to reduce the in-house AI efforts. I suppose this was a part of serious efforts to deliver in-house models but the directional changes on AI strategy made them to give up. What a shame... At least the approach itself seem interesting and hope others to take a look and use it for building something useful.