8 ms·
Animated Drawings
- hjessmith 4y agoHey! That’s my project! Code and dataset are here: https://github.com/facebookresearch/AnimatedDrawings https://github.com/facebookresearch/AnimatedDrawings And a browser-based version of it is here: http://sketch.metademolab.com/ http://sketch.metademolab.com/
- qainsights 4y agoCool project. Just tried with my kids. They enjoyed it. Is it possible to download the video?
- deleted 4y ago[deleted]
- hjessmith 4y agoI assume you're referring to the demo at sketch.metademolab.com? The server is returning an .mp4 file that is displayed by your browser, so you should be able to download it. On Chrome, if I maximize the video and click the three vertical dots, I get the option to download the video.
- qainsights 4y agoThanks. Yes, I figured it out :)
- Knucklebones 4y agoOn the online demo, you can also download by clicking 'share' at the end (there's an option in the menu that shows up).
- liveoneggs 4y agomy kids and I had a lot of fun with the sketch version while killing time in a hotel room a while ago.
- tomcam 4y agoAmazingly good demo. You rock. Also you're apparently very, very fit and flexible. I pulled a hamstring just watching the dance.
- hjessmith 4y agoIt's equally important to stretch before coding and stretch before dancing.
- tomcam 4y agoI’VE BEEN DOING IT WRONG THE WHOLE TIME
- marai2 4y agoI'm definitely going to start stretching before coding now! Maybe even 15 minutes of yoga before debugging.
- mind-blight 4y agoHonestly, that would be so good for your health. You'd go into your 30s, 40s, and 50s with much less MSK pain.
- doubtfuluser 4y agoThis is really a fun project! Is it actually possible to use the code and create an app for sale out of it. The MIT license says so I think, just wondering if anything else would speak against it.
- hjessmith 4y agoI don't know who would speak out against it. The MIT license certainly allows for it, and I don't actually own it... I just wrote it :)
- ac50hz 4y ago+1 This is a pleasant take. I may borrow the sentiment expressed here to help me describe some similar distinctions.
- upmostly 4y agoI'm working on an app called Monstdraw.com that uses this code if you're interested in collaborating? My email is in my bio.
- hjessmith 4y agoI’m curious to learn more.
- cstrat 4y agoThis is AMAZING, my 3 year old draws the cutest pictures and I can animate them! One thing with the browser version, it all works fine up until the animation and then its flipping my image upside down? Could this just be the photo metadata rotation/orientation??
- hjessmith 4y agoIt’s not metadata, you would see that when you upload the photo at the beginning. I’m guessing that the nose key point is located below the shoulders in your child’s drawing? That’s a known issue caused by how we ‘apply’ poses onto the characters. It’s fixed in the GitHub code but not in the browser version. Try moving the nose keypoint to the top of the characters head and that should resolve it
- cstrat 4y agoAhh yeah that is it, haha thanks :D cant wait to show her
- eurekin 4y agoImagine future job market, where a 5 year old will already be super familiar with the T-posing, which currently mostly CG artists now about
- tealpod 4y agoThank you Thank you Thank you, my kid loving it. I will be happy to volunteer and help in anyway I can.
- SanderNL 4y agoThis is great fun. Thank you for giving away this work and so accessibly. I got it working in minutes.
- the-mitr 4y agovery cool project!
- greggsy 4y agoThis is really cool, and refreshingly simple in the current landscape of generative tooling. I’m interested to hear your thoughts about the inevitable monetisation that will occur once someone wraps this into a horrible app with ads? Maybe someone else can answer the related question of how the respective app stores deal with attribution-based code licenses?
- hjessmith 4y agoYeah, I think that'll happen. I think it's already happening. And if someone wraps it in a terrible app with ads, makes millions of dollars, and I never see any of the money, yes I'll be very sad. But I'm trying to take a bigger perspective here. There's a lot of experience and tools that could be built off this, and I can't possibly build them all. And if I hid the code away so only I could use it, there's a good chance no one would ever create any of those experiences. Creating a successful app requires mobile UI people, devops, designers, marketing/SEO, monetization, etc. I don't want to do ANY of that... in the land of app creation, I'm more of a Tom Bombadil-type. This is also my first project out of graduate school. I feel like I'll probably have other ideas down the road, and maybe I'll try to monetize those ones with what I've learned from this adventure.
- sva_ 4y agoWould be nice to add a queue. Went through all the steps and then it said the servers are busy losing progress.
- ngram23 4y agoGreat work. Online comic strips are going to be next level!
- Awelton 4y agoThis is a really neat idea. My son got such a kick out of it that he has been drawing new characters for two hours now.
- yantrams 4y agoEdit: My bad it has to do with my Adblocker. Its all good now. Cheers Super cool! Thank you for bringing this to the world. Kids are gonna love this. I'm unable to get it running on the web demo unfortunately though :( Post image upload, I'm redirected to https://sketch.metademolab.com/canvas https://sketch.metademolab.com/canvas with a blank screen with the grid image. Without any error message, it isn't obvious if the issue is with the images I uploaded ( which probably is the case ). Will try the Python library and play around. PS: That silly dance thingy gif is so hilarious
- hjessmith 4y agoGlad you got it working. And glad you appreciate my sick dance moves :)
- yantrams 4y ago:) Can't wait for my 4 year old kiddo to wake up so I can tell her how her drawings come to life in the night and dance around if she behaves haha Thank you again (and Meta too) for making this happen. This is gonna be put a smile on millions of kids I tell you and make them draw more.
- Elias-Braun 4y agoI have a big smile in my face right now : D
- sebastianconcpt 4y agoBest use of computing power ever :D
- tomcam 4y agoAnalysis: true
- egonschiele 4y agoSo great to see more work on pose research! My ultimate dream is to have a tool where I can give it a cartoon character and a set of pose photos, and have the tool pose the character in those poses. This feels one step closer to that.
- hjessmith 4y agoYou might check out the technical paper, the background section lists some relevant related works on non-photorealistic pose estimation. One work that seems particularly pertinent to your use case is: Gesture3D: Posing 3D Characters via Gesture Drawings http://www.cs.ubc.ca/labs/imager/tr/2016/Gesture3D/ http://www.cs.ubc.ca/labs/imager/tr/2016/Gesture3D/
- Waterluvian 4y agoPhenomenal. For what it’s worth, I think this would be a killer app for my phone. My kids would easily spend hours drawing and animating their creations. I’d pay money for this.
- Galanwe 4y agoI would pay for that as well. All it needs is some kind of Android app.
- noduerme 4y agoGet them one of these. They'll actually get the experience pf animating, broken down into bare terms... https://the80sand90s.com/articles/etch-a-sketch-animator https://the80sand90s.com/articles/etch-a-sketch-animator that little toy I had as a kid did more to shape my career as than even hypercard. FTA (and relevant): > What truly made this the best toy ever made was the amount of patience and perseverance a kid needed to create these animations, just like real animators. This is also why it would later fall from popularity.
- mojomark 4y agoHoly crap - I totally forgot that I had one of those and spent an obscene number of hours creating on it. Thanks for the memory-blast post!
- noduerme 4y agoHah! Cool to meet another kid who grew up with that. I wish there were more actual devices like that now, with their own weird UIs and knobs and buttons, instead of everything being an app mixed in on a tablet with endless other apps. When you think about it, what a far-out thing for a company that made Etch-A-Sketches to engineer and sell...
- ShadowBanThis01 4y agoThat thing seemed really cool, but even at the time I passed on it because there was no way to save your work externally.
- upmostly 4y agoSmall anecdote: I discovered the codebase for this project last week and assumed that it had already been advertised and popularised. Apparently that's just happening now. I deployed the Docker container and got it working fine and I'm already thinking of plenty of cool use cases for the code. I get the impression that a few startups / apps are going to be launched off of the back of this. A few ideas: - Animated story book using GPT (I'm working on) - Animated greetings cards using photos (works really well)
- hjessmith 4y agoGlad to hear it was easy for you to get up and running, but where did come across the repo? It's been in 'stealth release' mode for a few months now, but nothing was publicly announced until today.
- upmostly 4y agoA friend told me about the project last week, so I did some Googling and discovered the GitHub repo by searching for AnimatedDrawings and Facebook.
- animatethrow 4y agoI want to animate video game characters with as much automation as possible because hand drawing frame-by-frame takes forever and ever. Is this my best bet?
- hjessmith 4y agoIt currently only outputs .mp4 and .gif files. You could probably use these as game assets, but I don't think it would be optimal. The rendering code utilizes an MVC framework. If you need the animation output in a .obj format or something, it's as simple as creating a new View that supports it. Feel free to request this in a Github issue if you want me to prioritize it.
- defaultcompany 4y agoYou could look at Adobe Character Animator.
- Jaxkr 4y agoI’ve been working on a project exactly for this: https://dreamlab.gg/ https://dreamlab.gg/. Currently it only allows for animating a character generated from a prompt. However, we’re working to use null text inversion [0] combined with inverse DDIM (used for pix2pix) to allow animating a pre-drawn character. [0]: https://arxiv.org/abs/2211.09794 https://arxiv.org/abs/2211.09794
- mholt 4y agoVery fun. Is there something like this but which can capture the motion from a webcam (without separate motion capture process as described in the readme) and project that onto the drawings?
- hjessmith 4y agoAs written now, it expects 3D motion data. Extending it to support 2D motion data is doable, but will take time. Feel free to open a github issue requesting it if you'd like me to prioritize it.
- simple10 4y agoThis is amazing! It's a bit buried in the repo readme files, but the author @hjessmith used Rokoko Video [1] to create the animation files (BVH) from a phone video. Then he used Animated Drawings to make the drawings replicate his motion captured from the video. Both Rokoko and Animated Drawings were new to me. I know what I'll be doing with my kids all weekend! Thanks for sharing the project. [1] https://www.rokoko.com/products/video https://www.rokoko.com/products/video
- hjessmith 4y agoIf you post your creation to social media, please use #FAIRAnimatedDrawings. I'd love to see what you create!
- nmfisher 4y agoCompletely tangential point - Rokoko Video is fine for a free tool, but if you need something better and you're willing to pay, there are other options. I spent a couple of days evaluating various video-only mocap and in the end went with DeepMotion. Rokoko Video wasn't very reliable (at one point I was standing perfectly still, but it thought I was on one leg twirling like a ballerina). Although it's very expensive, DeepMotion has been great (also if you were disappointed by DeepMotion 12+ months ago like I was, I'd suggest revisiting - it's leaps and bounds ahead of the earlier version I tried out). move.ai looks very good too, but needs multiple cameras which was overkill for my use case. Plask seems to have regressed (the platform seems like it doesn't even work half the time) and Radical's aggressive tactics really turned me off.
- simple10 4y agoIt looks like DeepMotion has a free tier. Did you just need more credits or is there some additional feature only available on the paid tiers?
- nmfisher 4y agoThe free tier only gives you 60 seconds of animation per month, and also isn't licensed for commercial use. Even for a hobbyist, 60 seconds wont go far!
- totetsu 4y agoI remember doing something similar to this with many manual steps with Photoshop and aftereffects many years ago.
- liendolucas 4y agoOut of all the ML projects submitted to HN this is one of the funniest and most entertaining I've seen so far! Also very neat demo and presentation in the website! KUDOS to the author(s) of the implementation and paper!
- Oras 4y agoThis is incredible! Fantastic presentation and easy to follow code.
- hjessmith 4y agoYou're the first stranger to ever compliment my coding style. I feel all warm and fuzzy now.
- Franzeus 4y agoVery cool tech and amazing dance! :) We are working on a startup (https://dibulo.com https://dibulo.com) which provides children an off-screen coloring experience combined with a digital reward. We have so many ideas and using pose-estimation is one of them. Definitely gonna play around with it!
- hjessmith 4y agoLove the idea. Feel free to reach out if you think there’s room for collaboration.
- finikytou 4y agothis is amazing. how about animating live from a video stream in AR :D?
- hjessmith 4y agoSounds awesome! Let’s do it.
- self_awareness 4y agoThe demo website is frozen and I can't click anything. > [Report Only] Refused to create a worker from 'blob:https://sketch.metademolab.com/923050e3-9390-4b78-9349-8159ad0136cf https://sketch.metademolab.com/923050e3-9390-4b78-9349-8159a...' because it violates the following Content Security Policy directive: "worker-src 'none'". > [Report Only] Refused to create a worker from 'blob:https://sketch.metademolab.com/d94425ea-c1b7-424f-a898-9070894135dc https://sketch.metademolab.com/d94425ea-c1b7-424f-a898-90708...' because it violates the following Content Security Policy directive: "worker-src 'none'". > [Report Only] Refused to create a worker from 'blob:https://sketch.metademolab.com/923050e3-9390-4b78-9349-8159ad0136cf https://sketch.metademolab.com/923050e3-9390-4b78-9349-8159a...' because it violates the following Content Security Policy directive: "worker-src 'none'". > Failed to load resource: net::ERR_UNKNOWN_URL_SCHEME > [Report Only] Refused to create a worker from 'blob:https://sketch.metademolab.com/d94425ea-c1b7-424f-a898-9070894135dc https://sketch.metademolab.com/d94425ea-c1b7-424f-a898-90708...' because it violates the following Content Security Policy directive: "worker-src 'none'". > Failed to load resource: net::ERR_UNKNOWN_URL_SCHEME > Uncaught (in promise) DOMException: The play method is not allowed by the user agent.
- umaar 4y agoThis is a great project and it was fun playing around with it. When Meta started discussing this in 2021, the HN sentiment [1] was quite...cautious, making connections to how Meta can't be trusted with kids drawings. Has something changed? [1] https://news.ycombinator.com/item?id=29580619 https://news.ycombinator.com/item?id=29580619
- devsda 4y agoThe terms of usage seems to remain the same and the demo page's privacy policy points to the common Facebook privacy policy. I hope the terms are limited to the demo and there's no telemetry within the source.
- aierou 4y agoWe've moved on to a different moral panic
- hutzlibu 4y agoI mean it gets exhausting. We had Covid (and Q), Musk taking over Twitter, war in Ukraine, fear of war in Taiwan, ... accelerated climate change and so on. There are real limits to how many things one can be upset about.
- blitzkrieg3 4y agoIf I had to guess it's because this one is unbranded and released as open source
- hvidevold 4y agoMade this using the tool: https://github.com/emnh/mantary-platformer/blob/main/mantary/images/walk.gif https://github.com/emnh/mantary-platformer/blob/main/mantary... I used the website, downloaded the mp4 and postprocessed to walk in place using a shell script: https://github.com/emnh/mantary-platformer/blob/main/animscript.sh https://github.com/emnh/mantary-platformer/blob/main/animscr... The character was generated using stable diffusion.
- s1mon 4y agoIt's cool that this is possible, but it's missing some very important subtleties which make professional animation look "right" or even "real". As it is, there's no sense of compression or snap at the right parts of the movement. 2D character animation has a lot of exaggerated distortions of objects to make them look like they land on the ground or bounce. The sense of weight and inertia is critical. These aren't necessarily great, but this page[0] does touch on some of the basic concepts. "Squash and stretch" is a key thing missing. Also while in the demo you're dancing not walking, a walk cycle is a very basic thing which is often done poorly by even supposedly professional companies. [1] It's really important for the "contact" part to seem like it's happening. Getting feet to plant themselves on the ground and look like they're not swimming or sliding around is a challenge. [0] https://idearocketanimation.com/13721-12-principles-of-animation-gifs/ https://idearocketanimation.com/13721-12-principles-of-anima... [1] https://www.angryanimator.com/word/2010/11/26/tutorial-2-walk-cycle/ https://www.angryanimator.com/word/2010/11/26/tutorial-2-wal... [my father worked in animation for much of his career and would complain bitterly about badly done animation, so I guess I can't unsee things with his critical eye]
- hipjiveguy 4y agoI agree, but 80/20 rule. Your day, for the context of his time, was in the 20% it sounds like, which is all we can ever hope for.
- hjessmith 4y agoI'm a student of good animation myself (know the 12 principles of animation by heart, got a copy of The Illusion of Life right next to my desk, etc.) and have great respect for what professional animators can do. And while I (maybe more than anyone else) am highly aware of all the ways this tool doesn't create high-quality animation, I'm okay with it. The point here isn't to create animations that rival those of professional animators; it's to lower the barrier-to-entry to the point where almost anyone can animate their own character. And if a tiny number of users become more interested in animation through this tool, learn how to do it the proper way, and create animations your father would approve of... well I think that would be a phenomenal outcome.
- dtjohnnymonkey 4y agoI have a bunch of my kids drawings that I want to turn into digital art paintings. Is there any product or service for this?
- polygamous_bat 4y agoHave you tried the browser app? https://sketch.metademolab.com/ https://sketch.metademolab.com/
- timini 4y agodid you get paid to make this?
- hjessmith 4y agoYes. I did most of this as a postdoc at FAIR, Meta's AI research group.
- yegle 4y agoOh wow, this is the animation I saw on Disney Cruise's Animator's Palate, but better! https://disneycruise.disney.go.com/onboard-activities/animators-palate-restaurant/ https://disneycruise.disney.go.com/onboard-activities/animat...
- josh_carterPDX 4y agoGuess I know what I'm doing with the kids this weekend. This looks awesome. Thank you!
- DiscourseFan 4y agoI hate to say it, but I think this could have a great application in horror games.
- lqlsing 4y agoBaby doll