8 ms·
BodyPix: Real-Time Person Segmentation in the Browser with Tensorflow.js
- minimaxir 7y agoThis update is for the 2.0 version. It's a shame there isn't a Python TF implementation of this (apparently due to a different file structure between the tf.js model and TF SavedModel?), and it doesn't sound like it's a priority for the team.
- m0zg 7y agoTF 2.0 in general (SavedModel included) has been a giant clusterfuck so far. As we used to say at Google "the old thing is deprecated, the new thing is not ready yet" - internally Google is pretty much entirely like this. I get that they had to ship it sooner or later, but IMO they shipped 2.0 a bit too soon. Shipping sooner is fine if you fix the bugs quickly, but that's not really happening either - some rather potent bugs remain open for half a year or more. So at the moment you have a choice of working with a version that's officially EOL (1.15) or using a version that has several critical features broken or missing, and on which TF Model Zoo doesn't work. Oh, and EOL version is also busted beyond belief. Worse yet, Google does not reward maintenance work, and since this is considered "launched", you can count on the bugs being there for an extended period of time until Google itself moves to 2.0, which I'd wager is unlikely to happen anytime soon. Facepalm.
- missosoup 7y agoSo basically everyone should move to pytorch and ONNX?
- amelius 7y agoThe best thing would be if everybody converged to the same framework.
- m0zg 7y agoONNX is a turd in is present state as well. Too many unsupported ops and you can't really run ONNX on e.g. a phone or really anywhere. You could theoretically convert it into something that does run where you need it to run, but then you have to deal with the converter's supported ops as well, and design your net around the intersection of the two sets of ops and their parameters. If you're doing anything more complicated than basic classifiers, you're going to run into issues. That said, I think in a few years with enough elbow grease ONNX could become pretty exciting. As to what people should use for research: just use PyTorch. I switched to it when it first came out, and never looked back, _except_ when I need my stuff to run on e.g. a phone or something, which is something PyTorch won't help you with.
- pjmlp 7y agoThe best part is the whole Tensorflow-Swift, with the halfbacked support for anything that isn't an Apple device. Meanwhile Julia works just fine on Windows.
- m0zg 7y agoProblem with Julia is that approximately nobody wants to learn yet another language just to do pretty much the same things that Python can do with the help of e.g. PyTorch or NumPy. Us programmers often overestimate the level of enthusiasm non-programmers have for learning programming languages. Heck, I'm kind of also running out of enthusiasm in this area myself.
- pjmlp 7y agoStill better to learning a language that only works properly on Apple platforms, without any kind of eco-system for scientific computing. I don't see Tensorflow-Swift going anywhere.
- m0zg 7y agoI agree. I looked into it briefly (since I sorta already know Swift anyway and like the language) but I can't even run this stuff on Linux easily, so it's kind of useless for me so far. This doesn't in any way mean that Julia has a snowball's chance in hell of taking off though. Besides the language, Python had a decade of head start in terms of "batteries included". And people have gotten used to that particular brand of batteries.
- pjmlp 7y agoJulia will add pressure for Python folks to actually take PyPy seriously. There is a large subcommunity that doesn't enjoy being forced to drop down to C for anything performance. Julia already counts some banks on their list, that tend to use ML languages instead of Python due to performance.
- 7y ago
- batmansmk 7y agoWhat can run client-side will run client-side.
- oh_sigh 7y agoCool demo but how can it figure out that all 3 guys are wearing briefs?
- Rebelgecko 7y agoWhat's weird about the pelvic area coloration is that as far as I can tell it doesn't line up with any of the body parts they say the segmentation is looking for (https://github.com/tensorflow/tfjs-models/tree/master/body-pix#the-body-parts https://github.com/tensorflow/tfjs-models/tree/master/body-p...)
- netwanderer3 7y agoCan this potentially be used in 2D animation replacing the manual skeletal animating processes?
- xsmasher 7y agoDo you mean 3D animation? It looks like the interface returns 2D coordinates for the joints, not 3D coordinates; so it can't directly create 3D animation data. It was also very jumpy in my browser, so it would require smoothing to be usable; or maybe if it wasn't running real-time it would be more accurate?
- soylentgraham 7y agoThat's how we did this; https://m.youtube.com/watch?v=7jfZ6zLkWF0 https://m.youtube.com/watch?v=7jfZ6zLkWF0 (was running live on-set)
- echelon 7y agoThat's so impressive! What company do you work for? This is super cool. Can you talk about the rest of your stack? Or other projects of a similar nature?
- soylentgraham 7y agohttp://newchromantics.com/ http://newchromantics.com/ is my own little company, (website is criminally out of date) I did the tech, http://analogstudio.co.uk/ http://analogstudio.co.uk/ did the art/comp/post work. The setup was essentially an app (c++ engine on osx running models), ingesting camera feeds, outputting skeletons, running sound->mouth poses, to unity which puppeteered 2D sprites. On-set we output to a monitor which mixed the real shots with our graphics. Nothing crazy! But it was a solid setup (except for overheating DSLRs)
- cryptozeus 7y agoReally cool ! Tensor js ?
- 7y ago
- mrieck 7y agoWow that's impressive. Is this library MIT license or something else? I would try using this to cutout people from images/gifs for my animation platform SuperAnimo: https://www.superanimo.com https://www.superanimo.com
- minimaxir 7y agoApache (which is sufficiently permissive), like most open-source Google projects.
- mrieck 7y agoMeh - it didn't work that well when I tried it. [1] Params I used were {architecture: 'ResNet50', outputStride: 32, quantBytes: 2}, with segmentMultiPerson function. The same images DeepLab handles perfectly. [2] If anyone sees a problem with the params/method I used let me know. [1] https://superanimo.com/bodypix/ https://superanimo.com/bodypix/ [2] Deeplab: https://github.com/tensorflow/models/tree/master/research/deeplab https://github.com/tensorflow/models/tree/master/research/de...
- deleted 7y ago[deleted]
- the8472 7y ago> Person segmentation Consider the defense applications!
- m_eiman 7y agoDoes this work in Safari for anyone? I've tried on iMac, iPhone and iPad. Only Firefox/MacOS works for me.
- lowdose 7y agoSince I quit smoking, I noticed how many tv-series and movies are loaded with cigarettes, alcohol and pharmacy pill consumption. Almost every social event good and bad on screen is dealt with by drinking a shot of liquor. Is this my personal paradigm or are we constantly fed with what we should view as our western culture? I wonder if it is possible to make an estimation per actor / producer / studio how many alcoholic, drug, cigarette they consume on average. Would be awesome to see if there is a correlation with project budget. And maybe even more important is there an increasing trend of average number of drug consumptions per hour.
- test1235 7y agocomment on the wrong article?
- asymmetric 7y agoThis seems like it can be used to lower the bar for (repressive) governments to identify people. E.g. by making it easier to distinguish parts of the body, identification via them, or gait analysis, becomes easier. Are there compelling use cases that make this risk worth it? Or are we just producing tech because we can?
- MayeulC 7y agoWhen I was a lot younger, I used to play softair sometimes with my friends (wargames with BBs gun). I seriously researched ways to build automatic turrets. Rules are simple: shoot at a foe until it raises its hand, that means the person was shot. Unfortunately (?), the available software at the time was subpar, and I forgot about it because of two problems: * How to identify a foe? It would need facial/body recognition from a few set of examples (team pictures before starting, better if it shots everything else). * How to know when to stop shooting? These new technologies would make almost trivial to build such an application, and make it quite reliable. I guess there is a market for this in softair... I'm even tempted to have a go at it, BUT... I don't think I will because I now have to ponder the ethics of building something that can take a gun, identify targets, aim at them accurately, and fire until they are down. I am usually all in on open source software, but that's just the kind of things that sound dangerous to share (same as defense distributed, TBH). I hope enough persons share the same misgivings, otherwise there is no point for me to refrain from building this. (Except perhaps spending my time on project that are actually useful for humankind).
- mark_l_watson 7y agoJavaScript is about my fifth favorite language - way down the scale of languages that I enjoy using. That said, I think TensorFlow.js is awesome on so many things: examples are easy understand and the build system makes them easy to run, many possibilities for using trained models in browser based apps, on my Linux GPU deep learning box training models is fast, great documentation, etc. In some ways I like the TensorFlow.js ecosystem better than the C++/Python ecosystem. So far, Swift TensorFlow has been a disappointment for me, but every few months I check it out again.
- error265 7y agois there a serverside version of this. I've been looking for somethign similar in python all day. Actually seems a little buggy seeing this test -> https://superanimo.com/bodypix/ https://superanimo.com/bodypix/