18 ms·

Background Features in Google Meet, Powered by Web ML

427 points by Marat_Dukhan 6y ago

ZephyrBlu 6y ago
Unrelated to the ML, but I don't like the fact I noticed that all the people in the clips are "diverse". No clue whether Google did this on purpose, but I feel like I've been trained to notice this due to the diversity movement thing currently occurring and I don't like it.
- ibuildthings 6y ago
  In this particular case, why is testing/demonstrating on diverse set of individuals a bad thing ? A personal anecdote is that a few years back the automatic door sensors in my university did not work on my skin tone.
  - ZephyrBlu 6y ago
    It's not a bad thing, I just dislike that I noticed it.
    - gexla 6y ago
      It's video conferencing software. Makes sense that they might put together imagery which might suggest people meeting from different corners of the planet. But sure, I get your point. I didn't notice this myself, but I have been living on side of the planet opposite from where I was born for the past decade.
  - easytiger 6y ago
    Automatic door sensors are in my experience universally infra red. In fact I don't think I've ever seen any camera technology used in that context. Are you saying they used cameras to open the doors?
- mdoms 6y ago
  Why don't you like it?
  - ZephyrBlu 6y ago
    Because I don't think I would have noticed a couple of years ago.
- dharma1 6y ago
  Sorry not everyone on the planet looks like you
  - hombre_fatal 6y ago
    They seem to be lamenting that they now notice skin color (due to all the racial essentialism of the last N years) where they used to not notice.
    - riffraff 6y ago
      I understand the feeling somewhat: it is very noticeable how TV shows for example now have expanded the default cast from "all white people + token black, maybe a gay individual" to "all white people + token black woman + token asian + token non-binary, maybe a transgender individual". It feels forced and ham-fisted, but I don't see how this could be made in a better way. Possibly people growing up with this will not notice it at all, and it will be good for them, it's only us old farts that need to adapt.
      - ZephyrBlu 6y ago
        Unfortunately I'm Gen Z, not an old fart :/. No clue how most Gen Z people feel about this type of thing though.
    - saurik 6y ago
      I think the issue is that, were it not for everyone being forced to notice, everyone was defaulting to caring so little that we just used the people around us or who we had existing connections with (for demos, training data, employees... whatever), and everything was horribly biased due to numerous reasons. So, while before you weren't being forced to pay attention, everything was more racist than now, where you are being forced to actually make an effort to be anti-racist. To put this into a programming metaphor, to me this is like being triggered by someone going out of their way to add a buffer overflow check due to a bunch of people spending the last decade screaming about buffer overflow security issues. Sure, before you didn't have to notice, but your code was probably also horribly insecure; now, everyone gets angry if you don't take at least minimal precautions, and people are even advocating that you use more secure languages from the start (to the point of questioning your architectural decisions if you don't), so you are being forced to pay attention--and sure, it seems a bit annoying and like extra work that a bunch of fanatics are foisting on you, and if someone had taken the time for their code to be secure before you might not have noticed (I mean, you weren't ever against security) but now it is screaming at you "this is because of those annoying security people rubbing our noses in our buffer overflows" so you get angry because this is taking time away from "getting the real work done" on your product--but the reality is that your code used to have glaring security issues that affected people who weren't you, and it sucked; everyone is better off for you paying attention now, and maybe one day we can fix the systemic problem and no one will have to put in such obvious effort to avoid being part of the problem, but we clearly aren't there yet. Being angry about this just comes off as not giving a shit about security: you didn't notice these checks before, because the code was just as good to you, for the criteria you were bothering to pay attention to, without the checks as it would have been with them, and the whole point is that that wasn't true... it was actually much worse; similarly, being angry about people making active efforts to have diversity in product advertising frankly just makes you come off as not giving a shit about minority representation :/.
mdoms 6y ago
Honestly it's the worst implementation I've seen of this technology yet. Just absolute and total garbage.
- yjftsjthsd-h 6y ago
  Why? What makes this one worse?
sillysaurusx 6y ago
Happy to see ML become mainstream. In the future, I don't think ML will be a separate field of programming. It'll just be "programming," the same way webdev is. There's a tendency to think of ML as "not programming," or something other than just plain programming. But as the tooling matures, that'll go away. (Lisp used to be considered "AI programming," till it became useful in many other contexts.)
- sltEvas 6y ago
  ML will become a library. It has about as much to do with programming as a compiler. You don't need to know what it does, you just need to know how to make it do things. The problem with ML currently is that nobody really knows how to do things and that you have a million parameters that need tuning and most algorithms need continuous improvement and fine tuning to the use case. There is nothing "mainstream" about ML at this point, except that everyone wants to use it. In maybe a decade, it might be found in standard libraries of programming languages and on top of things like `Math.abs`, we will have `ML.textToSpeech("Hello world")`, or `ML.isCat(image)`, etc. However, the problem I see with that is that no matter how far we wind the clock forward, we will only be able to put the most simplistic use cases into a library. `ML.isCat()` could be one of those, since most humans will be able to image categorization, it stands to reason that you could put this into a library. However, most industry application involved highly customized ML algorithms that are optimized for a very specific use-case. So there will always be a need for a research team in big companies at least. Maybe smaller companies will try to build their stuff by chaining libraries together.
  - fhaodhdms 6y ago
    Fwiw macs have had an equivalent functionality for both text to speech and speech to text for at least 17 years to my memory. The quality is poor compared to today's server-driven approaches, of course, but the functionality has been there if you're willing to articulate yourself clearly.
  - virgilp 6y ago
    There's never going to be a `ML.isCat(image)`, just like there isn't a `Math.solveProblem(hypothesis)`. Yes you do have `Math.abs` and you're going to have stuff `model.fit()` and `layers.dense()` - but something like `ML.isCat` is too specific to be used in a library
jtokoph 6y ago
The example video clips in the post look nothing like me and my team's view when using the new feature. Most of the time half of our hair gets blurred or replaced and hand gestures will cause either our hands or head to disappear.
- gundmc 6y ago
  It seems to have gotten a little better recently, but my experience matches yours. It really struggles when I wear over-ear headphones - they sort of phase in and out of existence. The other thing I've noticed is the background blur absolutely annihilates my CPU. To the point where I would rather just turn off my camera if I don't want my background visible.
- samtheprogram 6y ago
  I can vouch for this. I haven’t really needed the background blur feature personally, but I’ve tried it and both myself, colleagues, and friends — pretty much everyone I’ve talked to that has used it — loathe Google Meet’s background blur, and prefer Zoom’s by far. In my experience, it doesn’t completely cover the background most of the time, and if you move at all, as you point out, it can’t keep up. Kind of funny to see Google engineering blogging about it when it feels extremely half baked. This makes me sad, because in all other areas, I think Meet excels well beyond the competition. EDIT: removed my general sentiment on Google
  - judge2020 6y ago
    "Turns out our training set was comprised entirely of backdrops from Google HQ. Sorry everyone else!"
    - dontblink 6y ago
      Everyone is wfh @ Google
  - _pastel 6y ago
    "Half baked" misrepresents the difficulty this task. Yes, Zoom does it better, but it's _still_ an excellent and interesting engineering accomplishment. I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; this not only answers that question, but provides details down to the level of model architecture and the final latency and IOU benchmarks. Of course I'd be more interested to read how Zoom manages to do even better, but I'm not holding my breath for them to publish those details.
chdjakdkgb 6y ago
Wtf happened to hangouts? How many video products does google have?
- samtheprogram 6y ago
  Meet is business oriented and offers features that Hangouts does not, e.g. dialing in via phone. It also requires a G Suite account (or did before COVID, IIRC).
  - Polylactic_acid 6y ago
    Didn't hangouts have a business version as well?
    - Liquid_Fire 6y ago
      That's the same product. It was renamed to Meet.
- senectus1 6y ago
  Hangouts is EOL :-( We're being herded into the new more useless products.
  - toper-centage 6y ago
    Hangouts was always terrible and bloated. I miss Google Talk. Instead, we get something more terrible and bloated than Hangouts.
    - sumedh 6y ago
      Instead of focussing on Google Talk we have at least 5 different Apps from Google pretty much doing the same thing.
- hoveringhen 6y ago
  Hangouts is being replaced by Chat and Meet.
vinhboy 6y ago
Blur is awesome. Way better, less distracting, than backgrounds. Everyone who try it uses it permanently because it works so well. I also think it makes the subject look better for some reason.
- probably_wrong 6y ago
  Here's a tip: take a picture of your real, actual background from the POV of your webcam, and set that as your meeting background. Advantages: it looks natural, it covers whatever is going on behind you (in case you are not alone and people walks by, or if your living room is messy), and it blends better than fake backgrounds (because it's the same image behind it). I have a picture of my office that I use both at home and at my real office, and most people can't tell. And since I took the picture with my phone which has better resolution, my video feed looks better for cheap.
dharma1 6y ago
Sony released some software a couple of months ago that lets you use most of their DSLRs as webcams with USB. My goodness, paired with a fast lens, what a difference to my MacBook webcam, even with these ml blurred backgrounds! It's only 720p and around 15fps but real shallow dof, very little sensor noise, autofocus works. Well worth trying if you have a Sony camera from the last few years. Sensor size and good optics still wins. Having said that,the effort and detail gone into this feature is very impressive, enjoyed the blog post. Also webassembly SIMD looks super cool, looking forward to a new class of webapps using wasm.
- kenhwang 6y ago
  Canon did too! Definitely a huge upgrade over a typical webcam. I'm using my old T1i which can be had for less than $50 these days, plus you can pick up a 18-55mm kit lens for like $20 and the video quality blows away any webcam, especially for the same price. Also recommend battery->power adapter.
  - silly-silly 6y ago
    What is the canon software called ?
    - swang 6y ago
      EOS Webcam Utility https://www.usa.canon.com/internet/portal/us/home/support/self-help-center/eos-webcam-utility https://www.usa.canon.com/internet/portal/us/home/support/se...
  - callmeal 6y ago
    The T1i is not listed on their compatibility page. Which version of the utility did you get for it?
    - polpo 6y ago
      I have a T2i that I use with the EOS Webcam Utility and it works just fine despite not being listed as compatible.
neilpanchal 6y ago
Aside: Imagine you’re driving down the road and you need to make a right turn. Well, for some reason the steering wheel is stowed away and disappeared! You need to hover your hand around the center console in a specific area to be able to expose it. Out comes the steering wheel and now you can make a right turn. Google UX/UI team: Please fucking make the mute/unmute button visible at all times.
- askvictor 6y ago
  More important than the button is the status indicator - I need to know if the call is muted or not. Even better, promote it to an OS-level icon/badge/overlay. If my mic is actively in use, please make it blindingly obvious.
  - captn3m0 6y ago
    The color indicator for mic on/off is terribly for Meet.
    - nathancahill 6y ago
      Absolutely horrible. Both states are equally plausible in the UI.
  - neilpanchal 6y ago
    It's a toggle switch. The action and its current status is combined.
    - TeMPOraL 6y ago
      And then I have to keep hovering over the icon and guessing from the tooltip whether I'm muted or not. Unfortunately, different software tends to be inconsistent with toggle buttons - sometimes the icon tells you what is, sometimes it tells you what will happen if you click it.
  - odiroot 6y ago
    I already have it on my Thinkpad with KDE Plasma. Physical button to block the microphone, LED on the button itself and a tray icon with the microphone status displayed.
obilgic 6y ago
You all notice that this is a PR piece to get tech people interested in using google meet instead of zoom right.
- toper-centage 6y ago
  No, because tech people want software that works, has good UX etc. This is a PR piece for people that prefer software that has cutsie little backgrounds.
hota_mazi 6y ago
Google finally catching up to where Zoom was two years ago. Can we get a mute button visible at all times before 2024?
- amf12 6y ago
  > Can we get a mute button visible at all times before 2024? Is it just me or is the button visible at all times? I could see the button visible on the bottom of the screen at all times I used meet during a session with friends. I even tried it right now to make sure.
  - eCa 6y ago
    It’s not. Move the pointer outside of the window and the bottom bar should hide.
    - amf12 6y ago
      I just did. It did not hide. Maybe an experiment?
loosescrews 6y ago
Too bad it doesn't seem to be supported in Firefox.
- spurgu 6y ago
  Not yet at least. Jitsi also has background blur but it's only ok-ish on Chrome and unusably slow on Firefox.
nostromo 6y ago
I wish my coworkers would stop using background blur. It sucks and it’s distracting. Your hair and hands pop in and out of blur. Sometimes part of your face will blur. I don’t care if your workspace is messy or your kid walks in the room. I do care that we’re all being distracted by your weirdly blurred hair and hands.
- janekm 6y ago
  Your co-workers have a reasonable expectation of privacy regarding their home life and family members. Given that many had to start WfH with short notice meaning they couldn't relocate to circumstances enabling a dedicated home office space blurry hair and hands are a very reasonable compromise.
  - josalhor 6y ago
    > Your co-workers have a reasonable expectation of privacy regarding their home life and family members. I think you are overthinking it. I've seen people use it when it provides no real material benefit other than the placebo effect on the user to believe that the blur makes other people focus on their face.
  - fsloth 6y ago
    Yeah, this is why I use the background blur. I have my wife's and my hobbystuff behind me. Can't really align the video/pc setup any other way. Rather provide a blur than a confusion of guitars, sewing kit and such.
  - swiley 6y ago
    Is it really that hard to set up a greenscreen for this? I can look out accross the street and see a number of people who have done this in their tiny apartments. If I cared about people being able to see the room behind my WFH set up I would do it too. Thankfully for me it just points at the wall I use as a projector screen so there's nothing to see. Plus my team seems to have just given up on video anyway.
- josalhor 6y ago
  I find background blur even more distracting than background replacement. It's like my mind tries to picture the person that I am seeing in a particular environment and blur makes that process messy. But that's not always true tho, I have seen background replacement all over people's face (and yes, I seem to be the only one who thinks that's wrong).
arketyp 6y ago
They mention SIMD support, but It's unclear to me in what capacity the GPU is leveraged. The hair segmentation example on the MediaPipe webpage suggests it's evaluating the graph on the GPU though.
- jonex 6y ago
  The "Rendering Effects" section describes it in some detail: "Once segmentation is complete, we use OpenGL shaders for video processing and effect rendering" and some info on what that covers. (OpenGL parts runs on GPU)
kerng 6y ago
Interesting that this post made it to #1. It seems like Google marketing trick. Anyone who uses the blue realizes that it's far lacking in quality from other offerings and Google Meet UI is very bad also. Zoom, Teams, even WebEx are superior quality and usability wise.
- rplnt 6y ago
  Google Meet UX is the worst. Except WebEx which is somehow even worse. But WebEx is at least "complicated" and bad, Meet is just bad. Out of these I'm really surprised how "not as horrible" MS Teams are. Loads of functionality and the UX is bearable.
  - tantalor 6y ago
    Can you elaborate your criticism? Meet seems fine to me.
    - nexuist 6y ago
      My team does weekly Google Meet meetings along with WebEx. The biggest complaint I'd have is that Meet sacrifices functionality for cleanliness; everything useful is hidden behind some menu or popover, and you can only open one popover at a time (otherwise whatever you had open closes). This contrasts widely with WebEx, where most things (participants, controls, chat) can be shown at the same time, but also hidden if you don't want to see them. Meet seems complicated in comparison because views that are 0 clicks away on WebEx require 1-2 in Meet.
    - rplnt 6y ago
      Basically what other comments suggested. The popup menu that shows every time I move with the cursor and covers part of the screen. Can't be hidden on demand. Shows status I can't see without opening it (and covering part of the screen). Can't change my mute status without opening it (and covering part of the screen) or using a very obscure shortcut.
      - 6y ago
alblue 6y ago
Badly.
wdroz 6y ago
If you have a Windows computer with a RTX graphic card, you can use nvidia broadcast to get similar perks. It creates a virtual camera that you can select in whatever conference apps/browsers you are using. There are some works on OBS to get the green screen AI working, so I hope we will get that on GNU/Linux one day.
amq 6y ago
The single biggest missing feature compared to Zoom for my team is background noise cancellation. It's an unfortunate decision to limit it to Enterprise users.
hrktb 6y ago
I don’t understand this part: > In the current version, model inference is executed on the client’s CPU for low power consumption and widest device coverage. Naively I would think model inference done server side would have the lower CPU power (from the client point of view) and widest device coverage (client does nothing more), what am I missing ?
- blauditore 6y ago
  Probably increased latency due to additional round trips to the server. Perhaps image streams are sent between clients directly, but I'm not sure.
- jonex 6y ago
  It is done on the CPU instead of the GPU. GPU would seem like the natural choice for a convolution heavy model but was not used here for the mentioned reasons.
  - hrktb 6y ago
    Thanks, it makes sense
- Orphis 6y ago
  Some work needs to happen locally to show you a preview of what you're going to transmit, as it should for most video related work. If the segmentation is done server-side, then you need to sync it to the sender and reflect that quickly in the preview. It's probably not a great experience, at least for a launch.
mft_ 6y ago
As an aside, this example looks faked: https://1.bp.blogspot.com/-viEA4OY0sxA/X5s7IBwoXOI/AAAAAAAAGv0/nBYk9Nzxbc4Q_YEU6Ao2noAVCS4Ov9naACLcBGAsYHQ/s960/image5.gif https://1.bp.blogspot.com/-viEA4OY0sxA/X5s7IBwoXOI/AAAAAAAAG... As in, the blurred background looks totally different (light:dark, shapes, etc.) to the unblurred background. (I get that they’d need to do something funky to show blurred and unblurred backgrounds with the same foreground video, and faking it is likely easier than doing it programmatically, but this is just odd/sloppy.)
- kmisiunas 6y ago
  From the image caption "Background blur and background replacement, powered by MediaPipe on the web.".
- Jyaif 6y ago
  The fact that you think that the blurred background example is fake is a testament to how well background replacement work.
  - professor_v 6y ago
    Have you actually used it? In practice I wouldn't say it works at all really. The artifacts are terrible.
- germandude123 6y ago
  The left clip is an example of background blur. The right clip is an example of background replacement. This is why the blurred background on the left does not look anything like the unblurred background on the right.
jcims 6y ago
It would be nice if there was a webcam on the market that took actual lenses so you could get free, legit depth of field. Paying $700 for a used DSLR that has a clean hdmi out is not appealing, especially when I have a mirrorless from the same company that could probably do the same with a firmware update (that will never come)
- nucleardog 6y ago
  I think a cheaper solution would probably just be a depth sensing camera. Even a developer targeted Intel RealSense kit is only like $150. Consumer hardware could be much cheaper I imagine. Once you have depth information integrated with a camera, then it should be pretty trivial to do background removal. Whereas a 35mm f1.8 from Nikon is like $200 and whatever you mount it to is still going to need to do auto focusing and a bunch of other camera-y stuff to make it accessible to non photo geeks and then you’re going to need an off camera microphone so the entire call isn’t listening to your autofocus motor and...
mike_kamau 6y ago
Why are people replacing their backgrounds? I thought the whole point of having a video call is to see who you are talking to, and their environment to further enhance the effectiveness of the conversation. If you are in your kitchen, or under a tree, I definitely would like to see that because that environment will have an effect on how we communicate.
- gerbler 6y ago
  Sometimes people may not be comfortable sharing their backgrounds, and may not have convenient alternatives. For example, if you have a bed in the background it can be awkward and you might want to blur that out.
- adwww 6y ago
  I don't bother, but then I live in my own home and my background is an empty study. I have coworkers who are in house shares with 5 other adults all trying to work from home around tiny desks. Background blur for them is a nice way to hide some of the chaos of their living arrangements.
- spurgu 6y ago
  If the apartment is a mess in general. Table full of empty cans of beer. A dildo on a chair. Your wife randomly walking by in her underwear (not sure whether this would be unblurred?). In the above scenarios, if I'm not certain there aren't going to be ackward things in behind me, I'd want to blur or set a custom background. Back against a wall also works which is what a lot of people seem to be doing.
  - 0xffff2 6y ago
    Why not just turn off your camera? The blurring tech doesn't seem nearly reliable enough for me to trust it if my "office" was that much of a catastrophe.
    - Spooks 6y ago
      I think because seeing someone's face is more important than seeing their background
rkagerer 6y ago
Very awesome! Although there's a lot of blurring on the shoulder of the guy at the beach: https://i.imgur.com/D5ueGUh.png https://i.imgur.com/D5ueGUh.png
sercand 6y ago
I guess this is why when I open Google Meet my fan starts spinning and making noise.
adioe3 6y ago
Not supported in Firefox or Safari.
madeofpalk 6y ago
I wish one of the lovely people in the examples were wearing headphones.
kevingadd 6y ago
The listed CPU usage / elapsed time for the features in this article is obscene. Only 62FPS = maxing out at least one core on a 60hz display, just to replace/blur a background. Kiss your laptop's battery goodbye. How is this worth it?
The_rationalist 6y ago
What would be the inference time like on a modern smartphone?
consolelog2000 6y ago
Hi! I am Alexander from Work Reactor. We are interviewing potential business owners from the USA to understand their needs, problems or inconveniences while creating a business or a startup. If you take this interview, we guarantee to give you access to our product + its premium features and services as soon as it will be developed. We’d love to ask you several questions for our cust dev interview. You can answer the questions in any convenient form you want (via chat , zoom or as you prefer).
consolelog2000 6y ago
Hi! I am Alexander from Work Reactor. We are interviewing potential business owners from the USA to understand their needs, problems or inconveniences while creating a business or a startup. If you take this interview, we guarantee to give you access to our product + its premium features and services as soon as it will be developed. We’d love to ask you several questions for our cust dev interview. You can answer the questions in any convenient form you want (via chat , zoom or as you prefer).
acdha 6y ago
It’s funny how Google pours time into things like this but the last person I know who uses a Google chat product just stopped because it’s less reliable than Zoom. Losing 15 minutes with someone trying to get the sound working counts more than a gimmick many people never notice, not to mention now even normal people don’t want to yet install another app because they expect it to be cancelled soon.
- sjs7007 6y ago
  Most corporate employees probably don't even have a choice of which app they can use though. They are tied in by whatever their company chose.
  - acdha 6y ago
    This is only true to the extent that the IT department has complete control and is insulated from user opinion. That tends to result in choices like WebEx or the ever popular shadow IT option. This is especially true now that everyone is doing this and the odds approach certainty that if someone is having problems they’ll suggest switching to a different product they know works. Given the number of IT people I’ve heard express concerns about UI quality and eventual cancellation even for enterprise purchases, it’s also far from a given that the IT department is just blindly pushing a product.
tsycho 6y ago
Having used both Zoom and Meet extensively now for the past 6 months, my experience is: 1/ Your internet connection, especially upload bandwidth and latency matter a lot. 2/ Zoom's desktop app performs very well, but its web version is atrocious. Not just because of the dark patterns they use to force you to install the desktop app, but also its performance is terrible compared to its desktop version, as well as worse than almost everything else. Unfortunately, I don't trust them and refuse to use their desktop app on anything but my iPad. 3/ Meet used to be bad like Zoom on web 6 months ago, but has improved a lot and is slowly approaching Zoom desktop in performance. I have noticed that Meet on my work GSuite calls at work perform much better than on my personal account. This might be explained by #1 above I.e. my family has worse internet connections than my coworkers, but I am not sure if all improvements have been rolled out to personal accounts.
- izacus 6y ago
  Meet certanly rolls out improvements for GSuite before public ones. I think there's even a GSuite setting of "release channel" where you can control how early you get these improvements.
- boulos 6y ago
  Disclosure: I work on Google Cloud (but not Meet). For the last item, personal accounts (only?) default to send and receive video at lower resolution (360p). So if you meant that the quality is lower, you can set it on both sides to 720p. Edit: I don’t think Meet remembers those settings though, so you have to do it every time (and show your family members how to do so).
  - kevincox 6y ago
    It doesn't remember them, it is frustrating. As a legacy free GApps user it is even more confusing because the admin page gives me an option to default to higher quality video but that doesn't do anything.
  - ss3000 6y ago
    This is maddening... Why does Google, with all the resources at its disposal, choose to cheap out like this when competitors in the video chat space (from tiny startups to gigantic corporation of similar size) have offered near native resolution video chat for ages? Are they even _trying_ to compete?
daxfohl 6y ago
From the title I thought this was two distinct features running in google's background that used Web ML to figure out how to work together.
lern_too_spel 6y ago
Why isn't Mediapipe built on gstreamer? Nvidia gets this right. If you're slinging frame buffers around, use an API that there is already an ecosystem for.
Liskni_si 6y ago
A few people commented that the foreground/background detection cannot keep up with movements fast enough. Here's an idea that might help, although I'm not sure if it can realistically be done: When the video is encoded, the codec does motion estimation (among other things) to reduce the bandwidth required. So why don't we use the motion vectors from the video codec to modify the foreground/background mask in real time? Obviously this is going to create weird artifacts pretty soon, but it might just be good enough for a few frames before the ML model produces another accurate mask.
supernova87a 6y ago
I have a different issue with Google Meet. I have observed in the last couple months that whenever I create a Google Calendar invite with others, Google has started inserting a Google Meet conference as the location to meet. It was one thing to ask/offer this as an option if you'd like to use it, but now Google is positioning it as if you had chosen that. So if you left it empty, because you usually use some other understood method with your friends/colleagues, now your participants are confused and think you wanted to use Google Meet. I think that's going too far to get people to adopt your product.
- fx32s 6y ago
  I was able to change this default in the settings.
  - supernova87a 6y ago
    Ah, thanks for that! It's still shady that they turned that on by default to get people to use it...
- hongalex 6y ago
  I noticed this too, but I actually got a tooltip popup that notes this is something that can be disabled in the calendar settings. The specific checkbox is "Automatically add Google Meet video conferences to events I create" Disclaimer: I work at Google but not on these products. Edit: it seems the tooltip only appears the first time you try to add Meet. After that it doesn't appear and you have to go into settings.
Nimitz14 6y ago
I was going to point out that xnnpack was basically created by a single guy who also created qnnpack, and how amazing it is for the work of a single guy to have so much impact, then I realized he posted it! Congratz dude!
387032228 6y ago
387032228

427 points by Marat_Dukhan 6y ago

Unrelated to the ML, but I don't like the fact I noticed that all the people in the clips are "diverse". No clue whether Google did this on purpose, but I feel like I've been trained to notice this due to the diversity movement thing currently occurring and I don't like it.

ibuildthings 6y ago
In this particular case, why is testing/demonstrating on diverse set of individuals a bad thing ? A personal anecdote is that a few years back the automatic door sensors in my university did not work on my skin tone.
- ZephyrBlu 6y ago
  It's not a bad thing, I just dislike that I noticed it.
  - gexla 6y ago
    It's video conferencing software. Makes sense that they might put together imagery which might suggest people meeting from different corners of the planet. But sure, I get your point. I didn't notice this myself, but I have been living on side of the planet opposite from where I was born for the past decade.
- easytiger 6y ago
  Automatic door sensors are in my experience universally infra red. In fact I don't think I've ever seen any camera technology used in that context. Are you saying they used cameras to open the doors?
mdoms 6y ago
Why don't you like it?
- ZephyrBlu 6y ago
  Because I don't think I would have noticed a couple of years ago.
dharma1 6y ago
Sorry not everyone on the planet looks like you
- hombre_fatal 6y ago
  They seem to be lamenting that they now notice skin color (due to all the racial essentialism of the last N years) where they used to not notice.
  - riffraff 6y ago
    I understand the feeling somewhat: it is very noticeable how TV shows for example now have expanded the default cast from "all white people + token black, maybe a gay individual" to "all white people + token black woman + token asian + token non-binary, maybe a transgender individual". It feels forced and ham-fisted, but I don't see how this could be made in a better way. Possibly people growing up with this will not notice it at all, and it will be good for them, it's only us old farts that need to adapt.
    - ZephyrBlu 6y ago
      Unfortunately I'm Gen Z, not an old fart :/. No clue how most Gen Z people feel about this type of thing though.
  - saurik 6y ago
    I think the issue is that, were it not for everyone being forced to notice, everyone was defaulting to caring so little that we just used the people around us or who we had existing connections with (for demos, training data, employees... whatever), and everything was horribly biased due to numerous reasons. So, while before you weren't being forced to pay attention, everything was more racist than now, where you are being forced to actually make an effort to be anti-racist. To put this into a programming metaphor, to me this is like being triggered by someone going out of their way to add a buffer overflow check due to a bunch of people spending the last decade screaming about buffer overflow security issues. Sure, before you didn't have to notice, but your code was probably also horribly insecure; now, everyone gets angry if you don't take at least minimal precautions, and people are even advocating that you use more secure languages from the start (to the point of questioning your architectural decisions if you don't), so you are being forced to pay attention--and sure, it seems a bit annoying and like extra work that a bunch of fanatics are foisting on you, and if someone had taken the time for their code to be secure before you might not have noticed (I mean, you weren't ever against security) but now it is screaming at you "this is because of those annoying security people rubbing our noses in our buffer overflows" so you get angry because this is taking time away from "getting the real work done" on your product--but the reality is that your code used to have glaring security issues that affected people who weren't you, and it sucked; everyone is better off for you paying attention now, and maybe one day we can fix the systemic problem and no one will have to put in such obvious effort to avoid being part of the problem, but we clearly aren't there yet. Being angry about this just comes off as not giving a shit about security: you didn't notice these checks before, because the code was just as good to you, for the criteria you were bothering to pay attention to, without the checks as it would have been with them, and the whole point is that that wasn't true... it was actually much worse; similarly, being angry about people making active efforts to have diversity in product advertising frankly just makes you come off as not giving a shit about minority representation :/.

mdoms 6y ago

Honestly it's the worst implementation I've seen of this technology yet. Just absolute and total garbage.

yjftsjthsd-h 6y ago
Why? What makes this one worse?

sillysaurusx 6y ago

Happy to see ML become mainstream. In the future, I don't think ML will be a separate field of programming. It'll just be "programming," the same way webdev is. There's a tendency to think of ML as "not programming," or something other than just plain programming. But as the tooling matures, that'll go away. (Lisp used to be considered "AI programming," till it became useful in many other contexts.)

sltEvas 6y ago
ML will become a library. It has about as much to do with programming as a compiler. You don't need to know what it does, you just need to know how to make it do things. The problem with ML currently is that nobody really knows how to do things and that you have a million parameters that need tuning and most algorithms need continuous improvement and fine tuning to the use case. There is nothing "mainstream" about ML at this point, except that everyone wants to use it. In maybe a decade, it might be found in standard libraries of programming languages and on top of things like `Math.abs`, we will have `ML.textToSpeech("Hello world")`, or `ML.isCat(image)`, etc. However, the problem I see with that is that no matter how far we wind the clock forward, we will only be able to put the most simplistic use cases into a library. `ML.isCat()` could be one of those, since most humans will be able to image categorization, it stands to reason that you could put this into a library. However, most industry application involved highly customized ML algorithms that are optimized for a very specific use-case. So there will always be a need for a research team in big companies at least. Maybe smaller companies will try to build their stuff by chaining libraries together.
- fhaodhdms 6y ago
  Fwiw macs have had an equivalent functionality for both text to speech and speech to text for at least 17 years to my memory. The quality is poor compared to today's server-driven approaches, of course, but the functionality has been there if you're willing to articulate yourself clearly.
- virgilp 6y ago
  There's never going to be a `ML.isCat(image)`, just like there isn't a `Math.solveProblem(hypothesis)`. Yes you do have `Math.abs` and you're going to have stuff `model.fit()` and `layers.dense()` - but something like `ML.isCat` is too specific to be used in a library

jtokoph 6y ago

The example video clips in the post look nothing like me and my team's view when using the new feature. Most of the time half of our hair gets blurred or replaced and hand gestures will cause either our hands or head to disappear.

gundmc 6y ago
It seems to have gotten a little better recently, but my experience matches yours. It really struggles when I wear over-ear headphones - they sort of phase in and out of existence. The other thing I've noticed is the background blur absolutely annihilates my CPU. To the point where I would rather just turn off my camera if I don't want my background visible.
samtheprogram 6y ago
I can vouch for this. I haven’t really needed the background blur feature personally, but I’ve tried it and both myself, colleagues, and friends — pretty much everyone I’ve talked to that has used it — loathe Google Meet’s background blur, and prefer Zoom’s by far. In my experience, it doesn’t completely cover the background most of the time, and if you move at all, as you point out, it can’t keep up. Kind of funny to see Google engineering blogging about it when it feels extremely half baked. This makes me sad, because in all other areas, I think Meet excels well beyond the competition. EDIT: removed my general sentiment on Google
- judge2020 6y ago
  "Turns out our training set was comprised entirely of backdrops from Google HQ. Sorry everyone else!"
  - dontblink 6y ago
    Everyone is wfh @ Google
- _pastel 6y ago
  "Half baked" misrepresents the difficulty this task. Yes, Zoom does it better, but it's _still_ an excellent and interesting engineering accomplishment. I've always wondered what proportion of modern real-time video effects rely on ML vs. classical image processing; this not only answers that question, but provides details down to the level of model architecture and the final latency and IOU benchmarks. Of course I'd be more interested to read how Zoom manages to do even better, but I'm not holding my breath for them to publish those details.

chdjakdkgb 6y ago

Wtf happened to hangouts? How many video products does google have?

samtheprogram 6y ago
Meet is business oriented and offers features that Hangouts does not, e.g. dialing in via phone. It also requires a G Suite account (or did before COVID, IIRC).
- Polylactic_acid 6y ago
  Didn't hangouts have a business version as well?
  - Liquid_Fire 6y ago
    That's the same product. It was renamed to Meet.
senectus1 6y ago
Hangouts is EOL :-( We're being herded into the new more useless products.
- toper-centage 6y ago
  Hangouts was always terrible and bloated. I miss Google Talk. Instead, we get something more terrible and bloated than Hangouts.
  - sumedh 6y ago
    Instead of focussing on Google Talk we have at least 5 different Apps from Google pretty much doing the same thing.
hoveringhen 6y ago
Hangouts is being replaced by Chat and Meet.

vinhboy 6y ago

Blur is awesome. Way better, less distracting, than backgrounds. Everyone who try it uses it permanently because it works so well. I also think it makes the subject look better for some reason.

probably_wrong 6y ago
Here's a tip: take a picture of your real, actual background from the POV of your webcam, and set that as your meeting background. Advantages: it looks natural, it covers whatever is going on behind you (in case you are not alone and people walks by, or if your living room is messy), and it blends better than fake backgrounds (because it's the same image behind it). I have a picture of my office that I use both at home and at my real office, and most people can't tell. And since I took the picture with my phone which has better resolution, my video feed looks better for cheap.

dharma1 6y ago

Sony released some software a couple of months ago that lets you use most of their DSLRs as webcams with USB. My goodness, paired with a fast lens, what a difference to my MacBook webcam, even with these ml blurred backgrounds! It's only 720p and around 15fps but real shallow dof, very little sensor noise, autofocus works. Well worth trying if you have a Sony camera from the last few years. Sensor size and good optics still wins. Having said that,the effort and detail gone into this feature is very impressive, enjoyed the blog post. Also webassembly SIMD looks super cool, looking forward to a new class of webapps using wasm.

kenhwang 6y ago
Canon did too! Definitely a huge upgrade over a typical webcam. I'm using my old T1i which can be had for less than $50 these days, plus you can pick up a 18-55mm kit lens for like $20 and the video quality blows away any webcam, especially for the same price. Also recommend battery->power adapter.
- silly-silly 6y ago
  What is the canon software called ?
  - swang 6y ago
    EOS Webcam Utility https://www.usa.canon.com/internet/portal/us/home/support/self-help-center/eos-webcam-utility https://www.usa.canon.com/internet/portal/us/home/support/se...
- callmeal 6y ago
  The T1i is not listed on their compatibility page. Which version of the utility did you get for it?
  - polpo 6y ago
    I have a T2i that I use with the EOS Webcam Utility and it works just fine despite not being listed as compatible.

neilpanchal 6y ago

Aside: Imagine you’re driving down the road and you need to make a right turn. Well, for some reason the steering wheel is stowed away and disappeared! You need to hover your hand around the center console in a specific area to be able to expose it. Out comes the steering wheel and now you can make a right turn. Google UX/UI team: Please fucking make the mute/unmute button visible at all times.

askvictor 6y ago
More important than the button is the status indicator - I need to know if the call is muted or not. Even better, promote it to an OS-level icon/badge/overlay. If my mic is actively in use, please make it blindingly obvious.
- captn3m0 6y ago
  The color indicator for mic on/off is terribly for Meet.
  - nathancahill 6y ago
    Absolutely horrible. Both states are equally plausible in the UI.
- neilpanchal 6y ago
  It's a toggle switch. The action and its current status is combined.
  - TeMPOraL 6y ago
    And then I have to keep hovering over the icon and guessing from the tooltip whether I'm muted or not. Unfortunately, different software tends to be inconsistent with toggle buttons - sometimes the icon tells you what is, sometimes it tells you what will happen if you click it.
- odiroot 6y ago
  I already have it on my Thinkpad with KDE Plasma. Physical button to block the microphone, LED on the button itself and a tray icon with the microphone status displayed.

obilgic 6y ago

You all notice that this is a PR piece to get tech people interested in using google meet instead of zoom right.

toper-centage 6y ago
No, because tech people want software that works, has good UX etc. This is a PR piece for people that prefer software that has cutsie little backgrounds.

hota_mazi 6y ago

Google finally catching up to where Zoom was two years ago. Can we get a mute button visible at all times before 2024?

amf12 6y ago
> Can we get a mute button visible at all times before 2024? Is it just me or is the button visible at all times? I could see the button visible on the bottom of the screen at all times I used meet during a session with friends. I even tried it right now to make sure.
- eCa 6y ago
  It’s not. Move the pointer outside of the window and the bottom bar should hide.
  - amf12 6y ago
    I just did. It did not hide. Maybe an experiment?

loosescrews 6y ago

Too bad it doesn't seem to be supported in Firefox.

spurgu 6y ago
Not yet at least. Jitsi also has background blur but it's only ok-ish on Chrome and unusably slow on Firefox.

nostromo 6y ago

I wish my coworkers would stop using background blur. It sucks and it’s distracting. Your hair and hands pop in and out of blur. Sometimes part of your face will blur. I don’t care if your workspace is messy or your kid walks in the room. I do care that we’re all being distracted by your weirdly blurred hair and hands.

janekm 6y ago
Your co-workers have a reasonable expectation of privacy regarding their home life and family members. Given that many had to start WfH with short notice meaning they couldn't relocate to circumstances enabling a dedicated home office space blurry hair and hands are a very reasonable compromise.
- josalhor 6y ago
  > Your co-workers have a reasonable expectation of privacy regarding their home life and family members. I think you are overthinking it. I've seen people use it when it provides no real material benefit other than the placebo effect on the user to believe that the blur makes other people focus on their face.
- fsloth 6y ago
  Yeah, this is why I use the background blur. I have my wife's and my hobbystuff behind me. Can't really align the video/pc setup any other way. Rather provide a blur than a confusion of guitars, sewing kit and such.
- swiley 6y ago
  Is it really that hard to set up a greenscreen for this? I can look out accross the street and see a number of people who have done this in their tiny apartments. If I cared about people being able to see the room behind my WFH set up I would do it too. Thankfully for me it just points at the wall I use as a projector screen so there's nothing to see. Plus my team seems to have just given up on video anyway.
josalhor 6y ago
I find background blur even more distracting than background replacement. It's like my mind tries to picture the person that I am seeing in a particular environment and blur makes that process messy. But that's not always true tho, I have seen background replacement all over people's face (and yes, I seem to be the only one who thinks that's wrong).

arketyp 6y ago

They mention SIMD support, but It's unclear to me in what capacity the GPU is leveraged. The hair segmentation example on the MediaPipe webpage suggests it's evaluating the graph on the GPU though.

jonex 6y ago
The "Rendering Effects" section describes it in some detail: "Once segmentation is complete, we use OpenGL shaders for video processing and effect rendering" and some info on what that covers. (OpenGL parts runs on GPU)

kerng 6y ago

Interesting that this post made it to #1. It seems like Google marketing trick. Anyone who uses the blue realizes that it's far lacking in quality from other offerings and Google Meet UI is very bad also. Zoom, Teams, even WebEx are superior quality and usability wise.

rplnt 6y ago
Google Meet UX is the worst. Except WebEx which is somehow even worse. But WebEx is at least "complicated" and bad, Meet is just bad. Out of these I'm really surprised how "not as horrible" MS Teams are. Loads of functionality and the UX is bearable.
- tantalor 6y ago
  Can you elaborate your criticism? Meet seems fine to me.
  - nexuist 6y ago
    My team does weekly Google Meet meetings along with WebEx. The biggest complaint I'd have is that Meet sacrifices functionality for cleanliness; everything useful is hidden behind some menu or popover, and you can only open one popover at a time (otherwise whatever you had open closes). This contrasts widely with WebEx, where most things (participants, controls, chat) can be shown at the same time, but also hidden if you don't want to see them. Meet seems complicated in comparison because views that are 0 clicks away on WebEx require 1-2 in Meet.
  - rplnt 6y ago
    Basically what other comments suggested. The popup menu that shows every time I move with the cursor and covers part of the screen. Can't be hidden on demand. Shows status I can't see without opening it (and covering part of the screen). Can't change my mute status without opening it (and covering part of the screen) or using a very obscure shortcut.
    - 6y ago

alblue 6y ago

Badly.

wdroz 6y ago

If you have a Windows computer with a RTX graphic card, you can use nvidia broadcast to get similar perks. It creates a virtual camera that you can select in whatever conference apps/browsers you are using. There are some works on OBS to get the green screen AI working, so I hope we will get that on GNU/Linux one day.

amq 6y ago

The single biggest missing feature compared to Zoom for my team is background noise cancellation. It's an unfortunate decision to limit it to Enterprise users.

hrktb 6y ago

I don’t understand this part: > In the current version, model inference is executed on the client’s CPU for low power consumption and widest device coverage. Naively I would think model inference done server side would have the lower CPU power (from the client point of view) and widest device coverage (client does nothing more), what am I missing ?

blauditore 6y ago
Probably increased latency due to additional round trips to the server. Perhaps image streams are sent between clients directly, but I'm not sure.
jonex 6y ago
It is done on the CPU instead of the GPU. GPU would seem like the natural choice for a convolution heavy model but was not used here for the mentioned reasons.
- hrktb 6y ago
  Thanks, it makes sense
Orphis 6y ago
Some work needs to happen locally to show you a preview of what you're going to transmit, as it should for most video related work. If the segmentation is done server-side, then you need to sync it to the sender and reflect that quickly in the preview. It's probably not a great experience, at least for a launch.

mft_ 6y ago

As an aside, this example looks faked: https://1.bp.blogspot.com/-viEA4OY0sxA/X5s7IBwoXOI/AAAAAAAAGv0/nBYk9Nzxbc4Q_YEU6Ao2noAVCS4Ov9naACLcBGAsYHQ/s960/image5.gif https://1.bp.blogspot.com/-viEA4OY0sxA/X5s7IBwoXOI/AAAAAAAAG... As in, the blurred background looks totally different (light:dark, shapes, etc.) to the unblurred background. (I get that they’d need to do something funky to show blurred and unblurred backgrounds with the same foreground video, and faking it is likely easier than doing it programmatically, but this is just odd/sloppy.)

kmisiunas 6y ago
From the image caption "Background blur and background replacement, powered by MediaPipe on the web.".
Jyaif 6y ago
The fact that you think that the blurred background example is fake is a testament to how well background replacement work.
- professor_v 6y ago
  Have you actually used it? In practice I wouldn't say it works at all really. The artifacts are terrible.
germandude123 6y ago
The left clip is an example of background blur. The right clip is an example of background replacement. This is why the blurred background on the left does not look anything like the unblurred background on the right.

jcims 6y ago

It would be nice if there was a webcam on the market that took actual lenses so you could get free, legit depth of field. Paying $700 for a used DSLR that has a clean hdmi out is not appealing, especially when I have a mirrorless from the same company that could probably do the same with a firmware update (that will never come)

nucleardog 6y ago
I think a cheaper solution would probably just be a depth sensing camera. Even a developer targeted Intel RealSense kit is only like $150. Consumer hardware could be much cheaper I imagine. Once you have depth information integrated with a camera, then it should be pretty trivial to do background removal. Whereas a 35mm f1.8 from Nikon is like $200 and whatever you mount it to is still going to need to do auto focusing and a bunch of other camera-y stuff to make it accessible to non photo geeks and then you’re going to need an off camera microphone so the entire call isn’t listening to your autofocus motor and...

mike_kamau 6y ago

Why are people replacing their backgrounds? I thought the whole point of having a video call is to see who you are talking to, and their environment to further enhance the effectiveness of the conversation. If you are in your kitchen, or under a tree, I definitely would like to see that because that environment will have an effect on how we communicate.

gerbler 6y ago
Sometimes people may not be comfortable sharing their backgrounds, and may not have convenient alternatives. For example, if you have a bed in the background it can be awkward and you might want to blur that out.
adwww 6y ago
I don't bother, but then I live in my own home and my background is an empty study. I have coworkers who are in house shares with 5 other adults all trying to work from home around tiny desks. Background blur for them is a nice way to hide some of the chaos of their living arrangements.
spurgu 6y ago
If the apartment is a mess in general. Table full of empty cans of beer. A dildo on a chair. Your wife randomly walking by in her underwear (not sure whether this would be unblurred?). In the above scenarios, if I'm not certain there aren't going to be ackward things in behind me, I'd want to blur or set a custom background. Back against a wall also works which is what a lot of people seem to be doing.
- 0xffff2 6y ago
  Why not just turn off your camera? The blurring tech doesn't seem nearly reliable enough for me to trust it if my "office" was that much of a catastrophe.
  - Spooks 6y ago
    I think because seeing someone's face is more important than seeing their background

rkagerer 6y ago

Very awesome! Although there's a lot of blurring on the shoulder of the guy at the beach: https://i.imgur.com/D5ueGUh.png https://i.imgur.com/D5ueGUh.png

sercand 6y ago

I guess this is why when I open Google Meet my fan starts spinning and making noise.

adioe3 6y ago

Not supported in Firefox or Safari.

madeofpalk 6y ago

I wish one of the lovely people in the examples were wearing headphones.

kevingadd 6y ago

The listed CPU usage / elapsed time for the features in this article is obscene. Only 62FPS = maxing out at least one core on a 60hz display, just to replace/blur a background. Kiss your laptop's battery goodbye. How is this worth it?

The_rationalist 6y ago

What would be the inference time like on a modern smartphone?

consolelog2000 6y ago

Hi! I am Alexander from Work Reactor. We are interviewing potential business owners from the USA to understand their needs, problems or inconveniences while creating a business or a startup. If you take this interview, we guarantee to give you access to our product + its premium features and services as soon as it will be developed. We’d love to ask you several questions for our cust dev interview. You can answer the questions in any convenient form you want (via chat , zoom or as you prefer).

consolelog2000 6y ago

acdha 6y ago

It’s funny how Google pours time into things like this but the last person I know who uses a Google chat product just stopped because it’s less reliable than Zoom. Losing 15 minutes with someone trying to get the sound working counts more than a gimmick many people never notice, not to mention now even normal people don’t want to yet install another app because they expect it to be cancelled soon.

sjs7007 6y ago
Most corporate employees probably don't even have a choice of which app they can use though. They are tied in by whatever their company chose.
- acdha 6y ago
  This is only true to the extent that the IT department has complete control and is insulated from user opinion. That tends to result in choices like WebEx or the ever popular shadow IT option. This is especially true now that everyone is doing this and the odds approach certainty that if someone is having problems they’ll suggest switching to a different product they know works. Given the number of IT people I’ve heard express concerns about UI quality and eventual cancellation even for enterprise purchases, it’s also far from a given that the IT department is just blindly pushing a product.

tsycho 6y ago

Having used both Zoom and Meet extensively now for the past 6 months, my experience is: 1/ Your internet connection, especially upload bandwidth and latency matter a lot. 2/ Zoom's desktop app performs very well, but its web version is atrocious. Not just because of the dark patterns they use to force you to install the desktop app, but also its performance is terrible compared to its desktop version, as well as worse than almost everything else. Unfortunately, I don't trust them and refuse to use their desktop app on anything but my iPad. 3/ Meet used to be bad like Zoom on web 6 months ago, but has improved a lot and is slowly approaching Zoom desktop in performance. I have noticed that Meet on my work GSuite calls at work perform much better than on my personal account. This might be explained by #1 above I.e. my family has worse internet connections than my coworkers, but I am not sure if all improvements have been rolled out to personal accounts.

izacus 6y ago
Meet certanly rolls out improvements for GSuite before public ones. I think there's even a GSuite setting of "release channel" where you can control how early you get these improvements.
boulos 6y ago
Disclosure: I work on Google Cloud (but not Meet). For the last item, personal accounts (only?) default to send and receive video at lower resolution (360p). So if you meant that the quality is lower, you can set it on both sides to 720p. Edit: I don’t think Meet remembers those settings though, so you have to do it every time (and show your family members how to do so).
- kevincox 6y ago
  It doesn't remember them, it is frustrating. As a legacy free GApps user it is even more confusing because the admin page gives me an option to default to higher quality video but that doesn't do anything.
- ss3000 6y ago
  This is maddening... Why does Google, with all the resources at its disposal, choose to cheap out like this when competitors in the video chat space (from tiny startups to gigantic corporation of similar size) have offered near native resolution video chat for ages? Are they even _trying_ to compete?

daxfohl 6y ago

From the title I thought this was two distinct features running in google's background that used Web ML to figure out how to work together.

lern_too_spel 6y ago

Why isn't Mediapipe built on gstreamer? Nvidia gets this right. If you're slinging frame buffers around, use an API that there is already an ecosystem for.

Liskni_si 6y ago

A few people commented that the foreground/background detection cannot keep up with movements fast enough. Here's an idea that might help, although I'm not sure if it can realistically be done: When the video is encoded, the codec does motion estimation (among other things) to reduce the bandwidth required. So why don't we use the motion vectors from the video codec to modify the foreground/background mask in real time? Obviously this is going to create weird artifacts pretty soon, but it might just be good enough for a few frames before the ML model produces another accurate mask.

supernova87a 6y ago

I have a different issue with Google Meet. I have observed in the last couple months that whenever I create a Google Calendar invite with others, Google has started inserting a Google Meet conference as the location to meet. It was one thing to ask/offer this as an option if you'd like to use it, but now Google is positioning it as if you had chosen that. So if you left it empty, because you usually use some other understood method with your friends/colleagues, now your participants are confused and think you wanted to use Google Meet. I think that's going too far to get people to adopt your product.

fx32s 6y ago
I was able to change this default in the settings.
- supernova87a 6y ago
  Ah, thanks for that! It's still shady that they turned that on by default to get people to use it...
hongalex 6y ago
I noticed this too, but I actually got a tooltip popup that notes this is something that can be disabled in the calendar settings. The specific checkbox is "Automatically add Google Meet video conferences to events I create" Disclaimer: I work at Google but not on these products. Edit: it seems the tooltip only appears the first time you try to add Meet. After that it doesn't appear and you have to go into settings.

Nimitz14 6y ago

I was going to point out that xnnpack was basically created by a single guy who also created qnnpack, and how amazing it is for the work of a single guy to have so much impact, then I realized he posted it! Congratz dude!

387032228 6y ago

387032228