7 ms·

Can You See Me Now? A measurement study of Zoom, Webex, and Meet

fouc 5y ago
> Despite its significance, there has not been any systematic study characterizing the user-perceived performance of existing videoconferencing systems other than anecdotal reports. > We find that the existing videoconferencing systems vary in terms of geographic scope, which in turns determines streaming lag experienced by users. We also observe that streaming rate can change under different conditions (e.g., number of users in a session, mobile device status, etc), which affects user-perceived streaming quality. Beyond these findings, our measurement methodology can enable reproducible benchmark analysis for any types of comparative or longitudinal study on available videoconferencing systems. Seems like a very useful study at first glance!
travoc 5y ago
I worked at a company that blocked the native clients for every conference system except WebEx. The Zoom and Skype web experience was not good, particularly since all endpoints were loaded with half a dozen Infosec agents and sending all traffic through web proxies on the other coast. Surprise, the somewhat primitive Webex offered the best experience in that scenario and vendors who wanted to meet were inclined to accommodate us.
- azinman2 5y ago
  Webex interface is… not great. Zoom isn’t either and I’m quite surprised it’s as popular as it is given the UX. Surprising these things are as difficult as they are and yet get such adoption.
  - soylentnewsorg 5y ago
    The webex interface is perfectly fine, but if you are comparing it for the use case of zoom and skype, your target interface is wrong for the tool you're using. Webex is for working on something together. Multiple users using an application to manage some piece of hardware or software for example. Zoom and skype are for talking about something together. Let's say you get a new server. It needs several teams to set it up - storage to provision/rescan/configure disk, infosec to do domain/antivirus/dlp work, db team to get the database going. They're going to share an rdp session over webex and hand it off to each other to type commands, with everyone observing and catching mistakes. Yes, it's possible to give desktop control to someone on zoom, however that is a secondary afterthought of zoom, while for webex it's the primary function. Right tool for the job - don't transport a piano in a sports car then complain the sports car is.. not great.
    - blacksmith_tb 5y ago
      I am not sure you can answer a subjective assessment with a blanket dismissal - obviously at some level the webex UI can't be completely unusable (lots of people do use it), but apparently some people just don't like it. It has what strikes me as an unpleasant installer (like BlueJeans, it seems to only work if you download an installer and run it for every call, which seems excessive somehow).
    - azinman2 5y ago
      We use Webex at work (a very large tech company). I use it for it’s intended purposes. My issues: 1. The reaction buttons don’t indicate that anything is actually happening for you when you click them, you only can see others reactions. They’re also quite hidden when they come, make no noise so easy to miss, and don’t tie to the video. 2. The “people joined” ding is very non-intelligent. It used to be when 30+ people were queued for something to start, once it did it just dinged like crazy. Now it seems work has turned that off in a way I cannot turn back on, which means when people are late for a 1:1 they’re staring at me and I have no idea since I now have people blocking. A heuristic that would be aware of # participants, when people are joining, and batching up dings I cannot imagine would be groundbreaking research when this is your product you’re building. 3. The actual controls for video on mobile to switch camera is quite bad. 4. Screen sharing works _ok_, except it has no integration with the resolutions on both sides, Apple’s accessibility-based zoom, I’m given a red dot for a cursor but it doesn’t appear that way on the other side, and cursor tracking is extremely laggy and jumpy on the remote side despite my red cursor that’s giving me the impression of a laser pointer equivalent. 5. When someone is screen sharing, the controls for what to do with the video of others are not very good. Order is non-deterministic (afaik), not easily changed, provides few layouts, and not easily hidden. 6. When screen sharing I’m given a large list of all my windows to choose from. They’re in effectively a random order (not alphabetical, not by z-index on my Mac, etc), and the mini photo of them are usually unintelligible and difficult to pin point the one I want. These are just some of the bad UX that I encounter daily in work usage. I could probably come up with a list of 10+ more issues without even thinking much about it, and we haven’t even touched the “interface” level issues (eg actual design).
      - 5y ago
  - Pxtl 5y ago
    Having used MSTeams, I've stopped complaining about Zoom. It's stable and straightforward. The on-call UI has its warts, especially on mobile, but otherwise it's fine. Teams, by contrast, is buggy and inconsistent and weird and has the awful arbitrary rectangle subdividing of the chat window instead of just showing me the other people's full cameras.
- wenc 5y ago
  > Surprise, the somewhat primitive Webex offered the best experience in that scenario But it’s not a surprise in this scenario since WebEx is the only one not hobbled, no? I’ve used all 3 and WebEx is second to Zoom but better than Skype.
- dagw 5y ago
  The company I work for also blocks the native Zoom client, and I know of several other companies that do it as well. If we have a Zoom meeting we have to use the web client. Can anybody give me a reasonable rationale for why this is so common?
camgunz 5y ago
My prayers have been answered. Google Meet devs I beg you, make the app more efficient.
- Orphis 5y ago
  Ok, because you asked nicely!
- throwaway81523 5y ago
  Wait, Google Meet? I thought this was going to be about Jitsi Meet (not joking). Crap. I hadn't heard of Google Meet. So now I still want to know how Jitsi compares.
finnh 5y ago
I was on a webex recently, and after years of using Zoom I was all "oh, wow - is see Webex has a 'make my skin look worse' feature. Gotta compete with Zoom somehow"
- pempem 5y ago
  It can be downright startling!
athenot 5y ago
> In case of Webex, all sessions created in the US appear to be relayed via its infrastructure in US-east. This causes the sessions among users in US-west to be subject to artificial detour, inflating their streaming lag. Former webexer here. That depends on many criteria, sessions can go through multiple locations depending on where the customer is located, the features being used, the type of customer, etc. Since Webex also does transcoding (useful when a 4k wall system and a cell phone are on a same meeting), that can influence where the traffic flows. Webex, Zoom and Google also all have their own backbones to reduce the number of hops over the public internet. In this case the researchers appear to be based in New Jersey so they would be more likely to enter the internal network via the eastern POPs.
listenallyall 5y ago
Personally, I've been pretty impressed with Jitsi especially given that it's open source. So far it's been easy and the people on the other end have had no problem connecting, despite not having previously downloaded any client software. I had high hopes a ways back for appear.in but they unfortunately lost that domain and I have no idea what the new one is.
- kabes 5y ago
  Appear.in is now Whereby.com
  - listenallyall 5y ago
    ...and I already forgot the new name
    - pempem 5y ago
      The quality for free plans have tremendously decreased, to the point that I cannot recommend it to family/friends/non-corporate employees. Huge whereby/appear.in fan, used to run our family wknd catch-ups on there.
airstrike 5y ago
Zoom IMHO has nailed the workflow for clicking an autodial link on my phone, joining the call by video on my laptop and having the participant pin appear on the screen quick enough that I can type it right as the automated voice prompt asks for it on the phone When you're hopping from one call to the other, having the ability to quickly drop and join the next one [EDIT: with audio from my phone] is a big win That, the global Alt+A / Alt+V for the audio / video mute toggle respectively, and the "hold spacebar to talk" hotkeys make it the clear winner from an UX perspective IMHO, even if that X isn't really perfect At the end of the day, most calls are virtually lag free so I care more about my ability to efficiently interface with the app than I do about a perceived streaming quality [EDIT: except, of course, for those participants who don't join the audio via their phone, as their voices inevitably suffer from some lag and distortion]
- Macha 5y ago
  > Zoom IMHO has nailed the workflow for clicking an autodial link on my phone, joining the call by video on my laptop and having the participant pin appear on the screen quick enough that I can type it right as the automated voice prompt asks for it on the phone This sounds inconvenient? For Meet, I click the invite link from my calendar, email, or meeting list for an upcoming meeting list, or from the slack or other message a coworker sent me for the adhoc one, and I'm in, with video and audio going through the one device.
  - airstrike 5y ago
    You have the same option on Zoom. But then, just as in any other videoconferencing app, your audio is now VoIP - an acronym which stands for "heavily compressed, choppy and muffled" Dialing in from your phone increases the audio quality for everyone by a factor of approximately ∞
    - Macha 5y ago
      Are phone networks better where you're from? The audio compression on the phone network renders the sound quality/clarity inferior to a built in laptop mic in my experience.
propter_hoc 5y ago
As much as I'd like to save the money, Zoom is evidently still the best option. Hope the others continue to catch up. Neat to see that Meet is almost as good, according to this data - it was hot garbage a few years ago.
- notsureaboutpg 5y ago
  Meet has caught up. It used to suck a ton (back when it was Hangouts), but these days I'd say it's better than Zoom.

fouc 5y ago

> Despite its significance, there has not been any systematic study characterizing the user-perceived performance of existing videoconferencing systems other than anecdotal reports. > We find that the existing videoconferencing systems vary in terms of geographic scope, which in turns determines streaming lag experienced by users. We also observe that streaming rate can change under different conditions (e.g., number of users in a session, mobile device status, etc), which affects user-perceived streaming quality. Beyond these findings, our measurement methodology can enable reproducible benchmark analysis for any types of comparative or longitudinal study on available videoconferencing systems. Seems like a very useful study at first glance!

travoc 5y ago

I worked at a company that blocked the native clients for every conference system except WebEx. The Zoom and Skype web experience was not good, particularly since all endpoints were loaded with half a dozen Infosec agents and sending all traffic through web proxies on the other coast. Surprise, the somewhat primitive Webex offered the best experience in that scenario and vendors who wanted to meet were inclined to accommodate us.

azinman2 5y ago
Webex interface is… not great. Zoom isn’t either and I’m quite surprised it’s as popular as it is given the UX. Surprising these things are as difficult as they are and yet get such adoption.
- soylentnewsorg 5y ago
  The webex interface is perfectly fine, but if you are comparing it for the use case of zoom and skype, your target interface is wrong for the tool you're using. Webex is for working on something together. Multiple users using an application to manage some piece of hardware or software for example. Zoom and skype are for talking about something together. Let's say you get a new server. It needs several teams to set it up - storage to provision/rescan/configure disk, infosec to do domain/antivirus/dlp work, db team to get the database going. They're going to share an rdp session over webex and hand it off to each other to type commands, with everyone observing and catching mistakes. Yes, it's possible to give desktop control to someone on zoom, however that is a secondary afterthought of zoom, while for webex it's the primary function. Right tool for the job - don't transport a piano in a sports car then complain the sports car is.. not great.
  - blacksmith_tb 5y ago
    I am not sure you can answer a subjective assessment with a blanket dismissal - obviously at some level the webex UI can't be completely unusable (lots of people do use it), but apparently some people just don't like it. It has what strikes me as an unpleasant installer (like BlueJeans, it seems to only work if you download an installer and run it for every call, which seems excessive somehow).
  - azinman2 5y ago
    We use Webex at work (a very large tech company). I use it for it’s intended purposes. My issues: 1. The reaction buttons don’t indicate that anything is actually happening for you when you click them, you only can see others reactions. They’re also quite hidden when they come, make no noise so easy to miss, and don’t tie to the video. 2. The “people joined” ding is very non-intelligent. It used to be when 30+ people were queued for something to start, once it did it just dinged like crazy. Now it seems work has turned that off in a way I cannot turn back on, which means when people are late for a 1:1 they’re staring at me and I have no idea since I now have people blocking. A heuristic that would be aware of # participants, when people are joining, and batching up dings I cannot imagine would be groundbreaking research when this is your product you’re building. 3. The actual controls for video on mobile to switch camera is quite bad. 4. Screen sharing works _ok_, except it has no integration with the resolutions on both sides, Apple’s accessibility-based zoom, I’m given a red dot for a cursor but it doesn’t appear that way on the other side, and cursor tracking is extremely laggy and jumpy on the remote side despite my red cursor that’s giving me the impression of a laser pointer equivalent. 5. When someone is screen sharing, the controls for what to do with the video of others are not very good. Order is non-deterministic (afaik), not easily changed, provides few layouts, and not easily hidden. 6. When screen sharing I’m given a large list of all my windows to choose from. They’re in effectively a random order (not alphabetical, not by z-index on my Mac, etc), and the mini photo of them are usually unintelligible and difficult to pin point the one I want. These are just some of the bad UX that I encounter daily in work usage. I could probably come up with a list of 10+ more issues without even thinking much about it, and we haven’t even touched the “interface” level issues (eg actual design).
    - 5y ago
- Pxtl 5y ago
  Having used MSTeams, I've stopped complaining about Zoom. It's stable and straightforward. The on-call UI has its warts, especially on mobile, but otherwise it's fine. Teams, by contrast, is buggy and inconsistent and weird and has the awful arbitrary rectangle subdividing of the chat window instead of just showing me the other people's full cameras.
wenc 5y ago
> Surprise, the somewhat primitive Webex offered the best experience in that scenario But it’s not a surprise in this scenario since WebEx is the only one not hobbled, no? I’ve used all 3 and WebEx is second to Zoom but better than Skype.
dagw 5y ago
The company I work for also blocks the native Zoom client, and I know of several other companies that do it as well. If we have a Zoom meeting we have to use the web client. Can anybody give me a reasonable rationale for why this is so common?

camgunz 5y ago

My prayers have been answered. Google Meet devs I beg you, make the app more efficient.

Orphis 5y ago
Ok, because you asked nicely!
throwaway81523 5y ago
Wait, Google Meet? I thought this was going to be about Jitsi Meet (not joking). Crap. I hadn't heard of Google Meet. So now I still want to know how Jitsi compares.

finnh 5y ago

I was on a webex recently, and after years of using Zoom I was all "oh, wow - is see Webex has a 'make my skin look worse' feature. Gotta compete with Zoom somehow"

pempem 5y ago
It can be downright startling!

athenot 5y ago

> In case of Webex, all sessions created in the US appear to be relayed via its infrastructure in US-east. This causes the sessions among users in US-west to be subject to artificial detour, inflating their streaming lag. Former webexer here. That depends on many criteria, sessions can go through multiple locations depending on where the customer is located, the features being used, the type of customer, etc. Since Webex also does transcoding (useful when a 4k wall system and a cell phone are on a same meeting), that can influence where the traffic flows. Webex, Zoom and Google also all have their own backbones to reduce the number of hops over the public internet. In this case the researchers appear to be based in New Jersey so they would be more likely to enter the internal network via the eastern POPs.

listenallyall 5y ago

Personally, I've been pretty impressed with Jitsi especially given that it's open source. So far it's been easy and the people on the other end have had no problem connecting, despite not having previously downloaded any client software. I had high hopes a ways back for appear.in but they unfortunately lost that domain and I have no idea what the new one is.

kabes 5y ago
Appear.in is now Whereby.com
- listenallyall 5y ago
  ...and I already forgot the new name
  - pempem 5y ago
    The quality for free plans have tremendously decreased, to the point that I cannot recommend it to family/friends/non-corporate employees. Huge whereby/appear.in fan, used to run our family wknd catch-ups on there.

airstrike 5y ago

Zoom IMHO has nailed the workflow for clicking an autodial link on my phone, joining the call by video on my laptop and having the participant pin appear on the screen quick enough that I can type it right as the automated voice prompt asks for it on the phone When you're hopping from one call to the other, having the ability to quickly drop and join the next one [EDIT: with audio from my phone] is a big win That, the global Alt+A / Alt+V for the audio / video mute toggle respectively, and the "hold spacebar to talk" hotkeys make it the clear winner from an UX perspective IMHO, even if that X isn't really perfect At the end of the day, most calls are virtually lag free so I care more about my ability to efficiently interface with the app than I do about a perceived streaming quality [EDIT: except, of course, for those participants who don't join the audio via their phone, as their voices inevitably suffer from some lag and distortion]

Macha 5y ago
> Zoom IMHO has nailed the workflow for clicking an autodial link on my phone, joining the call by video on my laptop and having the participant pin appear on the screen quick enough that I can type it right as the automated voice prompt asks for it on the phone This sounds inconvenient? For Meet, I click the invite link from my calendar, email, or meeting list for an upcoming meeting list, or from the slack or other message a coworker sent me for the adhoc one, and I'm in, with video and audio going through the one device.
- airstrike 5y ago
  You have the same option on Zoom. But then, just as in any other videoconferencing app, your audio is now VoIP - an acronym which stands for "heavily compressed, choppy and muffled" Dialing in from your phone increases the audio quality for everyone by a factor of approximately ∞
  - Macha 5y ago
    Are phone networks better where you're from? The audio compression on the phone network renders the sound quality/clarity inferior to a built in laptop mic in my experience.

propter_hoc 5y ago

As much as I'd like to save the money, Zoom is evidently still the best option. Hope the others continue to catch up. Neat to see that Meet is almost as good, according to this data - it was hot garbage a few years ago.

notsureaboutpg 5y ago
Meet has caught up. It used to suck a ton (back when it was Hangouts), but these days I'd say it's better than Zoom.