8 ms·
Web Latency Benchmark: A new kind of browser benchmark
- MrBuddyCasino 13y agoLow latency is more important than overall throughput when it comes to UX, so that is a measurement that makes a lot of sense. Has anyone experience with it?
- 109876 13y ago1.1 out of 10 running the latest stable version of Chrome on a 2010 13" MacBook Pro with a 2.40GHz Core 2 Duo, 8GB of RAM, and a Samsung 840 SSD. Geez... Is there anything I can do to improve the score?
- hnriot 13y agoodd, i have much the same mac, same cpu/ram with a regular 7200rpm disk, same chrome and i get 2.5
- dmourati 13y agoI learned a new definition for jank today! 3.0/10 on MPBR early 2013 w 16GB and SSD. Authors should consider a way to collect and present the data.
- cheald 13y ago3.3 on a beefy Windows 8 machine running Chrome 31.0.1650.39 beta-m. Interestingly, running an Incognito tab with no extensions allowed to run in incognito results in a 2.7. Huh.
- dmourati 13y agoLower scores are better since we are measuring latency?
- gjm11 13y agoLooking at the source, higher scores are better. There are a bunch of tests, and each one produces a score from 0 (bad) to w (good) for some w. The total score is the total of all these. Given that all the other numbers on the page are ones for which smaller is better, I think the main score itself should be accompanied by a note saying explicitly that higher scores are better.
- modeless 13y agoGood point. I'll add a note next to the scores.
- lucideer 13y agokeep on getting https://github.com/google/latency-benchmark/blob/master/src/latency-benchmark.c#L505 https://github.com/google/latency-benchmark/blob/master/src/... on the first test... not sure why.
- benologist 13y agoI can't help but think this should be a test for websites, not computers. It's individual sites full of 3rd party shit that are slow and you can see the performance penalty by installing Ghostery.
- wtallis 13y agoThere are already good tools for measuring network delays. Firefox's web inspector does a beautiful job of graphing how the various requests contribute to overall page load time. This tool is about measuring how responsive the UI is, even when you're still fetching more data.
- modeless 13y agoIt's more of a test for browsers than computers. You can compare Chrome vs. Firefox vs. IE vs. Safari. Measuring the input latency of individual websites would be interesting too. I'd love to incorporate that into the benchmark if I could figure out a way to do it.
- Zikes 13y ago1.8/10 on a Dell Latitude laptop. Did anyone else have difficulty getting it to work on a multi-monitor setup? It gave me an error about trying to find the reference image and suggested I switch monitors, but that didn't work. I finally had to disconnect my secondary monitor altogether before it would work.
- modeless 13y ago(benchmark author here) Interesting. I will have to do more testing on Windows multi-monitor setups. I would not be surprised to learn that it's broken.
- Zikes 13y agoIf it helps at all, I'm on Windows 7 with my laptop screen as my primary monitor and an LCD screen to the left as my secondary. I was initially in Win7's "enhanced graphics" mode (can't remember actual feature name) but when I tried to run the benchmark I was thrown into Basic mode. I had a fair amount of other applications running, though, so I may have just run out of memory.
- modeless 13y agoBeing thrown into Basic mode is unfortunately expected. I'll bet the reason it didn't work is having the secondary monitor on the left. I think I only tested it with the secondary monitor on the right.
- euroclydon 13y agoMultiple monitors. Windows 7. Just got a black screen in both IE9 and Chrome Canary.
- madh 13y ago2.8 / 10.0 on a 1 week old 15" MBP (high-end) running latest stable Chrome.
- k3n 13y agoKeeps failing for my Win7x64/30.0.1599.101 m. > Failed to find test pattern on screen. Ensure that your browser's zoom level is set to "100%", and the top-left corner of the window is visible. If you have multiple displays, try moving the browser window to the main display. * Zoom 100%: check. * Main monitor (of 2): check. * Top-left visible: check. I give up.
- modeless 13y agoSorry to hear that. Due you have the Chrome flag for "Experimental Web Platform features" enabled? I recently learned that there's an issue with the test related to this flag. Edit: It's more likely that it's just buggy support for multiple monitors. I will look into fixing it; for now the only workaround would be disabling your second monitor.
- k3n 13y agoSorry for the late reply; I did notice a Github issue for it so I commented there. Thanks!
- andrethegiant 13y agoHas anyone else heard the term jank before? It's the first time I've seen it.
- mcpherrinm 13y agoThe Firefox team definitely used the term with this usage, since at least 2011.
- Pwntastic 13y agoI use/have used the term 'janky' before to describe similar kinds of things or things that seem off or strange. Grew up in the midwest fwiw
- azinman2 13y agoWe (at Empirical) use janky to describe animations that aren't butter as this author uses it. Not sure how standard but I suspect many people (native speakers) will understand when in context.
- makepanic 13y agoThere is a website [1] with more informations and talks about "jank". [1] - http://jankfree.org/ http://jankfree.org/
- rmccue 13y ago(I'd guess it's originally related to the concept of "jerk" in physics. http://en.wikipedia.org/wiki/Jerk_%28physics%29 http://en.wikipedia.org/wiki/Jerk_%28physics%29 ) It's been used in this context since at least 2010 by the Chrome team: http://en.wiktionary.org/wiki/Talk:janky http://en.wiktionary.org/wiki/Talk:janky
- codexon 13y agoNo, usually the term I have heard used for this is jitter.
- sp332 13y agoThe windows version disables the "Aero Glass" transparency effects on my Win7. A first glance looks like it might not be intentional, but it triggers some kind of compatibility mode in the OS. Not sure why a headless daemon would mess with transparency effects.
- modeless 13y agoBenchmark author here. This is unfortunately intentional due to the way the benchmark works. Measuring latency requires taking screenshots really fast, and on Windows 7 it is very difficult to impossible to take fast screenshots with Aero enabled. In Windows 8 they added a new DirectX screenshot API[1] that makes this unnecessary (which is good because it is impossible to disable Aero on Windows 8). [1] http://msdn.microsoft.com/en-us/library/windows/desktop/hh404487(v=vs.85).aspx http://msdn.microsoft.com/en-us/library/windows/desktop/hh40...
- azinman2 13y agoHas anyone assembled benchmarks across the different browsers yet? I like the aims of this benchmark a lot -- I've recently switched away from Firefox to Chrome simply because it feels less janky.
- modeless 13y ago(benchmark author here) I don't have comprehensive numbers compiled yet, but I can give you a general ranking of browsers. From best score to worst, the current ranking is roughly: Safari Firefox (except Mac) IE 11 Chrome and Mac Firefox
- Semaphor 13y agoI wonder if I'm using my browser differently than others or if there is something wrong with my machine. Whenever I try to switch from Chrome to FF it seems way behind Chrome in terms of speed and snappiness. edit: Windows 8.1, SSD, i3, 16GB RAM (63% used) Chrome 30.0.1599.101 m: 3.4/10 Chrome Incognito: 4/10 Firefox 25: 5.4/10 FF Private Browsing 3/10 O.o All tests done at least twice with similar numbers. But Chrome has been running for a few days while FF was just started. Guess I'll leave FF running a bit and see if it changes anything. Also interesting, FF takes around 2-3 times as long to run the tests than Chrome. The reason for the low Chrome score is this: JavaScript jank Tests responsiveness during JavaScript execution. CSS: 1.0 frames jank, Scrolling: 170.1 frames jank
- modeless 13y agoThere are many different ways to measure speed. For example this benchmark says nothing about page load times or JavaScript speed; Chrome might (hypothetically) be way faster than Firefox there and still score worse in this test. > FF takes around 2-3 times as long to run the tests than Chrome. There are a couple of reasons for this. Firstly, lower latency is more difficult to measure, so it takes more trials to get the same number of good measurements. Secondly, Firefox has longer scroll momentum than Chrome, and the benchmark waits for the scroll momentum to die down between each scroll latency measurement.
- rMBP 13y agoDoes not work in Safari on Mavericks using a rMBP @1680x1050 retina resolution. It suggests browser zooming be set to 100% which it is.
- modeless 13y agoThanks for the report; I see the problem. I'll look into fixing it, but in the meantime you can get it running by enabling WebGL in Safari (under the Develop menu, and if you don't have a Develop menu you can enable it in Preferences->Advanced). Spoiler: Safari actually gets a very good score (relative to other browsers).
- rMBP 13y agoEnabling WebGL works. My three tests scored 8.7, 6.7 and 6.3 respectively. I guess Safari, like me, hates repetitive tasks. :)
- modeless 13y ago:) There is a lot of run-to-run variance. Some of it is real in that browsers do give different results, and some of it is just test noise. I'm still working on improving the robustness. Recently I've been working on hardware latency measurement which gives more reliable numbers.
- selectodude 13y agoI do have WebGL enabled, and it can't pick up the test pattern.
- aaron42net 13y agoI also get "Failed to find test pattern on screen." in Safari with WebGL enabled on a Mid 2011 Mac Mini running 10.9 with dual monitors. View is set to "Actual Size", and I've tried moving the Safari window between monitors. Other scores - Chrome 30: 2.4/10. Firefox 24 and 25: 1.2/10.
- nej 13y agoGetting different scores when the test is ran at different browser sizes. 3.4 when the window is sized to it's smallest size. 2.5 when it's at maximum resolution (1600x1200). Running on OSX 10.9.
- modeless 13y agoYes, window size can definitely affect latency and jank. The benchmark page has a hidden full-window-sized layer that the browser must paint every frame, and a large browser window requires more CPU and GPU work to paint.
- carlob 13y agoI managed to get a 6.5 on Safari on Mavericks on a 3.5 year old MBP. All the other browser I've tried were below 2, except for Opera Next (2.2).
- nrp 13y agoAwesome to see a non-Virtual Reality use of our Latency Tester.
- modeless 13y agoIt's an awesome piece of hardware, thank you for making it! I'm currently brainstorming ways to use it more extensively in the tests. This benchmark does have a connection to virtual reality, as part of my inspiration to make it came from my experience using vr.js[1] and WebGL for virtual reality in browsers[2]. The Web isn't yet ready for VR due to latency issues, but I hope to change that! On that note, are you guys interested in browser support for the Rift? I believe the latency issues are surmountable with effort and some cooperation from browser vendors, and I'd love to see something official in that direction (in fact, I'd love to work on something official). [1] https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=753 https://developer.oculusvr.com/forums/viewtopic.php?f=20&t=7... [2] This is what I made: http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s http://www.youtube.com/watch?v=pWYlTIt4j0Y&hd=1&t=1m11s
- ygra 13y agoSomehow the second test always timeouts in IE 11 on Windows 8.1 for me.