16 ms·
New FCC Broadband Standards Should Consider Latency
- NelsonMinar 3y agoI've had Starlink since early beta days and am watching connection metrics very closely. Latency measurement is so much more useful than throughput. So is packet loss. FWIW I've averaged 0.6% packet loss, 38ms latency, and 130/20 MBit/s in Grass Valley, CA over the last month But averages obscure what's really important. The biggest indicator of Starlink congestion problems has been how packet loss increases in evenings. Average latency is interesting but much more interesting is the variance of latency or jitter. A steady 50ms is better than a connection varying 20-80ms all the time. As for bandwidth what I've found most useful is a measure of "hours under 20 Mbps download" (1 or 2 a day on my Starlink). IRTT is a fantastic tool for measuring latency and packet loss. Way better than simple pings. The hassle is you have to run a server too; I have one in the same datacenter as the Starlink terrestrial POP.
- mapt 3y agoStarlink seems to be somewhat limited in using a ~400 km^2 cell of shared bandwidth instead of a 4 km^2 cell typical of older mobile networks, a 0.04 km^2 cell which is more appropriate for newer 5G service, or a 0.0004km^2 wifi cell that covers most of the house. It's perfect for a sailboat or a wartime forward operating base, but not so much for a city. That may not be very relevant now, as barely anyone uses Starlink and as their bandwidth needs are 2024 bandwidth needs rather than 2034 or 2044. But eventually, starting with the densest cells, congestion will occur. If your city is tied to an electrical grid, it's not that much harder to run gobs of fiber. Starlink could shrink cell size with larger antennas (see Starlink 2.0), but I suspect it's abusing the deep magic of phased array antenna gain to expect a many-phased-array-N-to-many-phased-array-M network connection to scale feasibly at high SNR. Just taking a clever mathematical innovation and asking it to do things in too many dimensions with too many orders of magnitude improvements. ph'nglui Nyquist mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn The problems with physical/geometrical high-gain antennas are not so arcane; I wouldn't be surprised if we go back to dishes or equivalently free-space optical networking for fixed antennas. You could run an almost indefinite number of satellite-client connections simultaneously in something as high-gain as a laser/LED launch telescope.
- c2h5oh 3y agoAFAIK cities was never the target Starlink market as those are supposed to have cheaper and/or faster options either over copper or fiber. Starlink works well in my rural middle of nowhere and has no competition.
- ryandrake 3y agoI mean, Grass Valley is rural, but not "middle of nowhere" rural (source: fellow Gold Countryite). If broadband companies actually don't serve that town well, then they really have no good excuse. I also thought the target market for Starlink was boats and very remote outposts, not "minor towns that are just within civilization." It's shameful if ISPs can't be arsed to serve somewhere like that.
- NelsonMinar 3y agoIt's worse than that; I'm half a mile from a major Obama-era fiber loop with open access rules for any ISP to buy service there. There's no significant ISP selling connections to it. (There's a couple of tiny neighborhood co-ops like the Beckville Network.) Classic last mile cost problems combined with terrible regulation of the monopoly providers like Comcast and AT&T. We have a pretty robust local WISP. But that's expensive and not great performance. Starlink is really my best option.
- mapt 3y agoPoint to point wireless is an underappreciated tool. Building a tower half a mile away and pointing a hundred small dishes individually at various customers with their own dishes pointed at the tower, might be cheaper than running fiber to a hundred spread-out houses. Assuming no coordination problems. You just need a lot of extra signal slack built into the system if you want it resilient to weather.
- dtaht 3y agoSee: https://www.taranawireless.com/ https://www.taranawireless.com/
- jedberg 3y ago> But averages obscure what's really important. The data would be far more interesting if you indicated the distribution of the data, perhaps as ranges of one standard deviation. Then it would succinctly express your point as well as be easy for you to quickly find outliers.
- deleted 3y ago[deleted]
- icedchai 3y agoPacket loss is a real performance killer. Even a small percentage will result in enough TCP retransmissions to be noticeable. I had a cable DOCSIS connection for a while that would average 0.9% packet loss for long periods of time, with jumps to the 2 or 3% range. I remember there were days where I'd get like only a couple megabits due to extreme loss. It took a year to get it resolved, and even then I'm not sure if it was deliberate or accidentally fixed. You wind up with all sorts of excuses. "It must be your splitter." "We should recap the end of the cable." "It must be your router." They refuse to do basic diagnostics, like checking your neighbors connections for packet loss (which can be done remotely.)
- dtaht 3y agoSome packet loss (or ECN marking) is NEEDED to get good congestion control. Certainly random packet loss is a PITA.
- dtaht 3y agoGlad you use irtt! More should! The bufferbloat project does maintain a few public IRTT servers around the world: https://blog.cerowrt.org/post/flent_fleet/ https://blog.cerowrt.org/post/flent_fleet/ It would be great to add more to the pool.
- greyface- 3y agoLatency to where? To your ISP's edge? To the nearest IXP? To Google, Cloudflare, Netflix, or AWS? To any destination on the Internet? Latency is an end-to-end metric which will almost always involve path components beyond the control of your ISP - can and should they be held responsible for those paths?
- scq 3y agoLatency to IXP or to a Tier 1 network seem like reasonable measures.
- lalaithion 3y agoDoesn't the same question appear with bandwidth? I don't care about my bandwidth to my router or the backbone, I care about my bandwidth to Google, Cloudflare, Netflix, etc.
- godzillabrennus 3y agoYep, it’s always an on network test I’ve seen letting the carrier claim promised speeds are met, never an off network test.
- b112 3y agoSo the days of peopke begging to peer, by using cakes, will end? Because backend fiscal for profit plays will be mandated second hat, over connectivity?
- deleted 3y ago[deleted]
- paxys 3y agoBandwidth of the connection = bandwidth of the worst leg (which is usually between your house and the ISP). Latency of the connection = sum of latency of every hop along the route. So in practical terms your ISP directly controls your bandwidth, but latency is mostly out of their hands.
- WarOnPrivacy 3y agoBroadband funds helped deploy fiber here. Pings to 1.1.1.1 are ~2ms. I love considering latency now. edit: Cable is still hooked up and it's 17ms-20ms. Not awful. LTE: GSMA is 25ms-33ms, GSMT is 25ms-51ms. I don't have satellite, DSL, IDSN or dial up to compare.
- vardump 3y agoI've seen LTE latencies (not 5G) as low as 15 ms for 5+ years. Not saying it's typically that good, but that it can be significantly better than what you're saying.
- aaomidi 3y agoI don’t think it maintains this under load from your device.
- geraldhh 3y agolatency under load is next up for public awareness, hold on tight
- fourfour3 3y agoYeah - this is a big factor. My home internet gets ~7ms to London (I'm in the UK, London is the important IXP. My test is to one of my ISP's routers at LINX). 5G is about 40ms. When you add load, my home internet stays at about 7/8ms (CAKE is wonderful!), but my 5G goes to 200-600ms.
- vardump 3y agoNo doubt about that. Also my fiber optic connection latency increases a bit under load.
- gizmo686 3y agoI just tested on my phone (4G LTE) and it sees a ~28-35ms latency to 1.1.1.1. For reference, by cable internet gets ~12ms.
- 3y ago
- teeray 3y agoA full Nutrition Facts label would be nice. Downstream AND Upstream, latency numbers to headend and nearest IXP, availability numbers, then price stability between day 0 and 3-5 years.
- dtaht 3y agoThe biggest thing that makes me nuts about our speedtest regimes is that we test upload and download separately, rather than at the same time. People have 16+ wifi devices on their networks today. There IS upstream traffic going on at the same time, most of the time, and what happens when one side or the other also saturated or merely in use is widely misunderstood. For example a short packet fifo on the uplink starves the downlink, which you can see from Oleg´s recent starlink rrul tests here: https://olegkutkov.me/2024/02/12/starlink-terminal-revision-4-overview-and-tests/ https://olegkutkov.me/2024/02/12/starlink-terminal-revision-... If more of our tests tested up + down + latency - and ISPs and users deployed bufferbloat fixes like fq_codel, cake, and libreqos.io - the internet would be a lot less jittery and feel a lot more fast, all the time. https://blog.cerowrt.org/post/speedtests/ https://blog.cerowrt.org/post/speedtests/ has some ranting. More ranting about saner uses for fiber here, also: https://blog.cerowrt.org/post/towards_better_broadband/ https://blog.cerowrt.org/post/towards_better_broadband/ So the nutrition label scheme is seriously flawed without testing up+down.
- wmf 3y agohttps://www.fcc.gov/broadbandlabels https://www.fcc.gov/broadbandlabels
- JumpCrisscross 3y agoThose labels go into effect this year, when the "majority of providers must display at the point of sale clear, easy-to-understand, and accurate information about the cost and performance of broadband services by April 10, 2024; providers with 100,000 or fewer subscriber lines must do so by October 10, 2024. Those points of sale include online and in-store."
- rmbyrro 3y agoThere should be a measure of consistency as well. Like p90, p95, p99. It's a different measure, but something similar to how we measure system availability in SWE.
- NoPedantsThanks 3y ago[dead]
- Animats 3y agoMost web page latency problems are not network problems. They're the pages that load too many crap items. Look at the browser's display of network transactions. Pages with little content are making over a hundred transactions. Does anyone really need 34 trackers? But it takes a lot of round-trips to load all that crap. For a while, Google was penalizing slow-loading pages, but that got tied into AMP, which everybody hated. It only takes 25 Mbps to stream 4K TV, after all.
- asadotzler 3y agoUnless you're on Viasat or similar.
- Beached 3y agowebsites loaded faster in 1999 than they do today. web devs increasingly suck at their job and we end up loading half the internet every time I want to read a news article. internet became fast, but instead of faster load times, we got animated UIs. hyperbole obviously, but not entirely wrong.
- geraldhh 3y agonah, we can make websites that "instantly" display content with 56kbps and 100ms rtt
- asadotzler 3y agoThis. Viasat is not viable for Zoom so it's not viable as a modern broadband connection. It's that simple as I see it. If you can't do the stuff most people do with broadband regularly it's not broadband.
- acje 3y ago“Gil Tenne - How not to measure latency” some of the best infotainment on YouTube. Sorry no time to find exact link. There are many.
- laserlight 3y ago"How NOT to Measure Latency" by Gil Tene https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU
- acje 3y agoYes, that is a good one. I guess a TLDR for the FCC would be; a) No do not bother because most user will not be able to comprehend meaningful latency metrics. b) Report best case latency under idle load to nearest IXP because this would compose well with bandwidth properties when reasoning about the suitability of the service for different workloads. Of course there be dragons. Oversubscription in various parts of the network can have interesting behaviour. Especially when coordinated user behaviour or complicated packet processing is at play. Think encapsulation or deep packet inspection. My worst case experience includes Cisco Nexus M1 32x10Gbps line cards maxing out at 1-2 Gbps throughput and 1+ sec worst case latency because of OTP. This is a datacenter core switch. And F5 WAF eating random packets because something looks like a VISA card number, causing a retransmit, that again shows up as high latency at higher levels.
- davecb 3y agoOh goodness no, always measure latency under full load, in the middle of the bandwidth test. A convenient example is https://www.waveform.com/tools/bufferbloat https://www.waveform.com/tools/bufferbloat which can easily tell crappy ISPs from good ones
- acje 3y agoMeasuring latency under full load is like measuring how fast you can drive into a crossing and not making the turn. It is meaningless. See the fantastic explanation by Gil Tene in the link above. With best case latency you can determine if the service can be suitable for real-time or not. There will always be buffer effects and they will vary by other user activity and the complexity or the packet processing. These effects will largely be unknowable in advance and the part that is knowable is extremely difficult to communicate to an average user. They don’t really get HDR histograms https://github.com/HdrHistogram/HdrHistogram.NET/blob/master/README.md https://github.com/HdrHistogram/HdrHistogram.NET/blob/master...
- nomilk 3y agoWhen working remotely, latency matters because it's so awkward in video meetings when you're 1/4 second behind everyone else. But something else that matters (possibly not so much in the US, but definitely in many other countries) is mini internet 'brown outs', where there's a sudden internet drop out for just a few seconds. I don't know what statistic would be good for measuring that. But when it happens during a meeting it's quite annoying and noticeable. In some parts of the world they seem to occur about once an hour, although YMMV.
- Unfrozen0688 3y ago>where there's a sudden internet drop out for just a few seconds. I have had this on wifi for sure, but never wired with ethernet. Not an ISP issue. Sweden. So not USA. But western country where we decided to invest in broadband infra. I am on DOCSIS/COAX modem/router combo. 200 down 10 up. Ethernet wired to gaming PC and work laptop. Rock solid. Downspeed is better at work but it being WiFi there and shared with many people it still will suck despite being enterprise Cisco and Unifi hardware on two different networks.
- Beached 3y agosomeone correct me if I am wrong, but I believe these mini brown outs you are talking about are officially called Jitter and are measurable. Jitter is a big issue for IP based telecommunications and was a bigger concern than Mbps when I was implementing them in the past.
- c0balt 3y ago> But something else that matters (possibly not so much in the US, but definitely in many other countries) is mini internet 'brown outs', where there's a sudden internet drop out for just a few seconds. Availability, i.e., uptime of your link. Frequency of availability interruption etc.
- Dwedit 3y agoOn a somewhat related note, there are some channels on 802.11ac that are designated as "DFS" (dynamic frequency switching) channels, because they are sharing frequency ranges with weather radar equipment. Networking equipment will periodically monitor for radar signals. I think I heard somewhere that this monitoring for radar signals will temporarily add extra latency to the connection during this time. If any equipment detects a radar signal, it ends the wifi network and switches to another channel. But if another DFS channel is chosen at this time, there is a 60 second timeout before the new channel is confirmed. So if you're trying to avoid latency spikes, don't use DFS channels on 802.11ac. Page here has a list of all the Wifi channels on 5GHz: https://www.networkcomputing.com/wireless-infrastructure/channel-bonding-wifi-rules-and-regulations https://www.networkcomputing.com/wireless-infrastructure/cha... In particular, you can't get a 160Mhz wide channel (8 channels bonded together) on 802.11ac without being on a DFS channel.
- hoseja 3y agoThat's a pretty good DALL-E header image.
- secret-noun 3y ago> Up for consideration by the FCC on March 14, the draft report would increase the current national broadband speed definition of 25 * 3 Megabits per second (Mbps) established in 2015, raising it to 100 * 20 Mbps. What is this "25 * 3" and "100 * 20" notation? The next sentence just goes on to say "1 Gigabit" and "500 Mbps" directly.
- venkatagiri 3y agoThat's 25 Mbps download and 3 Mbps upload speed.
- scandum 3y agoWeb pages are increasingly bulky. A 3 MB page will take 1 second to load at 25 Mbps, so latency is often not the primary bottleneck. Part of the problem may be that companies who own network infrastructure, and get paid for data usage, are also the ones that are the largest content providers. This also comes with an electricity cost. We regulate efficiency for refrigerators, it might be time to add some sane limits to the largest content providers, which will also improve connectivity for those stuck with 2 Mbps.
- sdflhasjd 3y agoLatency is important because that 3MB isn't one 3MB transfer, its a tree of dependencies that can't be completely parallelised. On high latency Internet with otherwise OK speed, pages of "only" 3MB can take several seconds to load.
- Beached 3y agoam I the only person who sees this as a webdev problem and not ISP problem? I don't want my website to recursively load the entire freaking internet with Russian nesting doll dependency loading. when did we stop caring about optimization and good technical design?
- YetAnotherNick 3y agoWhich commonly used webpage is 3 MB for return requests excluding the images? Figma has everything and it's 317 KB transferred for small design for return return. Most of the content is cached.
- thfuran 3y agoWhy would you exclude images?
- YetAnotherNick 3y agoBecause they don't contribute to latency. I can start reading the content for 1 second they would take to load.
- freeAgent 3y agoIf latency doesn’t matter, can I create a service that cross-ships 1TB HDDs overnight and call it broadband?
- dtaht 3y agoThx for the laugh! No FCC regs demand sub 100ms latency, so unless you are taking a really big 24 hour average, that idea won't work. However the technology exists to aim for 20ms - even 4ms - of both idle and working latency today, and the 100ms target - as well as the 95% cutoff for measuring latency, are nutty. It is not every day that key members of the ISP industry actually call for stricter metrics - If you cut the ISP working latency off at 99% instead - well, jason livinggood from comcast reported: ... Looking at the FCC draft report, page 73, Figure 24 – I find it sort of ridiculous that the table describes things as “Low Latency Service” available or not. That is because they seem to really misunderstand the notion of working latency. The table instead seems to classify any network with idle latency <100 ms to be low latency – which as Dave and others close to bufferbloat know is silly. Lots of these networks that are in this report classified as low latency would in fact have working latencies of 100s to 1,000s of milliseconds – far from low latency. I looked at FCC MBA platform data from the last 6 months and here are the latency under load stats, 99th percentile for a selection of ten ISPs: ISP A 2470 ms ISP B 2296 ms ISP C 2281 ms ISP D 2203 ms ISP E 2070 ms ISP F 1716 ms ISP G 1468 ms ISP H 965 ms ISP I 909 ms ISP J 896 ms ... As for the insane 95% FCC working latency cutoff today - in terms of real time performance - what if, you got in your car, for a drive to work, and your steering wheel failed one time in 20. How long would you live? If we want a world with AR, VR, and other highly interactive experiences, 99.9% of no more than 20ms of consistent jitter and latency should the goal for the internet moving forward, and ideally, 4ms.
- zzz999 3y agoI never had a latency problem in the US but then again I never had to use satellite... But if I did, there wouldn't be any other options anyways.
- qwerty456127 3y agoExcept in some specific cases (like when your job is to regularly download and upload huge files) latency is even more important than bandwidth. Even fairly low bandwidth under 10 Mbps is enough for the majority of real-life office job cases and it feels this way if the latency and jitter are low yet high latency/jitter can make even a 100+ Mbps connection unbearable. It's low latency that makes Internet feel fast. ISPs should advertise latency and jitter levels alongside bandwidth and that's the most important numbers customers should look at. I usually explain this to non-techies by inviting them to imagine they own a train full of 1TB drives. This way they can move a million terabytes to a neighbor town in an hour. Naïvely this translates to 277.8 Tbps which seems super fast but there is a catch :-)
- dtaht 3y ago+10!
- ghaff 3y agoI admittedly don't have multiple (or even one) 4K video streams coming into my house at a time. But a lot of people obsess about 1+ gigabit bandwidth availability when nothing even approaching that matters to most people. Low/predictable latency and just reliability matters far more in general.
- davecb 3y agoI often get spikes of delay showing up in https://test.vsee.com/network/index.html https://test.vsee.com/network/index.html while on conference calls. Those correspond to periods in which speakers "sound like aliens", "have fallen down a well" or simply freeze up. Having a standard that ignores that is less than useful. It's the standards body showing disrespect for the people it's supposedly creating the standard _for_.
- vitus 3y ago> The comment submitted by Dave Taht, chief science officer of LibreQoS, argues that today’s applications are not typically bandwidth-limited, but are instead significantly limited by working latency. (for reference, Taht's comment is indirectly linked from the submission; https://www.fcc.gov/ecfs/document/12010651418616/1 https://www.fcc.gov/ecfs/document/12010651418616/1 which has much more detailed technical discussion.) Dave Taht... that's the guy behind a lot of the bufferbloat work over the past decade or so (and seems to be the submitter?), right? In my experience, predictable latency is more important than low latency. 50ms RTT can be fine for a video call; it's not fine if it spikes up to 500ms sporadically. Protocol work like QUIC that can reduce per-connection startup overhead definitely helps in this regard. On the other hand, if you have a highly interactive use case like SSH where every single keystroke must roundtrip between you and a server, that 50ms RTT is very visible. I think talking about absolute latency might be misguided in that it depends partially on the other end you're talking to. If I'm in the eastern US and communicating to a server in India, there's no way I'm getting < 100ms RTT (I'd be happy if I could get 200ms). On the other hand, if you're using legacy satellite internet like Viasat you might have 600ms latency talking to a server in the same city. (Starlink does better, but it still imposes sizable fixed latency costs on the order of 10s of ms.) If I had to come up with latency metrics, I might initially suggest "< 20ms within the same city; loaded latency must not impose a penalty greater than 40ms" (I'd love for these numbers to be lower, but they're already very ambitious for Starlink due to physical characteristics of the satellite constellation). edit: of course, in retrospect, that's my urban bias coming through -- what if you don't live in a city? or consider Hawaii -- is it good enough to just consider latency to Honolulu? Or do you want latency to www.google.com (the nearest frontends are in Los Angeles)?
- acje 3y agoHaving thought about this today and read some responses here, my best proposal would be to require endpoints available for the FCC to measure and then just create a service to show the results. That would give so many more vectors of improvement over time. How to measure, report such that users understand, capture change over time. No mater what number one puts on a label, that number will likely become misleading over time.
- 3y ago
- throw7 3y agoI never considered restrictions on running servers and the poor upload speeds "broadband", but apparently everyone does. So here we are.