4 ms·
I'm really confused by this : surely if you want to make a data set that has Internet speed binned by county surely the way to do that is as follows: 1. Go to
by dboreham 9y ago
I'm really confused by this : surely if you want to make a data set that has Internet speed binned by county surely the way to do that is as follows:
1. Go to a large Internet services provider (Amazon, Google, Akamai, Netflix).
2. Ask them to statistically sample the TCP flow rate observed in client traffic, by source IP address.
3. Get a data set that geolocates IP addresses to ZIP code (Amazon for example has this data).
4. Join the two.
- pmyteh 9y agoThey want data on Internet use (and specifically broadband Internet use) by county. That isn't a trivial ask. You can get reasonable rates at a larger scale using survey approaches (like OxIS, the British survey series from the Oxford Internet Institute) but these aren't granular enough to compare small areas. Otherwise you're using some kind of proxy data. Your approach has two issues. First, people without Internet access are not in the sampling frame at all, which is a critical problem if you're interested in network access. Secondly, the geolocation in step 3 is also not particularly reliable (mine tends to think I'm 100 miles away from where I am) so you'll get the same kinds of problems they found here in that the data look kinda plausible but aren't actually reliable enough to draw defensible conclusions from.
- deleted 9y ago[deleted]
- bri3d 9y agoGoogle do this with YouTube already: https://www.google.com/get/videoqualityreport/m/ https://www.google.com/get/videoqualityreport/m/
- throwawayjava 9y agoThis methodology would completely miss the point of the 538 story, which was to report on access.
- fjsolwmv 9y agoIt makes a reasonable assumption that everyone with access has similar usage profile, at the level of country averages
- throwawayjava 9y agoThat assumption isn't helpful, and even if it were, isn't necessarily reasonable. But I'll just fwd you to pmyteh's post instead of rehashing those claims here.
- TheCowboy 9y agoI've learned that getting a good data set is often the most challenging part of data analysis. You think "it should be easy to get this data" and usually it's not, unless someone else has already done the work. One flaw I can already identify with your data set is that it doesn't differentiate between traffic originating from workplaces, or cellphone traffic. A source like Netflix could be a heavily biased set since it relies on self-selected subscribers; there is no skipping research about if their user base provides an adequate sample at the county level. Asking a huge company like Google and getting them to pay attention is another challenge. The article writes about how the FCC does have more granular data, but "the commission is wary of “one carrier learning about another carrier’s market share or where their customers are,” Rosenberg said." So you are at the mercy of competitive businesses willingly releasing data that competitors might find useful.
- oxguy3 9y agoHow many people are represented by a single IP address? How do you deal with changing IP addresses (most residential ISPs assign dynamic addresses)? How do you distinguish between residential and commercial traffic? How do you deal with cellular devices (for which GeoIP data is much less accurate)? How do you account for people who don't use the service you got the data from? (Netflix in particular has a limited subscriber base, but all the providers would exclude, say, elderly folks who do nothing but email). Getting good data at nationwide scale is never as easy as it sounds in your head, unfortunately.
- dboreham 9y agoAll good questions: 1. Typically a single IP address represents a single connection. Yes there are providers who NAT multiple subscribers onto one IP but they are rare (because if they do that then they have to maintain NAT logs in order to identify criminal subscribers to law enforcement -- easier to just have 1:1 IP to subscriber). 2. Residential addresses are in fact not really dynamic. Yes they can change from time to time but for the most part they don't (see #1). 3. Cellular traffic can be identified because the cell carriers use specific identifiable netblocks. 4. It doesn't matter if not everyone uses the sampling service because that's the point of sampling.