11 ms·
We Used Broadband Data We Shouldn’t Have
- code4tee 9y agoNice to see they’re being so transparent about setting the record straight. Mistakes like this happen in the journalism business, but few outlets make a serious effort correct misreporting beyond burying corrections where nobody sees them.
- colemannugent 9y agoYeah, most news outlets will edit the original article with a small correction. Props to them for a full postmortem article.
- blensor 9y agoI am usually not one who favors government regulation of journalistic outlets, however I do have a proposal. Media outlets should be required to publish a correction of a story at least as visible and for at least as long as the initial incorrect version. This would prevent a site from first claiming something and then burying the correction somewhere where nobody finds it.
- FabHK 9y agoIn Germany, people covered by factual statements in the news have a right (under certain conditions) to demand a "Gegendarstellung" presenting their reply (consisting only of factual statements, not opinions), and that reply has to be in an equally prominent location/size/etc in the same media outlet. https://en.wikipedia.org/wiki/Right_of_reply https://en.wikipedia.org/wiki/Right_of_reply https://de.wikipedia.org/wiki/Gegendarstellung https://de.wikipedia.org/wiki/Gegendarstellung
- bonzini 9y agoOf course factual statements need not be true. There was a famous case in Italy where a politician denied having been incarcerated, while sending his reply from a state prison.
- blensor 9y agoDoes a "Gegendarstellung" not require a court order? I would very much love if the media outlets agree on self imposed "best practices for corrections". Which in the end should be a benefit for those complying because they can advertise it and be more believable
- hwh 9y agoThe right exists without a court order (and I'd wager it is more often granted than not by established outfits, with notable exceptions like tabloids - like Germany's largest newspaper Bild - and the yellow press). However, some will have to resort to go to the courts if the publishing entity does not comply. Which will take some effort, it being a civil rights matter - e.g. paying court fees up front (and claiming the money back in the same suit, leaving the applicant with the risk of an opposing party with no money).
- 18pfsmt 9y agoI think a specific subdomain or directory for all corrections would also be helpful. Visibility should be long term without an expiration or redirect.
- blensor 9y agoA separate list of corrections would definitely be a good counterbalance, but it would probably only matter to those who are already trying to verify the correctness of the story in the first place and not to those who took it at face value.
- colemannugent 9y agoThis would require the government determine "the truth" which is probably a really bad idea. I'm curious, how would you punish those who did not comply?
- bglusman 9y agowould it, or would it leave it up to the article publisher, but say IF AND WHEN you issue a retraction/correction, here is the way you must do it? I don't know that I agree with the proposal per se, but I think this specific criticism is misguided/wrong.
- colemannugent 9y agoSay the fed implemented this as you described: Why would any news company bother issuing corrections then? Publishers would just silently edit the article, forcing the federal government to monitor all news outlets content for changes. In the face of this daunting task, some naive senator would suggest that all news first be submitted to the fed for review. The only solution to this problem is the nationalization of the news media, which would be a very bad thing. Sometimes the only way to win is not to play.
- stusmall 9y agoOverall they do such a good job of that. They seem to really value self-improvement and reflection. Its part of what keeps me as a regular reader. One editor even does a yearly public review of the work he did that year, what he did wrong, and how he plans on improving it. https://fivethirtyeight.com/tag/mea-culpa/ https://fivethirtyeight.com/tag/mea-culpa/
- taoistextremist 9y agoHarry Enten is probably my favorite personality of FiveThirtyEight, mainly because of the amount of humility he has despite the fact that he's very good at his job from what I can tell.
- weberc2 9y agoI agree. This error has a pretty happy ending, but I think more political and ideological diversity in academia and journalism would help to spot these errors sooner and improve the overall quality of science and science journalism. In particular, in this case, a progressive research firm was trusted by academics who were trusted by journalists. Mistakes will happen regardless, but I wonder if a more skeptical eye somewhere in the pipeline would have caught this error sooner. Fortunately, there are bipartisan organizations like [Heterodox Academy][0] which are working to correct this. [0]: https://heterodoxacademy.org https://heterodoxacademy.org
- throwawayjava 9y agoThe major ISPs have a ridiculous amount of resources and are free to publish datasets answering this question at any moment. In fact, I'd be astounded if they don't already have this data, both for themselves and for their competitors. So until ISPs share that data, I'm terminally unsympathetic to any sort of bias in the public record on this particular topic. Also, the selection of left-leaning people doing publicly accessible research is, AFAICT, a self-selection. The problem isn't that our institutions are biased, but that conservatives (in general) choose not to pursue scientific careers in the same numbers as liberals. This suggests a fundamental limit to your proposed solution because there aren't enough "heterodox" collaborators to ensure viewpoint diversity. One of the studies published on the website you mentioned even suggests this underlying cause. Also, I'll note that in the vast majority of academia, internal politics (what research methodologies, maths, etc. someone likes to use) are far more of a problem than external politics (of the sort you'd typically call "politics" in general conversation). E.g. I'm not at all worried about "liberal bias" in physics or chemistry or CS; if it's a problem at all, then it's a minor one compared to other problems.
- weberc2 9y agoWhether or not the ISPs have this information is a fundamentally separate issue from whether ideological homogeneity biases science. There is no dichotomy here. The degree to which self selection plays a role is an interesting question. Firstly, there is a certain amount of overt harassment--for example, professors in the humanities and social sciences openly admit that they would be inclined to reject a candidate if they knew he or she was conservative. Also there are lots of other data, anecdotal or quantitative. Much more is needed, however. Importantly, the proportion of conservatives to liberals has been falling since the 90s, so something is driving the trend. It's also possible that conservatives are selecting out of a career that they (evidently correctly) perceive to be hostile to them or otherwise corrupt (more interested in furthering an agenda than seeking truth). The point is that even if it is self-selection, we may be able to improve the ratio enough to improve the science. And it's not like the ratio needs to be 50:50--probably 30:70 would suffice. Right now many fields are less than 10:90, and intuition suggests that the effect is nonlinear. If only 10% of faculty are conservative, intimidation causes them to suppress their already marginal voice, but 30% might be enough to allow them to feel secure in providing the sort of criticism those fields seem to desperately need. Of course, I'm picking on the academy, but the same probably holds true for journalism. Again, the point isn't to assign blame, but to improve the science.
- rayiner 9y agoYeah this is wonderful. Some of the original data just didn’t make sense. For example, London County, which is home to the Dulles technology corridor and has the highest median income of any county in the US, with a 3.5% poverty rate, has only 50% home broadband uptake?
- mobilefriendly 9y agoI live a few miles outside of Loudoun County, still in DC metro, and I have NO terrestrial internet access beyond dialup. It is quite possible that large sections of Loudoun don't have real broadband, just DSL. The county doesn't even have 100% cell phone coverage. There's Dulles/Leesburg, and everywhere else.
- Shivetya 9y agomost people don't realize how dismal even cellular coverage is in many parts of the country. they worry about broad band yet we cannot even guarantee you have coverage with any mobile device. worse people fight against it too
- stordoff 9y agoWhy is DSL not "real broadband"?
- beardicus 9y agoBecause the current US govt definition of "broadband" is a minimum of 25mbps down, and most DSL can't achieve those speeds.
- mynameisvlad 9y agoAccording to footnote 11, the FCC data set uses 10/1 as the definition of "broadband"
- maxsilver 9y ago
- wutbrodo 9y agoYea, 538 is one of the only outlets I can think of that quietly lives up to the standards that most media entities seem to think they hold themselves to. They're like Vox, minus the self-back-patting about intellectual honesty and plus the _actual_ intellectual honesty.
- jobu 9y agoUnfortunately they don't discuss two of the biggest disparities I've seen in reporting on broadband internet access: 1) How do they define "Access"? Does it mean actual subscriptions? Does it mean the building/home is connected? Or does it mean a line passes the household, but there's actually no way to connect to it? (Look up New York City's lawsuit against Verizon's FIOS rollout.) 2) How do they define "Broadband"? In 2010 the FCC defined it as 4 Mbit/s down, 1 Mbit/s up. In 2015 they redefined it as 25 Mbit/s down, 3 Mbit/s up. Currently I have 50Mbit/s down and 1Mbit/s up which Comcast absolutely defines as "Broadband", but it doesn't meet the FCC definition.
- danjoc 9y ago>In 2015 they redefined it as 25 Mbit/s down, 3 Mbit/s up. How can that be? That's the same time we achieved net neutrality. That's not net neutral. That's fast lane on download, slow lane on upload. Asymmetric bandwidth is a prime reason why we all can't host our own clouds and social networks.
- ahupp 9y agoI have never seen anyone define net neutrality as including symmetric bandwidth. I am also skeptical that lack up upload bandwidth has much to do with the lack of success for self-hosted social networks.
- narag 9y agoI've heard two theories about capped upload bandwith: making possible to upsell premium hosting services and P2P damage control.
- ahupp 9y agoI think the simplest explanation is that the transport uses a fixed allocation of bandwidth between upstream and downstream flows, and they've chosen an allocation that matches typical workloads. It looks like full-duplex DOCSIS (data-over-coax protocol) was just announced last year: https://en.wikipedia.org/wiki/DOCSIS https://en.wikipedia.org/wiki/DOCSIS
- cpsempek 9y agoI'm probably being too harsh but... Good on them for writing this, it's important to admit when you're wrong. However, I feel like this outlet has a larger responsibility to be actual data analysts as well as journalists (over say a more traditional journalist for a more traditional news outlet). As such, why was the analysis done in the postmortem article not done prior to publishing the original articles? A good analyst is one you can trust, and trust for an analyst is built by drawing conclusions from highly defensible data, and highly defensible data is data which has undergone sever scrutiny of the analyst before conclusions are drawn, not after. Also, they should probably update the now erroneous articles with a disclaimer indicating that much of the research is now invalid.
- misterbowfinger 9y agoI think the idea is that we all make mistakes. The best we can do is admit them and learn from them, just like any other postmortem. Also, they did update the erroneous articles: https://fivethirtyeight.com/features/lots-of-people-in-cities-still-cant-afford-broadband/ https://fivethirtyeight.com/features/lots-of-people-in-citie... https://fivethirtyeight.com/features/the-worst-internet-in-america/ https://fivethirtyeight.com/features/the-worst-internet-in-a...
- cpsempek 9y agoAlso, they did update the erroneous articles: Ah so they did, my mistake. I think the idea is that we all make mistakes. I am in total agreement, we all do make mistakes, and it's refreshing when people own up to them. I guess my main question still stands though. As a top-tier data journalism outlet, why was a data QA not performed prior to analysis, or, before publishing results? And if there was data QA performed, where did it fail? It just seems that they got bit by a time crunch to publish content amid the FCC repeal and sloppy analysis was the result.
- fjsolwmv 9y agoEveryone can do better. It's impossible to research every story from first principles of physics.
- dboreham 9y agoI'm really confused by this : surely if you want to make a data set that has Internet speed binned by county surely the way to do that is as follows: 1. Go to a large Internet services provider (Amazon, Google, Akamai, Netflix). 2. Ask them to statistically sample the TCP flow rate observed in client traffic, by source IP address. 3. Get a data set that geolocates IP addresses to ZIP code (Amazon for example has this data). 4. Join the two.
- pmyteh 9y agoThey want data on Internet use (and specifically broadband Internet use) by county. That isn't a trivial ask. You can get reasonable rates at a larger scale using survey approaches (like OxIS, the British survey series from the Oxford Internet Institute) but these aren't granular enough to compare small areas. Otherwise you're using some kind of proxy data. Your approach has two issues. First, people without Internet access are not in the sampling frame at all, which is a critical problem if you're interested in network access. Secondly, the geolocation in step 3 is also not particularly reliable (mine tends to think I'm 100 miles away from where I am) so you'll get the same kinds of problems they found here in that the data look kinda plausible but aren't actually reliable enough to draw defensible conclusions from.
- deleted 9y ago[deleted]
- bri3d 9y agoGoogle do this with YouTube already: https://www.google.com/get/videoqualityreport/m/ https://www.google.com/get/videoqualityreport/m/
- throwawayjava 9y agoThis methodology would completely miss the point of the 538 story, which was to report on access.
- fjsolwmv 9y agoIt makes a reasonable assumption that everyone with access has similar usage profile, at the level of country averages
- deleted 9y ago[deleted]
- alex95 9y agoWhy does every title have to be clickbait nowadays. Can't help but feel like it takes away from the legitimacy of a post if you have to use leading titles/clickbait.
- codezero 9y agoDoes anyone here know if there is a way to opt-out of being in the Catalist dataset?
- bluetwo 9y agoI don't recall which FCC action it was last year, but as I recall, the large providers are no longer required to at least show they attempted to offer broadband to all households. Previously they had to show state and fed gov't this info. Now they get to concentrate on providing access to the most profitable while ignoring the less profitable.
- outsidetheparty 9y agoSo if I'm reading this right, there are three data sources here: 1) US Census, which is based on surveying households "do you have broadband, Y/N" 2) FCC data, which is based on ISPs self-reporting (In a footnote the article says they're using Pai's new slower definition of broadband, 10Mbps, not the 2015 definition of 25Mbps.) 3) ASU/Iowa, which depends on a derived variable in commercially-purchased data which "denotes interest in ‘high tech’ products and/or services [including] personal computers and internet service providers" as a proxy for broadband ownership ...and the first two roughly match each other, while the third doesn't. The academics claim the company that sold them the data told them it was a reasonable proxy for broadband, the company says they didn't say that.
- CharlesMerriam2 9y agoResponsible adults; I approve. Notice we applaud the careful report on a research report in exactly the same way we applaud the post-outage report.
- zaroth 9y agoI few things I don't like about this; "After further reporting, we can no longer vouch for the academics’ data set. The preponderance of evidence we’ve collected has led us to conclude that it is fundamentally flawed.... The idea behind the stories was to demonstrate that broadband is not ubiquitous in the U.S. today, even as more of our lives and the economy go online. We stand by this sentiment and the on-the-ground reporting in the two stories even though we have lost confidence in the data set." If the data you used to reach a conclusion is fundamentally flawed, it's pretty disingenuous to claim you stand by the sentiment. So they started with a conclusion, set out to prove it, later found the data they used to prove it was flawed, but still believe it's true. The second thing I don't like is it seems readers are very confused between access and usage and their sloppy wording often conflates the two. It appears they were studying usage (actual subscriptions) not access (availability of a high speed connection). Lastly, they also seem to disregard an LTE wireless connection as usage of broadband, when I would have assumed it would clearly be considered. If LTE wireless is more commonly used as a form of access to broadband internet in certain areas (i.e. rural areas where density can't justify running the fiber, or dense metro where the LTE is so good there's no need for a wire) then not surprising you'll find broadband "usage" is low in those areas, even if those households are absolutely using broadband internet through an LTE hotspot.
- ghaff 9y agoWell, the other data they introduced and which isn't disputed supports the idea that broadband penetration is uneven. Just not at the widespread granular and detailed level that the faulty data purported to show. You do make a reasonable point about usage and access. Which is more relevant depends on what you're trying to show. Access is whether you can get it at all while usage could well bring in economics as well. It's a fair point about LTE. They don't seem to have included satellite either, which definitely gets used in rural areas where nothing else is available. I suspect though that neither of these would affect overall conclusions that much.
- hardwires 9y ago>> If the data you used to reach a conclusion is fundamentally flawed, it's pretty disingenuous to claim you stand by the sentiment. So they started with a conclusion, set out to prove it, later found the data they used to prove it was flawed, but still believe it's true. It's just a sentiment. I have a strong sentiment that LTE reception in the middle of the Sahara is terrible. I haven't collected any data whatsoever to support that sentiment, yet I fully stand by it. There's nothing disingeneous about that.
- ACow_Adonis 9y agoI just wanted to comment that I think this is brilliant and the kind of analysis and general skeptical of data we should see more of. Just for context,of its not obvious, I work with data. Both putting it together and analyzing it. And one of my chief frustrations with academia (and biggest lessons to people I advise) is a kind of "cultural reverence for the data set". Just because data is collected, in no way does that assure that's it's right or suitable, even if the valuable name says it is. Be skeptics. Private suppliers have incentive to sell you data. Private industries have incentives to keep data from you (it constitutes competitive advantage). Government data has political interference on what is collected, even if you're lucky enough to live in a world where the actual collection is independent and rigorous. Reporters and responses to surveys and interviews may be innacurate even when people thought they were being honest, and on socially contentious topics they usually don't have that. And even if you managed to avoid all that, it doesn't mean your data isn't problematic. Our census in my country, for example, is done in the winter time. How good is that at tracking information in seasonal towns? Proper data collection is some of the hardest work you can do, and proper analysis comes from measuring, corroborating, justifying, hypothesising on the data you have. It does not involve just calculating a stat or, god forbid, just testing things for statistical significance just because it's on your data set. For all those reasons, I highly commend this article. We need more of it.
- samstave 9y agoPersonally - there is something that I would like: A monitor of exactly how much traffic is used by ads vs content. So if I load a page, and say that page is just an article with text. What % of the content is the bandwidth-consumption I am interested vs the ads surrounding it? The reason why this number is important is Mobile. So a user signs up for "3 gigs of data" - how much of that 3 gigs is consumed by ads and shit they dont want/need? Actually - it would be good to have a standard on reporting for any given page "this page weighs in at 50KB for content and 500KB for ads..." Does this exist?
- bluetwo 9y agoLarge providers do scrape this kind of information, but I'm not sure anyone discloses it.
- deleted 9y ago[deleted]
- jinma 9y agoKudos to FiveThirtyEight on being transparent and analyzing what happened. But also...this was a series of mistakes, some of them pretty scary. FiveThirtyEight's biggest mistake seems to be trusting an academic dataset when they had no idea how it was collected. This is understandable, especially when the data was published on the Arizona State University's Center for Policy Informatics data portal. (You can go there right now and download the bad data - scroll to CATALIST DATA here https://policyinformatics.asu.edu/broadband-data-portal/dataaccess/countydata https://policyinformatics.asu.edu/broadband-data-portal/data...) A university should be a trusted source. But FiveThirtyEight took an unbelievable outlier from this dataset and wrote an entire post about it (https://fivethirtyeight.com/features/lots-of-people-in-cities-still-cant-afford-broadband/ https://fivethirtyeight.com/features/lots-of-people-in-citie...). The dataset claims that only 29% of Washington D.C.'s adults have broadband. (The real number according to the other datasets FiveThirtyEight looked at in the new post is closer to 70%.) They even make a point of how extreme the Washington D.C. datapoint is on the histogram in the article as the only large county with such a low percent. That should be a clue to question your data. What I find worse is that the academic researchers published this dataset. They bought behavioral marketing data and trusted a salesperson that the variable HTIA (“Denotes interest in ‘high tech’ products and/or services as reported via Share Force. This would include personal computers and internet service providers. Blended with modeled data.”) was a good proxy for broadband access. To be clear, HTIA includes modeled data, which means they took demographics, voting records, and whatever other individual data they could grab (maybe they have records of your purchases, I'm just guessing), and predicted whether each adult in the US was interested in tech. This is the kind of data companies buy for ad campaigns, figuring that if they advertise to these adults, it might be better than random. There's no reason to think the aggregates of these numbers would be accurate or calibrated correctly, especially for an entirely different purpose (broadband vs high tech). It's disturbing that these sort of datasets are floating out there in academia and really makes you wonder what other bad data is being blindly trusted to write blog posts, research papers, and news articles.