8 ms·

Sadly typical of HN today that a topic that is interesting technically and politically neutral (the challenge of accurate polling) is derailed up top with an in

Sadly typical of HN today that a topic that is interesting technically and politically neutral (the challenge of accurate polling) is derailed up top with an inaccurate culture war comment (there is no correlation between when a Dem condemned violent protest and whether they won their election). FWIW, Shor’s theory about poll respondents is shared by Robert Cahaly at Trafalgar, but Trafalgar polls missed badly too in 2020 but in the opposite direction. So either that’s not actually the critical factor, or it’s not possible to control for this factor in any sort of reliable way. I find that far more interesting than Shor’s personal HR problems.

SubiculumCode 6y ago
Agreed. HN is going the way of Slashdot.
- d1zzy 6y ago
  Oh wow, I'm so happy I'm not the only one thinking this. I noticed this transition over the past few years on these forums and I even commented about it in the past.
  - lanstin 6y ago
    It is the path of things on the internet that are interesting and then become popular. Same for Usenet :) Kuro5hin escaped by becoming uninteresting. Our descendants may find a solution.
    - sitkack 6y ago
      Things change over time. Also note that HN used to be over-run by libertarians that would make Ayn blush. Kuro5hin escaped by dying. There is lobsters, but it has almost zero discussion and overly focuses on "the rules". I think norms are better and more powerful. Be the norm you would like to see in the world. https://news.ycombinator.com/item?id=11609308 https://news.ycombinator.com/item?id=11609308
    - hanselot 6y ago
      the solution is 4chan. Moderate only the horrifying, appease nobody's personal belief that they have a right not to be offended, and make your voice heard. Everyone needs to learn to have a better bs filter in times of AI generated fake news. What use is trying to eradicate the things we disagree with? Clearly the answer is inoculation.
  - SubiculumCode 6y ago
    I should qualify my statement as I get voted down into oblivion. Hn still has higher quality comments and I'd ever seen it slashdot. However those high value comments usually come on particular subjects that lie within the hn community's zone of expertise. However for political topics or for science articles, hn comments usually lack expertise or neutrality. They're all falls into whether hn thinks of itself more as a repository of expertise or a forum for individuals to discuss topics where those individuals have expertise in some topic ie a professional which by and large the community is composed. Please excuse any strange typos I am voice typing this as I walk.
    - SubiculumCode 6y ago
      The truth hurts.
darawk 6y ago
> there is no correlation between when a Dem condemned violent protest and whether they won their election You got a source on that? It's clear that democrats lost ground, despite the Republican party's abject failure to control the coronavirus. This should have been the easiest win in world history, and somehow, the Democrats barely escaped with the presidency. There is pretty clearly something to explain here.
- deleted 6y ago
  [deleted]
- nxmnxm99 6y ago
  Every single demographic went further red than 2016 besides white men. There’s a reason for that.
  - systemvoltage 6y ago
    You’ve got data to back that up?
    - jimbobimbo 6y ago
      https://www.essence.com/news/politics/55-percent-white-women-trump-election-2020/ https://www.essence.com/news/politics/55-percent-white-women...
      - dragonwriter 6y ago
        This is based on exit polls. Exit polls only interview people who vote in person, which may be generally be useful, but in this election are an enormously systematically biased slice of the electorate. You really can't base any generalizations beyond the in-person voting segment based on them.
        AnthonyMouse 6y ago
        > This is based on exit polls. Are there some other measurements based on something else, or is it something we now can never know as a result of mail in ballots having no polling and an expected different statistical distribution?
newacct583 6y ago
Yeah, exactly. People who want to scream about how Shor got cancelled have zero interest in actually discussing anything Shor writes. That said, and to be perfectly clear: Civis should not have fired Shor. It was very bad. (To be fair in the other direction: the guy is a known genius in leftie political circles and my understanding is that he's done very well for himself since termination as an independent consultant.) But really it's the only good example of the "Cancel Culture" paranoia, so right wingers feel the need to trot out the argument every time his name comes up.
- klipt 6y ago
  > Civis should not have fired Shor. It was very bad. Why, because you agree with his politics? That doesn't seem like a very objective way to determine who we should or shouldn't cancel.
  - newacct583 6y ago
    No, because what he said was a factual analysis and the outrage was about its conclusions. I very much agree with his politics.
- alentist 6y ago
  > it's the only good example of the "Cancel Culture" Here's a few more: https://threadreaderapp.com/thread/1282404647160942598.html https://threadreaderapp.com/thread/1282404647160942598.html
  - joijoiwejrr 6y ago
    It astonishes me that anyone can think "cancel culture" is a right-wing myth.
nxmnxm99 6y ago
What? Trafalgar was one of the only polls calling a close race. In Trumps favour but close nonetheless. IIRC they were the only ones calling Florida going red. You’re being intentionally disingenuous saying they were equally as bad as the rest of the field predicting a blue wave.
- Green_man 6y ago
  Trafalgar almost certainly got more states wrong (counts aren't all finished, but we do know likely winners in all states). Other sites like 538 predicted all states correctly, except NC and FL (50.5%/48.8% and 50.9%/48.4% respectively, both predictions wrongly favoring Biden [1]). Meanwhile, Trafalgar's polls put Trump up in NV, GA, PA, AZ, and MI (from their website, all late Oct / early Nov polls [2]). Once counts are final, we can more accurately compare margins and errors more specifically, but Trafalgar does not have a good record for 2020. Comparing models to a single pollster isn't completely even, but it's important to note that the Trafalgar polls were mostly outliers, disagreeing with the consensus in many states [3]. [1]https://projects.fivethirtyeight.com/2020-election-forecast/ https://projects.fivethirtyeight.com/2020-election-forecast/ [2]https://www.thetrafalgargroup.org/ https://www.thetrafalgargroup.org/ [3]https://projects.fivethirtyeight.com/polls/president-general/national/ https://projects.fivethirtyeight.com/polls/president-general... https://www.realclearpolitics.com/epolls/latest_polls/ https://www.realclearpolitics.com/epolls/latest_polls/
  - zimablue 6y ago
    Number of states that you get wrong isn’t a great measure though right? If I call if for trump at 51-49 and you call it for Biden at 75-25 and Biden wins by 51-49 I was more correct than you were, to count “states called” is the wrong way to evaluate analysis in the same way that fptp is unrepresentatove of the popular vote, ironically.
    - Green_man 6y ago
      I agree with you, and that's why i made effort to differentiate between margins vs winning. In my opinion, 2020's polls were a lot worse than 2016's, but because of Biden's much bigger lead (in the polls and in key states), the polling errors did not cross the threshold to be "wrong" in the winner take all system. Finally, there is an argument that even a Biden 75-25 call gives the correct outcome because each state is a winner take all system (with the exception of NE and ME, iirc). From a stats standpoint, the delta in that example make it very inaccurate, but from a model/communication standpoint, getting each state "right" (in binary terms) is very valuable. Trafalgar had completely different results than the rest of the consensus, and once counts are more finalized in non swing states (places like CA and NY take forever, but we don't pay attention because they are always blue), we can accurately compare the polling and real results between different pollsters and polling methodologies. Some experts have cautioned against using exit polls for this purpose (what is usually done for a quick read of polling accuracy), because exit polls are only measuring election day in person votes, and thus trend really red this year. Trafalgar had unconventional methodologies like a "shy trump voter" bias where they arbitrarily shifted their numbers to the right, with weirdly consistent results of Trump +3 in a bunch of close/blue states. Perhaps there is more complicated justification of these numbers in the background, but I'm concerned that even if their deltas end up better than the "normal" pollsters, they are generating an inferior product with overbaked data.
rayiner 6y ago
Vox euphemized Schor’s departure in a way that was highly relevant to the subject of the article. The issue isn’t that pollsters couldn’t quantify the problem, it’s that they didn’t even realize the problem existed: https://www.washingtonpost.com/outlook/2020/11/02/shy-trump-voters-myth/ https://www.washingtonpost.com/outlook/2020/11/02/shy-trump-... > Trump voters are not liars. As a Democratic pollster and campaign strategist, I feel strange typing these words. But it is required to debunk the “shy Trump voter” myth. This was written days before the election. My post isn’t about “Shor’s personal HR problem.” It’s about a culture that caused very smart people to misperceive the facts on the ground, and misallocate funding as a result, in the middle of a high stakes situation. People in Silicon Valley should understand better than anyone that culture matters and bears on organizational and institutional effectiveness. Often these are pretty theoretical and abstract debates. But here is a case study that elucidates culture that works versus culture that doesn’t work. I’m not weighing in on the larger cultural issue. As I caveated, this is context sensitive. Political polling, news, academia, and the like are areas where free discourse with respect to sensitive issues is particularly important for the effectiveness of the institution. This is less true if you’re running a food delivery business.
- snowwrestler 6y ago
  I have a hard time believing your musings on culture are relevant to the topic of accurate polling when GOP-aligned pollsters, who firmly believed that Trump voters are less likely to answer polls and built that into their models, also missed badly.
  - thu2111 6y ago
    Were there any other than Trafalgar? And did Trafalgar miss as badly as their competitors? My understanding was no.
chmod600 6y ago
I think you missed the point: if people feel like their opinions will be held against them, they will not share their true opinions, and polling will fail. The comment seems on-topic to me regardless of whether the culture war represents reality or not
illiilliiililil 6y ago
There has never been such a thing as a politically neutral HN. It's just that there wasn't until recently a userbase to counteract the liberal tech groupthink.
- disgruntledphd2 6y ago
  That's more recent, to be honest. When I started reading HN, it was very very libertarian (but mostly non-political). As it's grown bigger, it's started to reflect the demographics of software/data/professional people, with a bias towards the West Coast of the US. That's presumably what you mean by the liberal tech groupthink (note that disliking Trump does not make one a liberal, which is a mistake a lot of conservatives have made over the past four years).
  - illiilliiililil 6y ago
    >That's more recent, to be honest. When I started reading HN, it was very very libertarian (but mostly non-political). Interesting. I came here a bit late.
  - ryathal 6y ago
    How long ago was the dominant libertarian scene? I've been reading for about 7 years and it's definitely been heavy liberal with the occasional libertarian belief even back then. Since 2016 it's been full throated liberalism though.
    - disgruntledphd2 6y ago
      2011 is when I started lurking here. i originally had disgruntledphd, but lost my password and there was no way to recover it (apparently you could contact paul graham, but I didn't know that) so I became disgruntledphd2.
newswasboring 6y ago
> there is no correlation between when a Dem condemned violent protest and whether they won their election The tweet shor got fired for was literally giving data for this. Did you see the article about his firing?
- snowwrestler 6y ago
  Neither that tweet, nor the paper it referred to, attempted to correlate the exact date at which each Democratic candidate condemned violence with their respective election results.
  - newswasboring 6y ago
    that is such an impossible ask. We can only look at these things in aggregate. Contrary to popular belief a whole country doesn't turn on a time. Also even if that is your standard, this finding is still relevant.
raxxorrax 6y ago
These HR problems have a negative influence on the quality of the work. To me it looks like "killing the messenger". Listeners want the most accurate data irrespective of the fact that some unknowns can be controlled for or not. Furthermore to respond to the culture war that you brought up, I think Shor's theory fits my personal experience that some people stopped stating their political opinion in the open. So it is on topic. That may very well be due to "HR" being a problem or those that play HR on public platforms. The explanation might be correct or wrong, it might resonate with people or not. Your comment is just a data point here.
qPM9l3XJrF 6y ago
>there is no correlation between when a Dem condemned violent protest and whether they won their election Even assuming this unsupported claim is true, it's not necessarily relevant because correlation isn't the same as causation. For example, it could be that Dems are pretty good at reading their local electorate, and left-leaning areas are more tolerant of violent protest in the name of left causes. So Dems in lefty areas did not condemn violence, whereas Dems in swing areas did condemn violence. If this was true, ceteris paribus you would expect Dems who condemn violence to be more likely to lose, just because they're running more competitive races. BTW, I think people are unfortunately quite tribal nowadays, and tend to assume that extremists have control of the other side even if that's not the case. And disavowing extremists is only partially effective at changing that perception. Think about it--how much have Trump's many disavowals of the far right shifted your perception that he's in bed with them? https://www.youtube.com/watch?v=Bd0cMmBvqWc https://www.youtube.com/watch?v=Bd0cMmBvqWc Probably not much. >So either that’s not actually the critical factor, or it’s not possible to control for this factor in any sort of reliable way. One failed attempt doesn't mean that something can't be done.

SubiculumCode 6y ago

Agreed. HN is going the way of Slashdot.

d1zzy 6y ago
Oh wow, I'm so happy I'm not the only one thinking this. I noticed this transition over the past few years on these forums and I even commented about it in the past.
- lanstin 6y ago
  It is the path of things on the internet that are interesting and then become popular. Same for Usenet :) Kuro5hin escaped by becoming uninteresting. Our descendants may find a solution.
  - sitkack 6y ago
    Things change over time. Also note that HN used to be over-run by libertarians that would make Ayn blush. Kuro5hin escaped by dying. There is lobsters, but it has almost zero discussion and overly focuses on "the rules". I think norms are better and more powerful. Be the norm you would like to see in the world. https://news.ycombinator.com/item?id=11609308 https://news.ycombinator.com/item?id=11609308
  - hanselot 6y ago
    the solution is 4chan. Moderate only the horrifying, appease nobody's personal belief that they have a right not to be offended, and make your voice heard. Everyone needs to learn to have a better bs filter in times of AI generated fake news. What use is trying to eradicate the things we disagree with? Clearly the answer is inoculation.
- SubiculumCode 6y ago
  I should qualify my statement as I get voted down into oblivion. Hn still has higher quality comments and I'd ever seen it slashdot. However those high value comments usually come on particular subjects that lie within the hn community's zone of expertise. However for political topics or for science articles, hn comments usually lack expertise or neutrality. They're all falls into whether hn thinks of itself more as a repository of expertise or a forum for individuals to discuss topics where those individuals have expertise in some topic ie a professional which by and large the community is composed. Please excuse any strange typos I am voice typing this as I walk.
  - SubiculumCode 6y ago
    The truth hurts.

darawk 6y ago

> there is no correlation between when a Dem condemned violent protest and whether they won their election You got a source on that? It's clear that democrats lost ground, despite the Republican party's abject failure to control the coronavirus. This should have been the easiest win in world history, and somehow, the Democrats barely escaped with the presidency. There is pretty clearly something to explain here.

deleted 6y ago
[deleted]
nxmnxm99 6y ago
Every single demographic went further red than 2016 besides white men. There’s a reason for that.
- systemvoltage 6y ago
  You’ve got data to back that up?
  - jimbobimbo 6y ago
    https://www.essence.com/news/politics/55-percent-white-women-trump-election-2020/ https://www.essence.com/news/politics/55-percent-white-women...
    - dragonwriter 6y ago
      This is based on exit polls. Exit polls only interview people who vote in person, which may be generally be useful, but in this election are an enormously systematically biased slice of the electorate. You really can't base any generalizations beyond the in-person voting segment based on them.
      - AnthonyMouse 6y ago
        > This is based on exit polls. Are there some other measurements based on something else, or is it something we now can never know as a result of mail in ballots having no polling and an expected different statistical distribution?

newacct583 6y ago

Yeah, exactly. People who want to scream about how Shor got cancelled have zero interest in actually discussing anything Shor writes. That said, and to be perfectly clear: Civis should not have fired Shor. It was very bad. (To be fair in the other direction: the guy is a known genius in leftie political circles and my understanding is that he's done very well for himself since termination as an independent consultant.) But really it's the only good example of the "Cancel Culture" paranoia, so right wingers feel the need to trot out the argument every time his name comes up.

klipt 6y ago
> Civis should not have fired Shor. It was very bad. Why, because you agree with his politics? That doesn't seem like a very objective way to determine who we should or shouldn't cancel.
- newacct583 6y ago
  No, because what he said was a factual analysis and the outrage was about its conclusions. I very much agree with his politics.
alentist 6y ago
> it's the only good example of the "Cancel Culture" Here's a few more: https://threadreaderapp.com/thread/1282404647160942598.html https://threadreaderapp.com/thread/1282404647160942598.html
- joijoiwejrr 6y ago
  It astonishes me that anyone can think "cancel culture" is a right-wing myth.

nxmnxm99 6y ago

What? Trafalgar was one of the only polls calling a close race. In Trumps favour but close nonetheless. IIRC they were the only ones calling Florida going red. You’re being intentionally disingenuous saying they were equally as bad as the rest of the field predicting a blue wave.

Green_man 6y ago
Trafalgar almost certainly got more states wrong (counts aren't all finished, but we do know likely winners in all states). Other sites like 538 predicted all states correctly, except NC and FL (50.5%/48.8% and 50.9%/48.4% respectively, both predictions wrongly favoring Biden [1]). Meanwhile, Trafalgar's polls put Trump up in NV, GA, PA, AZ, and MI (from their website, all late Oct / early Nov polls [2]). Once counts are final, we can more accurately compare margins and errors more specifically, but Trafalgar does not have a good record for 2020. Comparing models to a single pollster isn't completely even, but it's important to note that the Trafalgar polls were mostly outliers, disagreeing with the consensus in many states [3]. [1]https://projects.fivethirtyeight.com/2020-election-forecast/ https://projects.fivethirtyeight.com/2020-election-forecast/ [2]https://www.thetrafalgargroup.org/ https://www.thetrafalgargroup.org/ [3]https://projects.fivethirtyeight.com/polls/president-general/national/ https://projects.fivethirtyeight.com/polls/president-general... https://www.realclearpolitics.com/epolls/latest_polls/ https://www.realclearpolitics.com/epolls/latest_polls/
- zimablue 6y ago
  Number of states that you get wrong isn’t a great measure though right? If I call if for trump at 51-49 and you call it for Biden at 75-25 and Biden wins by 51-49 I was more correct than you were, to count “states called” is the wrong way to evaluate analysis in the same way that fptp is unrepresentatove of the popular vote, ironically.
  - Green_man 6y ago
    I agree with you, and that's why i made effort to differentiate between margins vs winning. In my opinion, 2020's polls were a lot worse than 2016's, but because of Biden's much bigger lead (in the polls and in key states), the polling errors did not cross the threshold to be "wrong" in the winner take all system. Finally, there is an argument that even a Biden 75-25 call gives the correct outcome because each state is a winner take all system (with the exception of NE and ME, iirc). From a stats standpoint, the delta in that example make it very inaccurate, but from a model/communication standpoint, getting each state "right" (in binary terms) is very valuable. Trafalgar had completely different results than the rest of the consensus, and once counts are more finalized in non swing states (places like CA and NY take forever, but we don't pay attention because they are always blue), we can accurately compare the polling and real results between different pollsters and polling methodologies. Some experts have cautioned against using exit polls for this purpose (what is usually done for a quick read of polling accuracy), because exit polls are only measuring election day in person votes, and thus trend really red this year. Trafalgar had unconventional methodologies like a "shy trump voter" bias where they arbitrarily shifted their numbers to the right, with weirdly consistent results of Trump +3 in a bunch of close/blue states. Perhaps there is more complicated justification of these numbers in the background, but I'm concerned that even if their deltas end up better than the "normal" pollsters, they are generating an inferior product with overbaked data.

rayiner 6y ago

Vox euphemized Schor’s departure in a way that was highly relevant to the subject of the article. The issue isn’t that pollsters couldn’t quantify the problem, it’s that they didn’t even realize the problem existed: https://www.washingtonpost.com/outlook/2020/11/02/shy-trump-voters-myth/ https://www.washingtonpost.com/outlook/2020/11/02/shy-trump-... > Trump voters are not liars. As a Democratic pollster and campaign strategist, I feel strange typing these words. But it is required to debunk the “shy Trump voter” myth. This was written days before the election. My post isn’t about “Shor’s personal HR problem.” It’s about a culture that caused very smart people to misperceive the facts on the ground, and misallocate funding as a result, in the middle of a high stakes situation. People in Silicon Valley should understand better than anyone that culture matters and bears on organizational and institutional effectiveness. Often these are pretty theoretical and abstract debates. But here is a case study that elucidates culture that works versus culture that doesn’t work. I’m not weighing in on the larger cultural issue. As I caveated, this is context sensitive. Political polling, news, academia, and the like are areas where free discourse with respect to sensitive issues is particularly important for the effectiveness of the institution. This is less true if you’re running a food delivery business.

snowwrestler 6y ago
I have a hard time believing your musings on culture are relevant to the topic of accurate polling when GOP-aligned pollsters, who firmly believed that Trump voters are less likely to answer polls and built that into their models, also missed badly.
- thu2111 6y ago
  Were there any other than Trafalgar? And did Trafalgar miss as badly as their competitors? My understanding was no.

chmod600 6y ago

I think you missed the point: if people feel like their opinions will be held against them, they will not share their true opinions, and polling will fail. The comment seems on-topic to me regardless of whether the culture war represents reality or not

illiilliiililil 6y ago

There has never been such a thing as a politically neutral HN. It's just that there wasn't until recently a userbase to counteract the liberal tech groupthink.

disgruntledphd2 6y ago
That's more recent, to be honest. When I started reading HN, it was very very libertarian (but mostly non-political). As it's grown bigger, it's started to reflect the demographics of software/data/professional people, with a bias towards the West Coast of the US. That's presumably what you mean by the liberal tech groupthink (note that disliking Trump does not make one a liberal, which is a mistake a lot of conservatives have made over the past four years).
- illiilliiililil 6y ago
  >That's more recent, to be honest. When I started reading HN, it was very very libertarian (but mostly non-political). Interesting. I came here a bit late.
- ryathal 6y ago
  How long ago was the dominant libertarian scene? I've been reading for about 7 years and it's definitely been heavy liberal with the occasional libertarian belief even back then. Since 2016 it's been full throated liberalism though.
  - disgruntledphd2 6y ago
    2011 is when I started lurking here. i originally had disgruntledphd, but lost my password and there was no way to recover it (apparently you could contact paul graham, but I didn't know that) so I became disgruntledphd2.

newswasboring 6y ago

> there is no correlation between when a Dem condemned violent protest and whether they won their election The tweet shor got fired for was literally giving data for this. Did you see the article about his firing?

snowwrestler 6y ago
Neither that tweet, nor the paper it referred to, attempted to correlate the exact date at which each Democratic candidate condemned violence with their respective election results.
- newswasboring 6y ago
  that is such an impossible ask. We can only look at these things in aggregate. Contrary to popular belief a whole country doesn't turn on a time. Also even if that is your standard, this finding is still relevant.

raxxorrax 6y ago

These HR problems have a negative influence on the quality of the work. To me it looks like "killing the messenger". Listeners want the most accurate data irrespective of the fact that some unknowns can be controlled for or not. Furthermore to respond to the culture war that you brought up, I think Shor's theory fits my personal experience that some people stopped stating their political opinion in the open. So it is on topic. That may very well be due to "HR" being a problem or those that play HR on public platforms. The explanation might be correct or wrong, it might resonate with people or not. Your comment is just a data point here.

qPM9l3XJrF 6y ago

>there is no correlation between when a Dem condemned violent protest and whether they won their election Even assuming this unsupported claim is true, it's not necessarily relevant because correlation isn't the same as causation. For example, it could be that Dems are pretty good at reading their local electorate, and left-leaning areas are more tolerant of violent protest in the name of left causes. So Dems in lefty areas did not condemn violence, whereas Dems in swing areas did condemn violence. If this was true, ceteris paribus you would expect Dems who condemn violence to be more likely to lose, just because they're running more competitive races. BTW, I think people are unfortunately quite tribal nowadays, and tend to assume that extremists have control of the other side even if that's not the case. And disavowing extremists is only partially effective at changing that perception. Think about it--how much have Trump's many disavowals of the far right shifted your perception that he's in bed with them? https://www.youtube.com/watch?v=Bd0cMmBvqWc https://www.youtube.com/watch?v=Bd0cMmBvqWc Probably not much. >So either that’s not actually the critical factor, or it’s not possible to control for this factor in any sort of reliable way. One failed attempt doesn't mean that something can't be done.