6 ms·

The Fall of Big Data

35 points by henridf 10y ago

forgetsusername 10y ago
The Fourth major failing is in assuming that the majority of the country faces the same concerns as the rich people piled into the coastal cities. There's nothing inherently wrong with calling upon like-minded people to change government policies as the author asks (it's called lobbying), but the assumption that the "other-half" are just mistaken bigots and racists is exactly how this mess started.
- michaelsbradley 10y ago
  Exactly. Frank Bruni over at the NYT wrote this yesterday: "Liberals miss this by being illiberal. They shame not just the racists and sexists who deserve it but all who disagree. A 64-year-old Southern woman not onboard with marriage equality finds herself characterized as a hateful boob. Never mind that Barack Obama and Hillary Clinton weren’t themselves onboard just five short years ago. "Political correctness has morphed into a moral purity that may feel exhilarating but isn’t remotely tactical. It’s a handmaiden to smugness and sanctimony, undermining its own goals."[+] And note too that some of us in tech who went to big-name private universities, maxed out our SATs, are good at math and love the physical sciences....... some of us "held our noses" and voted for Trump as well, finding, in comparison, the politics and platform of Hillary Clinton to be alienating and slightly worse, on the whole. [+] http://www.nytimes.com/2016/11/13/opinion/the-democrats-screwed-up.html http://www.nytimes.com/2016/11/13/opinion/the-democrats-scre...
deleted 10y ago
[deleted]
jimmydddd 10y ago
It does seem like a smart guy with a twitter account, an airplane and a lot of energy was able to beat large embedded organizations and large infrastructures. Maybe it's like a nimble startup overtaking a large status quo industry. Some people say that his party actually had a more sophisticated big data system, though. It's just that they used it, instead of talking about it.
sudoscript 10y ago
> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the country. There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen. They sure as hell didn't find it in us. Sure, we've given them lip service. We've looked at them through our rose-tinted Google Glasses, and at best, we've felt a twinge of regret at the way the world is going. Not enough to do anything. For all the efforts to teach girls to code (a problem deeply related to our own talent shortage), how much have we done to figure out what a blue-collar worker in rural Indiana is supposed to do when the sole factory in his town shuts down? Because let's face it -- we look down on them. We call them hicks and rednecks. We scoff at their lack of high-falutin education. We think they got left behind because they're not good enough. Because they didn't pass the tests we did, didn't have the ambition we do, didn't try as hard as we do. It's a nice thought if you want something to let you sleep at night, but it's not true. Those people aren't worse than us. They're not left behind because they're stupid. They're being left behind because people like us stopped giving a damn a long time ago. The solutions we did offer were exactly the ones you'd expect from a far-removed elite aristocracy. Teach everyone to code, or throw them money. Neither of which they want. So the people you laughed at, the people you thought yourselves better than, the people you offered little more to than pie in the sky techno-utopian dreams -- is it any surprise they turn around and elect someone you don't even understand? You never understood them anyway.
- loorinm 10y ago
  Yes. This is exactly what happened and what so many liberals (myself included) don't want to hear but have to come to terms with. It's in our best interest to share the prosperity we enjoy with those less fortunate, or we create a breeding ground for anger and ignorance that will come back to truly make our lives hell. Same thing applies in the middle east where we've bred terrorism from poverty. This is our collective fault. We can't ignore them anymore. The only way forward is to empathize with our fellow human beings whose suffering is real and whose anger is dangerous.
huffpopo 10y ago
A) This is NOT Big Data B) This election was 101 in ML. The biases were obvious and easy to undo. C) I called it and won a lot of money https://news.ycombinator.com/item?id=12863445 https://news.ycombinator.com/item?id=12863445 D) Best not to take advice from people who are constantly wrong E) Big Data and ML are still awsome :) See below from my linked post a week before the election: The problem with Bayesian is that it's often over confident in the model. Error bars don't take this into account and people over rely on error bars - especially with complex domains. Then there are the intentional and unintentional sampling biases in the polls that have to be taken into account. Then there are known human effects that add additional biases. E.g. the shy voter effect (e.g. Torres and Brexit), silent majorities (Nixon), momentum, dam breaking (Reagan), enthusiasm gap (Obama), the giant fk you to the establishment vote, pending indictments etc. In my view with the election being so emotional this year these effects are big enough to swing it to Trump making a poll based model wildly inaccurate. I've taken an actual position in the election. $4k on Trump to win at 15% odds made just after the tapes. At the time I felt Trump had at least 50% chance so the bet was positive expected return for me. I figured the public would get over it. My Bayesian friends had his odds at 2%. Today I consider Trump to be at least 70% with the whole FBI inditement saga picking up steam. In addition, the betting market behavior mirrors Brexit. A few very big bets on the status quo and many small bets against. It appears as if once again deep pocket punters are intentionality trying to manipulate the odds on the illiquid market in order to send a message that effects the vastly larger financial markets. So I think the odds are that there is some free money there.
- huffpopo 10y ago
  NOTE: The link only appears to work for my old account?? Was I shadow banned? Perhaps banning decenting opinions are a big reason why this election was such a supprise to everyone. https://news.ycombinator.com/threads?id=ttyltrful https://news.ycombinator.com/threads?id=ttyltrful
  - matt4077 10y ago
    Not banned / save the conspiracies for another day / "dissenting" / "surprise"
clifanatic 10y ago
Wow, talk about a click-bait headline... absolutely nothing to do with big data.
bitL 10y ago
I don't think this has any relation to Big Data. It's simply Trump supporters were clever enough to give out false signals due to being ostracized by media and worried about consequences of going public (fired from a job, removed from social circles etc.) So the massive pressure on conformity created a late-Soviet-Union-style dissonance, outwardly many supporting "approved" candidates, inwardly supporting Trump (maybe due to the this pressure from all sides as a simple vote against). Many would've probably voted for "nobody" if that were a choice alongside Hillary and Donald.
ramblenode 10y ago
Clickbait. The TLDR: A) Polls were biased and we don't have many elections on which to build good models. B) Social media created echo chambers. C) Other people's politics are sci-fi and more people should agree with me.
sluggg 10y ago
I urge everyone to read Social Physics by Alex Pentland, it addresses some of the topics in the article and provides a more holistic mental framework for big data. It's a good read. https://www.amazon.com/Social-Physics-Networks-Make-Smarter/dp/0143126334 https://www.amazon.com/Social-Physics-Networks-Make-Smarter/...
LordHog 10y ago
I stopped reading after the first few sentences that were politically driven. Click bait?

35 points by henridf 10y ago

The Fourth major failing is in assuming that the majority of the country faces the same concerns as the rich people piled into the coastal cities. There's nothing inherently wrong with calling upon like-minded people to change government policies as the author asks (it's called lobbying), but the assumption that the "other-half" are just mistaken bigots and racists is exactly how this mess started.

michaelsbradley 10y ago
Exactly. Frank Bruni over at the NYT wrote this yesterday: "Liberals miss this by being illiberal. They shame not just the racists and sexists who deserve it but all who disagree. A 64-year-old Southern woman not onboard with marriage equality finds herself characterized as a hateful boob. Never mind that Barack Obama and Hillary Clinton weren’t themselves onboard just five short years ago. "Political correctness has morphed into a moral purity that may feel exhilarating but isn’t remotely tactical. It’s a handmaiden to smugness and sanctimony, undermining its own goals."[+] And note too that some of us in tech who went to big-name private universities, maxed out our SATs, are good at math and love the physical sciences....... some of us "held our noses" and voted for Trump as well, finding, in comparison, the politics and platform of Hillary Clinton to be alienating and slightly worse, on the whole. [+] http://www.nytimes.com/2016/11/13/opinion/the-democrats-screwed-up.html http://www.nytimes.com/2016/11/13/opinion/the-democrats-scre...

deleted 10y ago

[deleted]

jimmydddd 10y ago

It does seem like a smart guy with a twitter account, an airplane and a lot of energy was able to beat large embedded organizations and large infrastructures. Maybe it's like a nimble startup overtaking a large status quo industry. Some people say that his party actually had a more sophisticated big data system, though. It's just that they used it, instead of talking about it.

sudoscript 10y ago

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the country. There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen. They sure as hell didn't find it in us. Sure, we've given them lip service. We've looked at them through our rose-tinted Google Glasses, and at best, we've felt a twinge of regret at the way the world is going. Not enough to do anything. For all the efforts to teach girls to code (a problem deeply related to our own talent shortage), how much have we done to figure out what a blue-collar worker in rural Indiana is supposed to do when the sole factory in his town shuts down? Because let's face it -- we look down on them. We call them hicks and rednecks. We scoff at their lack of high-falutin education. We think they got left behind because they're not good enough. Because they didn't pass the tests we did, didn't have the ambition we do, didn't try as hard as we do. It's a nice thought if you want something to let you sleep at night, but it's not true. Those people aren't worse than us. They're not left behind because they're stupid. They're being left behind because people like us stopped giving a damn a long time ago. The solutions we did offer were exactly the ones you'd expect from a far-removed elite aristocracy. Teach everyone to code, or throw them money. Neither of which they want. So the people you laughed at, the people you thought yourselves better than, the people you offered little more to than pie in the sky techno-utopian dreams -- is it any surprise they turn around and elect someone you don't even understand? You never understood them anyway.

loorinm 10y ago
Yes. This is exactly what happened and what so many liberals (myself included) don't want to hear but have to come to terms with. It's in our best interest to share the prosperity we enjoy with those less fortunate, or we create a breeding ground for anger and ignorance that will come back to truly make our lives hell. Same thing applies in the middle east where we've bred terrorism from poverty. This is our collective fault. We can't ignore them anymore. The only way forward is to empathize with our fellow human beings whose suffering is real and whose anger is dangerous.

huffpopo 10y ago

A) This is NOT Big Data B) This election was 101 in ML. The biases were obvious and easy to undo. C) I called it and won a lot of money https://news.ycombinator.com/item?id=12863445 https://news.ycombinator.com/item?id=12863445 D) Best not to take advice from people who are constantly wrong E) Big Data and ML are still awsome :) See below from my linked post a week before the election: The problem with Bayesian is that it's often over confident in the model. Error bars don't take this into account and people over rely on error bars - especially with complex domains. Then there are the intentional and unintentional sampling biases in the polls that have to be taken into account. Then there are known human effects that add additional biases. E.g. the shy voter effect (e.g. Torres and Brexit), silent majorities (Nixon), momentum, dam breaking (Reagan), enthusiasm gap (Obama), the giant fk you to the establishment vote, pending indictments etc. In my view with the election being so emotional this year these effects are big enough to swing it to Trump making a poll based model wildly inaccurate. I've taken an actual position in the election. $4k on Trump to win at 15% odds made just after the tapes. At the time I felt Trump had at least 50% chance so the bet was positive expected return for me. I figured the public would get over it. My Bayesian friends had his odds at 2%. Today I consider Trump to be at least 70% with the whole FBI inditement saga picking up steam. In addition, the betting market behavior mirrors Brexit. A few very big bets on the status quo and many small bets against. It appears as if once again deep pocket punters are intentionality trying to manipulate the odds on the illiquid market in order to send a message that effects the vastly larger financial markets. So I think the odds are that there is some free money there.

huffpopo 10y ago
NOTE: The link only appears to work for my old account?? Was I shadow banned? Perhaps banning decenting opinions are a big reason why this election was such a supprise to everyone. https://news.ycombinator.com/threads?id=ttyltrful https://news.ycombinator.com/threads?id=ttyltrful
- matt4077 10y ago
  Not banned / save the conspiracies for another day / "dissenting" / "surprise"

clifanatic 10y ago

Wow, talk about a click-bait headline... absolutely nothing to do with big data.

bitL 10y ago

I don't think this has any relation to Big Data. It's simply Trump supporters were clever enough to give out false signals due to being ostracized by media and worried about consequences of going public (fired from a job, removed from social circles etc.) So the massive pressure on conformity created a late-Soviet-Union-style dissonance, outwardly many supporting "approved" candidates, inwardly supporting Trump (maybe due to the this pressure from all sides as a simple vote against). Many would've probably voted for "nobody" if that were a choice alongside Hillary and Donald.

ramblenode 10y ago

Clickbait. The TLDR: A) Polls were biased and we don't have many elections on which to build good models. B) Social media created echo chambers. C) Other people's politics are sci-fi and more people should agree with me.

sluggg 10y ago

I urge everyone to read Social Physics by Alex Pentland, it addresses some of the topics in the article and provides a more holistic mental framework for big data. It's a good read. https://www.amazon.com/Social-Physics-Networks-Make-Smarter/dp/0143126334 https://www.amazon.com/Social-Physics-Networks-Make-Smarter/...

LordHog 10y ago

I stopped reading after the first few sentences that were politically driven. Click bait?