10 ms·

The fivethirtyeight R package

minimaxir 10y ago
It's worth nothing that per the linked talks, 538's data visualizations are indeed created with R and ggplot2; the fancy annotations are done in Illustrator after exporting from ggplot2 in SVG/PDF. The vignettes (https://mran.microsoft.com/web/packages/fivethirtyeight/vignettes/bechdel.html https://mran.microsoft.com/web/packages/fivethirtyeight/vign...) are a good tutorial in dplyr/ggplot2 too.
- KurtMueller 10y ago
  Side question: Can I create interactive visualizations with R? Something akin to D3 in javascript?
  - michaelsbradley 10y ago
    Check out ggvis and Shiny: http://ggvis.rstudio.com/ http://ggvis.rstudio.com/ http://shiny.rstudio.com/ http://shiny.rstudio.com/ http://rmarkdown.rstudio.com/flexdashboard/ http://rmarkdown.rstudio.com/flexdashboard/
    - apathy 10y ago
      Good lord, even after all this time, ggvis' demo is still rather a disaster on mobile. Oh well, it simply isn't possible for everything to be top priority. Rstudio has done some amazing work, I'm afraid ggvis is not yet up to the standards of their other projects.
      - fourthark 10y ago
        They've been busy!
  - arifyali 10y ago
    Well D3 is always gonna be the best (downside, you're messing around in JS). However the rBokeh and plotly packages can help make interactive graphs.
  - delphithrowaway 10y ago
    This article discusses many of the options to output interactive visualizations from R. https://www.r-bloggers.com/interactive-visualizations-with-r-a-minireview/ https://www.r-bloggers.com/interactive-visualizations-with-r... None of them are as flexible as D3, but there are packages to output from R to D3 out there.
  - minimaxir 10y ago
    I strongly recommend using Plotly, specifically ggplotly (https://plot.ly/ggplot2/ https://plot.ly/ggplot2/), which converts ggplot2 charts to interactive d3 charts with good parity. Plotly also has WebGL and 3D data visualization support. One of the Plotly developers has a very good book on using Plotly in R: https://cpsievert.github.io/plotly_book/ https://cpsievert.github.io/plotly_book/ Here's a higher-level demo of ggplotly in one of my R Notebooks: http://minimaxir.com/notebooks/breach-network/ http://minimaxir.com/notebooks/breach-network/ Shiny requires an external server and is often overkill for static data.
    - RockyMcNuts 10y ago
      You can build Shiny apps in RStudio and publish them with one click to shinyapps.io. Works very similarly to publishing with Plotly, even though Shiny is a server-side package vs. client-side for Plotly. (You can build a Plotly chart in ggplot, embed it in a web page and then script it with JS to get a visualization. Frankly, that's a PITA. You can write R/shiny code that will generate a visualization, with HTML controls etc, and you end up writing a web app in R and that's a different PITA. I would like to see Plotly generate a web wrapper that automates wiring up a plotly graph and scripting it from an app so you can update data on the fly to make it a visualization. Supposedly they are working on it with their dash framework, but it's not released yet.)
      - minimaxir 10y ago
        The pricing on shinyapps.io is prohibitive if the post gets any more than a trickle of page views. The embed-directly-into-HTML is the approach I use for my Jekyll blog and it does not require much effort. (I just have to set a YAML flag to load Plotly library)
        RockyMcNuts 10y ago
        Yes, for embedding a chart and getting some basic active functionality Plotly is very easy. The part that requires some custom Plotly coding is when you want to script the chart ... have HTML buttons, sliders, to filter, recompute, revisualize the data. It's beautiful that it even works, and you can programmatically update the JSON of an embedded chart. But it would be nice if you could give Plotly a dataset, plot the data, and say, give me a button to filter rows using these criteria for this column (or run some other code on the data) and re-render the chart. Right now you have to code that manually and update the JSON model. Visualization used to mean charts that were animated and/or would dynamically update but now it means any old chart LOL.
  - phillc73 10y ago
    htmlwidgets[0] and flexdashboard[1]. Essentially it's writing rmarkdown documents. Shiny integration is optional. [0] http://www.htmlwidgets.org/ http://www.htmlwidgets.org/ [1] http://rmarkdown.rstudio.com/flexdashboard/ http://rmarkdown.rstudio.com/flexdashboard/
lintiness 10y ago
here's nate silver's ingenious monthly odds generator: #odds [insert democrat here] wins sample(75:100, 12, replace=TRUE)
- lintiness 10y ago
  went negative pretty fast here! must be big demand for a package that helps you be massively wrong almost all the time as long as it reinforces your political religion.
  - sctb 10y ago
    We've banned this account for continuing to post unsubstantively after we've asked you multiple times to stop.
    - lintiness 10y ago
      "unsubstantively" = demonstrable political bunkering by the children around here. the ban is heartbreaking.
gfosco 10y ago
Very interesting, but the source of the data is so suspect that it'd be hard to have confidence in anything gleaned from it. It would be like CNN putting out a fake news package.
- 09bjb 10y ago
  Yep. Beautifully wrought confirmations of bad hypotheses. After this election I'm taking anything stats-related on 538 (most everything) with a huge grain of salt. Can't stand another "Here's why the numbers show Hillary will win in a landslide" etc.
  - ivraatiems 10y ago
    You do realize that of all the models available, 538's was the closest to the actual outcome and gave Trump by far the largest chance? 538 was very clear from the outset that the model's goal was predicting likelihoods, not being an oracle. It said there was a 1 in 3 chance of the election coming out like it did. That's not inaccurate - it was unlikely, but it still happened. Plus, 538 didn't make the data. It aggregated data others made. What exactly was bad in that hypothesis?
    - jackhack 10y ago
      Compsci 101. Garbage in, garbage out.
      - ivraatiems 10y ago
        The polls weren't garbage. They were closer this year than they were in 2012. They were just slightly wrong in the wrong direction.
        jackhack 10y ago
        It's not a pejorative. It's a common phrase meaning "the quality of your output is only as good as your input data". In this case, input data was skewed/biased/inaccurate for a number of reasons (sample choices, underreporting, etc.). Thus even the best predictions were wrong.
ivraatiems 10y ago
In my opinion, 538 did some of the best analytics work in the 2016 election. They were the only ones who came anywhere close to property estimating the chances of the outcome that actually occurred, and they were one of only a couple models who took sufficiently seriously how unusual things were - particularly, the massive unpopularity of both candidates. I'm getting really irritated with people (in this thread and other) attacking 538 along partisan or personal lines for not making the same predictions they did personally. 538 made a model, not a poll; they are only ever as good as the polls they represent, and even then, they still managed to be better than quite a lot of public polling and commentary. If Hillary Clinton had won, the same kinds of people on the other side would be attacking 538 for being too bullish on Trump. The worst part is that a lot of this is predicated on the idea that giving something a 1 in 3 chance is the same as saying it'll never happen, when it's almost the opposite. There are some critical misunderstandings of statistics going on here, on all sides.
- chimeracoder 10y ago
  > The worst part is that a lot of this is predicated on the idea that giving something a 1 in 3 chance is the same as saying it'll never happen, when it's almost the opposite. There are some critical misunderstandings of statistics going on here, on all sides. On the morning of the election, FiveThirtyEight said that the chances of Trump winning were roughly equal to the chances of a random coin flip coming up heads twice in a row (25%). In that situation, a good statistician would certainly say that the expected outcome is for at least one of the coin flips to land on "tails" (75% chance). But if the coin lands on heads twice, you wouldn't say that "statistics is broken"[0] or that the statistician predicting the outcome was wrong. Similarly, let's say that you had a separate person predicting the probability of the double-heads coin flip to be 90%. If it does happen, that doesn't mean they made a good prediction, just because the outcome was correct. They predicted the correct outcome, but the degree of confidence they expressed in it was completely wrong. Predictions aren't just about getting the outcome right - they're also about expressing the right degree of confidence in the results. [0] Which is literally what some otherwise-respectable journalists complained after the election
  - 10y ago
downrightmike 10y ago
My eyes, the small text.
Tempest1981 10y ago
I remember HuffingtonPost showing a 98% chance of Hillary winning. Seemed way too optimistic. I wonder how many voters that influenced to "safely" vote for a non-mainstream candidate... Here is their methodology -- apparently quite flawed? http://www.huffingtonpost.com/entry/high-probability-clinton-winning_us_581d0399e4b0e80b02ca2498 http://www.huffingtonpost.com/entry/high-probability-clinton...
nl 10y ago
I find it interesting that there isn't a single comment here about the package itself. There's some interesting datasets in here[1]! Everything from "How American's like their Steak"[2] to "The Most Common Unisex Names In America: Is Yours One Of Them?"[3] As a (mostly) Python person I'd like to see these in Python! [1] https://mran.microsoft.com/web/packages/fivethirtyeight/fivethirtyeight.pdf https://mran.microsoft.com/web/packages/fivethirtyeight/five... [2] https://fivethirtyeight.com/datalab/how-americans-like-their-steak/ https://fivethirtyeight.com/datalab/how-americans-like-their... [3] https://fivethirtyeight.com/features/there-are-922-unisex-names-in-america-is-yours-one-of-them/ https://fivethirtyeight.com/features/there-are-922-unisex-na...

It's worth nothing that per the linked talks, 538's data visualizations are indeed created with R and ggplot2; the fancy annotations are done in Illustrator after exporting from ggplot2 in SVG/PDF. The vignettes (https://mran.microsoft.com/web/packages/fivethirtyeight/vignettes/bechdel.html https://mran.microsoft.com/web/packages/fivethirtyeight/vign...) are a good tutorial in dplyr/ggplot2 too.

KurtMueller 10y ago
Side question: Can I create interactive visualizations with R? Something akin to D3 in javascript?
- michaelsbradley 10y ago
  Check out ggvis and Shiny: http://ggvis.rstudio.com/ http://ggvis.rstudio.com/ http://shiny.rstudio.com/ http://shiny.rstudio.com/ http://rmarkdown.rstudio.com/flexdashboard/ http://rmarkdown.rstudio.com/flexdashboard/
  - apathy 10y ago
    Good lord, even after all this time, ggvis' demo is still rather a disaster on mobile. Oh well, it simply isn't possible for everything to be top priority. Rstudio has done some amazing work, I'm afraid ggvis is not yet up to the standards of their other projects.
    - fourthark 10y ago
      They've been busy!
- arifyali 10y ago
  Well D3 is always gonna be the best (downside, you're messing around in JS). However the rBokeh and plotly packages can help make interactive graphs.
- delphithrowaway 10y ago
  This article discusses many of the options to output interactive visualizations from R. https://www.r-bloggers.com/interactive-visualizations-with-r-a-minireview/ https://www.r-bloggers.com/interactive-visualizations-with-r... None of them are as flexible as D3, but there are packages to output from R to D3 out there.
- minimaxir 10y ago
  I strongly recommend using Plotly, specifically ggplotly (https://plot.ly/ggplot2/ https://plot.ly/ggplot2/), which converts ggplot2 charts to interactive d3 charts with good parity. Plotly also has WebGL and 3D data visualization support. One of the Plotly developers has a very good book on using Plotly in R: https://cpsievert.github.io/plotly_book/ https://cpsievert.github.io/plotly_book/ Here's a higher-level demo of ggplotly in one of my R Notebooks: http://minimaxir.com/notebooks/breach-network/ http://minimaxir.com/notebooks/breach-network/ Shiny requires an external server and is often overkill for static data.
  - RockyMcNuts 10y ago
    You can build Shiny apps in RStudio and publish them with one click to shinyapps.io. Works very similarly to publishing with Plotly, even though Shiny is a server-side package vs. client-side for Plotly. (You can build a Plotly chart in ggplot, embed it in a web page and then script it with JS to get a visualization. Frankly, that's a PITA. You can write R/shiny code that will generate a visualization, with HTML controls etc, and you end up writing a web app in R and that's a different PITA. I would like to see Plotly generate a web wrapper that automates wiring up a plotly graph and scripting it from an app so you can update data on the fly to make it a visualization. Supposedly they are working on it with their dash framework, but it's not released yet.)
    - minimaxir 10y ago
      The pricing on shinyapps.io is prohibitive if the post gets any more than a trickle of page views. The embed-directly-into-HTML is the approach I use for my Jekyll blog and it does not require much effort. (I just have to set a YAML flag to load Plotly library)
      - RockyMcNuts 10y ago
        Yes, for embedding a chart and getting some basic active functionality Plotly is very easy. The part that requires some custom Plotly coding is when you want to script the chart ... have HTML buttons, sliders, to filter, recompute, revisualize the data. It's beautiful that it even works, and you can programmatically update the JSON of an embedded chart. But it would be nice if you could give Plotly a dataset, plot the data, and say, give me a button to filter rows using these criteria for this column (or run some other code on the data) and re-render the chart. Right now you have to code that manually and update the JSON model. Visualization used to mean charts that were animated and/or would dynamically update but now it means any old chart LOL.
- phillc73 10y ago
  htmlwidgets[0] and flexdashboard[1]. Essentially it's writing rmarkdown documents. Shiny integration is optional. [0] http://www.htmlwidgets.org/ http://www.htmlwidgets.org/ [1] http://rmarkdown.rstudio.com/flexdashboard/ http://rmarkdown.rstudio.com/flexdashboard/

lintiness 10y ago

here's nate silver's ingenious monthly odds generator: #odds [insert democrat here] wins sample(75:100, 12, replace=TRUE)

lintiness 10y ago
went negative pretty fast here! must be big demand for a package that helps you be massively wrong almost all the time as long as it reinforces your political religion.
- sctb 10y ago
  We've banned this account for continuing to post unsubstantively after we've asked you multiple times to stop.
  - lintiness 10y ago
    "unsubstantively" = demonstrable political bunkering by the children around here. the ban is heartbreaking.

gfosco 10y ago

Very interesting, but the source of the data is so suspect that it'd be hard to have confidence in anything gleaned from it. It would be like CNN putting out a fake news package.

09bjb 10y ago
Yep. Beautifully wrought confirmations of bad hypotheses. After this election I'm taking anything stats-related on 538 (most everything) with a huge grain of salt. Can't stand another "Here's why the numbers show Hillary will win in a landslide" etc.
- ivraatiems 10y ago
  You do realize that of all the models available, 538's was the closest to the actual outcome and gave Trump by far the largest chance? 538 was very clear from the outset that the model's goal was predicting likelihoods, not being an oracle. It said there was a 1 in 3 chance of the election coming out like it did. That's not inaccurate - it was unlikely, but it still happened. Plus, 538 didn't make the data. It aggregated data others made. What exactly was bad in that hypothesis?
  - jackhack 10y ago
    Compsci 101. Garbage in, garbage out.
    - ivraatiems 10y ago
      The polls weren't garbage. They were closer this year than they were in 2012. They were just slightly wrong in the wrong direction.
      - jackhack 10y ago
        It's not a pejorative. It's a common phrase meaning "the quality of your output is only as good as your input data". In this case, input data was skewed/biased/inaccurate for a number of reasons (sample choices, underreporting, etc.). Thus even the best predictions were wrong.

ivraatiems 10y ago

In my opinion, 538 did some of the best analytics work in the 2016 election. They were the only ones who came anywhere close to property estimating the chances of the outcome that actually occurred, and they were one of only a couple models who took sufficiently seriously how unusual things were - particularly, the massive unpopularity of both candidates. I'm getting really irritated with people (in this thread and other) attacking 538 along partisan or personal lines for not making the same predictions they did personally. 538 made a model, not a poll; they are only ever as good as the polls they represent, and even then, they still managed to be better than quite a lot of public polling and commentary. If Hillary Clinton had won, the same kinds of people on the other side would be attacking 538 for being too bullish on Trump. The worst part is that a lot of this is predicated on the idea that giving something a 1 in 3 chance is the same as saying it'll never happen, when it's almost the opposite. There are some critical misunderstandings of statistics going on here, on all sides.

chimeracoder 10y ago
> The worst part is that a lot of this is predicated on the idea that giving something a 1 in 3 chance is the same as saying it'll never happen, when it's almost the opposite. There are some critical misunderstandings of statistics going on here, on all sides. On the morning of the election, FiveThirtyEight said that the chances of Trump winning were roughly equal to the chances of a random coin flip coming up heads twice in a row (25%). In that situation, a good statistician would certainly say that the expected outcome is for at least one of the coin flips to land on "tails" (75% chance). But if the coin lands on heads twice, you wouldn't say that "statistics is broken"[0] or that the statistician predicting the outcome was wrong. Similarly, let's say that you had a separate person predicting the probability of the double-heads coin flip to be 90%. If it does happen, that doesn't mean they made a good prediction, just because the outcome was correct. They predicted the correct outcome, but the degree of confidence they expressed in it was completely wrong. Predictions aren't just about getting the outcome right - they're also about expressing the right degree of confidence in the results. [0] Which is literally what some otherwise-respectable journalists complained after the election
- 10y ago

downrightmike 10y ago

My eyes, the small text.

Tempest1981 10y ago

I remember HuffingtonPost showing a 98% chance of Hillary winning. Seemed way too optimistic. I wonder how many voters that influenced to "safely" vote for a non-mainstream candidate... Here is their methodology -- apparently quite flawed? http://www.huffingtonpost.com/entry/high-probability-clinton-winning_us_581d0399e4b0e80b02ca2498 http://www.huffingtonpost.com/entry/high-probability-clinton...

nl 10y ago

I find it interesting that there isn't a single comment here about the package itself. There's some interesting datasets in here[1]! Everything from "How American's like their Steak"[2] to "The Most Common Unisex Names In America: Is Yours One Of Them?"[3] As a (mostly) Python person I'd like to see these in Python! [1] https://mran.microsoft.com/web/packages/fivethirtyeight/fivethirtyeight.pdf https://mran.microsoft.com/web/packages/fivethirtyeight/five... [2] https://fivethirtyeight.com/datalab/how-americans-like-their-steak/ https://fivethirtyeight.com/datalab/how-americans-like-their... [3] https://fivethirtyeight.com/features/there-are-922-unisex-names-in-america-is-yours-one-of-them/ https://fivethirtyeight.com/features/there-are-922-unisex-na...