4 ms·

This is just a quick example - I would be grateful if people could recreate this brief look at UK COVID figures in another language: library(tidyverse) lib

by dm319 5y ago

This is just a quick example - I would be grateful if people could recreate this brief look at UK COVID figures in another language: library(tidyverse) library(scales) download.file(url = "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv", destfile = "./data.csv", method = "wget") read_csv("./data.csv") %>% pivot_longer(names_to = "Data", cols = c(newCasesBySpecimenDate, covidOccupiedMVBeds, newAdmissions, newDeaths28DaysByDeathDate)) %>% mutate(Data = factor(Data)) %>% mutate(Data = recode_factor(Data, newCasesBySpecimenDate = "New Cases", newAdmissions = "Admissions", newDeaths28DaysByDeathDate = "Deaths", covidOccupiedMVBeds = "Ventilated")) %>% ggplot(aes(y = value, x = date, colour = Data))+ geom_point(size = 1, colour = "gray", alpha = 0.6)+ geom_smooth(type = "LOESS", span = 0.1)+ labs(y = "Daily rate", x = "Date", colour = "UK COVID-19")+ scale_x_date(date_breaks = "months", date_labels = "%b-%y")+ scale_y_log10(labels = comma(10 ^ (0:5), accuracy = 1), breaks = 10 ^ (0:5))+ theme(axis.text.x = element_text(angle = 45, hjust = 1))

azalemeth 5y ago
You don't need the download file command -- read CSV works with URLs directly :-)
- dm319 5y ago
  Oh nice, that's very useful to know!
nuclearnice1 5y ago
I think these kind of common task challenges are great for comparison. You see a few approaches to a single task and you can do a more aligned and detailed comparison. Unfortunately, it’s also a lot of work. In this case, you’ve posted an intermediate stage artifact from R. If one of the many Python programmers reading this want to produce a comparable artifact they need to understand or run that code. That alone reduces your likelihood of getting any substantial replies. Maybe add a link to an image of the resulting plot?
- dm319 5y ago
  Good point, looks like I'm too late to edit, but here's a link [1] (excuse the R style indexing). Yes, I love these things and very curious to see what hackernews comes up with. Project Euler was an eye opener for how things could be optimised in different languages. [1] https://i.imgur.com/M8DX98I.png https://i.imgur.com/M8DX98I.png
_dain_ 5y ago
import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates import seaborn as sn df = (pd.read_csv("/tmp/overview_2022-03-21.csv") # i just used curl beforehand .assign(date=lambda x: pd.to_datetime(x["date"])) .set_index("date") .melt(value_vars=[ "newCasesBySpecimenDate", "covidOccupiedMVBeds", "newAdmissions", "newDeaths28DaysByDeathDate"], var_name="Data", ignore_index=False) .assign(Data=lambda x: x["Data"].replace({ "newCasesBySpecimenDate": "New Cases", "newAdmissions": "Admissions", "newDeaths28DaysByDeathDate": "Deaths", "covidOccupiedMVBeds": "Ventilated" })) ) ax = sn.scatterplot(data=df, x=df.index, y=df["value"], hue="Data") ax.set(xlabel="Date", ylabel="Daily rate", yscale="log") ax.xaxis.set_major_formatter(mdates.DateFormatter("%b")) plt.show() I spend 2 minutes on the pandas part and 20 minutes on the plotting part, which really says it all. Seaborn's support for smoothing is really bad and doesn't play nicely with datetimes for some reason, so if I wanted smoothing I'd need to do it myself. And the other stuff I left out requires going into matplotlib's documentation which I don't want to spend time on. pandas is as good or better than R's dataframe manipulation, but R's plotting tools are best in class. I hate all the python plotting libraries.
- nuclearnice1 5y ago
  This is great. You don't need the curl -- read_csv works with URLs directly. The lambda can be replaced by passing parse_dates=[“date”]
  - _dain_ 5y ago
    Thanks
- LosWochosWeek 5y ago
  > I hate all the python plotting libraries. I've given up discussing this. My personal opinion is that MPL is the assembly vs ggplots2's Python. Yes, maybe there are some things that are doable in MPL but not in ggplot2, but never (and I mean - never -) has any one of my colleagues found a single example that I couldnt recreate in ggplot2 with a more readable code that resulted in a better looking plot. MPL has this Latex like "once you get the hang of it, you will never use anything else in life for anything" and OOP's "If its good code then it is OOP and if its not, then you're not doing correct OOP" mythisque hanging over it. When confronted with bad MPL code that results in bad plots, its always one of those two. ggplot2 is the best plotting package out there and imo one of the best "end-user" packages in any language. Also, Hadley is a saint.
Hasnep 5y ago
A Julia solution with Chain and Gadfly might look something like this, although I've translated the R fairly directly so it might not be very idiomatic. import CSV using Chain: @chain using DataFrames import Downloads using Gadfly using Dates @chain begin Downloads.download( "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv", ) CSV.File DataFrame stack( [:newCasesBySpecimenDate, :covidOccupiedMVBeds, :newAdmissions, :newDeaths28DaysByDeathDate]; variable_name = :Data, ) transform( :Data => ( x -> replace( x, "newCasesBySpecimenDate" => "NewCases", "newAdmissions" => "Admissions", "newDeaths28DaysByDeathDate" => "Deaths", "covidOccupiedMVBeds" => "Ventilated", ) ) => :Data, ) subset(:value => ByRow(!ismissing)) # Can't plot Geom.smooth with missings plot( _, x = :date, y = :value, colour = :Data, layer(Geom.smooth(method = :loess, smoothing = 0.1)), layer(Geom.point), Scale.y_log10(), Guide.xlabel("Date"), Guide.ylabel("Daily rate"), Guide.xlabel("Angle"), Guide.colorkey(title = "UK COVID-19"), ) end
- dm319 5y ago
  Thank you for that, good to see there's an elegant Julia solution! The last time I was using 'pipes' with Julia, I think I was using DataFramesMeta. I also really like this interactive gadfly plot - reminds me of Matlab, but better. It's been a little while since using Julia, so I'd forgotten about the pre-compiling thing, but generally this code looks pretty nice and clear.
  - Hasnep 5y ago
    I used to use DataFramesMeta.jl, but eventually I found that the mini-DSL that DataFrames.jl has created is really powerful and not overly verbose. Now, going back to the Tidyverse's syntax makes me feel a little uneasy, like there's just too much magic going on behind the scenes, even though I used it for years with no problems.
hadley 5y ago
This was fun to play around with. I made some very minor changes and posted at https://gist.github.com/hadley/d54895557fbb0fe0402d2277b901120a https://gist.github.com/hadley/d54895557fbb0fe0402d2277b9011.... It revealed to me that there's a buglet in `forcats::last()` (https://github.com/tidyverse/forcats/issues/303 https://github.com/tidyverse/forcats/issues/303) and made me wonder if `pivot_longer()` should be able to rename the columns as you pivot them (https://github.com/tidyverse/tidyr/issues/1338 https://github.com/tidyverse/tidyr/issues/1338)
- dm319 5y ago
  This might be the most excited I've gotten about a comment in hackernews for a while! So interesting to see your style of code (I was actually on 4.0 so gave me the impetus to upgrade to get the new pipes). fct_reorder - what a hugely useful function I didn't know about. The chicks vignette was nicely illustrative. And label_date_short also super useful. Also curious generally about your bracket style (I just tend to pile them all up when closing, which is something I only do in R and probably shouldn't!). Renaming factors is one of those things that always seems a bit awkward. I think I've used several methods. Passing a list of named vectors into `levels(x)` allows a many-to-one mapping but was quite dangerous. I've used revalue and mapvalues from plyr. fct_recode is new to me. But yes, renaming while reshaping could be quite convenient. Just looking at fct_recode now, it looks really nice. Seems to support many-to-one and being able to pass it a name vector is very convenient. Learnt so much today, and this is even before trying out the python and julia examples! Many thanks for this and all your work in R!
- pbowyer 5y ago
  If you enjoyed this you might like https://github.com/VictimOfMaths/COVID-19 https://github.com/VictimOfMaths/COVID-19 :)

by dm319 5y ago

azalemeth 5y ago

You don't need the download file command -- read CSV works with URLs directly :-)

dm319 5y ago
Oh nice, that's very useful to know!

nuclearnice1 5y ago

I think these kind of common task challenges are great for comparison. You see a few approaches to a single task and you can do a more aligned and detailed comparison. Unfortunately, it’s also a lot of work. In this case, you’ve posted an intermediate stage artifact from R. If one of the many Python programmers reading this want to produce a comparable artifact they need to understand or run that code. That alone reduces your likelihood of getting any substantial replies. Maybe add a link to an image of the resulting plot?

dm319 5y ago
Good point, looks like I'm too late to edit, but here's a link [1] (excuse the R style indexing). Yes, I love these things and very curious to see what hackernews comes up with. Project Euler was an eye opener for how things could be optimised in different languages. [1] https://i.imgur.com/M8DX98I.png https://i.imgur.com/M8DX98I.png

_dain_ 5y ago

import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates import seaborn as sn df = (pd.read_csv("/tmp/overview_2022-03-21.csv") # i just used curl beforehand .assign(date=lambda x: pd.to_datetime(x["date"])) .set_index("date") .melt(value_vars=[ "newCasesBySpecimenDate", "covidOccupiedMVBeds", "newAdmissions", "newDeaths28DaysByDeathDate"], var_name="Data", ignore_index=False) .assign(Data=lambda x: x["Data"].replace({ "newCasesBySpecimenDate": "New Cases", "newAdmissions": "Admissions", "newDeaths28DaysByDeathDate": "Deaths", "covidOccupiedMVBeds": "Ventilated" })) ) ax = sn.scatterplot(data=df, x=df.index, y=df["value"], hue="Data") ax.set(xlabel="Date", ylabel="Daily rate", yscale="log") ax.xaxis.set_major_formatter(mdates.DateFormatter("%b")) plt.show() I spend 2 minutes on the pandas part and 20 minutes on the plotting part, which really says it all. Seaborn's support for smoothing is really bad and doesn't play nicely with datetimes for some reason, so if I wanted smoothing I'd need to do it myself. And the other stuff I left out requires going into matplotlib's documentation which I don't want to spend time on. pandas is as good or better than R's dataframe manipulation, but R's plotting tools are best in class. I hate all the python plotting libraries.

nuclearnice1 5y ago
This is great. You don't need the curl -- read_csv works with URLs directly. The lambda can be replaced by passing parse_dates=[“date”]
- _dain_ 5y ago
  Thanks
LosWochosWeek 5y ago
> I hate all the python plotting libraries. I've given up discussing this. My personal opinion is that MPL is the assembly vs ggplots2's Python. Yes, maybe there are some things that are doable in MPL but not in ggplot2, but never (and I mean - never -) has any one of my colleagues found a single example that I couldnt recreate in ggplot2 with a more readable code that resulted in a better looking plot. MPL has this Latex like "once you get the hang of it, you will never use anything else in life for anything" and OOP's "If its good code then it is OOP and if its not, then you're not doing correct OOP" mythisque hanging over it. When confronted with bad MPL code that results in bad plots, its always one of those two. ggplot2 is the best plotting package out there and imo one of the best "end-user" packages in any language. Also, Hadley is a saint.

Hasnep 5y ago

A Julia solution with Chain and Gadfly might look something like this, although I've translated the R fairly directly so it might not be very idiomatic. import CSV using Chain: @chain using DataFrames import Downloads using Gadfly using Dates @chain begin Downloads.download( "https://api.coronavirus.data.gov.uk/v2/data?areaType=overview&metric=covidOccupiedMVBeds&metric=newAdmissions&metric=newCasesBySpecimenDate&metric=newDeaths28DaysByDeathDate&metric=newPeopleReceivingFirstDose&format=csv", ) CSV.File DataFrame stack( [:newCasesBySpecimenDate, :covidOccupiedMVBeds, :newAdmissions, :newDeaths28DaysByDeathDate]; variable_name = :Data, ) transform( :Data => ( x -> replace( x, "newCasesBySpecimenDate" => "NewCases", "newAdmissions" => "Admissions", "newDeaths28DaysByDeathDate" => "Deaths", "covidOccupiedMVBeds" => "Ventilated", ) ) => :Data, ) subset(:value => ByRow(!ismissing)) # Can't plot Geom.smooth with missings plot( _, x = :date, y = :value, colour = :Data, layer(Geom.smooth(method = :loess, smoothing = 0.1)), layer(Geom.point), Scale.y_log10(), Guide.xlabel("Date"), Guide.ylabel("Daily rate"), Guide.xlabel("Angle"), Guide.colorkey(title = "UK COVID-19"), ) end

dm319 5y ago
Thank you for that, good to see there's an elegant Julia solution! The last time I was using 'pipes' with Julia, I think I was using DataFramesMeta. I also really like this interactive gadfly plot - reminds me of Matlab, but better. It's been a little while since using Julia, so I'd forgotten about the pre-compiling thing, but generally this code looks pretty nice and clear.
- Hasnep 5y ago
  I used to use DataFramesMeta.jl, but eventually I found that the mini-DSL that DataFrames.jl has created is really powerful and not overly verbose. Now, going back to the Tidyverse's syntax makes me feel a little uneasy, like there's just too much magic going on behind the scenes, even though I used it for years with no problems.

hadley 5y ago

This was fun to play around with. I made some very minor changes and posted at https://gist.github.com/hadley/d54895557fbb0fe0402d2277b901120a https://gist.github.com/hadley/d54895557fbb0fe0402d2277b9011.... It revealed to me that there's a buglet in `forcats::last()` (https://github.com/tidyverse/forcats/issues/303 https://github.com/tidyverse/forcats/issues/303) and made me wonder if `pivot_longer()` should be able to rename the columns as you pivot them (https://github.com/tidyverse/tidyr/issues/1338 https://github.com/tidyverse/tidyr/issues/1338)

dm319 5y ago
This might be the most excited I've gotten about a comment in hackernews for a while! So interesting to see your style of code (I was actually on 4.0 so gave me the impetus to upgrade to get the new pipes). fct_reorder - what a hugely useful function I didn't know about. The chicks vignette was nicely illustrative. And label_date_short also super useful. Also curious generally about your bracket style (I just tend to pile them all up when closing, which is something I only do in R and probably shouldn't!). Renaming factors is one of those things that always seems a bit awkward. I think I've used several methods. Passing a list of named vectors into `levels(x)` allows a many-to-one mapping but was quite dangerous. I've used revalue and mapvalues from plyr. fct_recode is new to me. But yes, renaming while reshaping could be quite convenient. Just looking at fct_recode now, it looks really nice. Seems to support many-to-one and being able to pass it a name vector is very convenient. Learnt so much today, and this is even before trying out the python and julia examples! Many thanks for this and all your work in R!
pbowyer 5y ago
If you enjoyed this you might like https://github.com/VictimOfMaths/COVID-19 https://github.com/VictimOfMaths/COVID-19 :)