12 ms·
The New York Times Releases Its Dataset of U.S. Confirmed Coronavirus Cases
- RcouF1uZ4gsC 7y agoAnd places it behind a login page, requiring people to sign up to see it.
- gmaster1440 7y agohttps://github.com/nytimes/covid-19-data https://github.com/nytimes/covid-19-data
- deleted 7y ago[deleted]
- sproketboy 7y agoWe don't have the Chinese ones since they kicked out the journalists.
- apearson 7y agoData: https://github.com/nytimes/covid-19-data https://github.com/nytimes/covid-19-data
- ABeeSea 7y agoOh boy, I can hear the the bootcamp data scientists firing up their medium blogs from here.
- airstrike 7y agoSadly all of this data assumes there are approximately as many beds as needed, and won't be very helpful `x` weeks from now because once we hit capacity, the situation is bound to look very different. I don't see enough people making this point: the young will be very vulnerable if left untreated, and you can't treat e.g. the entire state of NY with 3,800 ICU beds plus however many the wonderful people in the Army Corps of Engineer can put together in a pinch. It's not like young people are going to recover at home from their nasty case of pneumonia. With no ventilators and no medical staff, the mortality rate will markedly increase.
- tejohnso 7y ago> I don't see enough people making this point: the young will be very vulnerable if left untreated Why are you pointing out "the young" here?
- Footkerchief 7y agoBecause the prevailing narrative is that this disease is mainly dangerous to the elderly.
- paulmd 7y agoIt is primarily dangerous to the elderly. That is what the actual data shows. Some people are pushing a counter-narrative that youngs aren't invincible. Which is also true. 0.1% death rates doesn't mean zero deaths. A higher number will be hospitalized, and as hospitals become overwhelmed that means (a) quality of care will be degraded and death rates will increase somewhat, and (b) any youngs in a bed comes at the direct expense of an elderly in a bed. But if you are claiming that this disease is not mainly dangerous to the elderly, no, that is false, the deaths and hospitalizations are hugely weighted to the elderly and are quite low for young cohorts. Yes, it is important that everyone minimize contacts as much as possible. For some reason people love to really harp on this point with the youngs. I didn't go to florida for spring break, it was my boomer parents who did (or rather, were planning on going this summer and complained nonstop that Florida state parks closed and cut them off). Churches are still open in many states spreading disease. Boomers are still going to Home Depot and buying their home improvement goods. A lot of boomers aren't taking this seriously either but nobody chooses to bring that up. There are lots of changes that need to be made by everyone, but younger people are one of the primary targets of casual/unthinking ageism in this country. Avocado toast and all that. Focusing the blame on this onto the young is no more correct, helpful, or productive than focusing the blame on the Chinese, and I suspect is coming from most of the same media sources.
- Mediterraneo10 7y ago> Boomers are still going to Home Depot and buying their home improvement goods. It is worth pointing out that home-improvement stores were specifically exempt from the order to close businesses in several European countries. This is probably partly because those stores sell gardening products, and villagers need to be able to plant their vegetables now. However, the expectation on the part of public health authorities is presumably that that these stores remaining open, like supermarkets, will still not prevent the infection curve from flattening over the lockdown period.
- wolco 7y agoLet's try to not put people down for self learning. Knowledge can be found anywhere. Those that mock people like this will soon find themselves left behind with ancient university based education. Unless you plan to re-enroll every 10 years the world will pass you by if you are not self learning.
- throwawayjava 7y ago"Oh boy, I can hear the sophomore software engineering studio students firing up their word processors from here" works just as well. I don't think the critique was of self-learning or bootcamps. It was a critique of a certain sort of cringe-worthy self-promotion. And not just cringe-worthy. Spreading information based upon a cursory analysis of some CSV files without any training or experience in epidemiology, public health, public communications, etc. seems... irresponsible. Also, off-topic, but most of my peers at university taught themselves how to program years before starting college -- typically in middle/early high school. And I mean actually taught themselves, from books and zine tutorials, not 'attended a formal course of instruction that was offered by a venture-backed firm instead of a formal course of instruction at a traditional university'. I guess times have changed, but the characterization of university students in CS as 'not self-taught programmers' is definitely the opposite of my experience. You came into the CS degree knowing how to program and learned how to do CS. And the characterization of "anything not university" as "self-taught" -- even formal courses of instruction that cost five figures -- is even more strange.
- SpicyLemonZest 7y agoThat just doesn't seem like a fair standard. Why should only experts in a relevant field talk about the coronavirus? Nobody tries to stop people from sharing hospital stories from their friend's cousin's boyfriend who's a nurse.
- throwawayjava 7y agoOh, I think it's completely fine to play with the data and talk about it with friends. But SEOd medium posts have different reach potential than a facebook comment/phone call with friends/family. That's kind of the whole point of them.
- johnvega 7y agohttps://github.com/nytimes/covid-19-data https://github.com/nytimes/covid-19-data
- melling 7y ago“Math is boring. When am I ever going to use it?” The data can be used by a lot of people for a lot of learning. My project: I’d like to watch the growth state by state on a daily basis.
- acqq 7y agoIf you want an example of how to do that good, I find this the best: https://www.ft.com/coronavirus-latest https://www.ft.com/coronavirus-latest as a simpler start, see the logarithmic chart here: https://www.worldometers.info/coronavirus/country/us/ https://www.worldometers.info/coronavirus/country/us/ Note how easy is to estimate the time frame in which the next factor of 10 could be reached with the growth that continues. Compare with the logarithmic chart of China: https://www.worldometers.info/coronavirus/country/china/ https://www.worldometers.info/coronavirus/country/china/ Search for the date since when they had their measures applied in Wuhan. Then follow how long it took for them to be visible in the chart. Then mark on the US chart when the measures started. Compare. Also mark the day of the following quote: "And we’re prepared, and we’re doing a great job with it. And it will go away. Just stay calm. It will go away." One can learn really a lot just from the data.
- starpilot 7y agoWhen I see datasets like this, I just think of how 95% of my work as a DS has been just getting to that cleaned table at the end. After that the modeling is straightforward.
- ksk 7y agoI think its amazing that people are passionate, but the humility about not understanding something in its entirety is shrinking day by day. Today everyone wants to be the "explainer" or the "let me tell you how its done" person without actually having any hands-on experience or deep knowledge in the domain. The oldskool litmus test was "What have you done in this field that makes your opinion worth something?" and it is clearly no longer applicable in today's world.
- beefield 7y agoIs there anywhere (global) data about total number of tests made? The plain number of positive cases is quite difficult to make sense of.
- microdrum 7y agoAccording to [checks notes] the internet, you can't make this observation. If you do, you want people to die!!
- ddeck 7y agoFor the US, the COVID Tracking Project have tested, positive, negative numbers for most states. They also have the dataset available as a spreadsheet or a JSON or CSV API: https://covidtracking.com/data/ https://covidtracking.com/data/ The project was started by some journalists from The Atlantic. Seems to be a lot of duplicated effort in this area, although perhaps they are sharing data. https://covidtracking.com/about-team/ https://covidtracking.com/about-team/
- oli5679 7y agohttps://ourworldindata.org/coronavirus-testing-source-data https://ourworldindata.org/coronavirus-testing-source-data The key metric is tests/confirmed case. When I've looked, this is very predictive of confirmed cases (2 weeks ago) per death.
- beefield 7y agoThat was the best I had also found, but it is not of decent quality to me. sources seem to be quite out of date and often based on news articles, not actual daily data.
- mbostleman 7y agoI've been downloading this daily and watching some things. Mostly deaths in the US: https://www.ecdc.europa.eu/en/publications-data/download-todays-data-geographic-distribution-covid-19-cases-worldwide https://www.ecdc.europa.eu/en/publications-data/download-tod...
- chasebank 7y agohttps://coronavirus.jhu.edu/map.html https://coronavirus.jhu.edu/map.html
- 5cott0 7y agohttps://github.com/CSSEGISandData/COVID-19 https://github.com/CSSEGISandData/COVID-19 The NYT spent more effort patting themselves on the back than they did compiling the data.
- 16bytes 7y agoTheir data is of higher quality than upstream sources like the Johns Hopkins CSSE set. Notably, it's in long-form (one record per day per area) instead of wide-form and the schema is consistent. This takes some know-how and a bunch of ETL effort. There is significant value in their transformation of upstream data.
- tomrod 7y agousafacts.org has daily county data from 1/23 forward.
- reaperducer 7y agoThat map looks good from a global level, but when you zoom in, it kind of falls down. For example, two towns in my state have confirmed cases, are ported by the state health authorities and the local newspapers, but neither is on that map.
- ravenstine 7y ago> In light of the current public health emergency, The New York Times Company is providing this database under the following free-of-cost, perpetual, non-exclusive license. Anyone may copy, distribute, and display the database, or any part thereof, and make derivative works based on it, provided (a) any such use is for non-commercial purposes only and (b) credit is given to The New York Times in any public display of the database, in any publication derived in part or in full from the database, and in any other public use of the data contained in or derived from the database. How is this form of license valid for what's essentially just numbers from publicly available data? Do I get to take numbers from various sources and then require credit and that nobody can use them for commercial purposes?
- lukevp 7y agoSoftware is just numbers too.
- sjg007 7y agoYes.
- TallGuyShort 7y agoYour personal data is also just numbers. But it does seem weird. These numbers are already only authoritatively available from the government health services, right? The New York Times doesn't own the data either, if it's public it's public, they're providing a convenience. It's actually pretty analagous to news: they don't own the fact that an event happened, but they own their wording and they provide a service in getting that wording to you. edit: Yeah, "Over months, our journalists have recorded the details on newly confirmed cases as reported by state and local officials." It's all government, just aggregated by NYT instead of the federal government.
- reaperducer 7y agoI think it's just the Times firing a shot across the bows of other newspapers who will re-publish the data as their own without attribution. Newspapers sometimes do jerky things like that.
- santiagobasulto 7y agoAnybody know about data indicating age? It's always "confirmed", "dead", "recovered" but no info about age of patients.
- reaperducer 7y agoMy state publishes its statistics hourly. I looked at it just this morning, and was surprised to see that in the age breakdowns that there are more confirmed cases in people 20-40 than in 60-80.
- catalogia 7y agoIs that still true when you adjust for the relative size of those groups? 20-40 is a larger group of people than 60-80.
- reaperducer 7y agoIt's hard to say, because I don't have the time to do the math right now. But my state is generally considered to be retirement-friendly.
- MiroF 7y agoFlorida?
- Diederich 7y agoDo you have a link to that data? Thanks.
- bagacrap 7y agoI am not surprised. People in the 20-40 range are more likely to be out in public exposing themselves to the virus. The elderly are by default socially distanced in this country. There has been no evidence I've seen that age affects likelihood of infection, but it does affect severity. In countries like Italy where very few young people have tested positive, it's due to test triage. You aren't going to spend tests on asymptomatic or mildly symptomatic 25yos.
- yourapostasy 7y agoSo before I duplicate any effort, has anyone found a tracker that takes this dataset's death counts and updates a Pueyo/Khan Academy-style graph [1], at different granular levels (world, nation, province/state)? Pueyo's analysis technique uses deaths each day to impute a back-looking inferred actual infected count instead of relying upon reported testing numbers, and that in turn can be used to infer a two-week forward-looking infected and death count based upon currently-known stats on average time to hospitalization and mortality. After two weeks, it gets increasingly more speculative, but by the time you have number of known deaths today, you almost have a baked-in outcome two weeks from now. [1] https://www.youtube.com/watch?v=mCa0JXEwDEk https://www.youtube.com/watch?v=mCa0JXEwDEk
- JoeAltmaier 7y agoNot this data, but I like Johns Hopkins rendering of their data: https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6 https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.h...
- wolco 7y agoThis site has some of what you want. https://wuflu.live https://wuflu.live
- deleted 7y ago[deleted]
- bart_spoon 7y agoDoesn't that method assume we know the mortality rate, which we absolutely do not right now? It also would fail to account for any kind of containment measures.
- klaudius 7y agoHere's what I don't understand... If viruses and bacteria are all over your body and do some useful things, how can some be bad? What separates 'bad' virus from a good virus?
- JoeAltmaier 7y agoIs that really a question? If some people are good and do useful things, how can there be murderers? { edit: explanation } Viruses are tiny fragments of dna and proteins that are only there to throw a monkey wrench into your cells. Sometimes that monkey wrench does fairly harmless things. These viruses thrive, because they find lots of healthy hosts over a long period of time. But some break things badly, reproduce out of control or change your cells' functions to be less optimal. These don't do as well, since their hosts don't thrive. The viruses don't 'care' what they're doing; they're just bits of junk that mutate a lot. The ones that kill things quickly, tend to die out. The only reason the world isn't full of horrible murdering viruses (any more than it is) is, statistics. Not some grand plan or kindly act of nature.
- DoofusOfDeath 7y agoThe question seems pretty basic, but it's a good one. I realized I'm not 100% confident in the answer I was about to give. Hopefully we'll get an answer from an actual expert.
- catalogia 7y agoSome berries taste great but some berries poison you. Real puzzling isn't it.
- 7952 7y agoIt is interesting though. Why would a plant evolve to kill animals when the purpose of a berry is to spread seed through consumption?
- viklove 7y ago> purpose of a berry is to spread seed through consumption That's not the purpose of the berry. The purpose is just to ensure the survival of the seed. In drought-stricken environments, the seed is probably better off keeping its shell of water and fructose until it can form a root system.
- wrkronmiller 7y agoThis website appears to have more fine-grained statistics for the New York City area (ironically): https://accelerator.weather.com/bi/?perspective=dashboard&pathRef=.public_folders%2FCOVID19%2FDashboards%2FDS%2FCOVID-19%2B%2528Coronavirus%2529%2BGlobal%2BStatistics https://accelerator.weather.com/bi/?perspective=dashboard&pa...
- zaphod12 7y agoI don't understand why these lists make it so hard to find the USA. Every country is written out including the United Kingdom - but the USA is "US." Sometimes we're first because of ethnocentrism, sometimes last, but PLEASE BE CONSISTENT! Also, thanks for this link - rant aside, it's the best dataset I've seen.
- danso 7y agoGithub repo: https://github.com/nytimes/covid-19-data https://github.com/nytimes/covid-19-data
- pj_mukh 7y agoNoob Question: Anyone know an easy way to convert FIPS data (that this nytimes dataset uses) to Postal code?
- throwawayjava 7y agoproblem is somewhat ill-posed, but see https://raw.githubusercontent.com/bgruber/zip2fips/master/zip2fips.json https://raw.githubusercontent.com/bgruber/zip2fips/master/zi...
- jake-low 7y agoI don't think it's generally possible. The FIPS code just identifies the county (the first two digits are the state, e.g. 06 for California) and the final three identify a county in that state. So technically that column is redundant with the "county" and "state" columns; I suspect they've included it to make joining this dataset to other data that might use different name-formatting of counties/states easier. ZIP/postal codes are generally smaller (in the county I live in there are almost a hundred ZIP codes). I'm not sure they're even guaranteed to be entirely within one county either. We tend to think of ZIP codes as boundaries but they're actually delivery routes (which if you squint can be converted into boundaries by joining the properties that those routes serve together). You might be interested in this article: https://carto.com/blog/zip-codes-spatial-analysis/ https://carto.com/blog/zip-codes-spatial-analysis/
- chadlavi 7y agogod damn it NYT, "New York City" is not a county. Why can't I get borough by borough breakdowns? (each borough is its own county) edit: someone else posted this, which DOES give county-by-county breakdowns: https://accelerator.weather.com/bi/?perspective=dashboard&pathRef=.public_folders%2FCOVID19%2FDashboards%2FDS%2FCOVID-19%2B%2528Coronavirus%2529%2BGlobal%2BStatistics&id=iC2B38B09B142481EB83935F6419CA837&objRef=iC2B38B09B142481EB83935F6419CA837&options%5Bcollections%5D%5BcanvasExtension%5D%5Bid%5D=com.ibm.bi.dashboard.canvasExtension&options%5Bcollections%5D%5BfeatureExtension%5D%5Bid%5D=com.ibm.bi.dashboard.core-features&options%5Bcollections%5D%5Bbuttons%5D%5Bid%5D=com.ibm.bi.dashboard.buttons&options%5Bcollections%5D%5Bwidget%5D%5Bid%5D=com.ibm.bi.dashboard.widgets&options%5Bcollections%5D%5BcontentFeatureExtension%5D%5Bid%5D=com.ibm.bi.dashboard.content-features&options%5Bcollections%5D%5BboardModel%5D%5Bid%5D=com.ibm.bi.dashboard.boardModelExtension&options%5Bcollections%5D%5BsaveServices%5D%5Bid%5D=com.ibm.bi.dashboard.saveServices&options%5Bcollections%5D%5BserviceExtension%5D%5Bid%5D=com.ibm.bi.dashboard.serviceExtension&options%5Bcollections%5D%5BlayoutExtension%5D%5Bid%5D=com.ibm.bi.dashboard.layoutExtension&options%5Bcollections%5D%5BvisualizationExtension%5D%5Bid%5D=com.ibm.bi.dashboard.visualizationExtensionCA&options%5Bcollections%5D%5BcolorSetExtensions%5D%5Bid%5D=com.ibm.bi.dashboard.colorSetExtensions&options%5Bconfig%5D%5BsmartTitle%5D=false&options%5Bconfig%5D%5BeditPropertiesLabel%5D=true&options%5Bconfig%5D%5BnavigationGroupAction%5D=true&options%5Bconfig%5D%5BenableDataQuality%5D=false&options%5Bconfig%5D%5BmemberCalculation%5D=false&options%5Bconfig%5D%5BassetTags%5D%5B%5D=dashboard&options%5Bconfig%5D%5BfilterDock%5D=true&options%5Bconfig%5D%5BshowMembers%5D=true&options%5Bconfig%5D%5BassetType%5D=exploration&options%5Bconfig%5D%5BgeoService%5D=CA&isAuthoringMode=false&boardId=iC2B38B09B142481EB83935F6419CA837 https://accelerator.weather.com/bi/?perspective=dashboard&pa...
- deleted 7y ago[deleted]
- panpanna 7y agoGitHub link in case NYT is blocked at your... home-office? https://github.com/nytimes/covid-19-data https://github.com/nytimes/covid-19-data
- jashkenas 7y agoHere’s a quickie animated bubble map, pulling in this new NYT data from GitHub: https://observablehq.com/@jashkenas/united-states-coronavirus-daily-cases-map-covid-19 https://observablehq.com/@jashkenas/united-states-coronaviru... ... might be a useful starting point if you want to fork off your own visualizations and analyses.
- echelon 7y agoAt first I thought your animation wasn't working. Then it got to the last few days. Wow.
- jashkenas 7y agoYeah. Wow indeed. The dataset actually goes all the way back to the end of January, but I'm trimming it to start at the beginning of March, because the early days are pretty much invisible at the current scale. You can change the value of `startDate` in that notebook if you'd like to look back further in time.
- SAI_Peregrinus 7y agoYou could make the animation slow down exponentially, to make the growth look linear. Basically turn it into a log plot. Misleading if you don't know what you're looking at though.
- z3ugma 7y agoMy pet peeve with nearly all maps showing this data is that they are on geographical maps. A virus spread is only a weakly geographical phenomenon - most infections happen through human-to-human contact so it makes more sense to show on a population-weighted cartogram. Has anyone made a map like that? A great example is the 538 hexagonal-tile electoral college map: https://projects.fivethirtyeight.com/2016-election-forecast/ https://projects.fivethirtyeight.com/2016-election-forecast/ or one of these population-weighted projections of different areas: http://news.bbc.co.uk/2/hi/in_pictures/8284655.stm http://news.bbc.co.uk/2/hi/in_pictures/8284655.stm
- jake-low 7y agoThis is phenomenal. I've been scraping the data from primary sources for just Washington state for the past week [0], in order to make this chart which I hacked together last weekend [1]. [0]: https://github.com/jake-low/covid-19-wa-data https://github.com/jake-low/covid-19-wa-data [1]: https://observablehq.com/@jake-low/covid-19-in-washington-state https://observablehq.com/@jake-low/covid-19-in-washington-st... Doing this for just one state was a pretty substantial effort. I imagine there are multiple people at the Times who are spending several hours a day reviewing and cleaning scraped data (seems every couple of days some formatting change breaks your scripts, or a source publishes data that later needs to be retracted). The Times dataset appears to contain per-county case and death observations in a time series, going all the way back to the first confirmed U.S. case in January in Snohomish County, WA. This makes it by far the most comprehensive time series dataset of U.S. COVID-19 cases publicly available. Some people in this thread linked to the Johns Hopkins CSSE dataset; I've looked at this data but it doesn't go back very far in time for the U.S., and the tables are published as daily summaries with differing table schemas which makes them hard to use out of the box. For some days earlier in March, "sublocations" aren't even structured (for example the same column contains, "Boston, MA" and "Los Angeles County", making it very hard to use). No disrespect to the team behind the JHU dataset; it attempts to cover the whole world since the outbreak began which is an incredible and difficult goal. But for mapping and studying the outbreak in the U.S., the Times dataset will likely be the best choice right now. Huge kudos to the New York Times team for making this data freely available.
- bengebre 7y agoKudos to NYT, but I think the COVID Tracking Project data is probably better because it attempts to measures total testing as well (positives and negatives). I've been using it to report state-level testing statistics and new case/death curves: https://www.deptofnumbers.com/covid19/ https://www.deptofnumbers.com/covid19/ From the data I've learned that Washington state appears to be getting their arms around this thing: https://www.deptofnumbers.com/covid19/washington/ https://www.deptofnumbers.com/covid19/washington/
- peripitea 7y agoFor those interested in data, the COVID Tracking Project (http://covidtracking.com/ http://covidtracking.com/) doesn't have county data, but it does have test data (both total, positive, and negative). Very helpful given how relatively useless all of these case counts are without knowing how many tests were conducted.
- gboesel 7y agoI quickly hacked a website together this weekend based on the COVID Tracking Project data because I wanted to see how much testing was being done and where. I think that it's pretty interesting to sort by the columns and browse the data. The site's not great on mobile, but I'm working on that this weekend... http://VirusTracking.net http://VirusTracking.net
- s09dfhks 7y agoTheir data set shows only 1 case in san mateo county. That aint right