9 ms·
Why can't you get the data used to create the Consumer Price Index (CPI)?
The Consumer Price Index (CPI) measures inflation and, in the words of the US Bureau of Labor Statistics, "directly affects hundreds of millions of Americans".
I would have guessed the data used to construct the CPI would be open to everyone.
But no, it appears to be unavailable and it is instead a black box due to rules of confidentiality. What is strange is, it appears much of the data is publicly available data. For example, 8% comes from e-commerce sites, the USPS public website, etc. If you browse through the categories, you can also see that almost all the data could be collected from public sources - APIs, websites, etc.
So, does anyone know why this data cannot be obtained?
Reference: https://www.bls.gov/opub/hom/cpi/pdf/cpi.pdf
- alexmingoia 6y agoYou might be interested in The Billion Prices Project, which calculates price changes by scraping online product prices. They make the data available. http://www.thebillionpricesproject.com/ http://www.thebillionpricesproject.com/ Spoiler: Billion Prices Project has mostly tracked CPI within 1-2%
- ErikAugust 6y agoThank you - but I ask on the Spoiler Alert - it looks like they only have data through 2020 from two online retailers [1]? http://www.thebillionpricesproject.com/datasets/ http://www.thebillionpricesproject.com/datasets/
- RandyRanderson 6y ago1-2 pc every year or lifetime? Even a small difference in the yearly rate will add up to a huge difference over time.
- riffraff 6y agoThe difference does not need to be always in the same direction.
- Clewza313 6y agoIt would be if the thesis is that the CPI is systematically manipulated to suppress inflation.
- rsj_hn 6y agoThat would be pretty hard to do in our data rich environment. It's pretty easy to look up historical prices of things like oil, wheat, and other commodities, as well as labor, automobiles, and other consumer goods. Even a small systematic manipulation would become really obvious after a decade or two.
- totalZero 6y agoNot necessarily, if the manipulations are reverted smoothly over time. There are certain moments in which inflation is a focal point (now for example) and other moments where nobody really cares. Also, I think the inclusion of things like tuition and medical care kind of adulterates the data. I have read that people go to the hospital for non-emergency reasons less often than before the pandemic, and also that several schools slashed tuition due to the pandemic. That offsets price increases in other areas.
- imtringued 6y agoThe problem with this type of argument is that the fed doesn't want to exceed more than 3% inflation. It has no reason to lie and pretty much every central bank is struggling with lack of inflation. You can criticize the Fed for doing reckless policies that fail to increase inflation and only drive a bubble. Why would you even need an unproven conspiracy theory like CPI when the problems with the Fed are undeniable? You can even complain to the fed that inflation is too low and they should either calm down or do something else since the Fed is clearly not capable of driving inflation. Considering the mission statement of the Fed that's a huge blow by itself.
- totalZero 6y agoI don't think you understand what I am saying. 1. I'm not arguing that the fed is artificially inflating inflation. I'm saying that if they want to suppress the appearance of inflation in the short term so that they can keep rates low and prevent any panic about the dollar until they can react to it, they have some ability to do so because of the opaqueness of the data. 2. Bias in a signal doesn't have to be persistent to be problematic. The average reading does not have to be biased in the long term for there to be a problem with the spot reading. That's a major reason why more granular data is always better. 3. I'm not stating any conspiracy theory. The Fed is driven by data. Biases in data affect data-driven decisions. CPI includes various inputs, not all of which are experiencing the same demand. CPI is an agglomerate reference.
- Clewza313 6y agoLifetime: http://www.thebillionpricesproject.com/wp-content/uploads/2017/04/data-priceindices.jpg http://www.thebillionpricesproject.com/wp-content/uploads/20... Squinting at the graph, for 2008=100, in 2016 CPI was ~108 and BPP was ~109. (Couldn't find a newer graph with a quick look.)
- Retric 6y ago+8% vs +9% is a 12% divergence is just 8 years, and the numbers where even further apart in 2014. So, it looks like they are very different models.
- alexmingoia 6y agoThe dollar has lost more than 90% of its value in just 50 years. The point is the BPP is proof there is no nefarious conspiracy to manipulate CPI to hide “true” inflation. Amongst the gold-bug hyperinflationist crowd there is a commonly held belief that inflation is much higher than CPI or that CPI is manipulated to hide inflation. The fact that a simple model scraping a few online retailers results in nearly the same annual percentage change demonstrates that this conspiracy theory is a myth.
- rdtwo 6y agoCPI is probably near CPI the problem is that CPI does not actually measure inflation felt by the average American. The basket of goods is not reflective of actual allocation. Because of that the CPI value is significant lower than actual inflation
- antb123 6y agoEssentially inflation is a political measurement that drives policy decisions and that is cause for a lot of concern. The current inflationary model is mostly driven by salaries. Thus in the 70s unions drove up inflation due to oil shocks. The concern for me is that models do not include asset inflation for goods (housing) that most people would like to own and where there is little productivity improvement. Using an equivalent rent model ignores this preference.
- thorwasdfasdf 6y agoThe CPI is a weighted average of several categories. and some of those categories severely underestimate inflation. For instance, the CPI says new car prices haven't increased in the last 20 years but, that's obviously not true, if you calculate the real cpi for say a corrolla or a camaro or whatever model you choose, you can see it's gone up about 40 to 60% in that time which means for that category, they're undercounting CPI by almost 2 to 3 %. I think they're trying to prevent people from calculating the real CPI. there's alot at stake: the bond market, social security, federal interest payments, perception of real wages, real productivity per capita number, etc.
- itsdrewmiller 6y agoThey adjust car prices based on feature improvements: https://www.bls.gov/cpi/quality-adjustment/new-vehicles.pdf https://www.bls.gov/cpi/quality-adjustment/new-vehicles.pdf Certainly debatable since it is impossible to actually buy a new car that doesn't have a bunch of those features now.
- thorwasdfasdf 6y agoDebatable is an understatement. their discount model is completely disproportionate to the benefits that those features bring. If an auto now costs 50% more, and CPI really is 0% increase, then I expect to see a monetary benefit of 50%, either through reduced maintenance cost, increased MPG, or maybe it makes me breakfast of the equivalent cost. but, I don't see new cars have anywhere near this benefit. And it becomes even more preposterous when you look 50+ years back. CPI, says x3 increase in cost of car, but it's actually about x10. So, how does a car return +233% of it's value to a user when MPG hasn't even doubled over that time. i mean sure, monetary rewards are not the only advancements but they should the primary basis for the inflation discount model. those auto-dimming mirrors and countless other features just don't mean very much.
- tarlinian 6y agoA typical new car practically lasts way longer than a car built 30-50 years ago, early 2-3x the value there. Account for significant reduction in chance of death, MPG improvements, and these numbers make complete sense.
- deleted 6y ago[deleted]
- dsparkman 6y agoThe CPI is a propaganda tool. It is designed to keep the real inflation hidden to make it seem that inflation is not a real thing to worry about. Because it is formulated in secret, they can manipulate it to give a fairly rosy picture. The published CPI never panics.
- ausbah 6y agothe process is literally posted on the Bureau for Labor Statistics site [1], it's like the second result on Google https://www.bls.gov/cpi/questions-and-answers.htm https://www.bls.gov/cpi/questions-and-answers.htm
- imtringued 6y agoThe CPI absolutely crashed in April 2020 https://tradingeconomics.com/united-states/inflation-cpi https://tradingeconomics.com/united-states/inflation-cpi
- junippor 6y agoDo you have any actual evidence of this?
- spacemanmatt 6y agoI wish I had a dollar for every irrational (extremely emotionally charged!) response to this same question when I have posed it.
- zhdc1 6y agoFrom the text: > Multiple confidentiality issues arise in the production of the CPI. One set of issues arises out of the need to prevent unauthorized access of data that are embargoed, or yet to be released to the public. because the CPI data can affect financial markets, it is essential to ensure that no one without authorization has access to the data before release. BLS personnel who do see the data ahead of time are restricted by law from engaging in certain financial transactions during the period where they have seen data that are not public. Pre-release data are encrypted and always kept on secure servers, with any hard copies locked in secured areas.
- schoen 6y agoWait, so nobody who participates in CPI surveys chooses to publish the specific questions? Is it illegal for the respondents to do that? I know somebody who got a CPI survey last year -- maybe I can ask her.
- OldHand2018 6y agoYou can get access to some of the data through the University of Chicago, but it is super restricted (as required by their source): https://www.chicagobooth.edu/research/kilts/datasets/nielsen/policies https://www.chicagobooth.edu/research/kilts/datasets/nielsen... Good luck, this type of data is really really guarded as it is ridiculously valuable.
- astatine 6y agoDisclaimer: Sharing links to my own content I hear you. I tried recreating the CPI for India and couldn't get the data needed. (see https://arun.tanksali.com/topic/rpi/ https://arun.tanksali.com/topic/rpi/). I eventually thought I can come with a Personal Price Index which reflects how inflation is when seen from my personal perspective. I took a basket of goods derived from our household consumption and scraped prices for a year (till last month). See (2) for data. Obviously, CPI represents the average citizen and no individual is likely to fit that definition well. I am no economics expert but I expected some correlation between the published CPI and my own observations but there were times they were moving in opposite directions. 1: https://arun.tanksali.com/topic/rpi/ https://arun.tanksali.com/topic/rpi/ 2: https://realpi.tanksali.com/#/app/dashboard https://realpi.tanksali.com/#/app/dashboard
- parsimo2010 6y agoThe BLS makes nearly all of their products publicly available for free. You can poke around the old-school way here: https://download.bls.gov/pub/time.series/ https://download.bls.gov/pub/time.series/ They have an API with documentation here: https://www.bls.gov/developers/ https://www.bls.gov/developers/ And a few other options here: https://www.bls.gov/data/ https://www.bls.gov/data/ But I suspect that what you mean is that the raw data used to calculate these metrics isn’t available. Such as the “worker X on 25 Feb found that the price of a pound of flour at the Kroger store #357 is $1.” I suspect there are two reasons. (I can’t say for sure because I don’t work at the BLS, but I am a statistician, work for the government, and have been involved in decisions not to release data). 1. The BLS almost certainly contracts the leg work for the CPI. While the calculations are relatively simple (arguments about methodology excluded), the work to collect prices nationwide is nontrivial. Including permission to release the data publicly likely increased the cost of the contract. These data vendors also sell their price data to other “businesses intelligence” companies, and allowing the public release hurts their other business prospects, so they would charge a premium for it. And ironic as it may seem, government agencies/workers are interested being good stewards of the public’s tax dollars and had to make a value decision to not buy the rights to release the data. 2. For anything that isn’t legally/contractually prohibited from release, there may have been a conscious decision that the data at its most granular level either wasn’t useful to the public at all, or was likely to be misleading/misused because of critical gaps in what can be released. If the vendor for Alabama prices allowed release but the vendor for California prices did not, they may have just made the decision that releasing incomplete data set wasn’t a good idea.
- tacostakohashi 6y agoWhat exactly is it that you want, but can't obtain? Beyond the plethora of data that is published, you could submit a FOIA request for anything you feel is in the possession of the BLS, but not being published.
- Irongirl1 6y agoIsn't this it? https://catalog.data.gov/dataset/consumer-price-index-cpi https://catalog.data.gov/dataset/consumer-price-index-cpi
- simonblack 6y agoBecause if you could lay your hands on the data and calculate the CPI for yourself, you would find that the 'official' CPI is pure baloney.