6 ms·

Skimpflation is one problem with CPI -- falling under the umbrella of somewhat arbitrary hedonic adjustments. But the main problem is more fundamental. CPI is

Skimpflation is one problem with CPI -- falling under the umbrella of somewhat arbitrary hedonic adjustments. But the main problem is more fundamental. CPI is calculated by surveying what a bunch of median Americans bought, and weights it by HOW MUCH OF IT they bought. For example, the weight of gasoline oscillated ~100% from its low from 2003-2007 because oil prices increased significantly during this time (median Americans drove less/bought more fuel efficient cars). And therein lies the problem. If you poll the median, and the median tends to spend all of their money (just reallocating it according to priorities), what you are really measuring is the median household budget. Which is why any chart of "real wages" is a flat line, which is obviously not accurate. So CPI is a load of crap. Real inflation over the last 40 years has probably averaged closer to 3% (see the Big Mac index) than the <2% quoted by bureaucrats and politicians. To improve it, they could look at the 1000 highest volume items in the economy and measure changes in prices of those. Many of those are raw materials, so this would make hedonic adjustments a bit easier, too.

adgjlsfhk1 5y ago
What happens when the 1000 highest volume items change under your proposal?
- lend000 5y ago
  Same thing they do with the current system. Normalize and replace, and don't expect much of a shock from relatively low volume items breaking in and out of the 900-1000 rank.
ACow_Adonis 5y ago
as someone with a bit of an economics and official statistics background as well as tech (albeit not in the US), one of my frustrations with visiting hacker news are these threads on non-tech topics. for instance: CPI. how is it calculated? I haven't downvoted this post, but let's use it as an example. here's how CPI appears to be calculated: https://www.bls.gov/opub/hom/cpi/calculation.htm https://www.bls.gov/opub/hom/cpi/calculation.htm by my reading, the basic methodology is to obtain a basket of goods and consumer items and then track the price change in those items. yes, it's more complicated than that in the actual methodology in reality, there's likely some debatable things like hedonic adjustment and a real question over what the index should track, but the general idea seems sound. this post seems to suggest that CPI is based on some kind of longitudinal household survey methodology weighted for how much individual households consume over time, and that statisticians and economists are simpletons fooled by basic household budget substitution effects as prices and consumption patterns shift. CPI becomes "a load of crap". But the error seems to be in the original post's incorrect misunderstanding of how CPI is actually calculated. and the actual method is somewhat closer to his suggested base-item method.
- efaref 5y ago
  The key part is in "obtain a basket of goods". The ONS (UK statistics) are quite open about how they do this: https://www.ons.gov.uk/economy/inflationandpriceindices/articles/ukconsumerpriceinflationbasketofgoodsandservices/2021 https://www.ons.gov.uk/economy/inflationandpriceindices/arti... Specifically: > For example, if the price of tea rose dramatically during one year, consumers might switch their spending towards coffee, making it necessary to adjust the expenditure weights accordingly in the following year. Or put another way, if fuel goes up astronomically and people buy less of it, they will adjust its weight downwards, limiting the effects of its price increase on the headline inflation. It's not that statisticians and economists are fooled. The methodology works assuming that people have spare money and can choose to spend or not. But the reality is that things have got so expensive that most people are living paycheck to paycheck and spending everything they have, so their attempts to eradicate the bias of choice ends up just measuring how much money people have to spend (i.e. wage growth).
  - ACow_Adonis 5y ago
    Let's run that scenario in a simplified manner of how CPIs are likely actually going to be calculated. First, they have to determine the conceptual basket of goods to track. Ok, that's comparable to your 1000 "highest volume" items methodology (albeit yours is a bit simplified too, we'll ignore the problem that good or volume is complicated for everyone). Generally stats bodies do this by looking at what consumers actually spend things on. This seems emminently sensible to me, and no offense intended, superior to a top 1000 volume method. Fuel, presumably, gets included in both. Then they need to assign a weight for every good to determine how much its mixture of quantity + price movements contribute to the CPI figure. I put it to you this is necessary. Every item shouldn't be equally weighted even in your methodology (the distribution of volume of relative goods purcahsed/consumed is guaranteed to not be uniform across all 1000 goods, and it makes no sense for no. 1 to be given an equal weight to no. 999). Stats bodies do this too. This is what is meant by weight adjustment in this context. And both you and stat agencies should be doing it. At some point, your "top 1000" goods is going to change and you're going to have to figure out what is in a new basket at a new time. So too does the CPI basket of goods. You both need to do this. Note that your methodology does not actually fix the fuel-budget-substitution issue. Price movements in your method that invoke a strong enough substitution or consumption effect result in fuel dropping out of your top 1000 as well. By keeping your basket constant for a given period of time and just tracking the good's prices, you track expenditure somewhat consistently for a given period. That's a good thing. There's no reference to changes in a consumer's budget. But...CPI calculating stats agencies do this too for a set basket of goods just like you do. They fix the basket and track it over time, adjusting, like yours, only for re-inclusion the basket of goods under consideration. Lastly, comes the hedonic quality problem. Changes in quality of consumption/goods over time. Some substitutions seem just because of taste or culture (see for example, relative consumption of antimacassars over time, or certain culture's preference for certain foods or items). Others are obviously qualitative in nature (i.e. computer processing power), and most things are a complex mix of tech/economy/culture/price/quality. This is the part that captures and tries to control for your qualitative substitution problem (it's not without its controversies, but I argue that your method needs it as well because it suffers similar problems, and indeed all ones which try to standardise subjective consumption of a national economy into a single figure). Relevant to 'fuel' is its price, but also substitution to other new fuels that turn up (hydrogen, LPG, lithium-ion), but also technology efficiency gains that have changed the consumption to utility ratio. Again, stats agencies try to control for this too. Aside from the definition of the basket of goods (a weakness and methodology shared by your "fix"), there is generally no explicit reference to consumer's budget or assumed spending patterns when tracking within a given basket. There is nothing inherently equivalent to "assumed spare money or choosing to spend or not". (I suppose I'm happy to be proven wrong on this for a given countries methodology). it's not just a simplified survey of median households or expenditure. Now, you can (and many people justifiably do) take issue with ideas and quantification of hedonic quality problems: but it's a universal problem, your method doesn't fix it, and I think you're generally misconstruing how CPIs are calculated around the world. The problems you mention are already attempted to be tackled and controlled for in methodology, and the official methodology is already generally superior to your suggested replacement. Edit: it's also not like stats/economics agencies are doing this WITHOUT an accesible measures of nominal wages (that is wages and resource flows without inflation taken into account) available to them. edit 2: in the real world, further considerations like geography, seasonality, population, etc is additionally controlled and adjusted for. And many stat agencies, in my experience, offer additional breakdowns of other price series, such as geographical and categorical indexes, or various additional weightings and combinations, so if you take issue with headline CPI you can investigate other more appropriate measures for your specific use cases.

adgjlsfhk1 5y ago

What happens when the 1000 highest volume items change under your proposal?

lend000 5y ago
Same thing they do with the current system. Normalize and replace, and don't expect much of a shock from relatively low volume items breaking in and out of the 900-1000 rank.

ACow_Adonis 5y ago

as someone with a bit of an economics and official statistics background as well as tech (albeit not in the US), one of my frustrations with visiting hacker news are these threads on non-tech topics. for instance: CPI. how is it calculated? I haven't downvoted this post, but let's use it as an example. here's how CPI appears to be calculated: https://www.bls.gov/opub/hom/cpi/calculation.htm https://www.bls.gov/opub/hom/cpi/calculation.htm by my reading, the basic methodology is to obtain a basket of goods and consumer items and then track the price change in those items. yes, it's more complicated than that in the actual methodology in reality, there's likely some debatable things like hedonic adjustment and a real question over what the index should track, but the general idea seems sound. this post seems to suggest that CPI is based on some kind of longitudinal household survey methodology weighted for how much individual households consume over time, and that statisticians and economists are simpletons fooled by basic household budget substitution effects as prices and consumption patterns shift. CPI becomes "a load of crap". But the error seems to be in the original post's incorrect misunderstanding of how CPI is actually calculated. and the actual method is somewhat closer to his suggested base-item method.

efaref 5y ago
The key part is in "obtain a basket of goods". The ONS (UK statistics) are quite open about how they do this: https://www.ons.gov.uk/economy/inflationandpriceindices/articles/ukconsumerpriceinflationbasketofgoodsandservices/2021 https://www.ons.gov.uk/economy/inflationandpriceindices/arti... Specifically: > For example, if the price of tea rose dramatically during one year, consumers might switch their spending towards coffee, making it necessary to adjust the expenditure weights accordingly in the following year. Or put another way, if fuel goes up astronomically and people buy less of it, they will adjust its weight downwards, limiting the effects of its price increase on the headline inflation. It's not that statisticians and economists are fooled. The methodology works assuming that people have spare money and can choose to spend or not. But the reality is that things have got so expensive that most people are living paycheck to paycheck and spending everything they have, so their attempts to eradicate the bias of choice ends up just measuring how much money people have to spend (i.e. wage growth).
- ACow_Adonis 5y ago
  Let's run that scenario in a simplified manner of how CPIs are likely actually going to be calculated. First, they have to determine the conceptual basket of goods to track. Ok, that's comparable to your 1000 "highest volume" items methodology (albeit yours is a bit simplified too, we'll ignore the problem that good or volume is complicated for everyone). Generally stats bodies do this by looking at what consumers actually spend things on. This seems emminently sensible to me, and no offense intended, superior to a top 1000 volume method. Fuel, presumably, gets included in both. Then they need to assign a weight for every good to determine how much its mixture of quantity + price movements contribute to the CPI figure. I put it to you this is necessary. Every item shouldn't be equally weighted even in your methodology (the distribution of volume of relative goods purcahsed/consumed is guaranteed to not be uniform across all 1000 goods, and it makes no sense for no. 1 to be given an equal weight to no. 999). Stats bodies do this too. This is what is meant by weight adjustment in this context. And both you and stat agencies should be doing it. At some point, your "top 1000" goods is going to change and you're going to have to figure out what is in a new basket at a new time. So too does the CPI basket of goods. You both need to do this. Note that your methodology does not actually fix the fuel-budget-substitution issue. Price movements in your method that invoke a strong enough substitution or consumption effect result in fuel dropping out of your top 1000 as well. By keeping your basket constant for a given period of time and just tracking the good's prices, you track expenditure somewhat consistently for a given period. That's a good thing. There's no reference to changes in a consumer's budget. But...CPI calculating stats agencies do this too for a set basket of goods just like you do. They fix the basket and track it over time, adjusting, like yours, only for re-inclusion the basket of goods under consideration. Lastly, comes the hedonic quality problem. Changes in quality of consumption/goods over time. Some substitutions seem just because of taste or culture (see for example, relative consumption of antimacassars over time, or certain culture's preference for certain foods or items). Others are obviously qualitative in nature (i.e. computer processing power), and most things are a complex mix of tech/economy/culture/price/quality. This is the part that captures and tries to control for your qualitative substitution problem (it's not without its controversies, but I argue that your method needs it as well because it suffers similar problems, and indeed all ones which try to standardise subjective consumption of a national economy into a single figure). Relevant to 'fuel' is its price, but also substitution to other new fuels that turn up (hydrogen, LPG, lithium-ion), but also technology efficiency gains that have changed the consumption to utility ratio. Again, stats agencies try to control for this too. Aside from the definition of the basket of goods (a weakness and methodology shared by your "fix"), there is generally no explicit reference to consumer's budget or assumed spending patterns when tracking within a given basket. There is nothing inherently equivalent to "assumed spare money or choosing to spend or not". (I suppose I'm happy to be proven wrong on this for a given countries methodology). it's not just a simplified survey of median households or expenditure. Now, you can (and many people justifiably do) take issue with ideas and quantification of hedonic quality problems: but it's a universal problem, your method doesn't fix it, and I think you're generally misconstruing how CPIs are calculated around the world. The problems you mention are already attempted to be tackled and controlled for in methodology, and the official methodology is already generally superior to your suggested replacement. Edit: it's also not like stats/economics agencies are doing this WITHOUT an accesible measures of nominal wages (that is wages and resource flows without inflation taken into account) available to them. edit 2: in the real world, further considerations like geography, seasonality, population, etc is additionally controlled and adjusted for. And many stat agencies, in my experience, offer additional breakdowns of other price series, such as geographical and categorical indexes, or various additional weightings and combinations, so if you take issue with headline CPI you can investigate other more appropriate measures for your specific use cases.