This partially expresses what I wanted to reply. The other part is that food calories are the amount of energy needed to raise the temperature of one kilogram

This partially expresses what I wanted to reply. The other part is that food calories are the amount of energy needed to raise the temperature of one kilogram of water by one degree celsius. By that definition, calories in vs calories out seems pretty darn close to reality, and if it's not perfect, it's likely so close you're not going to notice a significant difference.