> The market is an oligopoly so you have 10-15 store chains. Not that prices aren't consistent across a chain, individual stores in one chain will have differe

> The market is an oligopoly so you have 10-15 store chains. Not that prices aren't consistent across a chain, individual stores in one chain will have different prices, calibrated to local supply and demand. In many cases they'll also have local products that aren't in most of the stores in that chain. I think you have to track prices in each and every store, not just per chain. Still, I think you're right about this fitting in memory. This isn't "big data".