> In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can ha

> In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."