The meaningful boundary between small data and large data is the difference whether the whole dataset/processing is expected to fit in the RAM of a single commo

The meaningful boundary between small data and large data is the difference whether the whole dataset/processing is expected to fit in the RAM of a single common machine. From that perspective, 1 billion rows a borderline case, which can be small or large depending on how large the rows are - and in this particular challenge the rows are tiny.