I thought this was an illustrative example of how to process big datasets. We could easily have a statistic per e.g. bitcoin transaction in a different problem,

I thought this was an illustrative example of how to process big datasets. We could easily have a statistic per e.g. bitcoin transaction in a different problem, see https://github.com/afiodorov/bitcoin_ancestries https://github.com/afiodorov/bitcoin_ancestries . I struggle a lot with this toy problem. Without constraints too trivial to pay attention to; then no one seems to agree on potential real-world constraints (stdlib only? no mmaps?). If we have to solve just this problem as is, shouldn't we be timing simple solutions using various frameworks (polars, pandas, spark, bigquery[^1], go, awk) to compare various frameworks? Once you have the answer, why would you try to get the same answer again but in 4 seconds the second time round? Comparing frameworks would at least indicate if a data practitioner should upskill and pick up yet another data framework. [^1]: https://medium.com/@shuvro_25220/the-one-billion-rows-challenge-with-bigquery-404538c5e3e6 https://medium.com/@shuvro_25220/the-one-billion-rows-challe...