3 ms·
You are optimizing for a hypothetical edge case. If there are few hundred thousand weather stations in world or 50,000 airports or 2 million hotels. Building sy
by blackoil 3y ago
You are optimizing for a hypothetical edge case. If there are few hundred thousand weather stations in world or 50,000 airports or 2 million hotels. Building system assuming they'll change to billions is impractical.
Also, even for this, original problem statement sets maximum to 10,000 different station names.
- afiodorov 3y agoI thought this was an illustrative example of how to process big datasets. We could easily have a statistic per e.g. bitcoin transaction in a different problem, see https://github.com/afiodorov/bitcoin_ancestries https://github.com/afiodorov/bitcoin_ancestries . I struggle a lot with this toy problem. Without constraints too trivial to pay attention to; then no one seems to agree on potential real-world constraints (stdlib only? no mmaps?). If we have to solve just this problem as is, shouldn't we be timing simple solutions using various frameworks (polars, pandas, spark, bigquery[^1], go, awk) to compare various frameworks? Once you have the answer, why would you try to get the same answer again but in 4 seconds the second time round? Comparing frameworks would at least indicate if a data practitioner should upskill and pick up yet another data framework. [^1]: https://medium.com/@shuvro_25220/the-one-billion-rows-challenge-with-bigquery-404538c5e3e6 https://medium.com/@shuvro_25220/the-one-billion-rows-challe...