4 ms·

Given this requirement, they would be wise to have the test data be different from the example data. That would prevent overfitting optimizations.

by Hugsun 3y ago

josephg 3y ago
I'm not sure that it is overfitting to optimize your code for the test set. The requirement is just that you don't break compatibility for arbitrary names in the process. This sort of thing shows up all the time in the real world - where, for example, 90% of traffic will hit one endpoint. Or 90% of a database is one specific table. Discovering and microoptimizing for the common case is an important skill.
- lifthrasiir 3y ago
  That's why I was so unsure. I think though, that it is generally better to have the input generator seeded and do not disclose the chosen seed in advance, which prevents overfitting to a single input and yet encourages overfitting to a much bigger class of inputs.
  - rocqua 3y ago
    That invites probing submissions to figure out a good hash for all names from that one seed. I think having the names for performance tests public is fine. But there should be correctness tests on sets with random names.
    - lifthrasiir 3y ago
      Of course, I assume names are also randomly generated. But it might be the case that they are all ASCII-only, which can be exploited.
- gunnarmorling 3y ago
  Their should be no optimizations which rely on a-priori knowledge of the dataset. I.e. if you calculate a perfect hash function at runtime by inspecting the dataset, that's fine (not sure whether it's beneficial), but hard coding it, is not.
  - johndough 3y ago
    That is a tricky rule to enforce. For example, some solutions assume that the temperature values fit into an int, which could be interpreted as relying on a priori knowledge.
    - Hugsun 3y ago
      That's true. Some assumptions always have to be made. In this case, we can assume place names that can be arbitrary strings and temperatures that occur naturally on earth. Optimizing around those constraints should be fine. We could probably limit the string length to something like 100 characters or so. Bad assumptions would for example be to assume that all possible place names are found within the example data.
    - sfn42 3y ago
      Somehow I doubt any weather station would register temperatures above 2,14 billion degrees

Given this requirement, they would be wise to have the test data be different from the example data. That would prevent overfitting optimizations.

by Hugsun 3y ago

Given this requirement, they would be wise to have the test data be different from the example data. That would prevent overfitting optimizations.

josephg 3y ago

I'm not sure that it is overfitting to optimize your code for the test set. The requirement is just that you don't break compatibility for arbitrary names in the process. This sort of thing shows up all the time in the real world - where, for example, 90% of traffic will hit one endpoint. Or 90% of a database is one specific table. Discovering and microoptimizing for the common case is an important skill.

lifthrasiir 3y ago
That's why I was so unsure. I think though, that it is generally better to have the input generator seeded and do not disclose the chosen seed in advance, which prevents overfitting to a single input and yet encourages overfitting to a much bigger class of inputs.
- rocqua 3y ago
  That invites probing submissions to figure out a good hash for all names from that one seed. I think having the names for performance tests public is fine. But there should be correctness tests on sets with random names.
  - lifthrasiir 3y ago
    Of course, I assume names are also randomly generated. But it might be the case that they are all ASCII-only, which can be exploited.
gunnarmorling 3y ago
Their should be no optimizations which rely on a-priori knowledge of the dataset. I.e. if you calculate a perfect hash function at runtime by inspecting the dataset, that's fine (not sure whether it's beneficial), but hard coding it, is not.
- johndough 3y ago
  That is a tricky rule to enforce. For example, some solutions assume that the temperature values fit into an int, which could be interpreted as relying on a priori knowledge.
  - Hugsun 3y ago
    That's true. Some assumptions always have to be made. In this case, we can assume place names that can be arbitrary strings and temperatures that occur naturally on earth. Optimizing around those constraints should be fine. We could probably limit the string length to something like 100 characters or so. Bad assumptions would for example be to assume that all possible place names are found within the example data.
  - sfn42 3y ago
    Somehow I doubt any weather station would register temperatures above 2,14 billion degrees