4 ms·

I'm struggling to understand how the methodology avoided peeking into the future. The author says he trained on companies that completed accelerators before 201

I'm struggling to understand how the methodology avoided peeking into the future. The author says he trained on companies that completed accelerators before 2014, leaving 2014-2016 as a holdout period. But then in the Outcomes section, he is focused on "post-incubator early-stage investment and late-stage exit". The author's claim of "predictably bad investment" requires a specific time at which this machine learning algorithm could have been trained based on past data and then predicted the future. What was that time here? It couldn't have been 2014-01-01, since the algorithm needs to be trained on an outcome, and it sounds like he's relying on post-incubator data to get that outcome. Meaning at least some of the data needed to train his model comes from after 2014-01-01... right? Am I missing anything here? Isn't he cheating by peeking into the future?

joshuamorton 4y ago
If you have two sets of companies, that who participated pre 2014, and those from 2014-2016, you can train based on the earlier ones and validate based on the later ones without peeking into the future. I haven't read enough to know that's what happened, but that's a simple way to avoid this.
- civilized 4y ago
  But if you need information from after 2014 to define the outcome metric that's used to train your model, you're still peeking into the future, even if you've limited yourself to companies that completed the incubator before 2014. He's relying on "post-incubator early-stage investment and late-stage exit" to train the model. Unless I'm missing something, that information would not have been available at the beginning of 2014, so the model could not have been trained at that time. In order to say the human investments were predictably bad, we need to specify a time at which all necessary data to train the model was available, and predict from that time point forward only.

joshuamorton 4y ago

If you have two sets of companies, that who participated pre 2014, and those from 2014-2016, you can train based on the earlier ones and validate based on the later ones without peeking into the future. I haven't read enough to know that's what happened, but that's a simple way to avoid this.

civilized 4y ago
But if you need information from after 2014 to define the outcome metric that's used to train your model, you're still peeking into the future, even if you've limited yourself to companies that completed the incubator before 2014. He's relying on "post-incubator early-stage investment and late-stage exit" to train the model. Unless I'm missing something, that information would not have been available at the beginning of 2014, so the model could not have been trained at that time. In order to say the human investments were predictably bad, we need to specify a time at which all necessary data to train the model was available, and predict from that time point forward only.