I thought about that, but decided against subtracting out startup time. This is a real-world benchmark of how long it takes to run some actual code that I care

I thought about that, but decided against subtracting out startup time. This is a real-world benchmark of how long it takes to run some actual code that I care about. Startup time counts, but the cumulative execution time will be dominated by time taken to run the slower programs, not startup time in the trivial ones, so I don't think I'm counting it too much. But I think next time I do this I'll increase the max runtime from 1 minute to 3. That will keep more of the slower programs in the benchmark, and make startup time count that much less. Without actually removing it, because that seems too artificial and contrived to me.