> - for some program, how long does it take to run that program. Start to finish. No ignoring warmup. This methodology likely comes from Java, which has long-r

> - for some program, how long does it take to run that program. Start to finish. No ignoring warmup. This methodology likely comes from Java, which has long-running server applications. "How long does it run" is often "until someone hits ^C". Here, startup cost can be slow as long as the peak performance is fine. It's accepted that the first minute or two of the server are slow, but that's small compared to the month or so that the server will be running for. > This tells you how good a VM is. I think papers like this approach it from the wrong angle. I don't care about the VM's theoretical peak performance. I care about being able to measure and track performance in a reliable way. Put simply, I'm fine with bad codegen as long as I can consistently measure it. Feel free to improve it, but adding to sometimes give me good codegen, unreliably, is much more frustrating than bad codegen. But this seems to be the way the VMs are going, with things like probabilistic profiling. If I refactor my code and replace for(let i = 0; i < L.length; i++) with for(const i of L), what's the cost? Will performance go up or down? We don't have tools or metrics to handle that right now. How can I ensure my codegen is good won't regress? I work on a particularly demanding website in my free time ( https://noclip.website/#smg/AstroGalaxy https://noclip.website/#smg/AstroGalaxy , unfortunately won't run in WebKit due to missing WebGL 2 ), and performance varies drastically from Chrome release to release, and I do extensive testing with node.js to make sure that I'm getting good codegen.