3 ms·

I'm not sure whether any of that timing precision will do any good on GitHub Actions, where this seems to be running. I don't think the machine is going to be q

I'm not sure whether any of that timing precision will do any good on GitHub Actions, where this seems to be running. I don't think the machine is going to be quiet enough to get a good measurement of anything.

jart 5y ago
Noise and jitter in the VM should cancel out with repeated measurements. Here's something interesting. If I run my test in a RHEL5 (Linux c. 2010) VirtualBox VM then it actually goes faster than a modern Linux kernel running on bare metal, taking only 14µs. If I run it in a RHEL7 VM then it needs 131µs. So >1ms hello world execution should be a red flag. If that's actually the latency of GitHub's kernels and can't be explained by something like a shell script doing the time interval measurements, then it'd be interesting to learn what it's doing.
- zwp 5y ago
  Is that due to SPECTRE & Meltdown mitigations?
  - loeg 5y ago
    You should be able to disable them for your VM guests, but I don’t know if VirtualBox does.
  - jart 5y ago
    I disable Spectre mitigations on my Linux workstation since it doesn't host VMs or browsers. I'm sure if those were enabled it'd go much slower. But either way it's pretty clear that a lot of stuff got added to the kernel since 2.6.18 that slows down process launches dramatically, then some of it probably got optimized, and the rest stuck. Would be interesting to know what it is.
- cormacrelf 5y ago
  It appears helloworld is the only test with any repeats, and it only has 5 repeats. https://github.com/hanabi1224/Programming-Language-Benchmarks/blob/a2e76950d4798906bda99e08e00fd5e9c57f4326/bench/bench.yaml#L18 https://github.com/hanabi1224/Programming-Language-Benchmark... (Also, you can't average out the VM scheduling something else during the test, if it is constantly scheduling other stuff as it is a very busy build machine.) Here's the measurement code, it appears to be significantly more complicated than a simple fork/exec/wait loop but that could just be all the C# getting in the way: https://github.com/hanabi1224/Programming-Language-Benchmarks/blob/ebed7a407f4dfcc291fb9efab04ca9c80cc70b93/bench/tool/ProcessUtils.cs#L149-L323 https://github.com/hanabi1224/Programming-Language-Benchmark... Note that we are definitely measuring the C# async runtime to some degree. Nevertheless you are probably right that the bulk of this 1.8ms is in the executable under test, and it truly is just bloat. Running `hyperfine ./empty-main-function` from rustc on my Mac gives 0.8ms.
  - jart 5y ago
    Looks like it might be async C# thread sleep code launching bash launching the benchmark.

jart 5y ago

Noise and jitter in the VM should cancel out with repeated measurements. Here's something interesting. If I run my test in a RHEL5 (Linux c. 2010) VirtualBox VM then it actually goes faster than a modern Linux kernel running on bare metal, taking only 14µs. If I run it in a RHEL7 VM then it needs 131µs. So >1ms hello world execution should be a red flag. If that's actually the latency of GitHub's kernels and can't be explained by something like a shell script doing the time interval measurements, then it'd be interesting to learn what it's doing.

zwp 5y ago
Is that due to SPECTRE & Meltdown mitigations?
- loeg 5y ago
  You should be able to disable them for your VM guests, but I don’t know if VirtualBox does.
- jart 5y ago
  I disable Spectre mitigations on my Linux workstation since it doesn't host VMs or browsers. I'm sure if those were enabled it'd go much slower. But either way it's pretty clear that a lot of stuff got added to the kernel since 2.6.18 that slows down process launches dramatically, then some of it probably got optimized, and the rest stuck. Would be interesting to know what it is.
cormacrelf 5y ago
It appears helloworld is the only test with any repeats, and it only has 5 repeats. https://github.com/hanabi1224/Programming-Language-Benchmarks/blob/a2e76950d4798906bda99e08e00fd5e9c57f4326/bench/bench.yaml#L18 https://github.com/hanabi1224/Programming-Language-Benchmark... (Also, you can't average out the VM scheduling something else during the test, if it is constantly scheduling other stuff as it is a very busy build machine.) Here's the measurement code, it appears to be significantly more complicated than a simple fork/exec/wait loop but that could just be all the C# getting in the way: https://github.com/hanabi1224/Programming-Language-Benchmarks/blob/ebed7a407f4dfcc291fb9efab04ca9c80cc70b93/bench/tool/ProcessUtils.cs#L149-L323 https://github.com/hanabi1224/Programming-Language-Benchmark... Note that we are definitely measuring the C# async runtime to some degree. Nevertheless you are probably right that the bulk of this 1.8ms is in the executable under test, and it truly is just bloat. Running `hyperfine ./empty-main-function` from rustc on my Mac gives 0.8ms.
- jart 5y ago
  Looks like it might be async C# thread sleep code launching bash launching the benchmark.